CharExtract Class Reference
The class is used to split a string into characters. More...
Public Types | |
| using | TokenType = int |
| The type of the token which the tokenizer extracts. More... | |
Public Member Functions | |
| int | operator() (boost::string_view &str) const |
| The function extracts the first character from the given string view and removes it from the view. More... | |
Static Public Member Functions | |
| static bool | IsTokenEmpty (const int token) |
| The function returns true if the given token is equal to EOF. More... | |
Detailed Description
The class is used to split a string into characters.
Definition at line 25 of file char_extract.hpp.
Member Typedef Documentation
◆ TokenType
| using TokenType = int |
The type of the token which the tokenizer extracts.
Definition at line 29 of file char_extract.hpp.
Member Function Documentation
◆ IsTokenEmpty()
|
inlinestatic |
The function returns true if the given token is equal to EOF.
- Parameters
-
token The given token.
Definition at line 56 of file char_extract.hpp.
◆ operator()()
|
inline |
The function extracts the first character from the given string view and removes it from the view.
Each charecter is casted to unsigned char i.e. it belongs to [0, 255]. The functon returns EOF provided that the input string is empty.
- Parameters
-
str String view to retrieve the next token from.
Definition at line 39 of file char_extract.hpp.
The documentation for this class was generated from the following file:
- /home/ryan/src/mlpack.org/_src/mlpack-git/src/mlpack/core/data/tokenizers/char_extract.hpp