Splits an input string into lexical tokens, i.e. smaller strings that are easily identifiable by tokens.tokenType().
tokens.tokenType()
Lexing starts always in a "stream" context. Incomplete input may be buffered until a complete token can be emitted.
In addition to slices of the original input, the following control characters may also be emitted:
\x02
\x18
\x1f
\u{FEFF}
Generate YAML tokens from the source string. If incomplete, a part of the last line may be left as a buffer for the next call.
source
incomplete
Optional
A generator of lexical tokens
Splits an input string into lexical tokens, i.e. smaller strings that are easily identifiable by
tokens.tokenType().Lexing starts always in a "stream" context. Incomplete input may be buffered until a complete token can be emitted.
In addition to slices of the original input, the following control characters may also be emitted:
\x02(Start of Text): A document starts with the next token\x18(Cancel): Unexpected end of flow-mode (indicates an error)\x1f(Unit Separator): Next token is a scalar value\u{FEFF}(Byte order mark): Emitted separately outside documents