Splits an input string into lexical tokens, i.e. smaller strings that are easily identifiable by tokens.tokenType().

Lexing starts always in a "stream" context. Incomplete input may be buffered until a complete token can be emitted.

In addition to slices of the original input, the following control characters may also be emitted:

  • \x02 (Start of Text): A document starts with the next token
  • \x18 (Cancel): Unexpected end of flow-mode (indicates an error)
  • \x1f (Unit Separator): Next token is a scalar value
  • \u{FEFF} (Byte order mark): Emitted separately outside documents



atEnd: any

Flag indicating whether the end of the current buffer marks the end of all input

atLineEnd: any
blockScalarIndent: any

Explicit indent set in block scalar header, as an offset from the current minimum indent, so e.g. set to 1 from a header |2+. Set to -1 if not explicitly set.

blockScalarKeep: any

Block scalars that include a + (keep) chomping indicator in their header include trailing empty lines, which are otherwise excluded from the scalar's contents.

buffer: any

Current input

charAt: any
continueScalar: any
flowKey: any

Flag noting whether the map value indicator : can immediately follow this node within a flow context.

flowLevel: any

Count of surrounding flow collection levels.

getLine: any
hasChars: any
indentNext: any

Minimum level of indentation required for next lines to be parsed as a part of the current scalar value.

indentValue: any

Indentation level of the current line.

lineEndPos: any

Position of the next \n character.

next: any

Stores the state of the lexer if reaching the end of incpomplete input

parseBlockScalar: any
parseBlockScalarHeader: any
parseBlockStart: any
parseDocument: any
parseFlowCollection: any
parseLineStart: any
parseNext: any
parsePlainScalar: any
parseQuotedScalar: any
parseStream: any
peek: any
pos: any

A pointer to buffer; the current position of the lexer.

pushCount: any
pushIndicators: any
pushNewline: any
pushSpaces: any
pushTag: any
pushToIndex: any
pushUntil: any
setNext: any


  • Generate YAML tokens from the source string. If incomplete, a part of the last line may be left as a buffer for the next call.


    • source: string
    • Optional incomplete: boolean

    Returns Generator<string, void, unknown>

    A generator of lexical tokens

Generated using TypeDoc