Header lexy/dsl.hpp

The rule DSL for specifying the grammar.

template <typename T>
concept rule = …;

The grammar in lexy is specified in several productions, where each one defines an associated rule. This rule is an object built from the objects and functions of namespace lexy::dsl that defines some (implementation-defined) parsing function. Parsing a rule takes the reader, which remembers the current position of the input, and the context, which stores information about the current production and whitespace rules, and is responsible for handling errors and values.

Parsing can have one of the following results:

  • Parsing can succeed. Then it consumes some input by advancing the reader position and produces zero or more values.

  • Parsing can fail. Then it reports an error, potentially after having consumed some input but without producing values. The parent rule can react to the failure by recovering from it or they fail itself.

  • Parsing can fail, but then recover. Then it has reported an error, but now it has consumed enough input to be in a known good state and parsing continues normally. See error recovery for details.

A branch rule is a special kind of rule that has an easy to check condition. They are used to guide decisions in the parsing algorithm. Every branch rule defines some (implementation defined) branch parsing function. It mostly behaves the same as the normal parse rule, but can have one additional result: branch parsing can backtrack. If it backtracks, it hasn’t consumed any input, raised errors or produced values. The parsing algorithm is then free to try another branch.

Note
The idea is that a branch rule can relatively quickly decide whether or not it should backtrack. If a branch rule does not backtrack, but fails instead, this failure is propagated and the parsing algorithm does not try another branch.

A token rule is a special kind of rule that describes the atomic elements. Parsing them never produces any values and can happen easily, as such they’re also branch rules where the entire rule is used as the condition. Because they’re atomic elements of the input, they also participate in automatic whitespace skipping: after every token, lexy will automatically skip whitespace, if one has been defined.

The parse context stores state that can be accessed during parsing. This includes things like the current recursion depth, see lexy::dsl::recurse, whether or not automatic whitespace skipping is currently enabled, see whitespace skipping, but also arbitrary user-defined variables, see lexy::dsl::context_flag, lexy::dsl::context_counter, and lexy::dsl::context_identifier.

When a rule modifies the context during parsing, by adding an additional context variable for example, this modification is available for all following rules in the current production and all child productions. In particular, the modification is no longer visible in any parent production. If a rule is parsed in a loop, e.g. by lexy::dsl::loop or lexy::dsl::list, any context modification does not persist between loop iterations, and is also not available outside the loop.

How to read the DSL documentation [1]

The behavior of a rule is described by the following sections.

Matching/parsing

This section describes what input is matched for the rule to succeed, and what is consumed. For token rules it is called "matching", otherwise "parsing".

It often delegates to the behavior of other rules: Here, the term "parsing" refers to the parsing operation of a rule, "branch parsing" or "try to parse" refers to the special parsing operation of a branch rule, which can backtrack, "matching" refers to the parsing operation of a token rule, which cannot produce values, and "try matching" refers to the branch parsing operation of a token rule, which cannot produce values or raise errors.

Branch parsing

This section describes what input is matched, consumed, and leads to a backtracking for a branch rule. Note that a rule can parse something different here than during non-branch parsing.

Errors

This section describes what errors are raised, when, and where. It also describes whether the rule can recover after the error.

Values

This section describes what values are produced during a successful parsing operation. It is omitted for token rules, which never produce values.

Parse tree

This section describes what nodes are created in the lexy::parse_tree. If omitted, a token rule creates a single token node covering everything consumed, and a rule produces no extra nodes besides the ones created by the other rules it parses.

If a rule parses another rule in a new context (e.g. lexy::dsl::peek), the other rule does not have access to context variables, and any context modification is not visible outside of the rule.

The rule DSL

Primitive rules
lexy::dsl::any

match anything

lexy::dsl::eof

match EOF

lexy::dsl::newline and lexy::dsl::eol

match the end of a line

Literal rules
lexy::dsl::lit_c

match a single character

lexy::dsl::lit and LEXY_LIT

match character sequences

lexy::dsl::lit_b

match a sequence of bytes

lexy::dsl::lit_cp

match a code point with the specified value

punctuators

match common punctuation

lexy::dsl::literal_set and LEXY_LITERAL_SET

match one of the specified literals

lexy::dsl::followed_by and lexy::dsl::not_followed_by

ensure a literal is (not) followed by a char class

lexy::dsl::ascii::case_folding and lexy::dsl::unicode::simple_case_folding

match a literal case-insensitively

Char classes
lexy::dsl::code_point

match specific Unicode code points

lexy::dsl::ascii

match ASCII char classes

lexy::dsl::unicode

match Unicode char classes

lexy::dsl::operator/ (char class), lexy::dsl::operator- (unary), lexy::dsl::operator-, lexy::dsl::operator&

combine char classes

LEXY_CHAR_CLASS

create a named char class

Branch conditions
lexy::dsl::operator>>

add a branch condition to a rule

lexy::dsl::else_

branch condition that is always taken

lexy::dsl::peek and lexy::dsl::peek_not

check whether something matches without consuming it

lexy::dsl::lookahead

check whether something matches somewhere in the input without consuming it

Combinators
lexy::dsl::token

turn a rule into a token

lexy::dsl::operator+

parse a sequence of rules

lexy::dsl::operator|

parse one of the specified (branch) rules

lexy::dsl::combination and lexy::dsl::partial_combination

parse all (some) of the (branch) rules in arbitrary order

lexy::dsl::if_ and lexy::dsl::opt

parse a branch rule if its condition matches

lexy::dsl::loop

parse a rule repeatedly

lexy::dsl::while_ and lexy::dsl::while_one

parse a branch rule while its condition matches

lexy::dsl::list

parse a list of things

lexy::dsl::times and lexy::dsl::repeat

parse a rule N times

lexy::dsl::until

skip everything until a rule matches

Brackets and delimited
lexy::dsl::terminator

parse something that ends with a terminator

lexy::dsl::brackets

parse something surrounded by brackets

lexy::dsl::delimited and lexy::dsl::escape

parse everything between two delimiters, with optional escape sequences

Productions
lexy::dsl::p and lexy::dsl::recurse

parse another production

lexy::dsl::inline_

parse another production’s rule inline

lexy::dsl::return_

exit early from parsing a production

lexy::dsl::subgrammar

parse a production defined in a different source file

Values
lexy::dsl::capture

capture everything consumed by a token rule

lexy::dsl::position

produce the current input position

lexy::dsl::nullopt

produce an empty placeholder value

lexy::dsl::member

parse something into a member variable

lexy::dsl::scan

parse a completely user-defined rule

lexy::dsl::parse_as

parses a rule ensuring it always produces a specific value

Errors and error recovery
lexy::dsl::error

explicitly raise an error

lexy::dsl::must

raise an error if a branch backtracks

lexy::dsl::try_

recover from a failed rule

lexy::dsl::recover

recover by looking and then continuing with some other rule

lexy::dsl::find

recover by looking for synchronization tokens

Whitespace
lexy::dsl::whitespace

explicitly skip whitespace

lexy::dsl::no_whitespace

do not skip whitespace

Identifiers
lexy::dsl::identifier

parse an identifier

lexy::dsl::keyword

parse a keyword

lexy::dsl::symbol

parse one of the specified symbols and produce their value

lexy::dsl::flag and lexy::dsl::flags

parses (multiple) symbols representing enum flags in any order

Numbers
lexy::dsl::zero

parse zero

lexy::dsl::digit

parse a digit

lexy::dsl::digits

parse one or more digits

lexy::dsl::n_digits

parse N digits

lexy::dsl::integer

convert digits to an integer

lexy::dsl::sign, lexy::dsl::plus_sign and lexy::dsl::minus_sign

parse a sign

lexy::dsl::code_point_id

convert N digits into a code point

Operator precedence parsing
lexy::dsl::op

parse an operator

lexy::dsl::operator/ (operator)

parse one of multiple operators

expression

parse an expression consisting of multiple operators

Context-sensitive parsing
lexy::dsl::context_flag

a boolean flag

lexy::dsl::context_counter

an integer counter

lexy::dsl::context_identifier

an identifier variable

Byte input
lexy::dsl::bytes and lexy::dsl::padding_bytes

parse N bytes

lexy::dsl::bint8, lexy::dsl::bint16, …​

parse a little/big endian integer

lexy::dsl::bits

parse a byte with specific bit patterns

lexy::dsl::bom

parse a byte-order mark (BOM)

Input and action specific rules
lexy::dsl::argv_separator

match the argument separator of a lexy::argv_input

lexy::dsl::tnode and lexy::dsl::pnode

match a node of a lexy::parse_tree_input

lexy::dsl::debug

generate a debug event that is visualized by lexy::trace