Header lexy/dsl/token.hpp

The basic token interface as well as the token token rule.

lexy/dsl/token.hpp
template <typename T>
concept token-rule = rule<T> && …;

A token is an atomic element of the input, which can be parsed by a token rule. Token rules are special rules that do not produce any values and can be matched efficiently. As such, every token rule is also a branch rule.

In a parse tree, every token rule produces a token node which directly captures a substring of the input, the part consumed by the rule. The token node also has a token kind, which identifies a token rule.

As token rules are the atomic elements, they also serve the basis for whitespace  handling: if automatic whitespace skipping has been enabled, it is skipped after every token rule.

Token interface .error<Tag>

lexy/dsl/token.hpp
class token-rule
{
public:
    template <typename Tag>
    static constexpr token-rule auto error;

    …
};

Every token rule token has a .error<Tag> member that overrides the error raised when token does not match.

Matching

Matches and consumes token.

Errors

Tag: if matching token fails. Its range covers everything already consumed by token. The rule then fails as well.

Example 1. A string literal with a nice error
struct production
{
    struct invalid_character
    {
        static constexpr auto name = "invalid character";
    };

    static constexpr auto rule = [] {
        // Arbitrary code points that aren't control characters.
        auto c = (-dsl::ascii::control).error<invalid_character>;

        return dsl::quoted(c);
    }();
};
Caution
.error<Tag> overrides all errors raised by a token rule to the same error. If a token can fail with different kinds of errors, this information is lost.

Token interface .kind<Tag>

lexy/dsl/token.hpp
class token-rule
{
public:
    template <auto Kind>
    static constexpr token-rule auto kind;

    …
};

Every token rule token has a .kind<Kind> member that overrides the token kind of its node in the parse tree.

Matching

Matches and consumes token.

Errors

All errors raised by token. The rule then fails.

Parse tree

A single token whose range covers everything consumed by token. However, its kind is now Kind.

Token rule lexy::dsl::token

lexy/dsl/token.hpp
namespace lexy
{
    struct missing_token {};
}

namespace lexy::dsl
{
    constexpr token-rule auto token(rule auto rule);
}

token is a rule that converts an arbitrary rule into a token rule.

Matching

Matches and consumes rule in a new context.

Errors

lexy::missing_token: if matching of rule has failed for any reason; its range covers everything rule has already consumed. The rule then fails.

Note
Whitespace skipping  is disabled while parsing rule, but it is skipped at the end, as it is a token rule.
Caution
While token is optimized itself, parsing rule can be arbitrarily expensive. As such, it should only be used when absolutely required.
Warning
rule does not have access to any context variables created by the context-sensitive parsing facilities and it can’t use recursion.

See also