Changelog

Add constructor to lexy::input_location.

lexy::error_context::production will not be a transparent production.

Fix lexy::production_info::operator== when the compiler doesn’t merge string literals.

Fix SWAR matching of dsl::ascii::print and dsl::ascii::graph.

Fix CMake target installation (#108).

Change dsl::scan: it will now be invoked with the previously produced values.

Add dsl::parse_as to ensure that a rule always produces a value (e.g. when combined with the dsl::scan change above).

Add lexy::lexeme_input to support multi-pass parsing.

Turn dsl::terminator(term)(branch) into a branch rule, as opposed to being a plain rule (#74).

Add dsl::ignore_trailing_sep() separator.

Add lexy::bounded<T, Max> for bounded integer parsing (#72).

Add dsl::code_unit_id rule.

Turn lexy::forward<void> into a sink.

Support references in lexy::parse_result and lexy::scan_result

Fix bug that prevented lexy::parse with a root production whose value is void.

Fix bug that caused infinite template instantiations for recursive scans.

Fix bug that didn’t skip whitespace in lexy::scanner for token productions.

BEHAVIOR CHANGE: lexy::token_production that define a ::whitespace member now skip whitespace in the direct rule as well. Previously, it would only apply the whitespace rule to child productions but not the production itself.

BEHAVIOR CHANGE: production rules that define a ::whitespace member now skip whitespace before parsing. This also applies to the root production, so whitespace at the beginning of the input is now skipped automatically.

dsl::p<Production> where Production defines a ::whitespace member is now longer a branch rule: as it will now skip whitespace first, it can’t be used as a branch condition.

Remove dsl::whitespace (no arguments); it’s now unnecessary as initial whitespace is skipped automatically.

BEHAVIOR CHANGE: dsl::capture_token() is now dsl::capture(), old dsl::capture() is removed. If you’re using dsl::capture_token() you need to rename it to dsl::capture() (compile error). If you’re using dsl::capture() on a non-token rule, you need to use dsl::scan instead and manually produce the value (compile error). If you’re using dsl::capture() on a token, this will no longer capture trailing whitespace (silent behavior change). I can’t imagine a situation where capturing trailing whitespace was intended.

BEHAVIOR CHANGE: if a non-root production defines a ::whitespace member, it will now also apply to all children. Previously, it would only apply to the production that defined the member, and not it’s children (except if it was a token production).

BEHAVIOR CHANGE: dsl::newline (and dsl::eol in the newline case) generate a token node with the lexy::literal_token_kind; lexy::newline_token_kind and lexy::eol_token_kind have been removed.

dsl::eof and dsl::eol are now branch rules: replace dsl::until(dsl::eol) by dsl::until(dsl::newline).or_eof().

Removed generic dsl::operator/ (alternative): use dsl::literal_set() or dsl::operator| instead.

Require a char class rule in .limit() of dsl::delimited(): instead of dsl::eol or dsl::newline, use dsl::ascii::newline.

Require literal rules in dsl::lookahead(), dsl::find(), and .limit() of error recovery rules.

Require literal rules in .reserve() and variants of dsl::identifier.

dsl::bom now generates a lexy::expected_literal error instead of lexy::expected_char_class.

BEHAVIOR CHANGE: the introduction of char class rules changes error messages and token kinds in some situations.

Renamed dsl::code_point.lit<Cp>() to dsl::lit_cp<Cp> and moved to dsl/literal.hpp.

Require char classes in operator- for tokens; removed dsl::contains() and dsl::prefix().

Require char classes in dsl::delimited() and dsl::identifier().

Renamed .character_class() of dsl::error to .name().

Added more pre-defined token kinds: for example, tokens created by LEXY_LIT() now have their own literal token kind. This breaks code that does not use user-defined token kinds and does matching on lexy::parse_tree.

dsl::delimited() now merges adjacent characters into a single lexy::lexeme that is passed to the sink.

lexy::token_production now longer merges adjacent tokens, but dsl::delimited() merges character tokens.

Terminator rules are no longer branch rules; this behavior was somewhat confusing. If you need branch rules, you can manually write the equivalent rules.

dsl::integer() now requires a token rule. This ensures the correct behavior in combination with whitespace skipping.

BEHAVIOR CHANGE: branch parsing an identifier will now backtrack without raising an error if it can match an identifier, but it is reserved. Previously, this would not backtrack and then raise an error (but trivially recover). This behavior is consistent with dsl::symbol().

Removed branch functionality of token sequence (again). It was already removed once as it was unimplementable due to automatic whitespace skipping, but then re-implemented later on. But as it turns out, it is in fact unimplementable and the current implementation was completely broken. Instead of tok1 + tok2 >> rule1 | tok1 + tok3 >> rule2 use tok1 >> (tok2 >> rule1 | tok3 >> rule2).

Removed dsl::encode(). The rule was completely broken in combination with dsl::capture() and rules built on top like dsl::identifier().

BEHAVIOR CHANGE: error recovery now produces a new error token in the parse tree. This ensures that the parse tree stays lossless even in the presence of errors.

Potential pitfall: dsl::recover() and dsl::find() now always raise the recovery events. If you’re using them outside of dsl::try_(), this is not what you want, so don’t do them - they’re not meant for it.

Moved lexy/match.hpp, lexy/parse.hpp, and lexy/validate.hpp to lexy/action/match.hpp, lexy/action/parse.hpp and lexy/action/validate.hpp.

Moved lexy::parse_as_tree() to new header lexy/action/parse_as_tree.hpp; lexy::parse_tree stayed in lexy/parse_tree.hpp.

Renamed lexy::parse_tree::builder::backtrack_production to cancel_production, and its production_state to marker.

Moved callback adapters and composition into new header files, but still implicitly included by callback.hpp.

Removed overload of lexy::bind that takes a sink; bind individual items in a separate production instead.

Removed unneeded overloads of lexy::as_sink and changed the transcoding behavior: It will now only use the pointer + size constructor if the character types match and no longer reinterpret_cast.

Simplified and minimized interface of the input classes, removing e.g. iterators from them.

Moved definition of lexy::code_point from encoding.hpp to new header code_point.hpp.

Turned dsl::else_ into a tag object that can only be used with operator>>, instead of a stand-alone rule.

BEHAVIOR CHANGE: dsl::peek[_not]() and dsl::lookahead() are no longer no-ops when used outside a branch condition. Instead, they will perform lookahead and raise an error if that fails.

Removed dsl::require/prevent(rule).error<tag>; use dsl::peek[_not](rule).error<tag> instead.

Improved and simplified interface for dsl::context_flag and dsl::context_counter: instead of .select()/.compare(), you now use .is_set()/.is() as a branch condition, and instead of .require(), you now use dsl::must() with .is[_set]().

Removed dsl::context_lexeme; use dsl::context_identifier instead.

lexy::fold[_inplace] is now longer a callback, only a sink; use lexy::callback(lexy::fold(…​)) to turn it into a callback if needed.

Removed dsl::opt_list(); use dsl::opt(dsl::list()) instead.

BEHAVIOR CHANGE: .opt_list() of dsl::terminator/dsl::brackets now produces lexy::nullopt instead of an empty sink result if the list has no items. If you’re using pre-defined callbacks like lexy::as_list, lexy::as_collection, or lexy::as_string, it continues to work as expected. If you’re using sink >> callback, callback now requires one overload that takes lexy::nullopt.

Removed .while[_one]() from dsl::terminator/dsl::brackets.

Removed dsl::label and dsl::id; use a separate production instead.

Removed lexy::sink; instead of lexy::sink<T>(fn) use lexy::fold_inplace<T>({}, fn).

BEHAVIOR CHANGE: dsl::times/dsl::twice no longer produce an array, but instead all values individually. Use lexy::fold instead of a loop.

Removed lexy::null_input.

Downgraded lexy/input/shell.hpp to lexy_ext/shell.hpp, with the namespace change to lexy_ext.

Removed .capture() from dsl::code_point; use dsl::capture() instead.

BEHAVIOR CHANGE: Don’t produce a tag value if no sign was present in dsl::[minus/plus_]sign. If you use lexy::as_integer as callback, this doesn’t affect you.

BEHAVIOR CHANGE: Don’t consume input in dsl::prevent.

BEHAVIOR CHANGE: Produce only a single whitespace node in parse tree, instead of the individual token nodes. Prohibited dsl::p/dsl::recurse inside the whitespace rule.

Changed dsl::[plus/minus_]sign to produce lexy::plus/minus_sign instead of +1/-1. Also changed callback lexy::as_integer to adapt.

Removed dsl::parse_state and dsl::parse_state_member; use lexy::bind() with lexy::parse_state instead.

Removed dsl::value_* rules; use lexy::bind() or dsl::id/dsl::label instead.

The alternative rule / now tries to find the longest match instead of the first one. If it was well-specified before, this doesn’t change anything.

Removed dsl::switch_(); use the new dsl::symbol() instead which is more efficient as well.

Removed .lit[_c]() from dsl::escape(); use the new .symbol() instead.

Restructure callback header files; an #include <lexy/callback.hpp> might be necessary now.

Support empty token nodes in the parse tree if they don’t have an unknown kind. In particular, the parse tree will now contain an EOF node at the end.

Turn lexy::unknown_token_kind into a value (as opposed to the type it was before).

Changed the return type of lexy::read_file() (and lexy_ext::read_file()) to use a new lexy::read_file_result over lexy::result.

Changed the return type of lexy::validate() and lexy::parse_as_tree() to a new lexy::validate_result type.

Changed the return type of lexy::parse() to a new lexy::parse_result type.

Removed lexy::result.

An error callback that returns a non-void type must now be a sink. Use lexy::collect<Container>(error_callback) to create a sink that stores all results in the container. If the error callback returns void, no change is required.

Removed dsl::no_trailing_sep(); dsl::sep() now has that behavior as well.

dsl::require() and dsl::prevent() now recover from errors, which might lead to worse error messages in certain situations. If they’re used as intended — to create a better error message if something didn’t work out — this shouldn’t happen.

Removed empty state from lexy::result. It was only added because it was useful internally, but this is no longer the case.

Reverted optimization that merged multiple lexemes in the sink/tokens of dsl::delimited(). Tokens are instead now automatically merged by the parse tree builder if direct children of a lexy::token_production.

dsl::switch_(rule).case_() now requires a branch of the form token >> rule, previously it could take an arbitrary branch.

Unified error interface:

BEHAVIOR CHANGE: if dsl::code_point_id overflows, the tag is now lexy::invalid_code_point instead of lexy::integer_overflow.

Replaced use of lexy::_detail::string_view by const char* in all user facing functions. As a consequence, automatic type name now requires GCC > 8.

Removed lexy::make_error_location(). It has been replaced by lexy_ext::find_input_location().

Replaced operator[] and dsl::whitespaced() by new dsl::whitespace rule. Whitespace can now be parsed manually or automatically.

Removed support for choices in while, i.e. dsl::while_(a | b | c). This can be replaced by dsl::loop(a | b | c | dsl::break_).

Removed .check() from dsl::context_flag and .check_eq/lt/gt from dsl::context_counter due to implementation problems. Use .select() and .compare() instead.

A sequence rule using operator+ is now longer a branch. Previously, it was a branch if it consisted of only tokens. However, this was unimplementable in combination with automatic whitespace skipping.

Removed callback from lexy::as_list and lexy::as_collection; they’re now only sink. lexy::construct can be used in most cases instead.

Merged ::list and ::value callbacks from productions. There are three cases:

lexy::result now has an empty state. It is only used internally and never exposed to the user. As a related change, the default constructor has been removed due to unclear semantics. Use lexy::result(lexy::result_error) to restore its behavior of creating a default constructed error.

Replaced Pattern concept with a new Token and Branch concept (See #10). A Branch is a rule that can make branching decision (it is required by choices and can be used as branch condition). A Token is an atomic parse unit; it is also a Branch.

Removed dsl::match(); use dsl::token() instead. If you previously had dsl::peek(dsl::match(rule)) >> then you can now even use dsl::peek(rule) >> then, as dsl::peek[_not]() have learned to support arbitrary rules.

Removed dsl::try_<Tag>(pattern). If pattern is now a token, you can use rule.error<Tag>() instead. Otherwise, use dsl::token(pattern).error<Tag>().

Removed .capture() on dsl::sep(pattern) and dsl::trailing_sep(pattern). You can now use dsl::sep(dsl::capture(pattern)), as dsl::capture() is now a branch and the separators have learned to support branches.

Removed .zero() and .non_zero() from dsl::digit<Base>. Use dsl::zero instead of dsl::digit<Base>.zero(). Use dsl::digit<Base> - dsl::zero (potentially with a nice error specified using .error()) instead of dsl::digit<Base>.non_zero().

Removed dsl::success, as it is now longer needed internally. It can be added back if needed.

BEHAVIOR CHANGE: As part of the branch changes, dsl::peek(), dsl::peek_not() and dsl::lookahead() are now no-ops if not used as branch condition. For example, prefix + dsl::peek(rule) + suffix is equivalent to prefix + suffix. In most cases, this is only a change in the error message as they don’t consume characters. Use dsl::require() and dsl::prevent() if the lookahead was intended.

BEHAVIOR CHANGE: Errors in whitespace are currently not reported. For example, if you have /* unterminated C comment int i; and support space and C comments as whitespace, this would previously raise an error about the unterminated C comment. Right now, it will try to skip the C comment, fail, and then just be done with whitespace skipping. The error for the unterminated C comment then manifests as expected 'int', got '/*'.

Removed dsl::build_list() and dsl::item(). They were mainly used to implement dsl::list(), and became unnecessary after an internal restructuring.

Removed support for choices in lists, i.e. dsl::list(a | b | c). This can be added back if needed.

Removed dsl::operator! due to implementation problems. Existing uses of dsl::peek(!rule) can be replaced by dsl::peek_not(rule); existing uses of !rule >> do_sth can be replaced using dsl::terminator().