Erez Shinan
a49df1b2e4
Added option to provide a custom lexer (with example)
6 years ago
Erez Shinan
5c6df8e825
Moved and restructured exceptions
* All exceptions are now under exceptions.py
* UnexpectedInput is now superclass of UnexpectedToken and UnexpectedCharacters,
all of which support the get_context() and match_examples() methods.
6 years ago
Erez Shinan
c0cf1b3176
Added some docstrings, removed is_terminal from common
6 years ago
Erez Shinan
34cd792ffc
Fixed Python grammars, and a bug in newline detection
6 years ago
Erez Shinan
ac0d49e7ab
Added %declare
6 years ago
Erez Shinan
2b4ef11ebf
Columns now start at 1
6 years ago
Erez Shinan
4a5aa745ea
All tests passing
6 years ago
Erez Shinan
67f372c994
Symbols instead of strings - initial
6 years ago
Erez Shinan
0f0776c0fa
BUGIX in lexer: Embedding strings overwrote priority (Issue #121 )
6 years ago
Parker
138f1d5d76
Fix order of members when pickling Token
I found this while porting Token to C, essentially the value and pos_in_stream members of Token were swapped in ``__reduce__``, which means running ``pickle.loads`` and ``pickle.dumps`` would result in unpickled tokens whose value was the original's position in stream, and vice versa. In my C extension this caused a TypeError exception, but the behavior will have to be corrected in both.
6 years ago
Erez Shinan
f960c1b8ac
Initial: Added transformers.py, and Meta to tree
6 years ago
Erez Shinan
ba0dc789a3
Significantly better memory performance (Thanks @drslump!)
Added __slots__ to RulePtr and Token, resulting in significantly lower memory consumption.
As suggested by @drslump .
6 years ago
night199uk
7be2c04637
Ensure Tokens can be pickled correctly
7 years ago
Erez Shinan
327cca8c00
Added 'considered_rules' to exceptions, to help users debug
7 years ago
Erez Shinan
710cb6d20b
Added more information in UnexpectedInput exception (Issue #78 )
7 years ago
Erez Shinan
22e525f53e
Fixed propagate positions. Added lexer_callbacks option.
7 years ago
Erez Shinan
8acd77d7ff
Minor fixes in lexer
7 years ago
Erez Shinan
9b0672fda6
Standalone tools now supports postlex
7 years ago
Erez Shinan
da1910f5b6
More refactoring towards standalone
7 years ago
Erez Shinan
7182ba3991
Minor refactoring for the standalone tool (in progress)
7 years ago
Erez Shinan
38c5fd244a
Improved grammar validation and refactored the lexers
7 years ago
Erez Shinan
1cc4c965e8
Big Refactor: Grammars now build in half the time. Code shorter & cleaner.
7 years ago
Erez Shinan
f63174a22c
BUGFIX: Create unless returned duplicate tokens (Issue #56 )
7 years ago
Erez Shinan
852607b978
BUGFIX: Tokens of different type were equal, causing disambiguation errors (Issue #21 )
7 years ago
Erez Shinan
34449651bf
Added UnexpectedInput exception (with line & column) to xearley (Issue #43 )
7 years ago
Erez Shinan
08a8a747b8
Fixed escaping for all tests
7 years ago
Erez Shinan
7373993102
Oops!! Important bugfix to last commit
7 years ago
Erez Shinan
ac9ea26b58
Added token priority for standard lexers.
Big Thanks to Seth G of the mappyfile project for financing this feature!
7 years ago
Erez Shinan
188386cf04
Fixed main bug in test_token_collision2.
7 years ago
Erez Shinan
88242f10d7
Tiny fix in lexer
7 years ago
Erez Shinan
a588a70a7a
Added the experimental "propagate_positions" feature (only for standard lexer for now).
7 years ago
Erez Shinan
794a1c4965
More flags work
8 years ago
Erez Shinan
9d6b496f3e
Fixed bug in unless: Now uses regexp, not string matching
8 years ago
Erez Shinan
2dbf5d7b26
Little stuff
8 years ago
Erez Shinan
680449fb6b
A little bit of cleanup
8 years ago
Erez Shinan
5c115acbd9
Final fixes for scanless (hopefully)
8 years ago
Erez Shinan
96ebe94963
Solved issue #3 : infinite loop due to zero-length tokens
8 years ago
Erez Shinan
d230e7744c
Pylint
8 years ago
Erez Shinan
a60c339ff7
Scanless Earley now working for all tests!
8 years ago
Erez Shinan
a5a20a423a
Changed parser/lexer interface in lark. Bumped minor version
8 years ago
Erez Shinan
5236e4a32a
Refactored TokenDef to store Pattern(Str/RE)
8 years ago
Erez Shinan
c70355389f
Initial support for EBNF in tokens (automatic compilation to regexps)
8 years ago
Erez Shinan
e4d3e74f6a
Finished refactoring
8 years ago
Erez Shinan
d4425887d6
Added conf.py example and indenter support in contextual lexing
8 years ago
Erez Shinan
520ab88cd8
Added ContextualLexer for LALR. Working, but doesn't seem to improve speed.
Also some issues with python_parser
8 years ago
Erez Shinan
f31482c236
Lint treatment
8 years ago
Erez Shinan
387fa07818
Moved unless handling to lexer and improved code
8 years ago
Erez Shinan
a438255193
Fixed bug in lexer. Fixed calc example
8 years ago
Erez Shinan
772f1cc0aa
Improved error reporting
8 years ago
Erez Shinan
a98d3c93b1
Small additions to the docs
8 years ago