Erez Shinan
dd69b95338
Cleanup after pylint
6 years ago
Erez Shinan
a49df1b2e4
Added option to provide a custom lexer (with example)
6 years ago
Erez Shinan
6ea4588bcf
Dynamic lexer is now returns the maximum match only. Complete lexing beahvior moved to "dynamic_complete"
Changed dynamic lexer behavior to only match terminals to their maximum length (i.e. greedy match), emulating the standard lexer.
The original dynamic lexer behavior, that attempts to match all appearances of a terminal, has been moved to the "dynamic_complete" lexer.
For example, when applying a terminal "a"+ to the text "aaa":
- dynamic: ["aaa"]
- dynamic_complete: ["a", "aa", "aaa"]
6 years ago
Erez Shinan
5c6df8e825
Moved and restructured exceptions
* All exceptions are now under exceptions.py
* UnexpectedInput is now superclass of UnexpectedToken and UnexpectedCharacters,
all of which support the get_context() and match_examples() methods.
6 years ago
Erez Shinan
7b32ffd83a
Fixed token visibility rules (Issue #109 )
Anonymous tokens would become visible if they had the same value as named tokens.
That's because they are merged for the lexer. But after this change, the rules for
visibility are based on their use in the rule, and not their name or identity.
6 years ago
Erez Shinan
33caa391d5
Breaking backwards compatibility:
* Removed the scanless parsing feature (dynamic lexing is king)
* Default LALR lexer is now contextual
6 years ago
Erez Shinan
4a5aa745ea
All tests passing
6 years ago
Erez Shinan
cf7ddeee88
Earley working too
6 years ago
Erez Shinan
b951b5f79c
Significant reduction in memory consumption (Saving only parse-table instead of analysis instance)
6 years ago
Erez Shinan
6578742c06
BUGFIX: Undefined symbols (Issue #87 )
7 years ago
Erez Shinan
07f45a0bd3
Style fixes
7 years ago
Erez Shinan
6c28010581
Simplified grammar analysis. Improved performance of FIRST/FOLLOW calculations
7 years ago
Erez Shinan
22e525f53e
Fixed propagate positions. Added lexer_callbacks option.
7 years ago
Erez Shinan
3d8b8b4725
A few more adaptations
7 years ago
Erez Shinan
b36b8165c7
Small fix
7 years ago
Erez Shinan
748e9b7248
All relevant tests passing. Also indentation and other refactoring.
7 years ago
Erez Shinan
d173d6d66b
Validate against zero-width terminals in XEarley (Issue #63 )
7 years ago
Erez Shinan
37c1c0f65f
Better error message for bad regexps (Issue #62 )
7 years ago
Ehud Tamir
7c253b9372
Merge CYK parser.
7 years ago
Erez Shinan
39e58cb8fd
Post-refactor cleanup
7 years ago
Erez Shinan
1cc4c965e8
Big Refactor: Grammars now build in half the time. Code shorter & cleaner.
7 years ago
Erez Shinan
0ee80e675a
Refactoring for LALR, added the ParseTable class
7 years ago
Erez Shinan
5748920df4
BUGFIX in xearley + Feature: earley__predict_all
7 years ago
Erez Shinan
c93140386d
BUGFIX: The Earley parser collected repeating derivations, and filtered them out much later than needed. This resulted in very long lists and many unnecessary comparisons.
7 years ago
Erez Shinan
aede340449
Added earley__all_derivations due to performance concerns
7 years ago
Erez Shinan
5d41371fb3
Added resolve__antiscore_sum
7 years ago
Erez Shinan
d9b490f322
Refactoring for ambiguity resolution
7 years ago
Erez Shinan
5c4c320c26
Small fixes
7 years ago
Erez Shinan
0d5dd2bd0d
Improved README. Dynamic lexer now generates tokens.
7 years ago
Erez Shinan
dda0719375
Post-merge cleanup
7 years ago
Erez Shinan
4eec924403
Added prioritization to Earley. Use rule.1 etc. Highest priority will be selected in case of ambiguity.
7 years ago
Erez Shinan
e7edae0604
Working. All tests passing but two. Minor fixes probably.
7 years ago
Erez Shinan
8e2221b2fd
Mid-work. A nice first try
7 years ago
Erez Shinan
f374e70b2c
Feature: Added explicit ambiguity option for Earley
7 years ago
Erez Shinan
00811a7c16
Don't construct a list of tokens, to reduce memory footprint
8 years ago
Erez Shinan
a73cc9ad90
Re-wrote the Earley parser to use a parse-forest
It now knows how to resolve ambiguity! And in a memory-efficient way!
8 years ago
Erez Shinan
680449fb6b
A little bit of cleanup
8 years ago
Erez Shinan
5c115acbd9
Final fixes for scanless (hopefully)
8 years ago
Erez Shinan
d230e7744c
Pylint
8 years ago
Erez Shinan
a60c339ff7
Scanless Earley now working for all tests!
8 years ago
Erez Shinan
c9b45823ed
Better support for scanless parsing
8 years ago
Erez Shinan
a5a20a423a
Changed parser/lexer interface in lark. Bumped minor version
8 years ago
Erez Shinan
4076256faa
Reconstruct working again
8 years ago
Erez Shinan
c17558dd91
Official switched to my Earley implementation
8 years ago
Erez Shinan
538f944602
My Earley parser is working
8 years ago
Erez Shinan
e4d3e74f6a
Finished refactoring
8 years ago
Erez Shinan
d5123812b0
Another step of refactoring
8 years ago
Erez Shinan
0b902b1d03
Added the '!' prefix to keep all tokens in rule. Also removed tools (it's now in lark/tools)
8 years ago
Erez Shinan
d4425887d6
Added conf.py example and indenter support in contextual lexing
8 years ago
Erez Shinan
520ab88cd8
Added ContextualLexer for LALR. Working, but doesn't seem to improve speed.
Also some issues with python_parser
8 years ago