Erez Shinan
688c453456
Corrections
6 years ago
night199uk
80a09697fe
Rebuild the way Earley prioritizes ambiguities
- Makes rule ordering the default ambiguity tie breaker.
E.g.
start: a | b
a: "A"
b: "A"
will return:
start
a
start: b | a
a: "A"
b: "A"
will return
start
b
- Replaces the ambiguity='resolve__antiscore_sum' with a separate option: 'priority'.
The priority option has 4 values: 'auto', 'none', 'normal', 'invert'.
'Auto' maps to 'Normal' for CYK and Earley and 'None' for LALR.
'None' filters your priorities and ignores them. This saves some extra tree walking on Earley.
'Normal' uses your priorities untouched, mimicing the old behaviour.
'Invert' negates your priorities, emulating the old 'resolve__antiscore_sum' behaviour.
This allows you to use priority logic even when ambiguity=='explicit', to get a better idea
of the shape of your tree; and to easily disable priorities without removing them from the
grammar for testing (or performance).
- ambiguity='explicit' now correctly returns an ambiguous tree again, as 0.6 did.
6 years ago
evandrocoan
6c013b9a1d
Fix misspelling on lark/lark.py documentation
6 years ago
Erez Shinan
222df5bab4
New Feature: Added maybe_placeholders option (Issue #285 )
6 years ago
Erez Shinan
6290b1647d
Lark now works under Python -OO (Issue #283 )
6 years ago
Jan Rydzewski
3ef84318b1
Removed unused imports and variables
6 years ago
night199uk
1d5fd7301a
Heavy modifications to the Earley parser to try and make it handle more
types of ambiguity.
- Rewritten along the lines of Elizabeth Scott's parser.
https://www.sciencedirect.com/science/article/pii/S1571066108001497
- Implement SPPF trees per Elizabeth Scott and Bram van der Sanden's work.
http://www.bramvandersanden.com/post/2014/06/shared-packed-parse-forest/
6 years ago
evandrocoan
6fc2f5a6ba
Fix #231
6 years ago
Erez Shinan
cffd610e77
Fixed reconstructor. All tests passing
6 years ago
Erez Shinan
f99e2ecc86
Lark grammars are now utf8 by default (Issue #179 )
6 years ago
Erez Shinan
a49df1b2e4
Added option to provide a custom lexer (with example)
6 years ago
Erez Shinan
6ea4588bcf
Dynamic lexer is now returns the maximum match only. Complete lexing beahvior moved to "dynamic_complete"
Changed dynamic lexer behavior to only match terminals to their maximum length (i.e. greedy match), emulating the standard lexer.
The original dynamic lexer behavior, that attempts to match all appearances of a terminal, has been moved to the "dynamic_complete" lexer.
For example, when applying a terminal "a"+ to the text "aaa":
- dynamic: ["aaa"]
- dynamic_complete: ["a", "aa", "aaa"]
6 years ago
Erez Shinan
c0cf1b3176
Added some docstrings, removed is_terminal from common
6 years ago
Erez Shinan
33caa391d5
Breaking backwards compatibility:
* Removed the scanless parsing feature (dynamic lexing is king)
* Default LALR lexer is now contextual
6 years ago
Erez Shinan
4a7a66d773
.lark (preparing)
6 years ago
Erez Shinan
209a3fe8fd
Interface improvements for the Lark instance
6 years ago
Erez Shinan
25c3c51b1c
Fixed bug in Earley: A tree builder optimization clashed with explicit ambiguity
6 years ago
Erez Shinan
22e525f53e
Fixed propagate positions. Added lexer_callbacks option.
7 years ago
Ehud Tamir
7c253b9372
Merge CYK parser.
7 years ago
Erez Shinan
da1910f5b6
More refactoring towards standalone
7 years ago
Erez Shinan
07b5469e86
More refactoring, untangling grammar compilation and parse-tree creation
7 years ago
Erez Shinan
1cc4c965e8
Big Refactor: Grammars now build in half the time. Code shorter & cleaner.
7 years ago
Erez Shinan
5748920df4
BUGFIX in xearley + Feature: earley__predict_all
7 years ago
Erez Shinan
c93140386d
BUGFIX: The Earley parser collected repeating derivations, and filtered them out much later than needed. This resulted in very long lists and many unnecessary comparisons.
7 years ago
Erez Shinan
b3cb2f433b
Towards an introspectable tree-builder
7 years ago
Erez Shinan
19a9c9c206
Towards an introspectable tree-builder. Also added tests.
7 years ago
Erez Shinan
aede340449
Added earley__all_derivations due to performance concerns
7 years ago
Erez Shinan
7373993102
Oops!! Important bugfix to last commit
7 years ago
Erez Shinan
5d41371fb3
Added resolve__antiscore_sum
7 years ago
Erez Shinan
d9b490f322
Refactoring for ambiguity resolution
7 years ago
Erez Shinan
0c5acaab8a
Refactoring load_grammar
7 years ago
Erez Shinan
feb95ec64d
Earley now uses dynamic lexer by default. Bump to version 0.3.0
7 years ago
Erez Shinan
1da851516c
Added option: Keep all tokens
7 years ago
Erez Shinan
a588a70a7a
Added the experimental "propagate_positions" feature (only for standard lexer for now).
7 years ago
Erez Shinan
8e2221b2fd
Mid-work. A nice first try
7 years ago
Erez Shinan
f374e70b2c
Feature: Added explicit ambiguity option for Earley
7 years ago
Erez Shinan
a73cc9ad90
Re-wrote the Earley parser to use a parse-forest
It now knows how to resolve ambiguity! And in a memory-efficient way!
8 years ago
Erez Shinan
c1fbb3214c
Earley.lexer = None as default now
8 years ago
Erez Shinan
d230e7744c
Pylint
8 years ago
Erez Shinan
a60c339ff7
Scanless Earley now working for all tests!
8 years ago
Erez Shinan
df4d471641
Updated docs to match v0.2
8 years ago
Erez Shinan
c9b45823ed
Better support for scanless parsing
8 years ago
Erez Shinan
a5a20a423a
Changed parser/lexer interface in lark. Bumped minor version
8 years ago
Erez Shinan
c70355389f
Initial support for EBNF in tokens (automatic compilation to regexps)
8 years ago
Erez Shinan
f31482c236
Lint treatment
8 years ago
Erez Shinan
7af3de208d
Moved lexing responsibility to parser frontend
8 years ago
Erez Shinan
387fa07818
Moved unless handling to lexer and improved code
8 years ago
Erez Shinan
e7445a11ae
Added a docs/reference
8 years ago
Erez Shinan
8b9c5801da
Improved lexer, added profiler option to Lark
8 years ago
Erez Shinan
8b182b37c0
Added an automatic 'unless' mechanism for (some) colliding tokens
8 years ago