Browse Source

Fixed bug where identical anonymous tokens got separate names

tags/gm/2021-09-23T00Z/github.com--lark-parser-lark/0.5.1
Erez Shinan 7 years ago
parent
commit
ee798f81f2
2 changed files with 4 additions and 1 deletions
  1. +1
    -1
      README.md
  2. +3
    -0
      lark/load_grammar.py

+ 1
- 1
README.md View File

@@ -73,7 +73,7 @@ These features are planned to be implemented in the near future:

## Comparison to other parsers

This is a feature comparison. For benchmarks vs pyparsing, check out the [JSON tutorial](/docs/json_tutorial.md#conclusion).
This is a feature comparison. For benchmarks vs other parsers, check out the [JSON tutorial](/docs/json_tutorial.md#conclusion).

| Library | Algorithm | LOC | Grammar | Builds tree?
|:--------|:----------|:----|:--------|:------------


+ 3
- 0
lark/load_grammar.py View File

@@ -261,6 +261,7 @@ class ExtractAnonTokens(InlineTransformer):

elif token.type == 'REGEXP':
token_name = 'ANONRE_%d' % self.i
value = token.value
self.i += 1
else:
assert False, x
@@ -268,6 +269,8 @@ class ExtractAnonTokens(InlineTransformer):
if token_name not in self.token_set:
self.token_set.add(token_name)
self.tokens.append((token_name, token, []))
assert value not in self.token_reverse
self.token_reverse[value] = token_name

return Token('TOKEN', token_name, -1)



Loading…
Cancel
Save