Browse Source

Added a JSON parser tutorial

tags/gm/2021-09-23T00Z/github.com--lark-parser-lark/0.5.1
Erez Shinan 7 years ago
parent
commit
1fc41c9d02
1 changed files with 416 additions and 0 deletions
  1. +416
    -0
      docs/json_tutorial.md

+ 416
- 0
docs/json_tutorial.md View File

@@ -0,0 +1,416 @@
# Lark Tutorial - JSON parser

Lark is a parser - a program that accepts a grammar and text, and produces a structured tree that represents that text.

In this tutorial we will write a JSON parser in Lark, and explore Lark's various features in the process.

It has 5 parts.

1. Writing the grammar
2. Creating the parser
3. Shaping the tree
4. Evaluating the tree
5. Optimizing

Knowledge assumed:
- Using Python
- A basic understanding of how to use regular expressions

## Part 1 - The Grammar

Lark accepts its grammars in a format called [EBNF](https://www.wikiwand.com/en/Extended_Backus%E2%80%93Naur_form). It basically looks like this:

rule_name: some rules and TOKENS
| or others

TOKEN: "some text to match"

(*a token is a string or a regular expression*)

How to structure those rules is beyond the scope of this tutorial, but it's often enough to follow one's intuition.

In the case of JSON, the structure is simple: A json document is either a list, or a dictionary, or a string/number/etc.

The dictionaries and lists are recursive, and contain other json documents (or "values").

Let's write this structure in EBNF form:

value: dict
| list
| string
| number
| "true" | "false" | "null"

list : "[" [value ("," value)*] "]"

dict : "{" [pair ("," pair)*] "}"
pair : string ":" value


A quick explanation of the syntax:
- Parenthesis let us group rules together.
- rule\* means *any amount*. That means, zero or more instances of that rule.
- [rule] means *optional*. That means zero or one instance of that rule.

Lark also supports the rule+ operator, meaning one or more instances.

Of course, we still haven't defined "string" and "number".

We'll do that now, and also take care of the white-space, which is part of the text.

number : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
string : /".*?(?<!\\)"/

WS.ignore: /[ \t\n]+/

These regular-expressions are a bit complex, but there's no simple way around it. We want to match "3.14" and also "-2e10", and that's just how it's done.

Notice that WS, which matches whitespace, gets flagged with "ignore". This tells Lark not to pass it to the parser. Otherwise, we'd have to fill our grammar with WS tokens.

## Part 2 - Creating the Parser

Once we have our grammar, creating the parser is very simple.

We simply instanciate Lark, and tell it to accept a "value":

```python
from lark import Lark
json_parser = Lark(r"""
value: dict
| list
| string
| number
| "true" | "false" | "null"

list : "[" [value ("," value)*] "]"

dict : "{" [pair ("," pair)*] "}"
pair : string ":" value

number : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
string : /".*?(?<!\\)"/

WS.ignore: /[ \t\n]+/

""", start='value')
```

It's that simple! Let's test it out:

```python
>>> text = '{"key": ["item0", "item1", 3.14]}'
>>> json_parser.parse(text)
Tree(value, [Tree(dict, [Tree(pair, [Tree(string, [Token(ANONRE_1, "key")]), Tree(value, [Tree(list, [Tree(value, [Tree(string, [Token(ANONRE_1, "item0")])]), Tree(value, [Tree(string, [Token(ANONRE_1, "item1")])]), Tree(value, [Tree(number, [Token(ANONRE_0, 3.14)])])])])])])])
>>> print( _.pretty() )
value
dict
pair
string "key"
value
list
value
string "item0"
value
string "item1"
value
number 3.14
```

As promised, Lark automagically creates a tree that represents the parsed text.

But something is suspiciously missing from the tree. Where are the curly braces, the commas and all the other punctuation tokens?

Lark automatically filters out tokens from the tree, based on the following criteria:

- Filter out string tokens without a name, or with a name that starts with an underscore.
- Keep regex tokens, even unnamed ones, unless their name starts with an underscore.

This tutorial won't give an example of named tokens, but you can find such use in the [calculator example](/examples/calc.py).

Unfortunately, this means that it will also filter out tokens like "true" and "false", and we will lose that information. The next section, "Shaping the tree" deals with this issue, and others.

## Part 3 - Shaping the Tree

We now have a parser that can create a parse tree (or: AST), but the tree has some issues:

1. "true", "false" and "null" are filtered out (test it out yourself!)
2. Is has useless branches, like *value*, that clutter-up our view.

I'll present the solution, and then explain it:

?value: dict
| list
| string
| number
| "true" -> true
| "false" -> false
| "null" -> null

1. Those little arrows signify *aliases*. An alias is a name for a specific part of the rule. In this case, we will name *true/false/null* matches, and this way we won't lose the information.

2. The question mark prefixing *value* ("?value") tells the tree-builder to inline this branch if it has only one member. In this case, *value* will always have only one member.

Here is the new grammar:

```python
from lark import Lark
json_parser = Lark(r"""
?value: dict
| list
| string
| number
| "true" -> true
| "false" -> false
| "null" -> null

list : "[" [value ("," value)*] "]"

dict : "{" [pair ("," pair)*] "}"
pair : string ":" value

number : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
string : /".*?(?<!\\)"/

WS.ignore: /[ \t\n]+/

""", start='value')
```

And let's test it out:

```python
>>> text = '{"key": ["item0", "item1", 3.14, true]}'
>>> print( json_parser.parse(text).pretty() )
dict
pair
string "key"
list
string "item0"
string "item1"
number 3.14
true
```

Ah! That is much much nicer.

## Part 4 - Evaluating the tree

It's nice to have a tree, but what we really want is a JSON object.

The way to do it is to evaluate the tree, using a Transformer.

A transformer is a class with methods corresponding to branch names. For each branch, the appropriate method will be called with the children of the branch as its argument, and its return value will replace the branch in the tree.

So let's write a partial transformer, that handles lists and dictionaries:

```python
from lark import Transformer

class MyTransformer(Transformer):
def list(self, items):
return list(items)
def pair(self, (k,v)):
return k, v
def dict(self, items):
return dict(items)
```

And when we run it, we get this:
```python
>>> tree = json_parser.parse(text)
>>> MyTransformer().transform(tree)
{Tree(string, [Token(ANONRE_1, "key")]): [Tree(string, [Token(ANONRE_1, "item0")]), Tree(string, [Token(ANONRE_1, "item1")]), Tree(number, [Token(ANONRE_0, 3.14)]), Tree(true, [])]}
```

This is pretty close. Let's write a full transformer that can handle the tokens too.

Also, our definitions of list and dict are a bit verbose. We can do better:

```python
from lark.tree import Transformer

class TreeToJson(Transformer):
def string(self, (s,)):
return s[1:-1]
def number(self, (n,)):
return float(n)

list = list
pair = tuple
dict = dict

null = lambda self, _: None
true = lambda self, _: True
false = lambda self, _: False
```

And when we run it:

```python
>>> tree = json_parser.parse(text)
>>> TreeToJson().transform(tree)
{u'key': [u'item0', u'item1', 3.14, True]}
```
Magic!

## Part 5 - Optimizing

### Step 1 - Benchmark

By now, we have a fully working JSON parser, that can accept a string of JSON, and return its Pythonic representation.

But how fast is it?

Now, of course there are JSON libraries for Python written in C, and we can never compete with them. But since this is applicable to any parser you would write in Lark, let's see how far we can take this.

The first step for optimizing is to have a benchmark. For this benchmark I'm going to take data from [json-generator.com/](http://www.json-generator.com/). I took their default suggestion and changed it to 5000 objects. The result is a 6.6MB sparse JSON file.

Our first program is going to be just a concatanation of everything we've done so far:

```python
import sys
from lark import Lark, Transformer

json_grammar = r"""
?value: dict
| list
| string
| number
| "true" -> true
| "false" -> false
| "null" -> null

list : "[" [value ("," value)*] "]"

dict : "{" [pair ("," pair)*] "}"
pair : string ":" value

number : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
string : /".*?(?<!\\)"/

WS.ignore: /[ \t\n]+/
"""

class TreeToJson(Transformer):
def string(self, (s,)):
return s[1:-1]
def number(self, (n,)):
return float(n)

list = list
pair = tuple
dict = dict

null = lambda self, _: None
true = lambda self, _: True
false = lambda self, _: False

json_parser = Lark(json_grammar, start='value')

if __name__ == '__main__':
with open(sys.argv[1]) as f:
tree = json_parser.parse(f.read())
print(TreeToJson().transform(tree))
```

We run it and get this:

$ time python tutorial_json.py json_data > /dev/null

real 0m36.257s
user 0m34.735s
sys 0m1.361s


That's unsatisfactory time for a 6MB file. Maybe if we were parsing configuration or a small DSL, but we're trying to handle large amount of data here.

Well, turns out there's quite a bit we can do about it!

### Step 2 - LALR(1)

So far we've been using the Earley algorithm, which is the default in Lark. Earley is powerful but slow. But it just so happens that our grammar is LR-compatible, and specifically LALR(1) compatible.

So let's switch to LALR(1) and see what happens:

```python
json_parser = Lark(json_grammar, start='value', parser='lalr')
```
$ time python tutorial_json.py json_data > /dev/null

real 0m7.722s
user 0m7.504s
sys 0m0.175s

Ah, that's much better. The resulting JSON is of course exactly the same. You can run it for yourself an see.

It's important to note that not all grammars are LR-compatible, and so you can't always switch to LALR(1). But there's no harm in trying! If Lark lets you build the grammar, it means you're good to go.

### Step 3 - Tree-less LALR(1)

So far, we've built a full parse tree for our JSON, and then transformed it. It's a convenient method, but it's not the most efficient in terms of speed and memory. Luckily, Lark lets us avoid building the tree when parsing with LALR(1).

Here's the way to do it:

```python
json_parser = Lark(json_grammar, start='value', parser='lalr', transformer=TreeToJson())

if __name__ == '__main__':
with open(sys.argv[1]) as f:
print( json_parser.parse(f.read()) )
```

We've used the transformer we've already written, but this time we plug it straight into the parser. Now it can avoid building the parse tree, and just send the data straight into our transformer. The *parse()* method now returns the transformed JSON, instead of a tree.

Let's benchmark it:

real 0m4.866s
user 0m4.722s
sys 0m0.121s

That's a measurable improvement! Also, this way is more memory efficient. Check out the benchmark table at the end to see just how much.

As a general practice, it's recommended to work with parse trees, and only skip the tree-builder when your transformer is already working.

### Step 4 - PyPy

PyPy is a JIT engine for running Python, and it's designed to be a drop-in replacement.

Lark is written purely in Python, which makes it very suitable for PyPy.

Let's get some free performance:

$ time pypy tutorial_json.py json_data > /dev/null

real 0m1.397s
user 0m1.296s
sys 0m0.083s

PyPy is awesome!

### Conclusion

We've brought the run-time down from 36 seconds to 1.4 seconds, in a series of small and simple steps.

Now let's compare the benchmarks in a nicely organized table.

I measured memory consumption using a little script called [memusg](https://gist.github.com/netj/526585)

| Code | CPython Time | PyPy Time | CPython Mem | PyPy Mem
|:-----|:-------------|:------------|:----------|:---------
| Lark - Earley | 36s | 4s | 6.2M | 1.2M |
| Lark - LALR(1) | 7.2s | 1.3s | 0.6M | 0.3M |
| Lark - LALR(1) tree-less | 4.3s | 1.2s | 0.4M | 0.3M |
| PyParsing ([Parser](http://pyparsing.wikispaces.com/file/view/jsonParser.py)) | 32s | 4.1s | 0.4M | 0.2M |

I added PyParsing for comparison. It fairs pretty well in its memory usage, but it can't compete with the run-time speed of LALR(1).

Once again, shout-out to PyPy for being so effective.

## Afterword

This is the end of the tutorial. I hoped you liked it and learned a little about Lark.

To see what else you can do with Lark, check out the [examples](/examples).

For questions or any other subject, feel free to email me at erezshin at gmail dot com.


Loading…
Cancel
Save