Added a JSON parser tutorial

7 years ago · 1fc41c9d02
--- a/docs/json_tutorial.md
+++ b/docs/json_tutorial.md
@@ -0,0 +1,416 @@
 # Lark Tutorial - JSON parser

 Lark is a parser - a program that accepts a grammar and text, and produces a structured tree that represents that text.

 In this tutorial we will write a JSON parser in Lark, and explore Lark's various features in the process.

 It has 5 parts.

  1. Writing the grammar
  2. Creating the parser
  3. Shaping the tree
  4. Evaluating the tree
  5. Optimizing

 Knowledge assumed:
 - Using Python
 - A basic understanding of how to use regular expressions

 ## Part 1 - The Grammar

 Lark accepts its grammars in a format called [EBNF](https://www.wikiwand.com/en/Extended_Backus%E2%80%93Naur_form). It basically looks like this:

    rule_name: some rules and TOKENS
             | or others

    TOKEN: "some text to match"

 (*a token is a string or a regular expression*)

 How to structure those rules is beyond the scope of this tutorial, but it's often enough to follow one's intuition.

 In the case of JSON, the structure is simple: A json document is either a list, or a dictionary, or a string/number/etc.

 The dictionaries and lists are recursive, and contain other json documents (or "values").

 Let's write this structure in EBNF form:

    value: dict
         | list
         | string
         | number
         | "true" | "false" | "null"

    list : "[" [value ("," value)*] "]"

    dict : "{" [pair ("," pair)*] "}"
    pair : string ":" value


 A quick explanation of the syntax:
 - Parenthesis let us group rules together.
 - rule\* means *any amount*. That means, zero or more instances of that rule.
 - [rule] means *optional*. That means zero or one instance of that rule.

 Lark also supports the rule+ operator, meaning one or more instances.

 Of course, we still haven't defined "string" and "number".

 We'll do that now, and also take care of the white-space, which is part of the text.

    number : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
    string : /".*?(?<!\\)"/

    WS.ignore: /[ \t\n]+/

 These regular-expressions are a bit complex, but there's no simple way around it. We want to match "3.14" and also "-2e10", and that's just how it's done.

 Notice that WS, which matches whitespace, gets flagged with "ignore". This tells Lark not to pass it to the parser. Otherwise, we'd have to fill our grammar with WS tokens.

 ## Part 2 - Creating the Parser

 Once we have our grammar, creating the parser is very simple.

 We simply instanciate Lark, and tell it to accept a "value":

 ```python
 from lark import Lark
 json_parser = Lark(r"""
    value: dict
         | list
         | string
         | number
         | "true" | "false" | "null"

    list : "[" [value ("," value)*] "]"

    dict : "{" [pair ("," pair)*] "}"
    pair : string ":" value

    number : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
    string : /".*?(?<!\\)"/

    WS.ignore: /[ \t\n]+/

    """, start='value')
 ```

 It's that simple! Let's test it out:

 ```python
 >>> text = '{"key": ["item0", "item1", 3.14]}'
 >>> json_parser.parse(text)
 Tree(value, [Tree(dict, [Tree(pair, [Tree(string, [Token(ANONRE_1, "key")]), Tree(value, [Tree(list, [Tree(value, [Tree(string, [Token(ANONRE_1, "item0")])]), Tree(value, [Tree(string, [Token(ANONRE_1, "item1")])]), Tree(value, [Tree(number, [Token(ANONRE_0, 3.14)])])])])])])])
 >>> print( _.pretty() )
 value
  dict
    pair
      string	"key"
      value
        list
          value
            string	"item0"
          value
            string	"item1"
          value
            number	3.14
 ```

 As promised, Lark automagically creates a tree that represents the parsed text.

 But something is suspiciously missing from the tree. Where are the curly braces, the commas and all the other punctuation tokens?

 Lark automatically filters out tokens from the tree, based on the following criteria:

 - Filter out string tokens without a name, or with a name that starts with an underscore.
 - Keep regex tokens, even unnamed ones, unless their name starts with an underscore.

 This tutorial won't give an example of named tokens, but you can find such use in the [calculator example](/examples/calc.py).

 Unfortunately, this means that it will also filter out tokens like "true" and "false", and we will lose that information. The next section, "Shaping the tree" deals with this issue, and others.

 ## Part 3 - Shaping the Tree

 We now have a parser that can create a parse tree (or: AST), but the tree has some issues:

 1. "true", "false" and "null" are filtered out (test it out yourself!)
 2. Is has useless branches, like *value*, that clutter-up our view.

 I'll present the solution, and then explain it:

    ?value: dict
          | list
          | string
          | number
          | "true"             -> true
          | "false"            -> false
          | "null"             -> null

 1. Those little arrows signify *aliases*. An alias is a name for a specific part of the rule. In this case, we will name *true/false/null* matches, and this way we won't lose the information.

 2. The question mark prefixing *value* ("?value") tells the tree-builder to inline this branch if it has only one member. In this case, *value* will always have only one member.

 Here is the new grammar:

 ```python
 from lark import Lark
 json_parser = Lark(r"""
    ?value: dict
          | list
          | string
          | number
          | "true"             -> true
          | "false"            -> false
          | "null"             -> null

    list : "[" [value ("," value)*] "]"

    dict : "{" [pair ("," pair)*] "}"
    pair : string ":" value

    number : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
    string : /".*?(?<!\\)"/

    WS.ignore: /[ \t\n]+/

    """, start='value')
 ```

 And let's test it out:

 ```python
 >>> text = '{"key": ["item0", "item1", 3.14, true]}'
 >>> print( json_parser.parse(text).pretty() )
 dict
  pair
    string	"key"
    list
      string	"item0"
      string	"item1"
      number	3.14
      true
 ```

 Ah! That is much much nicer.

 ## Part 4 - Evaluating the tree

 It's nice to have a tree, but what we really want is a JSON object.

 The way to do it is to evaluate the tree, using a Transformer.

 A transformer is a class with methods corresponding to branch names. For each branch, the appropriate method will be called with the children of the branch as its argument, and its return value will replace the branch in the tree.

 So let's write a partial transformer, that handles lists and dictionaries:

 ```python
 from lark import Transformer

 class MyTransformer(Transformer):
    def list(self, items):
        return list(items)
    def pair(self, (k,v)):
        return k, v
    def dict(self, items):
        return dict(items)
 ```

 And when we run it, we get this:
 ```python
 >>> tree = json_parser.parse(text)
 >>> MyTransformer().transform(tree)
 {Tree(string, [Token(ANONRE_1, "key")]): [Tree(string, [Token(ANONRE_1, "item0")]), Tree(string, [Token(ANONRE_1, "item1")]), Tree(number, [Token(ANONRE_0, 3.14)]), Tree(true, [])]}
 ```

 This is pretty close. Let's write a full transformer that can handle the tokens too.

 Also, our definitions of list and dict are a bit verbose. We can do better:

 ```python
 from lark.tree import Transformer

 class TreeToJson(Transformer):
    def string(self, (s,)):
        return s[1:-1]
    def number(self, (n,)):
        return float(n)

    list = list
    pair = tuple
    dict = dict

    null = lambda self, _: None
    true = lambda self, _: True
    false = lambda self, _: False
 ```

 And when we run it:

 ```python
 >>> tree = json_parser.parse(text)
 >>> TreeToJson().transform(tree)
 {u'key': [u'item0', u'item1', 3.14, True]}
 ```
 Magic!

 ## Part 5 - Optimizing

 ### Step 1 - Benchmark

 By now, we have a fully working JSON parser, that can accept a string of JSON, and return its Pythonic representation.

 But how fast is it?

 Now, of course there are JSON libraries for Python written in C, and we can never compete with them. But since this is applicable to any parser you would write in Lark, let's see how far we can take this.

 The first step for optimizing is to have a benchmark. For this benchmark I'm going to take data from [json-generator.com/](http://www.json-generator.com/). I took their default suggestion and changed it to 5000 objects. The result is a 6.6MB sparse JSON file.

 Our first program is going to be just a concatanation of everything we've done so far:

 ```python
 import sys
 from lark import Lark, Transformer

 json_grammar = r"""
    ?value: dict
          | list
          | string
          | number
          | "true"             -> true
          | "false"            -> false
          | "null"             -> null

    list : "[" [value ("," value)*] "]"

    dict : "{" [pair ("," pair)*] "}"
    pair : string ":" value

    number : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
    string : /".*?(?<!\\)"/

    WS.ignore: /[ \t\n]+/
    """

 class TreeToJson(Transformer):
    def string(self, (s,)):
        return s[1:-1]
    def number(self, (n,)):
        return float(n)

    list = list
    pair = tuple
    dict = dict

    null = lambda self, _: None
    true = lambda self, _: True
    false = lambda self, _: False

 json_parser = Lark(json_grammar, start='value')

 if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        tree = json_parser.parse(f.read())
        print(TreeToJson().transform(tree))
 ```

 We run it and get this:

    $ time python tutorial_json.py json_data > /dev/null

    real	0m36.257s
    user	0m34.735s
    sys         0m1.361s


 That's unsatisfactory time for a 6MB file. Maybe if we were parsing configuration or a small DSL, but we're trying to handle large amount of data here.

 Well, turns out there's quite a bit we can do about it!

 ### Step 2 - LALR(1)

 So far we've been using the Earley algorithm, which is the default in Lark. Earley is powerful but slow. But it just so happens that our grammar is LR-compatible, and specifically LALR(1) compatible.

 So let's switch to LALR(1) and see what happens:

 ```python
 json_parser = Lark(json_grammar, start='value', parser='lalr')
 ```
    $ time python tutorial_json.py json_data > /dev/null

    real	0m7.722s
    user	0m7.504s
    sys 	0m0.175s

 Ah, that's much better. The resulting JSON is of course exactly the same. You can run it for yourself an see.

 It's important to note that not all grammars are LR-compatible, and so you can't always switch to LALR(1). But there's no harm in trying! If Lark lets you build the grammar, it means you're good to go.

 ### Step 3 - Tree-less LALR(1)

 So far, we've built a full parse tree for our JSON, and then transformed it. It's a convenient method, but it's not the most efficient in terms of speed and memory. Luckily, Lark lets us avoid building the tree when parsing with LALR(1).

 Here's the way to do it:

 ```python
 json_parser = Lark(json_grammar, start='value', parser='lalr', transformer=TreeToJson())

 if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        print( json_parser.parse(f.read()) )
 ```

 We've used the transformer we've already written, but this time we plug it straight into the parser. Now it can avoid building the parse tree, and just send the data straight into our transformer. The *parse()* method now returns the transformed JSON, instead of a tree.

 Let's benchmark it:

    real	0m4.866s
    user	0m4.722s
    sys 	0m0.121s

 That's a measurable improvement! Also, this way is more memory efficient. Check out the benchmark table at the end to see just how much.

 As a general practice, it's recommended to work with parse trees, and only skip the tree-builder when your transformer is already working.

 ### Step 4 - PyPy

 PyPy is a JIT engine for running Python, and it's designed to be a drop-in replacement.

 Lark is written purely in Python, which makes it very suitable for PyPy.

 Let's get some free performance:

    $ time pypy tutorial_json.py json_data > /dev/null

    real	0m1.397s
    user	0m1.296s
    sys 	0m0.083s

 PyPy is awesome!

 ### Conclusion

 We've brought the run-time down from 36 seconds to 1.4 seconds, in a series of small and simple steps.

 Now let's compare the benchmarks in a nicely organized table.

 I measured memory consumption using a little script called [memusg](https://gist.github.com/netj/526585)

 | Code | CPython Time | PyPy Time | CPython Mem | PyPy Mem
 |:-----|:-------------|:------------|:----------|:---------
 | Lark - Earley  | 36s | 4s | 6.2M | 1.2M |
 | Lark - LALR(1) | 7.2s | 1.3s | 0.6M | 0.3M |
 | Lark - LALR(1) tree-less | 4.3s | 1.2s | 0.4M | 0.3M |
 | PyParsing ([Parser](http://pyparsing.wikispaces.com/file/view/jsonParser.py)) | 32s | 4.1s | 0.4M | 0.2M |

 I added PyParsing for comparison. It fairs pretty well in its memory usage, but it can't compete with the run-time speed of LALR(1).

 Once again, shout-out to PyPy for being so effective.

 ## Afterword

 This is the end of the tutorial. I hoped you liked it and learned a little about Lark.

 To see what else you can do with Lark, check out the [examples](/examples).

 For questions or any other subject, feel free to email me at erezshin at gmail dot com.