Merge branch 'swuecho-transformer_doc'

docs/classes.md

@@ -1,15 +1,13 @@
-# Classes - Reference
+# Classes Reference
 
 This page details the important classes in Lark.
 
 ----
 
-## Lark
+## lark.Lark
 
 The Lark class is the main interface for the library. It's mostly a thin wrapper for the many different parsers, and for the tree constructor.
 
-### Methods
-
 #### \_\_init\_\_(self, grammar, **options)
 
 The Lark class accepts a grammar string or file object, and keyword options:
@@ -50,14 +48,10 @@ If a transformer is supplied to `__init__`, returns whatever is the result of th
 
 The main tree class
 
-### Properties
-
 * `data` - The name of the rule or alias
 * `children` - List of matched sub-rules and terminals
 * `meta` - Line & Column numbers, if using `propagate_positions`
 
-### Methods
-
 #### \_\_init\_\_(self, data, children)
 
 Creates a new tree, and stores "data" and "children" in attributes of the same name.
@@ -92,102 +86,6 @@ Trees can be hashed and compared.
 
 ----
 
-## Transformers & Visitors
-
-Transformers & Visitors provide a convenient interface to process the parse-trees that Lark returns.
-
-They are used by inheriting from the correct class (visitor or transformer), and implementing methods corresponding to the rule you wish to process. Each methods accepts the children as an argument. That can be modified using the `v_args` decorator, which allows to inline the arguments (akin to `*args`), or add the tree `meta` property as an argument.
-
-See: https://github.com/lark-parser/lark/blob/master/lark/visitors.py
-
-### Visitors
-
-Visitors visit each node of the tree, and run the appropriate method on it according to the node's data.
-
-They work bottom-up, starting with the leaves and ending at the root of the tree.
-
-**Example**
-```python
-class IncreaseAllNumbers(Visitor):
-  def number(self, tree):
-    assert tree.data == "number"
-    tree.children[0] += 1
-
-IncreaseAllNumbers().visit(parse_tree)
-```
-
-There are two classes that implement the visitor interface:
-
-* Visitor - Visit every node (without recursion)
-
-* Visitor_Recursive - Visit every node using recursion. Slightly faster.
-
-### Transformers
-
-Transformers visit each node of the tree, and run the appropriate method on it according to the node's data.
-
-They work bottom-up (or: depth-first), starting with the leaves and ending at the root of the tree.
-
-Transformers can be used to implement map & reduce patterns.
-
-Because nodes are reduced from leaf to root, at any point the callbacks may assume the children have already been transformed (if applicable).
-
-Transformers can be chained into a new transformer by using multiplication.
-
-**Example:**
-```python
-from lark import Tree, Transformer
-
-class EvalExpressions(Transformer):
-    def expr(self, args):
-            return eval(args[0])
-
-t = Tree('a', [Tree('expr', ['1+2'])])
-print(EvalExpressions().transform( t ))
-
-# Prints: Tree(a, [3])
-```
-
-
-Here are the classes that implement the transformer interface:
-
-- Transformer - Recursively transforms the tree. This is the one you probably want.
-- Transformer_InPlace - Non-recursive. Changes the tree in-place instead of returning new instances
-- Transformer_InPlaceRecursive - Recursive. Changes the tree in-place instead of returning new instances
-
-### v_args
-
-`v_args` is a decorator.
-
-By default, callback methods of transformers/visitors accept one argument: a list of the node's children. `v_args` can modify this behavior.
-
-When used on a transformer/visitor class definition, it applies to all the callback methods inside it.
-
-`v_args` accepts one of three flags:
-
-- `inline` - Children are provided as `*args` instead of a list argument (not recommended for very long lists).
-- `meta` - Provides two arguments: `children` and `meta` (instead of just the first)
-- `tree` - Provides the entire tree as the argument, instead of the children.
-
-Examples:
-
-```python
-@v_args(inline=True)
-class SolveArith(Transformer):
-    def add(self, left, right):
-        return left + right
-
-
-class ReverseNotation(Transformer_InPlace):
-    @v_args(tree=True):
-    def tree_node(self, tree):
-        tree.children = tree.children[::-1]
-```
-
-### Discard
-
-When raising the `Discard` exception in a transformer callback, that node is discarded and won't appear in the parent.
-
 ## Token
 
 When using a lexer, the resulting tokens in the trees will be of the Token class, which inherits from Python's string. So, normal string comparisons and operations will work as expected. Tokens also have other useful attributes:
@@ -199,17 +97,25 @@ When using a lexer, the resulting tokens in the trees will be of the Token class
 * `end_line` - The line where the token ends
 * `end_column` - The next column after the end of the token. For example, if the token is a single character with a `column` value of 4, `end_column` will be 5.
 
+## Transformer
+## Visitor
+## Interpreter
+
+See the [visitors page](visitors.md)
+
 
 ## UnexpectedInput
 
+## UnexpectedToken
+
+## UnexpectedException
+
 - `UnexpectedInput`
     - `UnexpectedToken` - The parser recieved an unexpected token
     - `UnexpectedCharacters` - The lexer encountered an unexpected string
 
 After catching one of these exceptions, you may call the following helper methods to create a nicer error message:
 
-### Methods
-
 #### get_context(text, span)
 
 Returns a pretty string pinpointing the error in the text, with `span` amount of context characters around it.

docs/visitors.md

@@ -0,0 +1,117 @@
+## Transformers & Visitors
+
+Transformers & Visitors provide a convenient interface to process the parse-trees that Lark returns.
+
+They are used by inheriting from the correct class (visitor or transformer), and implementing methods corresponding to the rule you wish to process. Each method accepts the children as an argument. That can be modified using the `v_args` decorator, which allows to inline the arguments (akin to `*args`), or add the tree `meta` property as an argument.
+
+See: <a href="https://github.com/lark-parser/lark/blob/master/lark/visitors.py">visitors.py</a>
+
+### Visitors
+
+Visitors visit each node of the tree, and run the appropriate method on it according to the node's data.
+
+They work bottom-up, starting with the leaves and ending at the root of the tree.
+
+**Example**
+```python
+class IncreaseAllNumbers(Visitor):
+  def number(self, tree):
+    assert tree.data == "number"
+    tree.children[0] += 1
+
+IncreaseAllNumbers().visit(parse_tree)
+```
+
+There are two classes that implement the visitor interface:
+
+* Visitor - Visit every node (without recursion)
+
+* Visitor_Recursive - Visit every node using recursion. Slightly faster.
+
+### Transformers
+
+Transformers visit each node of the tree, and run the appropriate method on it according to the node's data.
+
+They work bottom-up (or: depth-first), starting with the leaves and ending at the root of the tree.
+
+Transformers can be used to implement map & reduce patterns.
+
+Because nodes are reduced from leaf to root, at any point the callbacks may assume the children have already been transformed (if applicable).
+
+Transformers can be chained into a new transformer by using multiplication.
+
+`Transformer` can do anything `Visitor` can do, but because it reconstructs the tree, it is slightly less efficient.
+
+
+**Example:**
+```python
+from lark import Tree, Transformer
+
+class EvalExpressions(Transformer):
+    def expr(self, args):
+            return eval(args[0])
+
+t = Tree('a', [Tree('expr', ['1+2'])])
+print(EvalExpressions().transform( t ))
+
+# Prints: Tree(a, [3])
+```
+
+All these classes implement the transformer interface:
+
+- Transformer - Recursively transforms the tree. This is the one you probably want.
+- Transformer_InPlace - Non-recursive. Changes the tree in-place instead of returning new instances
+- Transformer_InPlaceRecursive - Recursive. Changes the tree in-place instead of returning new instances
+
+### visit_tokens
+
+By default, transformers only visit rules. `visit_tokens=True` will tell Transformer to visit tokens as well. This is a slightly slower alternative to `lexer_callbacks`, but it's easier to maintain and works for all algorithms (even when there isn't a lexer).
+
+Example:
+
+```python
+class T(Transformer):
+    INT = int
+    NUMBER = float
+    def NAME(self, name):
+        return lookup_dict.get(name, name)
+
+
+T(visit_tokens=True).transform(tree)
+```
+
+
+### v_args
+
+`v_args` is a decorator.
+
+By default, callback methods of transformers/visitors accept one argument: a list of the node's children. `v_args` can modify this behavior.
+
+When used on a transformer/visitor class definition, it applies to all the callback methods inside it.
+
+`v_args` accepts one of three flags:
+
+- `inline` - Children are provided as `*args` instead of a list argument (not recommended for very long lists).
+- `meta` - Provides two arguments: `children` and `meta` (instead of just the first)
+- `tree` - Provides the entire tree as the argument, instead of the children.
+
+Examples:
+
+```python
+@v_args(inline=True)
+class SolveArith(Transformer):
+    def add(self, left, right):
+        return left + right
+
+
+class ReverseNotation(Transformer_InPlace):
+    @v_args(tree=True):
+    def tree_node(self, tree):
+        tree.children = tree.children[::-1]
+```
+
+### Discard
+
+When raising the `Discard` exception in a transformer callback, that node is discarded and won't appear in the parent.
+
+

lark/visitors.py

@@ -186,6 +186,11 @@ class Visitor(VisitorBase):
             self._call_userfunc(subtree)
         return tree
 
+    def visit_topdown(self,tree):
+        for subtree in tree.iter_subtrees_topdown():
+            self._call_userfunc(subtree)
+        return tree        
+
 class Visitor_Recursive(VisitorBase):
     """Bottom-up visitor, recursive
 
@@ -198,8 +203,16 @@ class Visitor_Recursive(VisitorBase):
             if isinstance(child, Tree):
                 self.visit(child)
 
-        f = getattr(self, tree.data, self.__default__)
-        f(tree)
+        self._call_userfunc(tree)
+        return tree
+
+    def visit_topdown(self,tree):
+        self._call_userfunc(tree)
+
+        for child in tree.children:
+            if isinstance(child, Tree):
+                self.visit_topdown(child)
+        
         return tree
 
 

mkdocs.yml

@@ -9,5 +9,6 @@ pages:
     - How To Develop (Guide): how_to_develop.md
     - Grammar Reference: grammar.md
     - Tree Construction Reference: tree_construction.md
+    - Visitors and Transformers: visitors.md
     - Classes Reference: classes.md
     - Recipes: recipes.md

tests/test_trees.py

@@ -7,7 +7,7 @@ import pickle
 import functools
 
 from lark.tree import Tree
-from lark.visitors import Transformer, Interpreter, visit_children_decor, v_args, Discard
+from lark.visitors import Visitor, Visitor_Recursive, Transformer, Interpreter, visit_children_decor, v_args, Discard
 
 
 class TestTrees(TestCase):
@@ -34,6 +34,43 @@ class TestTrees(TestCase):
         nodes = list(self.tree1.iter_subtrees_topdown())
         self.assertEqual(nodes, expected)
 
+    def test_visitor(self):
+        class Visitor1(Visitor):
+            def __init__(self):
+                self.nodes=[]
+
+            def __default__(self,tree):
+                self.nodes.append(tree)
+        class Visitor1_Recursive(Visitor_Recursive):
+            def __init__(self):
+                self.nodes=[]
+
+            def __default__(self,tree):
+                self.nodes.append(tree)
+
+        visitor1=Visitor1()
+        visitor1_recursive=Visitor1_Recursive()
+
+        expected_top_down = [Tree('a', [Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z')]),
+                    Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z')]
+        expected_botton_up= [Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z'),
+                    Tree('a', [Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z')])]
+
+        visitor1.visit(self.tree1)
+        self.assertEqual(visitor1.nodes,expected_botton_up)
+
+        visitor1_recursive.visit(self.tree1)
+        self.assertEqual(visitor1_recursive.nodes,expected_botton_up)
+
+        visitor1.nodes=[]
+        visitor1_recursive.nodes=[]
+
+        visitor1.visit_topdown(self.tree1)
+        self.assertEqual(visitor1.nodes,expected_top_down)
+
+        visitor1_recursive.visit_topdown(self.tree1)
+        self.assertEqual(visitor1_recursive.nodes,expected_top_down)
+
     def test_interp(self):
         t = Tree('a', [Tree('b', []), Tree('c', []), 'd'])