Merge branch 'BlankSpruce-improved-iter-subtrees'

README.md

@@ -93,7 +93,7 @@ See more [examples here](https://github.com/lark-parser/lark/tree/master/example
  - **LALR(1)** parser
     - Fast and light, competitive with PLY
     - Can generate a stand-alone parser
- - **CYK** parser, for highly ambiguous grammars (NEW! Courtesy of [ehudt](https://github.com/ehudt))
+ - **CYK** parser, for highly ambiguous grammars
  - **EBNF** grammar
  - **Unicode** fully supported
  - **Python 2 & 3** compatible

docs/visitors.md

@@ -12,7 +12,7 @@ Visitors visit each node of the tree, and run the appropriate method on it accor
 
 They work bottom-up, starting with the leaves and ending at the root of the tree.
 
-**Example**
+**Example:**
 ```python
 class IncreaseAllNumbers(Visitor):
   def number(self, tree):
@@ -67,7 +67,7 @@ All these classes implement the transformer interface:
 
 By default, transformers only visit rules. `visit_tokens=True` will tell Transformer to visit tokens as well. This is a slightly slower alternative to `lexer_callbacks`, but it's easier to maintain and works for all algorithms (even when there isn't a lexer).
 
-Example:
+**Example:**
 
 ```python
 class T(Transformer):
@@ -95,7 +95,7 @@ When used on a transformer/visitor class definition, it applies to all the callb
 - `meta` - Provides two arguments: `children` and `meta` (instead of just the first)
 - `tree` - Provides the entire tree as the argument, instead of the children.
 
-Examples:
+**Examples:**
 
 ```python
 @v_args(inline=True)
@@ -105,11 +105,19 @@ class SolveArith(Transformer):
 
 
 class ReverseNotation(Transformer_InPlace):
-    @v_args(tree=True):
+    @v_args(tree=True)
     def tree_node(self, tree):
         tree.children = tree.children[::-1]
 ```
 
+### `__default__` and `__default_token__`
+These are the functions that are called on if a function with a corresponding name has not been found.
+
+- The `__default__` method has the signature `(data, children, meta)`, with `data` being the data attribute of the node. It defaults to reconstruct the Tree
+
+- The `__default_token__` just takes the `Token` as an argument. It defaults to just return the argument.
+
+
 ### Discard
 
 When raising the `Discard` exception in a transformer callback, that node is discarded and won't appear in the parent.

lark/tree.py

@@ -4,6 +4,7 @@ except ImportError:
     pass
 
 from copy import deepcopy
+from collections import OrderedDict
 
 
 ###{standalone
@@ -58,25 +59,15 @@ class Tree(object):
         return hash((self.data, tuple(self.children)))
 
     def iter_subtrees(self):
-        # TODO: Re-write as a more efficient version
-
-        visited = set()
-        q = [self]
-
-        l = []
-        while q:
-            subtree = q.pop()
-            l.append( subtree )
-            if id(subtree) in visited:
-                continue    # already been here from another branch
-            visited.add(id(subtree))
-            q += [c for c in subtree.children if isinstance(c, Tree)]
-
-        seen = set()
-        for x in reversed(l):
-            if id(x) not in seen:
-                yield x
-                seen.add(id(x))
+        queue = [self]
+        subtrees = OrderedDict()
+        for subtree in queue:
+            subtrees[id(subtree)] = subtree
+            queue += [c for c in reversed(subtree.children)
+                      if isinstance(c, Tree) and id(c) not in subtrees]
+
+        del queue
+        return reversed(list(subtrees.values()))
 
     def find_pred(self, pred):
         "Find all nodes where pred(tree) == True"