complete sphinx autodoc

docs/classes.rst

@@ -24,33 +24,23 @@ Token
 
 .. autoclass:: lark.Token
 
-Visitor
--------
+Transformer, Vistor & Interpretor
+---------------------------------
 
-.. autoclass:: lark.visitors.VisitorBase
+See :doc:`visitors`.
 
-.. autoclass:: lark.visitors.Visitor
+UnexpectedInput
+---------------
 
-.. autoclass:: lark.visitors.Visitor_Recursive
+.. autoclass:: lark.exceptions.UnexpectedInput
+    :members: get_context, match_examples
 
-Interpreter
------------
-
-.. autoclass:: lark.visitors.Interpreter
-
-
-Transformer
------------
-
-.. autoclass:: lark.visitors.Transformer
-    :members: __default__, __default_token__
-
-v_args
-------
+.. autoclass:: lark.exceptions.UnexpectedToken
 
-.. autofunction:: lark.visitors.v_args
+.. autoclass:: lark.exceptions.UnexpectedCharacters
 
-Discard
--------
+ParserPuppet
+------------
 
-.. autoclass:: lark.visitors.Discard 
+.. autoclass:: lark.parsers.lalr_puppet.ParserPuppet
+    :members: choices, feed_token, copy, pretty, resume_parse

docs/visitors.md

@@ -1,148 +0,0 @@
-# Transformers & Visitors
-
-Transformers & Visitors provide a convenient interface to process the parse-trees that Lark returns.
-
-They are used by inheriting from the correct class (visitor or transformer), and implementing methods corresponding to the rule you wish to process. Each method accepts the children as an argument. That can be modified using the `v_args` decorator, which allows to inline the arguments (akin to `*args`), or add the tree `meta` property as an argument.
-
-See: <a href="https://github.com/lark-parser/lark/blob/master/lark/visitors.py">visitors.py</a>
-
-### Visitors
-
-Visitors visit each node of the tree, and run the appropriate method on it according to the node's data.
-
-They work bottom-up, starting with the leaves and ending at the root of the tree.
-
-**Example:**
-```python
-class IncreaseAllNumbers(Visitor):
-  def number(self, tree):
-    assert tree.data == "number"
-    tree.children[0] += 1
-
-IncreaseAllNumbers().visit(parse_tree)
-```
-
-There are two classes that implement the visitor interface:
-
-* Visitor - Visit every node (without recursion)
-
-* Visitor_Recursive - Visit every node using recursion. Slightly faster.
-
-### Interpreter
-
-The interpreter walks the tree starting at the root (top-down).
-
-For each node, it calls the method corresponding with its `data` attribute.
-
-Unlike Transformer and Visitor, the Interpreter doesn't automatically visit its sub-branches.
-The user has to explicitly call `visit`, `visit_children`, or use the `@visit_children_decor`.
-This allows the user to implement branching and loops.
-
-**Example:**
-```python
-class IncreaseSomeOfTheNumbers(Interpreter):
-  def number(self, tree):
-    tree.children[0] += 1
-
-  def skip(self, tree):
-      # skip this subtree. don't change any number node inside it.
-      pass
-
-IncreaseSomeOfTheNumbers().visit(parse_tree)
-```
-
-### Transformers
-
-Transformers visit each node of the tree, and run the appropriate method on it according to the node's data.
-
-They work bottom-up (or: depth-first), starting with the leaves and ending at the root of the tree.
-
-Transformers can be used to implement map & reduce patterns.
-
-Because nodes are reduced from leaf to root, at any point the callbacks may assume the children have already been transformed (if applicable).
-
-Transformers can be chained into a new transformer by using multiplication.
-
-`Transformer` can do anything `Visitor` can do, but because it reconstructs the tree, it is slightly less efficient.
-
-
-**Example:**
-```python
-from lark import Tree, Transformer
-
-class EvalExpressions(Transformer):
-    def expr(self, args):
-            return eval(args[0])
-
-t = Tree('a', [Tree('expr', ['1+2'])])
-print(EvalExpressions().transform( t ))
-
-# Prints: Tree(a, [3])
-```
-
-All these classes implement the transformer interface:
-
-- Transformer - Recursively transforms the tree. This is the one you probably want.
-- Transformer_InPlace - Non-recursive. Changes the tree in-place instead of returning new instances
-- Transformer_InPlaceRecursive - Recursive. Changes the tree in-place instead of returning new instances
-
-### visit_tokens
-
-By default, transformers only visit rules. `visit_tokens=True` will tell Transformer to visit tokens as well. This is a slightly slower alternative to `lexer_callbacks`, but it's easier to maintain and works for all algorithms (even when there isn't a lexer).
-
-**Example:**
-
-```python
-class T(Transformer):
-    INT = int
-    NUMBER = float
-    def NAME(self, name):
-        return lookup_dict.get(name, name)
-
-
-T(visit_tokens=True).transform(tree)
-```
-
-
-### v_args
-
-`v_args` is a decorator.
-
-By default, callback methods of transformers/visitors accept one argument: a list of the node's children. `v_args` can modify this behavior.
-
-When used on a transformer/visitor class definition, it applies to all the callback methods inside it.
-
-`v_args` accepts one of three flags:
-
-- `inline` - Children are provided as `*args` instead of a list argument (not recommended for very long lists).
-- `meta` - Provides two arguments: `children` and `meta` (instead of just the first)
-- `tree` - Provides the entire tree as the argument, instead of the children.
-
-**Examples:**
-
-```python
-@v_args(inline=True)
-class SolveArith(Transformer):
-    def add(self, left, right):
-        return left + right
-
-
-class ReverseNotation(Transformer_InPlace):
-    @v_args(tree=True)
-    def tree_node(self, tree):
-        tree.children = tree.children[::-1]
-```
-
-### `__default__` and `__default_token__`
-These are the functions that are called on if a function with a corresponding name has not been found.
-
-- The `__default__` method has the signature `(data, children, meta)`, with `data` being the data attribute of the node. It defaults to reconstruct the Tree
-
-- The `__default_token__` just takes the `Token` as an argument. It defaults to just return the argument.
-
-
-### Discard
-
-When raising the `Discard` exception in a transformer callback, that node is discarded and won't appear in the parent.
-
-

docs/visitors.rst

@@ -0,0 +1,46 @@
+Transformers & Visitors
+=======================
+
+Transformers & Visitors provide a convenient interface to process the
+parse-trees that Lark returns.
+
+They are used by inheriting from the correct class (visitor or transformer),
+and implementing methods corresponding to the rule you wish to process. Each
+method accepts the children as an argument. That can be modified using the
+`v_args` decorator, which allows to inline the arguments (akin to `*args`),
+or add the tree `meta` property as an argument.
+
+See: `visitors.py`_
+
+.. _visitors.py: https://github.com/lark-parser/lark/blob/master/lark/visitors.py
+
+Visitor
+-------
+
+.. autoclass:: lark.visitors.VisitorBase
+
+.. autoclass:: lark.visitors.Visitor
+
+.. autoclass:: lark.visitors.Visitor_Recursive
+
+
+Transformer
+-----------
+
+.. autoclass:: lark.visitors.Transformer
+    :members: __default__, __default_token__
+
+Interpreter
+-----------
+
+.. autoclass:: lark.visitors.Interpreter
+
+v_args
+------
+
+.. autofunction:: lark.visitors.v_args
+
+Discard
+-------
+
+.. autoclass:: lark.visitors.Discard 

lark/exceptions.py

@@ -24,9 +24,24 @@ class UnexpectedEOF(ParseError):
 
 
 class UnexpectedInput(LarkError):
+    """UnexpectedInput Error.
+
+    - ``UnexpectedToken``: The parser recieved an unexpected token
+    - ``UnexpectedCharacters``: The lexer encountered an unexpected string
+
+    After catching one of these exceptions, you may call the following
+    helper methods to create a nicer error message.
+    """
     pos_in_stream = None
 
     def get_context(self, text, span=40):
+        """Returns a pretty string pinpointing the error in the text,
+        with span amount of context characters around it.
+
+        Note:
+            The parser doesn't hold a copy of the text it has to parse,
+            so you have to provide it again
+        """
         pos = self.pos_in_stream
         start = max(pos - span, 0)
         end = pos + span
@@ -40,11 +55,22 @@ class UnexpectedInput(LarkError):
             return (before + after + b'\n' + b' ' * len(before) + b'^\n').decode("ascii", "backslashreplace")
 
     def match_examples(self, parse_fn, examples, token_type_match_fallback=False, use_accepts=False):
-        """ Given a parser instance and a dictionary mapping some label with
-            some malformed syntax examples, it'll return the label for the
-            example that bests matches the current error.
-
-            It's recommended to call this with `use_accepts=True`. The default is False for backwards compatibility.
+        """Allows you to detect what's wrong in the input text by matching
+        against example errors.
+        
+        Given a parser instance and a dictionary mapping some label with
+        some malformed syntax examples, it'll return the label for the
+        example that bests matches the current error. The function will
+        iterate the dictionary until it finds a matching error, and
+        return the corresponding value.
+
+        For an example usage, see examples/error_reporting_lalr.py
+
+        Args:
+            parse_fn: parse function (usually ``lark_instance.parse``)
+            examples: dictionary of ``{'example_string': value}``.
+            use_accepts: Recommended to call this with ``use_accepts=True``.
+                The default is ``False`` for backwards compatibility.
         """
         assert self.state is not None, "Not supported for this exception"
 
@@ -109,8 +135,13 @@ class UnexpectedCharacters(LexError, UnexpectedInput):
         super(UnexpectedCharacters, self).__init__(message)
 
 
-
 class UnexpectedToken(ParseError, UnexpectedInput):
+    """When the parser throws UnexpectedToken, it instanciates a puppet
+    with its internal state. Users can then interactively set the puppet to
+    the desired puppet state, and resume regular parsing.
+
+    see: ``ParserPuppet``.
+    """
     def __init__(self, token, expected, considered_rules=None, state=None, puppet=None):
         self.line = getattr(token, 'line', '?')
         self.column = getattr(token, 'column', '?')
@@ -132,6 +163,7 @@ class UnexpectedToken(ParseError, UnexpectedInput):
 
         super(UnexpectedToken, self).__init__(message)
 
+
 class VisitError(LarkError):
     """VisitError is raised when visitors are interrupted by an exception
 

lark/parsers/lalr_puppet.py

@@ -7,6 +7,12 @@ from .. import Token
 
 
 class ParserPuppet(object):
+    """ParserPuppet gives you advanced control over error handling when
+    parsing with LALR.
+
+    For a simpler, more streamlined interface, see the ``on_error``
+    argument to ``Lark.parse()``.
+    """
     def __init__(self, parser, state_stack, value_stack, start, stream, set_state):
         self.parser = parser
         self._state_stack = state_stack
@@ -18,8 +24,10 @@ class ParserPuppet(object):
         self.result = None
 
     def feed_token(self, token):
-        """Advance the parser state, as if it just received `token` from the lexer
+        """Feed the parser with a token, and advance it to the next state,
+        as if it recieved it from the lexer.
 
+        Note that ``token`` has to be an instance of ``Token``.
         """
         end_state = self.parser.parse_table.end_states[self._start]
         state_stack = self._state_stack
@@ -59,6 +67,10 @@ class ParserPuppet(object):
         value_stack.append(token)
 
     def copy(self):
+        """Create a new puppet with a separate state.
+
+        Calls to feed_token() won't affect the old puppet, and vice-versa.
+        """
         return type(self)(
             self.parser,
             list(self._state_stack),
@@ -69,6 +81,7 @@ class ParserPuppet(object):
         )
 
     def pretty(self):
+        """Print the output of ``choices()`` in a way that's easier to read."""
         out = ["Puppet choices:"]
         for k, v in self.choices().items():
             out.append('\t- %s -> %s' % (k, v))
@@ -76,6 +89,12 @@ class ParserPuppet(object):
         return '\n'.join(out)
 
     def choices(self):
+        """Returns a dictionary of token types, matched to their action in
+        the parser. Only returns token types that are accepted by the
+        current state.
+
+        Updated by ``feed_token()``.
+        """
         return self.parser.parse_table.states[self._state_stack[-1]]
 
     def accepts(self):
@@ -91,4 +110,8 @@ class ParserPuppet(object):
         return accepts
 
     def resume_parse(self):
-        return self.parser.parse(self._stream, self._start, self._set_state, self._value_stack, self._state_stack)
+        """Resume parsing from the current puppet state."""
+        return self.parser.parse(
+            self._stream, self._start, self._set_state,
+            self._value_stack, self._state_stack
+        )

lark/visitors.py

@@ -263,9 +263,7 @@ class VisitorBase:
     
     Run the appropriate method on it according to the node’s data.
     They work bottom-up, starting with the leaves and ending at the root
-    of the tree.
-
-    There are two classes that implement the visitor interface:
+    of the tree. There are two classes that implement the visitor interface:
 
     - ``Visitor``: Visit every node (without recursion)
     - ``Visitor_Recursive``: Visit every node using recursion. Slightly faster.