One function

Transformer in lark-parser/lark

The author described this change as Fix for issue #1394. It counts as a record because the check below fails on the code as it stood at 7646fb316 and passes on 544893115, with nothing else changed between the two runs.

Fix saved2024-02-25
Sharing licenceMIT · LICENSE
Change size+5 1

What the code was meant to do, written into the code itself as a docstring

Transformers work bottom-up (or depth-first), starting with visiting the leaves and working their way up until ending at the root of the tree. For each node visited, the transformer will call the appropriate method (callbacks), according to the node's `data`, and use the returned value to replace the node, thereby creating a new tree structure. Transformers can be used to implement map & reduce patterns. Because nodes are reduced from leaf to root, at any point the callbacks may assume the children have already been transformed (if applicable). If the transformer cannot find a method with the right name, it will instead call `__default__`, which by default creates a copy of the node. To discard a node, return Discard (`lark.visitors.Discard`). `Transformer` can do anything `Visitor` can do, but because it reconstructs the tree, it is slightly less efficient. A transformer without methods essentially performs a non-memoized partial deepcopy. All these classes implement the transformer interface: - `Transformer` - Recursively transforms the tree. This is the one you probably want. - `Transformer_InPlace` - Non-recursive. Changes the tree in-place instead of returning new instances - `Transformer_InPlaceRecursive` - Recursive. Changes the tree in-place instead of returning new instances Parameters: visit_tokens (bool, optional): Should the transformer visit tokens in addition to rules. Setting this to `False` is slightly faster. Defaults to `True`. (For processing ignored tokens, use the `lexer_callbacks` options)

The change

8585
8686 def transform(self, tree: Tree[_Leaf_T]) -> _Return_T:
8787 "Transform the given tree, and return the final result"
88- return self._transform_tree(tree)
88+ res = list(self._transform_children([tree]))
89+ if not res:
90+ return None # type: ignore[return-value]
91+ assert len(res) == 1
92+ return res[0]
8993
9094 def __mul__(
9195 self: 'Transformer[_Leaf_T, Tree[_Leaf_U]]',

The check that tells the two apart

failpass·tests/test_trees.py::TestTrees::test_transform_token

Check file tests/test_trees.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.

Origin and history

The code before it7646fb31609c6c83e0998b121eeffb908f5ec5ba
Broken version dated2024-02-02
Modulelark.visitors
Units changedTransformer
Fingerprintddbb413f71e602a7
Checked2026-08-18 by goldset/0.1

Every field above is generated by our program. None of it is written by hand.

Other bugs found in lark-parser/lark