One function
_tokenize in justinmayer/smartypants.py
The author described this change as “Fix the primitive tokenizer to handle HTML comments that contain tags, otherwise this generates completely broken HTML by turning the end-comment into an en-dash.”. It counts as a record because the check below fails on the code as it stood at 19c07ad54 and passes on 901336f55, with nothing else changed between the two runs.
Projectjustinmayer/smartypants.py
Fix saved2013-09-19
Sharing licenceBSD-2-Clause · COPYING
Change size+1 −1
What the code was meant to do, written into the code itself as a docstring
Reference to an array of the tokens comprising the input string. Each token is either a tag (possibly with nested, tags contained therein, such as `<a href="<MTFoo>">`, or a run of text between tags. Each element of the array is a two-element array; the first is either 'tag' or 'text'; the second is the actual value. Based on the _tokenize() subroutine from `Brad Choate's MTRegex plugin`__. __ http://www.bradchoate.com/past/mtregex.php
The change
| 13 | 13 | ||
| 14 | 14 | tokens = [] | |
| 15 | 15 | ||
| 16 | - | tag_soup = re.compile('([^<]*)(<[^>]*>)') | |
| 16 | + | tag_soup = re.compile(r"""(?s)([^<]*)(<!--.*?--\s*>|<[^>]*>)""") | |
| 17 | 17 | ||
| 18 | 18 | token_match = tag_soup.search(text) | |
| 19 | 19 |
The check that tells the two apart
fail→pass·tests/test.py::SmartyPantsTestCase::test_comments
Check file tests/test.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before it19c07ad54c02c2b1f0802cb2dcff8e5cd450d4c6
Broken version dated2013-09-18
Modulesmartypants
Units changed_tokenize
Fingerprint5a98a9c85ffbf11e
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.
Other bugs found in justinmayer/smartypants.py
- 2018-05-19convert_quotes
- 2013-09-21_tokenize
- 2013-08-12fix convert_quote, it was also turned on since False != "0"