One function
_tokenize in justinmayer/smartypants.py
The author described this change as “fix hyphens do not cause a comment not a comment, add more tests”. It counts as a record because the check below fails on the code as it stood at 0134cea88 and passes on b00f2b51b, with nothing else changed between the two runs.
Projectjustinmayer/smartypants.py
Fix saved2013-09-21
Sharing licenceBSD-2-Clause · COPYING
Change size+22 −2
What the code was meant to do, written into the code itself as a docstring
Reference to an array of the tokens comprising the input string. Each token is either a tag (possibly with nested, tags contained therein, such as `<a href="<MTFoo>">`, or a run of text between tags. Each element of the array is a two-element array; the first is either 'tag' or 'text'; the second is the actual value. Based on the _tokenize() subroutine from `Brad Choate's MTRegex plugin`__. __ http://www.bradchoate.com/past/mtregex.php
The change
| 13 | 13 | ||
| 14 | 14 | tokens = [] | |
| 15 | 15 | ||
| 16 | - | tag_soup = re.compile(r"""(?s)([^<]*)(<!--.*?--\s*>|<[^>]*>)""") | |
| 16 | + | tag_soup = re.compile(r'([^<]*)(<!--.*?--\s*>|<[^>]*>)', re.S) | |
| 17 | 17 | ||
| 18 | 18 | token_match = tag_soup.search(text) | |
| 19 | 19 | ||
| ⋯ | |||
| 22 | 22 | if token_match.group(1): | |
| 23 | 23 | tokens.append(['text', token_match.group(1)]) | |
| 24 | 24 | ||
| 25 | - | tokens.append(['tag', token_match.group(2)]) | |
| 25 | + | # if -- in text part of comment, then it's not a comment, therefore it | |
| 26 | + | # should be converted. | |
| 27 | + | # | |
| 28 | + | # In HTML4 [1]: | |
| 29 | + | # [...] Authors should avoid putting two or more adjacent hyphens | |
| 30 | + | # inside comments. | |
| 31 | + | # | |
| 32 | + | # In HTML5 [2]: | |
| 33 | + | # [...] the comment may have text, with the additional restriction | |
| 34 | + | # that the text must not [...], nor contain two consecutive U+002D | |
| 35 | + | # HYPHEN-MINUS characters (--) | |
| 36 | + | # | |
| 37 | + | # [1]: http://www.w3.org/TR/REC-html40/intro/sgmltut.html#h-3.2.4 | |
| 38 | + | # [2]: http://www.w3.org/TR/html5/syntax.html#comments | |
| 39 | + | tag = token_match.group(2) | |
| 40 | + | type_ = 'tag' | |
| 41 | + | if tag.startswith('<!--'): | |
| 42 | + | # remove --[white space]> from the end of tag | |
| 43 | + | if '--' in tag[4:].rstrip('>').rstrip().rstrip('-'): | |
| 44 | + | type_ = 'text' | |
| 45 | + | tokens.append([type_, tag]) | |
| 26 | 46 | ||
| 27 | 47 | previous_end = token_match.end() | |
| 28 | 48 | token_match = tag_soup.search(text, token_match.end()) | |
The check that tells the two apart
fail→pass·tests/test.py::SmartyPantsTestCase::test_comments
Check file tests/test.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before it0134cea881640d9583339d997d517652de32450e
Broken version dated2013-09-20
Modulesmartypants
Units changed_tokenize
Fingerprint9e1ac7c0581f46e6
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.
Other bugs found in justinmayer/smartypants.py
- 2018-05-19convert_quotes
- 2013-09-19_tokenize
- 2013-08-12fix convert_quote, it was also turned on since False != "0"