Whole file
asottile/tokenize-rt
The author described this change as “fix token offsets following multiline string”. It counts as a record because the check below fails on the code as it stood at cc52eed14 and passes on 0c1b21317, with nothing else changed between the two runs.
Projectasottile/tokenize-rt
Fix saved2021-10-21
Sharing licenceMIT · LICENSE
Change size+4 −1
What the code was meant to do, written into the code itself as a save note
fix token offsets following multiline string
The change
| 126 | 126 | else: | |
| 127 | 127 | tokens.append(Token(tok_name, tok_text, sline, end_offset)) | |
| 128 | 128 | last_line, last_col = eline, ecol | |
| 129 | - | end_offset += len(tok_text.encode()) | |
| 129 | + | if sline != eline: | |
| 130 | + | end_offset = len(lines[last_line][:last_col].encode()) | |
| 131 | + | else: | |
| 132 | + | end_offset += len(tok_text.encode()) | |
| 130 | 133 | ||
| 131 | 134 | return tokens | |
| 132 | 135 |
The check that tells the two apart
fail→pass·tests/tokenize_rt_test.py::test_src_to_tokens_multiline_string
Check file tests/tokenize_rt_test.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before itcc52eed143799d7b3d3b300aa7a9b0db10d6ed95
Broken version dated2021-10-21
Moduletokenize_rt
Units changedsrc_to_tokens
Fingerprinte0b95c7d7d4eee6b
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.