Whole file
asottile/tokenize-rt
The author described this change as “fix position of empty NEWLINE at no-eol eof”. It counts as a record because the check below fails on the code as it stood at 7f89433be and passes on 0d214dc28, with nothing else changed between the two runs.
Projectasottile/tokenize-rt
Fix saved2021-10-21
Sharing licenceMIT · LICENSE
Change size+8 −6
What the code was meant to do, written into the code itself as a save note
fix position of empty NEWLINE at no-eol eof
The change
| 88 | 88 | last_line += 1 | |
| 89 | 89 | if newtok: | |
| 90 | 90 | tokens.append(Token(UNIMPORTANT_WS, newtok, sline, 0)) | |
| 91 | + | end_offset = len(newtok.encode()) | |
| 92 | + | else: | |
| 93 | + | end_offset = 0 | |
| 91 | 94 | ||
| 92 | 95 | elif scol > last_col: | |
| 93 | - | tokens.append( | |
| 94 | - | Token(UNIMPORTANT_WS, line[last_col:scol], sline, end_offset), | |
| 95 | - | ) | |
| 96 | + | newtok = line[last_col:scol] | |
| 97 | + | tokens.append(Token(UNIMPORTANT_WS, newtok, sline, end_offset)) | |
| 98 | + | end_offset += len(newtok.encode()) | |
| 96 | 99 | ||
| 97 | 100 | tok_name = tokenize.tok_name[tok_type] | |
| 98 | - | utf8_byte_offset = len(line[:scol].encode()) | |
| 99 | 101 | # when a string prefix is not recognized, the tokenizer produces a | |
| 100 | 102 | # NAME token followed by a STRING token | |
| 101 | 103 | if ( | |
| ⋯ | |||
| 122 | 124 | ): | |
| 123 | 125 | tokens[-1] = tokens[-1]._replace(src=tokens[-1].src + tok_text) | |
| 124 | 126 | else: | |
| 125 | - | tokens.append(Token(tok_name, tok_text, sline, utf8_byte_offset)) | |
| 127 | + | tokens.append(Token(tok_name, tok_text, sline, end_offset)) | |
| 126 | 128 | last_line, last_col = eline, ecol | |
| 127 | - | end_offset = utf8_byte_offset + len(tok_text.encode()) | |
| 129 | + | end_offset += len(tok_text.encode()) | |
| 128 | 130 | ||
| 129 | 131 | return tokens | |
| 130 | 132 | ||
The check that tells the two apart
fail→pass·tests/tokenize_rt_test.py::test_src_to_tokens_no_eol_eof
Check file tests/tokenize_rt_test.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before it7f89433be049d73e189a903485a93bdf3e56f1a0
Broken version dated2021-10-21
Moduletokenize_rt
Units changedsrc_to_tokens
Fingerprint20a8147ed00206e8
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.