Whole file
jawah/charset_normalizer
The author described this change as “:bug: automatically fallback on valid UTF-16 or UTF-32 even if the md says it's noisy”. It counts as a record because the check below fails on the code as it stood at 37397c179 and passes on 15ae24110, with nothing else changed between the two runs.
Projectjawah/charset_normalizer
Fix saved2025-08-09
Sharing licenceMIT · LICENSE
Change size+3 −2
What the code was meant to do, written into the code itself as a save note
:bug: automatically fallback on valid UTF-16 or UTF-32 even if the md says it's noisy
The change
| 369 | 369 | # Preparing those fallbacks in case we got nothing. | |
| 370 | 370 | if ( | |
| 371 | 371 | enable_fallback | |
| 372 | - | and encoding_iana in ["ascii", "utf_8", specified_encoding] | |
| 372 | + | and encoding_iana | |
| 373 | + | in ["ascii", "utf_8", specified_encoding, "utf_16", "utf_32"] | |
| 373 | 374 | and not lazy_str_hard_failure | |
| 374 | 375 | ): | |
| 375 | 376 | fallback_entry = CharsetMatch( | |
| 376 | 377 | sequences, | |
| 377 | 378 | encoding_iana, | |
| 378 | 379 | threshold, | |
| 379 | - | False, | |
| 380 | + | bom_or_sig_available, | |
| 380 | 381 | [], | |
| 381 | 382 | decoded_payload, | |
| 382 | 383 | preemptive_declaration=specified_encoding, |
The check that tells the two apart
fail→pass·tests/test_base_detection.py::test_md_triggered_but_with_bom_or_sig[\xff\xfe\x00\x00\xb6%\x00\x00h\x00\x00\x00e\x00\x00\x00l\x00\x00\x00l\x00\x00\x00o\x00\x00\x00-utf_32]
Check file tests/test_base_detection.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before it37397c1799a05454ab42fbbc94a7643cdb277924
Broken version dated2025-08-09
Modulecharset_normalizer.api
Units changedfrom_bytes
Fingerprintc2891bf2fdd12b40
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.