Whole file

jawah/charset_normalizer

The author described this change as “fix: unicodedecodeerror in fallback path”. It counts as a record because the check below fails on the code as it stood at be252d70c and passes on 2bc26076a, with nothing else changed between the two runs.

Projectjawah/charset_normalizer

Fix saved2026-07-07

Sharing licenceMIT · LICENSE

Change size+25 −0

What the code was meant to do, written into the code itself as a save note

fix: unicodedecodeerror in fallback path

The change

529	529		in ["ascii", "utf_8", specified_encoding, "utf_16", "utf_32"]
530	530		and not lazy_str_hard_failure
531	531		):
	532	+	# Always fully decode payload before.
	533	+	# We've missed a UnicodeDecodeError proof
	534	+	# while issuing release 3.4.8
	535	+	# see https://github.com/jawah/charset_normalizer/issues/771
	536	+	if decoded_payload is None:
	537	+	try:
	538	+	decoded_payload = str(
	539	+	(
	540	+	sequences
	541	+	if not strip_sig_or_bom
	542	+	else sequences[len(sig_payload) :]
	543	+	),
	544	+	encoding=encoding_iana,
	545	+	)
	546	+	except (UnicodeDecodeError, LookupError):
	547	+	logger.log(
	548	+	TRACE,
	549	+	"%s does not decode the whole payload: fallback entry withheld.",
	550	+	encoding_iana,
	551	+	)
	552	+	continue
	553	+	if is_too_large_sequence:
	554	+	# Don't retain huge payload in RAM.
	555	+	decoded_payload = None
	556	+
532	557		fallback_entry = CharsetMatch(
533	558		sequences,
534	559		encoding_iana,

The check that tells the two apart

fail→pass·tests/test_edge_case.py::test_regression_gh771_fallback_entry_on_undecodable_payload

Check file tests/test_edge_case.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.

Origin and history

The fix2bc26076a86704fc19bc2ae811615b169373dea6

The code before itbe252d70c9f45846f0e851cc4d1829328540c825

Broken version dated2026-07-06

Modulecharset_normalizer.api

Units changedfrom_bytes

Fingerprint9ac39af9810a0a81

Checked2026-08-18 by goldset/0.1

Every field above is generated by our program. None of it is written by hand.

What the code was meant to do, written into the code itself as a save note

The change

The check that tells the two apart

Origin and history

Other bugs found in jawah/charset_normalizer