Whole file

ekzhu/datasketch

The author described this change as “fix pickling for minhash”. It counts as a record because the check below fails on the code as it stood at dff39ee3f and passes on 50d2e334b, with nothing else changed between the two runs.

Projectekzhu/datasketch

Fix saved2015-04-01

Sharing licenceMIT · LICENSE

Change size+16 −7

What the code was meant to do, written into the code itself as a save note

fix pickling for minhash

The change

16	16
17	17		def _create_permutation():
18	18		'''
19		-	Create a random bijective permutation function that maps a 32-bit
20		-	hash value to another 32-bit hash value.
	19	+	Create parameters for a random bijective permutation function
	20	+	that maps a 32-bit hash value to another 32-bit hash value.
21	21		http://en.wikipedia.org/wiki/Universal_hashing
22	22		'''
23	23		a = random.randint(1, _max_hash)
24	24		b = random.randint(0, _max_hash)
25		-	return lambda x : ((a * x + b) % _mersenne_prime) % _hash_range
	25	+	return (a, b)
26	26
27	27
	28	+	_permutation_func = lambda x, a, b: ((a * x + b) % _mersenne_prime) % _hash_range
	29	+
	30	+
28	31		class MinHash(object):
29	32		'''
30	33		The MinHash object.
⋯
53	56		'''
54	57		# Digest the hash object to get the hash value
55	58		hv = int(hashobj.hexdigest()[:8], 16)
56		-	for i, p in enumerate(self.permutations):
57		-	phv = p(hv)
	59	+	for i, (a, b) in enumerate(self.permutations):
	60	+	phv = _permutation_func(hv, a, b)
58	61		if phv < self.hashvalues[i]:
59	62		self.hashvalues[i] = phv
60	63
⋯
91	94		'''
92	95		Serializes this MinHash object into bytes, store in `buffer`
93	96		starting at `offset` position.
94		-	The size of `buffer` must equal to the size returned by
95		-	the `bytesize` method.
96	97		'''
97	98		if len(buffer) - offset < self.bytesize():
98	99		raise MinHashException("The buffer does not have enough space\
⋯
113	114		mh.hashvalues[i] = struct.unpack_from('I', buffer, offset)[0]
114	115		offset += struct.calcsize('I')
115	116		return mh
	117	+
	118	+	# Implement the following 2 methods to make pickling happy in python 2
	119	+	def __getstate__(self):
	120	+	return dict([x, getattr(self, x)] for x in self.__slots__)
	121	+
	122	+	def __setstate__(self, d):
	123	+	for key in d:
	124	+	setattr(self, key, d[key])
116	125
117	126
118	127		def jaccard(mhs):

The check that tells the two apart

fail→pass·test/minhash_test.py::TestMinHash::test_init

Check file test/minhash_test.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.

Origin and history

The fix50d2e334b8c578f3c66ccd8c14a9a17515e8eca1

The code before itdff39ee3f8279b690544274d096f90de444dae34

Broken version dated2015-04-01

Moduledatasketch.minhash

Units changedMinHash, _create_permutation

Fingerprint3f259f0564c4e3cb

Checked2026-08-18 by goldset/0.1

Every field above is generated by our program. None of it is written by hand.