xxHash v0.7.4 - Finalizing XXH3 and XXH128
xxHash v0.7.4
is the last evolution of xxh3
and xxh128
, primarily designed to finalize the algorithm.
It is considered release candidate for v0.8.0
, which means that if all goes right, this version will rebranded v0.8.0
, almost "as is", within the next few weeks, after receiving sufficient feedback.
v0.8.0
is the official version after which XXH3
and XXH128
are considered "stabilized", meaning that return values will never change given the same input
and seed
, making the hash suitable for long-term storage and transmission.
Beyond these "final touches", the new version also brings a few notable improvements.
Automatic vector detection
x86
/x64
systems can enjoy a new unit, xxh_x86dispatch
, which can detect at runtime the best vector instruction set present on host system (none, sse2
, avx2
or avx512
), thanks to a cpu feature detector designed by @easyaspi314. It then automatically runs the appropriate vector code.
This makes it safer to deploy a single binary with advanced vector instruction sets, such as AVX2
, since there is no hard requirement for all target systems to actually support it : the binary can automatically switch to SSE2
instead.
As a proof of concept, the windows builds provided alongside this release are compiled with this new capability.
AVX512
support
A new vector instruction set is supported, thanks to @gzm55 : AVX512
. It can be applied on XXH3
and XXH128
, using some of the most recent Intel cpus, such as IceLake on laptop. It typically offers +50% more performance compared to AVX2
.
Secret Generator
Advanced users can be interested in the highly customizable variant _withSecret()
, which makes it possible to run XXH3
and XXH128
algorithms using one's own secret
.
However, the quality of the hash depends on the high entropy (randomness) of the secret
. And sometimes, it can be difficult to ensure that the candidate secret
is "random enough".
In order to produce a secret
of high quality, a new function XXH3_generateSecret()
is proposed in the advanced API section. It will convert any blob of bytes, named customSeed
, into a high quality secret
which respects all conditions expected by XXH3
and XXH128
. This is true even if customSeed
itself is of poor quality, such as a bunch of \0
bytes or some short or repeated common sequence.
No API modification
The existing API present in 0.7.3 has remained unchanged in 0.7.4. Any programs linking with 0.7.3 should continue to work as-is.
Note however that xxh3
/xxh128
return values are not comparable across these versions.
0.7.x are labelled development versions, and should only be used for ephemeral data (hash produced and consume in the same local session).
(note : this limitation does not extend to XXH32
and XXH64
, which are considered fully stable and specified).
Changelist
There are multiple smaller bug fixes and minor improvements that have been brought to this repository by great contributors. Here is a summarized list:
- perf: automatic vector detection and selection at runtime (
xxh_x86dispatch.h
), initiated by @easyaspi314 - perf: added
AVX512
support, by @gzm55 - api : new: secret generator
XXH_generateSecret()
, suggested by @koraa - api : fix:
XXH3_state_t
is movable, identified by @koraa - api : fix: state is correctly aligned in AVX mode (unlike
malloc()
), by @easyaspi314 - api : fix: streaming generated wrong values in some combination of random ingestion lengths, reported by @WayneD
- cli : fix unicode print on Windows, by @easyaspi314
- cli : can
-c
check file generated bysfv
- build:
make DISPATCH=1
generatesxxhsum
andlibxxhash
with runtime vector detection (x86/x64 only) - install: cygwin installation support
- doc : Cryptol specification of
XXH32
andXXH64
, by @weaversa