6 ms·

Use fast data algorithms (2021)

221 points by Ice_cream_suit 4y ago

mastax 4y ago
> Try zstd --adapt. This feature automatically adapts the compression ratio as the IO conditions change to make the current optimal tradeoff between CPU and “keeping the pipe fed”. I didn't know about that, that's neat. These types of articles come up often, and it's good to proselytize about better algorithms. However the end of the article hints at an issue. Most of the hashing and compression in my life are done embedded in some system or protocol that I can't easily change. Yeah, Docker and Debian and Firefox should use zstd but there's not much I can do about it. I may reach for zstd when I'm moving a big file between systems, but I'd have to install it first and much of the time that's not worthwhile.
- jdc 4y ago
  Why not try a self-extracting archive? see https://makeself.io https://makeself.io
HWR_14 4y ago
There are a lot of interesting algorithms, but as the author points out, he had to move the data to a RAM drive to avoid the disk access being the limiting factor. For a lot of use cases, it's not the CPU that is going to limit you. I did love the anecdote about adding gzip moving the bottleneck to the cpu from the network, and actually slowing down the whole system.
- jiggawatts 4y ago
  Storage is the limit until it isn't. Mechanical disks stagnated in performance for so long that people just "internalised" the rule that storage is always vastly slower than compute, and that just about any compression algorithm is faster than nothing, literally always. Similarly, hashing overheads could be safely ignored. Meanwhile, my current laptop (not server!) has a single NVMe SSD that can easily do 7 GB/s reads... but only in benchmarks. Why benchmarks you ask? Because nearly 100% of software has been written with the "disk is slow anyway, don't bother optimising I/O" assumption. I was recently trying to process some bulk data on my laptop, and there were practically no tools available that could ingest data that fast! The fastest I saw was about 2 GB/s. Similarly, I could not find any way to accelerate I/O further using compression without multithreading. That is, even LZ4 can only decompress at around 5GB/s, which would slow down reads. Algorithm choice -- while important -- doesn't even begin to approach the gains that can be made via data formats that enable parallelism. Chunking the data so that multiple CPU cores can process a stream is critical. Again, my laptop has 8 cores and 16 threads. Using only 1 core is throwing away at least 90% of the available performance. So if your code looks like this you have made a mistake: var doc = new Parser( new XmlParser( new Utf8Reader( new ZipStream( File.Open( filename ))))); Even if forced to use that sequence because of a legacy format, ideally that sequence should be distributed between CPU cores so that one core is responsible for zip decode while another core is doing the low-level parsing, and the third core is building the object model. In an ideal world, the on-disk format would be in ~1MB chunks so that decode could be parallelised to almost any degree. Unfortunately, this has to be done end-to-end. Even if, say, the compression is chunked, this won't help much if the file that is stored is a sequentially parsed format such as a huge JSON or XML file. The industry needs a renaissance of serialization formats where the "serial" part is taken out.
  - adgjlsfhk1 4y ago
    Note that there are totally valid reason to not saturate NVME read speeds. Given the read vs write speeds of modern SSDs, if you have an algorithm that makes your reads 2x slower, but reduces the number of random writes, that can be a major win overall.
  - thesz 4y ago
    Your comment kind of support notion "I/O is slow" you seemingly oppose - the I/O that is needed by real world programs may be different from I/O that is fast in benchmarks. E.g., B-tree reads need pretty much random access after many non-sequential data insertions. And we are talking about read-optimized data structures here.
    - nine_k 4y ago
      I/O hardware is fact. The algorithms did not adapt, so I/O in software is slow. Exactly what the author complaints about. BTW network is also fast; your typical server has at least a 10G interface, sometimes 40G or even 100G. Transfers within the same rack may be faster than your NVMe, and within the same DC, comparable.
      - thesz 4y ago
        Oh, those pesky algorithms, they did not adapt to the good, good hardware that is so fast. I do not like that stance at all. Consider CPUs, out-of-order ones. They can adapt to the algorithms being thrown at them and most of the time algorithm author is not even aware it can be done. Improvements in this area are steady, about two-to-five of percents per year. Apple's M1 is able to schedule almost 8 instructions per cycle, for example, in quite latency-sensitive task [1]. [1] https://lemire.me/blog/2021/03/24/counting-cycles-and-instructions-on-the-apple-m1-processor/ https://lemire.me/blog/2021/03/24/counting-cycles-and-instru... Storage systems development is unable to produce something that is similar to OoO CPUs. You can't throw algorithm at storage so that it'll adapt. Algorithms should be adapted to hardware. Circling back to CPUs, situation with storage hardware is very much like situation with the IBM's Cell BE architecture: hardware is fast, but nobody knows how to make concrete algorithms fast on that hardware. In my opinion, Cell BE is slow, just like contemporary storage systems are slow. They are fast only in benchmarks.
- jandrewrogers 4y ago
  Storage is rarely the bottleneck in modern systems, a lot of software is just written as if it was. It is completely ordinary these days to have 4-12+ GB/sec of sustained storage bandwidth available, and that is available to the application if you write the code correctly. Even with serious performance engineering, it is difficult to drive compression, parsers, codecs, etc with throughput comparable to modern storage. There are several non-cryptographic hashing algorithms that can be driven that hard, but none of them are mentioned in the article.
  - jolynch 4y ago
    Agree with everything you say except that the post didn't mention non-cryptographic hashing algos that can be driven that hard. xxHash[1] (and especially XXH3) is almost always the fastest hashing choice, as it both is fast and has wide language support. Sure there are some other fast ones out there like cityhash[2] but there aren't good Java/Python bindings I'm aware of and I wouldn't recommend using it in production given the lack of wide-spread use versus xxhash which is used by LZ4 internally and in databases all over the place. [1] https://github.com/Cyan4973/xxHash https://github.com/Cyan4973/xxHash [2] https://github.com/google/cityhash https://github.com/google/cityhash
theden 4y ago
AWS S3 right now supports CRC32, CRC32C, SHA-1, and SHA-256 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/checki...), which is interesting given the announcement to support the four wasn't long ago (https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/)—it https://aws.amazon.com/blogs/aws/new-additional-checksum-alg... seems like they went with what's more widely used than actually faster?
- jandrewrogers 4y ago
  The performance of these algorithms isn't entirely dependent on software when running at the scale of S3. Dedicated hashing hardware or hardware support will likely have reasonable throughput. You don't need the maximum possible performance, just sufficient to achieve objectives as inexpensively as possible.
- adrian_b 4y ago
  All modern Intel/AMD CPUs and 64-bit ARM CPUs have hardware instructions for these 4 algorithms (polynomial multiplication for CRC32, dedicated instructions for the other 3). Therefore these are very fast when using the hardware support, so they are a natural choice versus other algorithms that must be implemented in software, which may be slower even when they seem much simpler.
- foota 4y ago
  These are probably chosen based on popularity than anything.
jonstewart 4y ago
These are some of my favorite tricks. SHA2-256 is never “fast” but it can be really slow if you’re using a non-SIMD implementation (openssl uses SIMD). A trick not mentioned here: for Python devs, import “orjson” instead of the json standard library; it is usually a drop-in replacement.
- scaramanga 4y ago
  orjson is great, it's not quite a drop-in replacement, but in a good way... With orjson, encoding produces a bytes object instead of a string object, and when you're writing to a file that avoids a bunch of extra memory management overhead and a str.encode() when the text-file wrapper converts that string to a bytes behind the scenes. So that interface change is quite a big performance win over and above just having the faster JSON encoder.
- adrian_b 4y ago
  On all sufficiently new CPUs (all AMD Zen, most Intel Atom, Intel Core since Ice Lake, almost all 64-bit ARM), both SHA-1 and SHA2-256 are extremely fast, because there are special hardware instructions implementing them. They may be slow only in programs which do not check for hardware support and which do not use the dedicated hardware instructions (which is the case in many programs, because Intel Skylake derivatives have been the most popular CPUs during many years, and they were the only modern CPUs without hardware support for SHA-1/SHA2-256, so most developers did not bother to optimize their programs for the other CPUs). OpenSSL is one of the few programs which use the hardware instructions when available. This makes, e.g., "openssl dgst -sha256" much faster than "sha256sum" on recent CPUs. When using the hardware instructions, SHA-1 and SHA2-256 are faster than many non-cryptographic hashes and only 1 cryptographic hash is faster: BLAKE3. However, it must be kept in mind that BLAKE3 is much faster than any other cryptographic hash only because it distributes the computation on all CPU cores. So the much higher speed is accompanied by a much higher CPU utilization. If you have something else that should be done in parallel with the hash computation, using BLAKE3 does not necessarily reduce the total execution time for your entire application, even if the hash is computed much faster, because other concurrent activities may be stalled until the hash computation is completed.
getup8 4y ago
We are considering using MD5 for user id + salt hashing for randomizing users for A/B testing. Should blake3 or xxhash work as a replacement for that use case?
- Robin_Message 4y ago
  This article is focussed on performance of large data sets. Unless you have gigabytes of user IDs to hash on a regular basis, it doesn't matter what hash you use aside from achieving the randomness you desire, which I imagine almost any function, even x%2==0, would achieve.
- akx 4y ago
  xxhash would work fine for this, especially since it has a seed value option (which would be your salt).
oconnor663 4y ago
> For example, one situation where you want an intentionally slow algorithm is when dealing with passwords. This is true, but there's also more to the story. Modern password hashes like Argon2 force attackers into a "time-memory tradeoff", to try to reduce the advantage that specialized hardware has over the general-purpose computers that human beings use. I find that a lot of folks have memorized a summary like "slow hashes are good", and that's doubly unfortunate, because 1) like the article says, fast hashes are good, and 2) slow hashes in and of themselves are no longer the best we can do for password security. I often wish that password hashes weren't even called "hashes", because it's just a very different problem that they're solving. > or even just a high number of rounds of SHA-512 Please god no :)
- CodesInChaos 4y ago
  > or even just a high number of rounds of SHA-512 PBKDF1 (which is just iterated hashing) has the same security as PBKDF2 when used with fixed length salts, and avoids the numerous pitfalls of the latter. Though it's still not as good as bcrypt, scrypt or argon2.
- espadrine 4y ago
  > I often wish that password hashes weren't even called "hashes" I agree! And same for crypto hash (as in collision-resistant function) vs. “mixer” hash (as in diffuse permutation)[0]. We should make “PBKDF” a more common name. [0]: https://cr.yp.to/salsa20.html https://cr.yp.to/salsa20.html
- jolynch 4y ago
  >> or even just a high number of rounds of SHA-512 > Please god no :-) Heh I didn't mean it as a recommendation per say, but I'm pretty sure Linux uses repeated SHA-512 on at least some common distros. At least on my Ubuntu Focal machine my /etc/shadow appears to be using SHA-512.
_vvhw 4y ago
This is a great list—to add two categories: If you need bloom filters, then use split block bloom filters [1] which use SIMD to increase speed from 30%-450%. If you need erasure coding, then use Cauchy Reed-Solomon instead of Reed-Solomon. Cauchy Reed-Solomon uses pure XOR so you can do erasure coding at the speed of per-core memory bandwidth. [1] https://arxiv.org/abs/2101.01719 https://arxiv.org/abs/2101.01719
ur-whale 4y ago
There's always a huge focus on speed in these type of articles, and rightfully so for most applications. However, I often find myself in situations where I care neither about speed nor any other of the traditional performance metrics (memory consumption, latency, bandwidth, parallelizability, etc ...). In these situations, what I actually care about is: a) code size (as in: fits compiled in 512 bytes) b) code simplicity (as in: fits in head, takes up around 20 C++ LOC) Very unfortunately, there are very, very few algorithms that are designed to optimize along these lines. Exceptions: TEA : https://en.wikipedia.org/wiki/Tiny_Encryption_Algorithm https://en.wikipedia.org/wiki/Tiny_Encryption_Algorithm Speck: https://en.wikipedia.org/wiki/Speck_(cipher) https://en.wikipedia.org/wiki/Speck_(cipher) Both of these are ciphers, and both can be perverted into becoming hashes for various scenarios. But, there aren't any compression or native crypto-hard hashing algorithms that I know of that are specifically designed to optimize along that particular dimension. Blake3 or zstd are large pieces of code.
- oconnor663 4y ago
  Most of the BLAKE3 code size is optimized implementations for four different flavors of SIMD. But if you only compile the portable implementation, I don't think it's substantially more code than e.g. BLAKE2 or ChaCha. (Though tbh I haven't measured.) In particular, the tree mode adds very little extra code.

> Try zstd --adapt. This feature automatically adapts the compression ratio as the IO conditions change to make the current optimal tradeoff between CPU and “keeping the pipe fed”. I didn't know about that, that's neat. These types of articles come up often, and it's good to proselytize about better algorithms. However the end of the article hints at an issue. Most of the hashing and compression in my life are done embedded in some system or protocol that I can't easily change. Yeah, Docker and Debian and Firefox should use zstd but there's not much I can do about it. I may reach for zstd when I'm moving a big file between systems, but I'd have to install it first and much of the time that's not worthwhile.

jdc 4y ago
Why not try a self-extracting archive? see https://makeself.io https://makeself.io

HWR_14 4y ago

There are a lot of interesting algorithms, but as the author points out, he had to move the data to a RAM drive to avoid the disk access being the limiting factor. For a lot of use cases, it's not the CPU that is going to limit you. I did love the anecdote about adding gzip moving the bottleneck to the cpu from the network, and actually slowing down the whole system.

jiggawatts 4y ago
Storage is the limit until it isn't. Mechanical disks stagnated in performance for so long that people just "internalised" the rule that storage is always vastly slower than compute, and that just about any compression algorithm is faster than nothing, literally always. Similarly, hashing overheads could be safely ignored. Meanwhile, my current laptop (not server!) has a single NVMe SSD that can easily do 7 GB/s reads... but only in benchmarks. Why benchmarks you ask? Because nearly 100% of software has been written with the "disk is slow anyway, don't bother optimising I/O" assumption. I was recently trying to process some bulk data on my laptop, and there were practically no tools available that could ingest data that fast! The fastest I saw was about 2 GB/s. Similarly, I could not find any way to accelerate I/O further using compression without multithreading. That is, even LZ4 can only decompress at around 5GB/s, which would slow down reads. Algorithm choice -- while important -- doesn't even begin to approach the gains that can be made via data formats that enable parallelism. Chunking the data so that multiple CPU cores can process a stream is critical. Again, my laptop has 8 cores and 16 threads. Using only 1 core is throwing away at least 90% of the available performance. So if your code looks like this you have made a mistake: var doc = new Parser( new XmlParser( new Utf8Reader( new ZipStream( File.Open( filename ))))); Even if forced to use that sequence because of a legacy format, ideally that sequence should be distributed between CPU cores so that one core is responsible for zip decode while another core is doing the low-level parsing, and the third core is building the object model. In an ideal world, the on-disk format would be in ~1MB chunks so that decode could be parallelised to almost any degree. Unfortunately, this has to be done end-to-end. Even if, say, the compression is chunked, this won't help much if the file that is stored is a sequentially parsed format such as a huge JSON or XML file. The industry needs a renaissance of serialization formats where the "serial" part is taken out.
- adgjlsfhk1 4y ago
  Note that there are totally valid reason to not saturate NVME read speeds. Given the read vs write speeds of modern SSDs, if you have an algorithm that makes your reads 2x slower, but reduces the number of random writes, that can be a major win overall.
- thesz 4y ago
  Your comment kind of support notion "I/O is slow" you seemingly oppose - the I/O that is needed by real world programs may be different from I/O that is fast in benchmarks. E.g., B-tree reads need pretty much random access after many non-sequential data insertions. And we are talking about read-optimized data structures here.
  - nine_k 4y ago
    I/O hardware is fact. The algorithms did not adapt, so I/O in software is slow. Exactly what the author complaints about. BTW network is also fast; your typical server has at least a 10G interface, sometimes 40G or even 100G. Transfers within the same rack may be faster than your NVMe, and within the same DC, comparable.
    - thesz 4y ago
      Oh, those pesky algorithms, they did not adapt to the good, good hardware that is so fast. I do not like that stance at all. Consider CPUs, out-of-order ones. They can adapt to the algorithms being thrown at them and most of the time algorithm author is not even aware it can be done. Improvements in this area are steady, about two-to-five of percents per year. Apple's M1 is able to schedule almost 8 instructions per cycle, for example, in quite latency-sensitive task [1]. [1] https://lemire.me/blog/2021/03/24/counting-cycles-and-instructions-on-the-apple-m1-processor/ https://lemire.me/blog/2021/03/24/counting-cycles-and-instru... Storage systems development is unable to produce something that is similar to OoO CPUs. You can't throw algorithm at storage so that it'll adapt. Algorithms should be adapted to hardware. Circling back to CPUs, situation with storage hardware is very much like situation with the IBM's Cell BE architecture: hardware is fast, but nobody knows how to make concrete algorithms fast on that hardware. In my opinion, Cell BE is slow, just like contemporary storage systems are slow. They are fast only in benchmarks.
jandrewrogers 4y ago
Storage is rarely the bottleneck in modern systems, a lot of software is just written as if it was. It is completely ordinary these days to have 4-12+ GB/sec of sustained storage bandwidth available, and that is available to the application if you write the code correctly. Even with serious performance engineering, it is difficult to drive compression, parsers, codecs, etc with throughput comparable to modern storage. There are several non-cryptographic hashing algorithms that can be driven that hard, but none of them are mentioned in the article.
- jolynch 4y ago
  Agree with everything you say except that the post didn't mention non-cryptographic hashing algos that can be driven that hard. xxHash[1] (and especially XXH3) is almost always the fastest hashing choice, as it both is fast and has wide language support. Sure there are some other fast ones out there like cityhash[2] but there aren't good Java/Python bindings I'm aware of and I wouldn't recommend using it in production given the lack of wide-spread use versus xxhash which is used by LZ4 internally and in databases all over the place. [1] https://github.com/Cyan4973/xxHash https://github.com/Cyan4973/xxHash [2] https://github.com/google/cityhash https://github.com/google/cityhash

theden 4y ago

AWS S3 right now supports CRC32, CRC32C, SHA-1, and SHA-256 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/checki...), which is interesting given the announcement to support the four wasn't long ago (https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/)—it https://aws.amazon.com/blogs/aws/new-additional-checksum-alg... seems like they went with what's more widely used than actually faster?

jandrewrogers 4y ago
The performance of these algorithms isn't entirely dependent on software when running at the scale of S3. Dedicated hashing hardware or hardware support will likely have reasonable throughput. You don't need the maximum possible performance, just sufficient to achieve objectives as inexpensively as possible.
adrian_b 4y ago
All modern Intel/AMD CPUs and 64-bit ARM CPUs have hardware instructions for these 4 algorithms (polynomial multiplication for CRC32, dedicated instructions for the other 3). Therefore these are very fast when using the hardware support, so they are a natural choice versus other algorithms that must be implemented in software, which may be slower even when they seem much simpler.
foota 4y ago
These are probably chosen based on popularity than anything.

jonstewart 4y ago

These are some of my favorite tricks. SHA2-256 is never “fast” but it can be really slow if you’re using a non-SIMD implementation (openssl uses SIMD). A trick not mentioned here: for Python devs, import “orjson” instead of the json standard library; it is usually a drop-in replacement.

scaramanga 4y ago
orjson is great, it's not quite a drop-in replacement, but in a good way... With orjson, encoding produces a bytes object instead of a string object, and when you're writing to a file that avoids a bunch of extra memory management overhead and a str.encode() when the text-file wrapper converts that string to a bytes behind the scenes. So that interface change is quite a big performance win over and above just having the faster JSON encoder.
adrian_b 4y ago
On all sufficiently new CPUs (all AMD Zen, most Intel Atom, Intel Core since Ice Lake, almost all 64-bit ARM), both SHA-1 and SHA2-256 are extremely fast, because there are special hardware instructions implementing them. They may be slow only in programs which do not check for hardware support and which do not use the dedicated hardware instructions (which is the case in many programs, because Intel Skylake derivatives have been the most popular CPUs during many years, and they were the only modern CPUs without hardware support for SHA-1/SHA2-256, so most developers did not bother to optimize their programs for the other CPUs). OpenSSL is one of the few programs which use the hardware instructions when available. This makes, e.g., "openssl dgst -sha256" much faster than "sha256sum" on recent CPUs. When using the hardware instructions, SHA-1 and SHA2-256 are faster than many non-cryptographic hashes and only 1 cryptographic hash is faster: BLAKE3. However, it must be kept in mind that BLAKE3 is much faster than any other cryptographic hash only because it distributes the computation on all CPU cores. So the much higher speed is accompanied by a much higher CPU utilization. If you have something else that should be done in parallel with the hash computation, using BLAKE3 does not necessarily reduce the total execution time for your entire application, even if the hash is computed much faster, because other concurrent activities may be stalled until the hash computation is completed.

getup8 4y ago

We are considering using MD5 for user id + salt hashing for randomizing users for A/B testing. Should blake3 or xxhash work as a replacement for that use case?

Robin_Message 4y ago
This article is focussed on performance of large data sets. Unless you have gigabytes of user IDs to hash on a regular basis, it doesn't matter what hash you use aside from achieving the randomness you desire, which I imagine almost any function, even x%2==0, would achieve.
akx 4y ago
xxhash would work fine for this, especially since it has a seed value option (which would be your salt).

oconnor663 4y ago

> For example, one situation where you want an intentionally slow algorithm is when dealing with passwords. This is true, but there's also more to the story. Modern password hashes like Argon2 force attackers into a "time-memory tradeoff", to try to reduce the advantage that specialized hardware has over the general-purpose computers that human beings use. I find that a lot of folks have memorized a summary like "slow hashes are good", and that's doubly unfortunate, because 1) like the article says, fast hashes are good, and 2) slow hashes in and of themselves are no longer the best we can do for password security. I often wish that password hashes weren't even called "hashes", because it's just a very different problem that they're solving. > or even just a high number of rounds of SHA-512 Please god no :)

CodesInChaos 4y ago
> or even just a high number of rounds of SHA-512 PBKDF1 (which is just iterated hashing) has the same security as PBKDF2 when used with fixed length salts, and avoids the numerous pitfalls of the latter. Though it's still not as good as bcrypt, scrypt or argon2.
espadrine 4y ago
> I often wish that password hashes weren't even called "hashes" I agree! And same for crypto hash (as in collision-resistant function) vs. “mixer” hash (as in diffuse permutation)[0]. We should make “PBKDF” a more common name. [0]: https://cr.yp.to/salsa20.html https://cr.yp.to/salsa20.html
jolynch 4y ago
>> or even just a high number of rounds of SHA-512 > Please god no :-) Heh I didn't mean it as a recommendation per say, but I'm pretty sure Linux uses repeated SHA-512 on at least some common distros. At least on my Ubuntu Focal machine my /etc/shadow appears to be using SHA-512.

_vvhw 4y ago

This is a great list—to add two categories: If you need bloom filters, then use split block bloom filters [1] which use SIMD to increase speed from 30%-450%. If you need erasure coding, then use Cauchy Reed-Solomon instead of Reed-Solomon. Cauchy Reed-Solomon uses pure XOR so you can do erasure coding at the speed of per-core memory bandwidth. [1] https://arxiv.org/abs/2101.01719 https://arxiv.org/abs/2101.01719

ur-whale 4y ago

There's always a huge focus on speed in these type of articles, and rightfully so for most applications. However, I often find myself in situations where I care neither about speed nor any other of the traditional performance metrics (memory consumption, latency, bandwidth, parallelizability, etc ...). In these situations, what I actually care about is: a) code size (as in: fits compiled in 512 bytes) b) code simplicity (as in: fits in head, takes up around 20 C++ LOC) Very unfortunately, there are very, very few algorithms that are designed to optimize along these lines. Exceptions: TEA : https://en.wikipedia.org/wiki/Tiny_Encryption_Algorithm https://en.wikipedia.org/wiki/Tiny_Encryption_Algorithm Speck: https://en.wikipedia.org/wiki/Speck_(cipher) https://en.wikipedia.org/wiki/Speck_(cipher) Both of these are ciphers, and both can be perverted into becoming hashes for various scenarios. But, there aren't any compression or native crypto-hard hashing algorithms that I know of that are specifically designed to optimize along that particular dimension. Blake3 or zstd are large pieces of code.

oconnor663 4y ago
Most of the BLAKE3 code size is optimized implementations for four different flavors of SIMD. But if you only compile the portable implementation, I don't think it's substantially more code than e.g. BLAKE2 or ChaCha. (Though tbh I haven't measured.) In particular, the tree mode adds very little extra code.