4 ms·

Do the runtime tests decouple from I/O times e.g. file and memory caching aren't affecting the results? Surprised because I'd expect a big file like that to be

by mintyc 7y ago

Do the runtime tests decouple from I/O times e.g. file and memory caching aren't affecting the results? Surprised because I'd expect a big file like that to be I/o bound in practice

thephyber 7y ago
There is a tech talk about Bao (I think it’s the same author) where he prefaced the talk by describing a few caveats like “benchmarks are lies” and he mentioned that you should always run a timed process a second time to tease out bias from cached/pages instructions and data. The core reason this method is faster than traditional SHAs is because Blake3 is a binary tree where the leaf blocks can be hashed in parallel and SHA is entirely sequential
- lucb1e 7y ago
  > you should always run a timed process a second time I should hope that anyone doing benchmarks does them multiple times to avoid crap like popcon or adobe reader update randomly keeping the system busy during one of the two algorithms' benchmarks. I don't expect that the author ran this only once and decided to make a blog post based on that, even if the post doesn't show multiple runs.
  - kvlr 7y ago
    Author here, I did run it a few times but you don’t have to take my word for it, you can rerun the notebook yourself if you sign up for Nextjournal and remix my notebook. Full disclosure: I’m also a Cofounder of Nextjournal. But I wouldn’t suggest to use Nextjournal for serious benchmarking (yet). We’re running on Google Cloud and it’s not suited for benchmarking unless you pay for a (big) sole tenant instance. In the future we plan to offer dedicated instances for benchmarking.
- oconnor663 7y ago
  To be clear, "benchmarks are nothing but lies" is an exaggeration to get my point across. (And a good way to get a laugh at the start of a talk.) But take a look at Figure 3 from the Performance section of the BLAKE3 spec: https://i.imgur.com/smGHAKA.png https://i.imgur.com/smGHAKA.png. Even ignoring things like caching and compiler settings and different hardware, that's a complicated graph! The lines cross all over each other. I count 5 different functions in that graph that are, for at least some input, the "fastest hash function". (I think BLAKE2sp just baaarely passes below BLAKE2bp, just prior to 8 KiB.) It's hard to give a simple summary of a complicated picture. For the big red bar chart at the top of the BLAKE3 repo, we certainly cherry-picked a point where BLAKE3 looks good. But you could also cherry-pick points where it looks bad.

by mintyc 7y ago

Do the runtime tests decouple from I/O times e.g. file and memory caching aren't affecting the results? Surprised because I'd expect a big file like that to be I/o bound in practice

thephyber 7y ago

There is a tech talk about Bao (I think it’s the same author) where he prefaced the talk by describing a few caveats like “benchmarks are lies” and he mentioned that you should always run a timed process a second time to tease out bias from cached/pages instructions and data. The core reason this method is faster than traditional SHAs is because Blake3 is a binary tree where the leaf blocks can be hashed in parallel and SHA is entirely sequential

lucb1e 7y ago
> you should always run a timed process a second time I should hope that anyone doing benchmarks does them multiple times to avoid crap like popcon or adobe reader update randomly keeping the system busy during one of the two algorithms' benchmarks. I don't expect that the author ran this only once and decided to make a blog post based on that, even if the post doesn't show multiple runs.
- kvlr 7y ago
  Author here, I did run it a few times but you don’t have to take my word for it, you can rerun the notebook yourself if you sign up for Nextjournal and remix my notebook. Full disclosure: I’m also a Cofounder of Nextjournal. But I wouldn’t suggest to use Nextjournal for serious benchmarking (yet). We’re running on Google Cloud and it’s not suited for benchmarking unless you pay for a (big) sole tenant instance. In the future we plan to offer dedicated instances for benchmarking.
oconnor663 7y ago
To be clear, "benchmarks are nothing but lies" is an exaggeration to get my point across. (And a good way to get a laugh at the start of a talk.) But take a look at Figure 3 from the Performance section of the BLAKE3 spec: https://i.imgur.com/smGHAKA.png https://i.imgur.com/smGHAKA.png. Even ignoring things like caching and compiler settings and different hardware, that's a complicated graph! The lines cross all over each other. I count 5 different functions in that graph that are, for at least some input, the "fastest hash function". (I think BLAKE2sp just baaarely passes below BLAKE2bp, just prior to 8 KiB.) It's hard to give a simple summary of a complicated picture. For the big red bar chart at the top of the BLAKE3 repo, we certainly cherry-picked a point where BLAKE3 looks good. But you could also cherry-pick points where it looks bad.