7 ms·

The One Billion Row Challenge in CUDA

241 points by tspeterkim 2y ago

Lucas-YANG 2y ago
[flagged]
MuffinFlavored 2y ago
> GPU Hash Table? How bad would performance have suffered if you sha256'd the lines to build the map? I'm going to guess "badly"? Maybe something like this in CUDA: https://github.com/Cyan4973/xxHash https://github.com/Cyan4973/xxHash ?
- tspeterkim 2y ago
  So performance would increase since hashing is faster than binary-searching. However, the problem of collisions across threads and dealing with concurrent map key insertions still remains. e.g. when two different cities produce the same hash (one at each thread), how can we atomically compare 100 byte city strings and correctly do collision-solving (using linear probe, for example - https://nosferalatu.com/SimpleGPUHashTable.html https://nosferalatu.com/SimpleGPUHashTable.html) Atomic operations are limited to 32-bits.
  - convivialdingo 2y ago
    You could hash the city names using something like a djb2a algorithm?
  - mandarax8 2y ago
    > Atomic operations are limited to 32-bits. I'm using 64bit atomics at work, are you on an old version of cuda or are some operations only supported on 32bit?
    - tspeterkim 2y ago
      I misspoke. (got confused with the key limit in my link above) Atomics work up to 128-bits (https://docs.nvidia.com/cuda/cuda-c-programming-guide/#atomic-functions https://docs.nvidia.com/cuda/cuda-c-programming-guide/#atomi...). Regardless, it's still less than 100 bytes, which is the max length of city strings.
  - Sesse__ 2y ago
    If the set of cities is known (as your binary search algorithm assumes), you can insert all of them first, on the CPU. That will resolve all collisions ahead of time, making the structure of the hash table essentially read-only. (Of course, you would still need atomics for the values.)
bhouston 2y ago
Interesting approach. I wonder if one could use a reduce operation across the cities or similar rather than atomic operations? GPUs are amazing at reduce operations.
- tspeterkim 2y ago
  Agreed. I tried reducing across cities first. The problem was that the work of gathering all the temperatures for each city (before I could launch the reduction CUDA kernels) required a full parsing through the input data. My final solution would be slower than the C++ baseline since the baseline already does the full parsing anyways.
cavisne 2y ago
The Programming Massively Parallel Processors textbook has a chapter on histograms. You have basically mentioned the main trick though (privatization). For this problem I think coursening would also help (so everything isn’t queued up on the copy to global memory)
- tspeterkim 2y ago
  The PMPP book is great. I reread the histogram chapter after finishing the blog, and realized I could use privatization. You got me! By coarsening, do you mean making the threads handle more file parts, and reducing the number of private copies (of histogram or stats here) to globally commit at the end?
kevincox 2y ago
> These offsets are obtained iteratively, stepping through the entire file buffer by the desired split size (= total file size / desired number of parts), and marking the position of a new line character: In many cases it is better to just pass the raw byte offsets to the workers, then they can skip to the first newline and progress until the next newline after their "end offset". This way the launcher doesn't need to access the input data at all. This can be ineffective when boundaries are not self-describing or when the minimum processable chunk is near the expected batch size (as you will have significant swings in batch size) but I think would work quite well for this case.
- tspeterkim 2y ago
  By "launcher", do you mean the CUDA kernel? How can it avoid accessing the input data since it needs access to the characters based on the offsets? I also already pass these offsets to the threads as `Part* parts`. I also probably didn't understand your suggestion and am drawing a blank here. So pls feel free to elaborate and correct me.
  - tromp 2y ago
    He's proposing to skip the whole computation of the parts[] array and within the kernel, replace for (int i = 0; i < parts[bx].length; i++) { // bx is the global thread index char c = buffer[parts[bx].offset-buffer_offset + i]; by something like long long split_size = size / num_parts; long long offset = bx * split_size; while (buffer[offset++] != '\n') ; for (int i = 0; buffer[offset+i] != '\n'; i++) { char c = buffer[offset + i]; (ignoring how to deal with buffer_offset for now).
    - tspeterkim 2y ago
      for (int i = 0; buffer[offset+i] != '\n'; i++) { This would only process the current line, though. Here, each thread processes ~split_size bytes (multiple lines). Even if were to read multiple lines, how would a thread know when to stop? (at which offset?) And when it does, it should communicate where it stopped with the other threads to prevent re-reading the buffer. My brain's hurting now.
konstantinua00 2y ago
am I reading correctly that your AtomicMin stops only when CAS succeeds? can you not stop early if extracted value becomes smaller than val?
- konstantinua00 2y ago
  also, iirc, floats' comparison is compatible with signed integers' comparison due to representation? so just using cuda's AtomicMin for ints would be enough
  - Sesse__ 2y ago
    Positive finite floats compare like ints. If you can have negative values, things get slightly more complicated, since they use sign-magnitude (you basically shift the sign bit to the right and use that as an XOR mask on all the other bits). If you need to care about -0.0 == +0.0 and NaN != NaN, most bets are off. :-)
formerly_proven 2y ago
Well that's a very pessimistic C++ baseline - using iostreams. The CUDA solution presented here seems to me like it's a bit of an old-timey approach, treating the GPU as an accelerator. The implementation first prepares a million tasks, then submits them as kernel invocations which perform the work. The first phase already takes over two seconds, the second phase takes over ten seconds. Resulting runtime is 16s, 60x faster than iostreams. This is similar to how GPU acceleration is bolted on in a ton of commercial software, you look at hotspots and then move just those over to the accelerator. For example outsourcing some linear algebra in a FEM solver. That's a sensible approach for trying to accelerate legacy software, but it's not very effective and leads to poor utilization ("copy mat, copy mat, copy mat copy copy copy mat" - Dave Cutler about GPU acceleration). I think a) you can do all of this on the GPU b) this should be limited by PCIe bandwidth for getting the file contents into VRAM. The 1BRC data set is 14 GB. So this is processing that file at about 1 GB/s using both CPU and GPU.
- taspeotis 2y ago
  Yeah <iostream> has terrible performance, and also I find the use of a map rather than an unordered_map a bit suspect.
  - jodleif 2y ago
    Its much easier to improve is the baseline is terrible
  - dkersten 2y ago
    Standard library maps/unordered_maps are themselves notoriously slow anyway. A flat_hash_map from abseil or parallel-hashmaps[1] would be better. [1] https://github.com/greg7mdp/parallel-hashmap https://github.com/greg7mdp/parallel-hashmap
- deleted 2y ago
  [deleted]
muep 2y ago
Instead of desiring atomic maps, would it work here to give each task its own map and then merge those after the parallel processing has ended? I have have absolutely no experience of CUDA, so no idea how applicable that approach would be. However, it seemed pretty practical and effective when I tried a parallel implementation of this challenge on plain CPUs.
- speps 2y ago
  Looks like TFA already uses partitioning per thread.
- SJC_Hacker 2y ago
  You can definitely have thread local storage
- tspeterkim 2y ago
  This is the possible optimization that I mention at the end of the blog - using a private map for each thread block. The catch is that this map must fit in shared memory, which is pretty limited on all current hardware: ~100KB. I originally thought that my map (stats array) was too big to fit into this shared memory. Now, however, I realize it can. It'll interesting to see how much speedup (or not!) this optimization can bring.
_zoltan_ 2y ago
I think this is very slow for some reason. I'll bookmark this and try on an A100 and a H100 box and then will try with multigpu on larger data.
- tspeterkim 2y ago
  Please do. My hope with this blog was to rile up other CUDA enthusiasts. Making them wanna bring in bigger and better hardware that I don't have access to. + Someone in the CUDA MODE community got 6 seconds on a 4090.
pama 2y ago
There are some good ideas for this type of problem here: https://github.com/dannyvankooten/1brc https://github.com/dannyvankooten/1brc After you deal with parsing and hashes, basically you are IO limited, so mmap helps. The C code takes less than 1.4s without any CUDA access. Because there is no compute to speak of, other than parsing and a hashmap, a reasonable guess is that even for the optimal CUDA implementation, the starting of kernels and transfer of data to the GPU would likely add a noticeable bottleneck and make the optimal CUDA code slower than this pure C code.
- _zoltan_ 2y ago
  this is true for small datasets. on large datasets, once loaded into GPU memory, cross GPU shuffling with NVLink is going to be much faster than CPU to RAM. on the H100 boxes with 8x400Gbps, IO with GDS is also pretty fast. for truly IObound tasks I think a lot of GPUs beats almost anything :-)
  - 15155 2y ago
    Try an array of FPGAs.
    - _zoltan_ 2y ago
      I've never been a fan plus good luck getting 800GB/s across.
    - dan-robertson 2y ago
      Why does that help with IO bandwidth?
      - 15155 2y ago
        Because every transceiver pair can do 32Gb/s or 56Gb/s and you have dozens of them.
  - candido_heavyai 2y ago
    I am testing a gh200 and the speed you can access the system memory is amazing.. Assuming you have already encoded the station into a smallint and the size of the dataset would be around 6gb that on such system takes just 20 ms to be transfered (I am sure about that because I'm observing transfer a 9.5gb that took about 33ms right now).
jbochi 2y ago
Pretty cool. Shameless plug: My team at Anthropic is hiring people that can write accelerator kernels. Please reach out to <my username>@gmail.com if you want to make state of the art models faster :)
- winwang 2y ago
  Curious, do you guys use Triton? I was surprised to find out PyTorch slowdown is palpable even compared to basic CUDA kernels.
- petermcneeley 2y ago
  qq: Would this be in CUDA?
  - jbochi 2y ago
    Not sure what I can share publicly, but we use TPUs as well: https://cloud.google.com/blog/products/compute/announcing-cloud-tpu-v5e-in-ga https://cloud.google.com/blog/products/compute/announcing-cl...
    - fpgamlirfanboy 2y ago
      It's funny - former Google devs (whom I just maintain a good relationship with their former employer) are ideally positioned to profitably take advantage of the arb of TPU over GPU.
      - bee_rider 2y ago
        Google should improve their abstractions until there isn’t room for that anymore, haha.
- trashtensor 2y ago
  not looking for a job at this time but i do this kind of work - what is the name of the team that you are working on / typically works on this?
Aznable 2y ago
The query itself it's a perfect hash (assuming the station is dictionary encoded) and takes around 100ms on a gh-200 (the Gpu is a single h100 with 132 SMs) with a query that's concurrency constrained. The same level of performance can be obtained using an Ada like an L40s or and RTX 4090. The transfer across the nvlink connecting the Cpu and GPU on a gh-200 after the parse and encoding of the source CSV takes a negligible amount of time given the 500 gb/sec of system memory bandwidth and the 900 GB /sec interconnection between Cpu and Gpu. So the problem is disk bandwidth that's going to limit the performance of the Gpu kernel. The faster solution should be parse the Csv with a gpu kernel using namp and Managed memory (?) encode the station into a interger or a small integer. The min and max value can be used to create keyless perfect hash table for each SM to limit the concurrency on global memory using 32bit atomic operations for min, max, count and sum and then do a final reduction on the Gpu. I don't think that is needed more then 1 modern gpu for this, especially if you are on a modern hardware like the gh-200. I'm running this kind of aggregates on a gh-200 using 10 Billion of records having the data in Gpu or Cpu memory using our software (heavydb) for testing purposes in the last two weeks
candido_heavyai 2y ago
The query itself it's a perfect hash (assuming the station is dictionary encoded) and takes around 100ms on a gh-200 (the Gpu is a single h100 with 132 SMs) with a query that's concurrency constrained. The same level of performance can be obtained using an Ada like an L40s or and RTX 4090. The transfer across the nvlink connecting the Cpu and GPU on a gh-200 after the parse and encoding of the source CSV takes a negligible amount of time given the 500 gb/sec of system memory bandwidth and the 900 GB /sec interconnection between Cpu and Gpu. So the problem is disk bandwidth that's going to limit the performance of the Gpu kernel. The faster solution should be parse the Csv with a gpu kernel using namp and Managed memory (?) encode the station into a interger or a small integer. The min and max value can be used to create keyless perfect hash table for each SM to limit the concurrency on global memory using 32bit atomic operations for min, max, count and sum and then do a final reduction on the Gpu. I don't think that is needed more then 1 modern gpu for this, especially if you are on a modern hardware like the gh-200. I'm running this kind of aggregates on a gh-200 using 10 Billion of records having the data in Gpu or Cpu memory using our software (heavydb) for testing purposes in the last two weeks
- winwang 2y ago
  Great context for us GPGPU fans! Do you think the L4 should have similar perf? Considering it has similar features and specs to L40, and I doubt the lower 300 GB/s bandwidth woudl be an issue.
- fulafel 2y ago
  The 1brc benchmark has the data in a RAM disk[0], there's no disk IO. [0] https://github.com/gunnarmorling/1brc#evaluating-results https://github.com/gunnarmorling/1brc#evaluating-results
ehsankia 2y ago
I have a hard time believing the basic C++ implementation took 16 minutes... I just did the most trivial non-parallel implementation in python and it took less than 5 minute with pypy.
- winwang 2y ago
  I think the author wanted to compare with a similar base algorithm. They should have stated that, however, instead of implying that a basic C++ algo would typically take that long. Something else to consider is cost of cloud hardware, though it could be extrapolated from the author's results.
- anthony88 2y ago
  The basic Java implementation was also around 5 minutes

241 points by tspeterkim 2y ago

[flagged]

> GPU Hash Table? How bad would performance have suffered if you sha256'd the lines to build the map? I'm going to guess "badly"? Maybe something like this in CUDA: https://github.com/Cyan4973/xxHash https://github.com/Cyan4973/xxHash ?

tspeterkim 2y ago
So performance would increase since hashing is faster than binary-searching. However, the problem of collisions across threads and dealing with concurrent map key insertions still remains. e.g. when two different cities produce the same hash (one at each thread), how can we atomically compare 100 byte city strings and correctly do collision-solving (using linear probe, for example - https://nosferalatu.com/SimpleGPUHashTable.html https://nosferalatu.com/SimpleGPUHashTable.html) Atomic operations are limited to 32-bits.
- convivialdingo 2y ago
  You could hash the city names using something like a djb2a algorithm?
- mandarax8 2y ago
  > Atomic operations are limited to 32-bits. I'm using 64bit atomics at work, are you on an old version of cuda or are some operations only supported on 32bit?
  - tspeterkim 2y ago
    I misspoke. (got confused with the key limit in my link above) Atomics work up to 128-bits (https://docs.nvidia.com/cuda/cuda-c-programming-guide/#atomic-functions https://docs.nvidia.com/cuda/cuda-c-programming-guide/#atomi...). Regardless, it's still less than 100 bytes, which is the max length of city strings.
- Sesse__ 2y ago
  If the set of cities is known (as your binary search algorithm assumes), you can insert all of them first, on the CPU. That will resolve all collisions ahead of time, making the structure of the hash table essentially read-only. (Of course, you would still need atomics for the values.)

bhouston 2y ago

Interesting approach. I wonder if one could use a reduce operation across the cities or similar rather than atomic operations? GPUs are amazing at reduce operations.

tspeterkim 2y ago
Agreed. I tried reducing across cities first. The problem was that the work of gathering all the temperatures for each city (before I could launch the reduction CUDA kernels) required a full parsing through the input data. My final solution would be slower than the C++ baseline since the baseline already does the full parsing anyways.

cavisne 2y ago

The Programming Massively Parallel Processors textbook has a chapter on histograms. You have basically mentioned the main trick though (privatization). For this problem I think coursening would also help (so everything isn’t queued up on the copy to global memory)

tspeterkim 2y ago
The PMPP book is great. I reread the histogram chapter after finishing the blog, and realized I could use privatization. You got me! By coarsening, do you mean making the threads handle more file parts, and reducing the number of private copies (of histogram or stats here) to globally commit at the end?

kevincox 2y ago

> These offsets are obtained iteratively, stepping through the entire file buffer by the desired split size (= total file size / desired number of parts), and marking the position of a new line character: In many cases it is better to just pass the raw byte offsets to the workers, then they can skip to the first newline and progress until the next newline after their "end offset". This way the launcher doesn't need to access the input data at all. This can be ineffective when boundaries are not self-describing or when the minimum processable chunk is near the expected batch size (as you will have significant swings in batch size) but I think would work quite well for this case.

tspeterkim 2y ago
By "launcher", do you mean the CUDA kernel? How can it avoid accessing the input data since it needs access to the characters based on the offsets? I also already pass these offsets to the threads as `Part* parts`. I also probably didn't understand your suggestion and am drawing a blank here. So pls feel free to elaborate and correct me.
- tromp 2y ago
  He's proposing to skip the whole computation of the parts[] array and within the kernel, replace for (int i = 0; i < parts[bx].length; i++) { // bx is the global thread index char c = buffer[parts[bx].offset-buffer_offset + i]; by something like long long split_size = size / num_parts; long long offset = bx * split_size; while (buffer[offset++] != '\n') ; for (int i = 0; buffer[offset+i] != '\n'; i++) { char c = buffer[offset + i]; (ignoring how to deal with buffer_offset for now).
  - tspeterkim 2y ago
    for (int i = 0; buffer[offset+i] != '\n'; i++) { This would only process the current line, though. Here, each thread processes ~split_size bytes (multiple lines). Even if were to read multiple lines, how would a thread know when to stop? (at which offset?) And when it does, it should communicate where it stopped with the other threads to prevent re-reading the buffer. My brain's hurting now.

konstantinua00 2y ago

am I reading correctly that your AtomicMin stops only when CAS succeeds? can you not stop early if extracted value becomes smaller than val?

konstantinua00 2y ago
also, iirc, floats' comparison is compatible with signed integers' comparison due to representation? so just using cuda's AtomicMin for ints would be enough
- Sesse__ 2y ago
  Positive finite floats compare like ints. If you can have negative values, things get slightly more complicated, since they use sign-magnitude (you basically shift the sign bit to the right and use that as an XOR mask on all the other bits). If you need to care about -0.0 == +0.0 and NaN != NaN, most bets are off. :-)

formerly_proven 2y ago

Well that's a very pessimistic C++ baseline - using iostreams. The CUDA solution presented here seems to me like it's a bit of an old-timey approach, treating the GPU as an accelerator. The implementation first prepares a million tasks, then submits them as kernel invocations which perform the work. The first phase already takes over two seconds, the second phase takes over ten seconds. Resulting runtime is 16s, 60x faster than iostreams. This is similar to how GPU acceleration is bolted on in a ton of commercial software, you look at hotspots and then move just those over to the accelerator. For example outsourcing some linear algebra in a FEM solver. That's a sensible approach for trying to accelerate legacy software, but it's not very effective and leads to poor utilization ("copy mat, copy mat, copy mat copy copy copy mat" - Dave Cutler about GPU acceleration). I think a) you can do all of this on the GPU b) this should be limited by PCIe bandwidth for getting the file contents into VRAM. The 1BRC data set is 14 GB. So this is processing that file at about 1 GB/s using both CPU and GPU.

taspeotis 2y ago
Yeah <iostream> has terrible performance, and also I find the use of a map rather than an unordered_map a bit suspect.
- jodleif 2y ago
  Its much easier to improve is the baseline is terrible
- dkersten 2y ago
  Standard library maps/unordered_maps are themselves notoriously slow anyway. A flat_hash_map from abseil or parallel-hashmaps[1] would be better. [1] https://github.com/greg7mdp/parallel-hashmap https://github.com/greg7mdp/parallel-hashmap
deleted 2y ago
[deleted]

muep 2y ago

Instead of desiring atomic maps, would it work here to give each task its own map and then merge those after the parallel processing has ended? I have have absolutely no experience of CUDA, so no idea how applicable that approach would be. However, it seemed pretty practical and effective when I tried a parallel implementation of this challenge on plain CPUs.

speps 2y ago
Looks like TFA already uses partitioning per thread.
SJC_Hacker 2y ago
You can definitely have thread local storage
tspeterkim 2y ago
This is the possible optimization that I mention at the end of the blog - using a private map for each thread block. The catch is that this map must fit in shared memory, which is pretty limited on all current hardware: ~100KB. I originally thought that my map (stats array) was too big to fit into this shared memory. Now, however, I realize it can. It'll interesting to see how much speedup (or not!) this optimization can bring.

_zoltan_ 2y ago

I think this is very slow for some reason. I'll bookmark this and try on an A100 and a H100 box and then will try with multigpu on larger data.

tspeterkim 2y ago
Please do. My hope with this blog was to rile up other CUDA enthusiasts. Making them wanna bring in bigger and better hardware that I don't have access to. + Someone in the CUDA MODE community got 6 seconds on a 4090.

pama 2y ago

There are some good ideas for this type of problem here: https://github.com/dannyvankooten/1brc https://github.com/dannyvankooten/1brc After you deal with parsing and hashes, basically you are IO limited, so mmap helps. The C code takes less than 1.4s without any CUDA access. Because there is no compute to speak of, other than parsing and a hashmap, a reasonable guess is that even for the optimal CUDA implementation, the starting of kernels and transfer of data to the GPU would likely add a noticeable bottleneck and make the optimal CUDA code slower than this pure C code.

_zoltan_ 2y ago
this is true for small datasets. on large datasets, once loaded into GPU memory, cross GPU shuffling with NVLink is going to be much faster than CPU to RAM. on the H100 boxes with 8x400Gbps, IO with GDS is also pretty fast. for truly IObound tasks I think a lot of GPUs beats almost anything :-)
- 15155 2y ago
  Try an array of FPGAs.
  - _zoltan_ 2y ago
    I've never been a fan plus good luck getting 800GB/s across.
  - dan-robertson 2y ago
    Why does that help with IO bandwidth?
    - 15155 2y ago
      Because every transceiver pair can do 32Gb/s or 56Gb/s and you have dozens of them.
- candido_heavyai 2y ago
  I am testing a gh200 and the speed you can access the system memory is amazing.. Assuming you have already encoded the station into a smallint and the size of the dataset would be around 6gb that on such system takes just 20 ms to be transfered (I am sure about that because I'm observing transfer a 9.5gb that took about 33ms right now).

jbochi 2y ago

Pretty cool. Shameless plug: My team at Anthropic is hiring people that can write accelerator kernels. Please reach out to <my username>@gmail.com if you want to make state of the art models faster :)

winwang 2y ago
Curious, do you guys use Triton? I was surprised to find out PyTorch slowdown is palpable even compared to basic CUDA kernels.
petermcneeley 2y ago
qq: Would this be in CUDA?
- jbochi 2y ago
  Not sure what I can share publicly, but we use TPUs as well: https://cloud.google.com/blog/products/compute/announcing-cloud-tpu-v5e-in-ga https://cloud.google.com/blog/products/compute/announcing-cl...
  - fpgamlirfanboy 2y ago
    It's funny - former Google devs (whom I just maintain a good relationship with their former employer) are ideally positioned to profitably take advantage of the arb of TPU over GPU.
    - bee_rider 2y ago
      Google should improve their abstractions until there isn’t room for that anymore, haha.
trashtensor 2y ago
not looking for a job at this time but i do this kind of work - what is the name of the team that you are working on / typically works on this?

Aznable 2y ago

The query itself it's a perfect hash (assuming the station is dictionary encoded) and takes around 100ms on a gh-200 (the Gpu is a single h100 with 132 SMs) with a query that's concurrency constrained. The same level of performance can be obtained using an Ada like an L40s or and RTX 4090. The transfer across the nvlink connecting the Cpu and GPU on a gh-200 after the parse and encoding of the source CSV takes a negligible amount of time given the 500 gb/sec of system memory bandwidth and the 900 GB /sec interconnection between Cpu and Gpu. So the problem is disk bandwidth that's going to limit the performance of the Gpu kernel. The faster solution should be parse the Csv with a gpu kernel using namp and Managed memory (?) encode the station into a interger or a small integer. The min and max value can be used to create keyless perfect hash table for each SM to limit the concurrency on global memory using 32bit atomic operations for min, max, count and sum and then do a final reduction on the Gpu. I don't think that is needed more then 1 modern gpu for this, especially if you are on a modern hardware like the gh-200. I'm running this kind of aggregates on a gh-200 using 10 Billion of records having the data in Gpu or Cpu memory using our software (heavydb) for testing purposes in the last two weeks

candido_heavyai 2y ago

winwang 2y ago
Great context for us GPGPU fans! Do you think the L4 should have similar perf? Considering it has similar features and specs to L40, and I doubt the lower 300 GB/s bandwidth woudl be an issue.
fulafel 2y ago
The 1brc benchmark has the data in a RAM disk[0], there's no disk IO. [0] https://github.com/gunnarmorling/1brc#evaluating-results https://github.com/gunnarmorling/1brc#evaluating-results

ehsankia 2y ago

I have a hard time believing the basic C++ implementation took 16 minutes... I just did the most trivial non-parallel implementation in python and it took less than 5 minute with pypy.

winwang 2y ago
I think the author wanted to compare with a similar base algorithm. They should have stated that, however, instead of implying that a basic C++ algo would typically take that long. Something else to consider is cost of cloud hardware, though it could be extrapolated from the author's results.
anthony88 2y ago
The basic Java implementation was also around 5 minutes