3 ms·

Sounds really interesting for some of the things I'm working on. I would love to see some very basic stats on performance before trying it out though. A fairly

by pbnjay 14y ago

Sounds really interesting for some of the things I'm working on. I would love to see some very basic stats on performance before trying it out though. A fairly trivial example would be to store 8 billion random integers, then step through them in order and XOR them. Then you can have a baseline to compare monolithic file performance to blaze.

csense 14y ago
This would be a worst-case benchmark. Since random integers aren't compressible, you'll end up with the overhead of the extra logic for compression, but none of the benefits since it won't actually reduce the size of your data.
- faltet 14y ago
  That's right. But even in this case, Blosc, the internal compressor used in Blaze, can detect whether the data is compressible or not pretty early in the compression pipeline, and decide to stop compressing and start just copying (how early that decision is taked depends on the compression level). The good news is that Blosc can still use threads in parallel for doing the copy, and this normally gives significantly better speed than a non-threaded memcpy() (the default for many modern systems).
  - csense 14y ago
    > significantly better speed than a non-threaded memcpy() Really? I always thought that a single core could always saturate available memory bandwidth (unless you have some weird architecture like NUMA). If you're seeing a multithread memcpy that has better performance speed, maybe you're just stealing memory bandwidth from other processes (since AFAIK memory bandwidth is probably done on a per-thread basis), or maybe you're getting more CPU cache allocated to you because you're running on multiple cores? This would be interesting to investigate.
    - faltet 14y ago
      I'm not sure why, but yes, I'm seeing these speedups. You can see them too in: http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks and paying attention to compression ratios of 1 (compression disabled). If you find any good reason on why this is happening, I'm all ears.

by pbnjay 14y ago

csense 14y ago

This would be a worst-case benchmark. Since random integers aren't compressible, you'll end up with the overhead of the extra logic for compression, but none of the benefits since it won't actually reduce the size of your data.

faltet 14y ago
That's right. But even in this case, Blosc, the internal compressor used in Blaze, can detect whether the data is compressible or not pretty early in the compression pipeline, and decide to stop compressing and start just copying (how early that decision is taked depends on the compression level). The good news is that Blosc can still use threads in parallel for doing the copy, and this normally gives significantly better speed than a non-threaded memcpy() (the default for many modern systems).
- csense 14y ago
  > significantly better speed than a non-threaded memcpy() Really? I always thought that a single core could always saturate available memory bandwidth (unless you have some weird architecture like NUMA). If you're seeing a multithread memcpy that has better performance speed, maybe you're just stealing memory bandwidth from other processes (since AFAIK memory bandwidth is probably done on a per-thread basis), or maybe you're getting more CPU cache allocated to you because you're running on multiple cores? This would be interesting to investigate.
  - faltet 14y ago
    I'm not sure why, but yes, I'm seeing these speedups. You can see them too in: http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks and paying attention to compression ratios of 1 (compression disabled). If you find any good reason on why this is happening, I'm all ears.