4 ms·

Good piece, goes into such depth that probably not all scientists must know all of this :) That said, my own experience of noticing what people's machines are

Good piece, goes into such depth that probably not all scientists must know all of this :) That said, my own experience of noticing what people's machines are wasting time on around me is very different. For scientists at firms (outside of academia) a major bottleneck is downloading, decompressing and shuffling data around. Unfortunately the lessons in the piece are in fact detrimental to this workload: 1. decompression is usually not faster than 500MB/s, which is much slower than SSD or network, but I still see compression everywhere with the idea that "cpu is faster". 2. Standard tools for downloading data, like awscli, are crazy slow compared to the actual ability of the network cards that are attached to the machines we (in big firms) use in the cloud. Specifically awscli caps at about 250MB/s no matter the configuration or network bandwidth, and people think it's normal because "network io is slow". (this of course doesn't apply to scientists who work out of the cloud, with much lower download speeds) None of this contradicts anything in the piece. It's that the units of work for different sectors are different. Whereas a geoscientist working on modeling the ocean has bottlenecks in computation, scientists at firms often have them in data.

ReleaseCandidat 5y ago
> 1. decompression is usually not faster than 500MB/s, which is much slower than SSD or network, There are some, but you have to select the right one. https://github.com/inikep/lzbench#benchmarks https://github.com/inikep/lzbench#benchmarks
throwaway984393 5y ago
I think the distinction is in systems programming versus system programming. Systems programming involves multiple systems with many components. It's not uncommon to find you simply can't make a solution work any better than it's designed, and you need to develop brand new solutions to problems (which makes the system even bigger). System programming is simpler because it's one system and one or two codebases. It can still be quite complex, but you can often just optimize the one solution rather than having to come up with new ones.
criticaltinker 5y ago
If you’re working with time series data and using a vectorized algorithm implemented with SIMD instructions you can achieve decompression speeds approaching 15 GB/s [1]. Anecdotally when working with biological signals at my day job, compression is a massive win and an absolute no brainer tradeoff when shuffling data across the network or even just storing in memory. That said, I think your first point is still reasonable for certain types of data and compression algorithms. [1] https://github.com/lemire/SIMDCompressionAndIntersection https://github.com/lemire/SIMDCompressionAndIntersection

ReleaseCandidat 5y ago

> 1. decompression is usually not faster than 500MB/s, which is much slower than SSD or network, There are some, but you have to select the right one. https://github.com/inikep/lzbench#benchmarks https://github.com/inikep/lzbench#benchmarks

throwaway984393 5y ago

I think the distinction is in systems programming versus system programming. Systems programming involves multiple systems with many components. It's not uncommon to find you simply can't make a solution work any better than it's designed, and you need to develop brand new solutions to problems (which makes the system even bigger). System programming is simpler because it's one system and one or two codebases. It can still be quite complex, but you can often just optimize the one solution rather than having to come up with new ones.

criticaltinker 5y ago

If you’re working with time series data and using a vectorized algorithm implemented with SIMD instructions you can achieve decompression speeds approaching 15 GB/s [1]. Anecdotally when working with biological signals at my day job, compression is a massive win and an absolute no brainer tradeoff when shuffling data across the network or even just storing in memory. That said, I think your first point is still reasonable for certain types of data and compression algorithms. [1] https://github.com/lemire/SIMDCompressionAndIntersection https://github.com/lemire/SIMDCompressionAndIntersection