5 ms·

What scientists must know about hardware to write fast code (2020)

109 points by spekcular 5y ago

yesenadam 5y ago
Seems like a good article, thank you! It's like a long sequel to the performance tips section of the Julia documentation. Only read a bit so far, found this error: > 0.001321 seconds > about 13 miliseconds Uh no, that's about 1.3 milliseconds.
jakobnissen 5y ago
Hey all, author here. That link is to an older version of the post - an updated version can be found here: https://viralinstruction.com/posts/hardware/ https://viralinstruction.com/posts/hardware/
- junon 5y ago
  @dang can it be replaced?
- rolph 5y ago
  i can read >>that link<< -but, the updated version is inaccessable to me, you may want to fix that if you want everybody to be able to read what you have published.
- branko_d 5y ago
  Excellent write up, thank you! It starts kind of slow, but then somehow morphs into effortlessly explaining advanced topics. I knew bits and pieces of this before, but I found information that was genuinely new to me, like this gem: ”Going back to the original example, that is why the perfectly predicted copy_odds_branches! performs better than code_odds_branchless!. Even though the latter has fewer instructions, it has a memory dependency: The index of dst where the odd number gets stored to depends on the last loop iteration. So fewer instructions can be executed at a time compared to the former function, where the branch predictor predicts several iterations ahead and allow for the parallel computation of multiple iterations.”
- jacobolus 5y ago
  CSS/layout note: the floating table of contents to the right covers part of the main body column (in Safari, Firefox, and Chrome here), making this document illegible unless I use a narrow enough browser window that the table of contents goes away entirely. Also, this document doesn’t load whatsoever when Javascript is disabled.
  - jakobnissen 5y ago
    Thanks for the heads-up. I've filed a bug report for the table of content issue. Wrt. Javascript, I think the presentation style of the notebook is worth requiring Javascript.
- BlueTemplar 5y ago
  Curious, looks like you're assuming that most scientists are using laptops rather than desktops ? Or are the researchers that I know atypical ? Such a great and condensed write up... and then at the last moment it looks like one is finally going to learn about (dedicated) GPU programming, and... nope ! :P
  - jakobnissen 5y ago
    Thanks! Well every one of my colleagues use laptops - but I think that detail should not matter for the content of the blog.
belval 5y ago
This is a lot of words to essentially repeat the "Latency Numbers every developer should know" by Peter Norvig: https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832 EDIT: That was an uncharitable take and I apologize to the author.
- jakobnissen 5y ago
  What about the parts about SIMD, struct of arrays, the heap vs the stack, multithreading, the short introduction to assembly code, inlining and unrolling, and the actual explanations of what all those phrases in the doc you linked means? You know, 80% of the content of my blog post?
  - zdyn5 5y ago
    Great article. Ignore the (if I may, arrogant) lossy compression attempted above.
    - aeroman 5y ago
      I agree - this post is really useful, thanks for writing it!
  - deleted 5y ago
    [deleted]
- ChrisRackauckas 5y ago
  What? That post has none of the information in this post. Did you even read it?
vlovich123 5y ago
> Both your operating system and your programming language will make use of caches, however, sometimes it is necessary to manually buffer your files. Hmmm? This part stands out. It sounds like the author is here. Can you help me understand this advice? It feels like it should have a million caveats and not be general advice. Often times memory mapping the file is going to be better. Unless you’re an expert you’re probably not going to do better than the page cache.
- jandrewrogers 5y ago
  Using mmap() is popular because it is easy despite its many sharp edges, but outside a limited number of access patterns and hardware setups it is often quite slow because the page cache behavior is poor for many use cases. Manually buffering is often simpler, and with better performance, than trying to make the page cache perform well. Furthermore, mmap() doesn't work for large storage volumes generally, so you might find yourself doing your own buffering regardless. If you have a relatively simple access pattern, and most applications do, then you don't need to be a database kernel engineer to design a trivial manual buffering system that can significantly outperform mmap(). It is a handful of lines of code in most languages.
  - vlovich123 5y ago
    Hmmm… this does not match my experience at all. Some user space buffering helps but that’s just because it’s easier to write your application logic (ie read 4 bytes, read 17 bytes, read 46kb). I have not heard the advice to layer another layer of buffer on top of if that. The reason is that it lets you write such code efficiently without paying the syscall overhead. These buffers are very stupid and io_uring or mmap (with a read ahead option) will perform better for larger files because there’s either no extra memory copy or the cost is hidden better. I’ve seen people struggle and fail to outperform the page cache. The only time I’ve beaten it is when it was caching the wrong thing (it was caching compressed and uncompressed pages via fuse so telling it to forget about the compressed pages after decompression eased memory pressure).
    - 5y ago
henrydark 5y ago
Good piece, goes into such depth that probably not all scientists must know all of this :) That said, my own experience of noticing what people's machines are wasting time on around me is very different. For scientists at firms (outside of academia) a major bottleneck is downloading, decompressing and shuffling data around. Unfortunately the lessons in the piece are in fact detrimental to this workload: 1. decompression is usually not faster than 500MB/s, which is much slower than SSD or network, but I still see compression everywhere with the idea that "cpu is faster". 2. Standard tools for downloading data, like awscli, are crazy slow compared to the actual ability of the network cards that are attached to the machines we (in big firms) use in the cloud. Specifically awscli caps at about 250MB/s no matter the configuration or network bandwidth, and people think it's normal because "network io is slow". (this of course doesn't apply to scientists who work out of the cloud, with much lower download speeds) None of this contradicts anything in the piece. It's that the units of work for different sectors are different. Whereas a geoscientist working on modeling the ocean has bottlenecks in computation, scientists at firms often have them in data.
- ReleaseCandidat 5y ago
  > 1. decompression is usually not faster than 500MB/s, which is much slower than SSD or network, There are some, but you have to select the right one. https://github.com/inikep/lzbench#benchmarks https://github.com/inikep/lzbench#benchmarks
- throwaway984393 5y ago
  I think the distinction is in systems programming versus system programming. Systems programming involves multiple systems with many components. It's not uncommon to find you simply can't make a solution work any better than it's designed, and you need to develop brand new solutions to problems (which makes the system even bigger). System programming is simpler because it's one system and one or two codebases. It can still be quite complex, but you can often just optimize the one solution rather than having to come up with new ones.
- criticaltinker 5y ago
  If you’re working with time series data and using a vectorized algorithm implemented with SIMD instructions you can achieve decompression speeds approaching 15 GB/s [1]. Anecdotally when working with biological signals at my day job, compression is a massive win and an absolute no brainer tradeoff when shuffling data across the network or even just storing in memory. That said, I think your first point is still reasonable for certain types of data and compression algorithms. [1] https://github.com/lemire/SIMDCompressionAndIntersection https://github.com/lemire/SIMDCompressionAndIntersection
spekcular 5y ago
If I'm reading the table under "fast instruction, slow instruction" correctly, float multiplication has half the reciprocal throughput of integer multiplication, which means float multiplication has twice the throughput. So, if I want to multiply a bunch of integers together, does this mean it's faster for me to represent them as floats (even though I don't need the additional precision)?
- NohatCoder 5y ago
  It depends on the processor, but yes in some cases using floats would be faster for this reason. Please note that floats don't have "additional" precision, they have a different structure that means they can express a different set of numbers. Integers should in general always be preferred when the data you deal with is of integral nature. Even if your CPU can do some instructions faster with floats it is generally not the kind of optimization you would want to make. You would end up making things more complicated for little practical benefit. Funny thing by the way, integer arithmetic units are generally cheaper to make than float units, but most multiply/division heavy code is floating point, so CPU makers prioritise the floating point units.
  - spekcular 5y ago
    Thank you for the detailed explanation.
dang 3y ago
That site now directs to malware, but the article is readable here: http://web.archive.org/web/20221208105716/https://biojulia.net/post/hardware/ http://web.archive.org/web/20221208105716/https://biojulia.n...

109 points by spekcular 5y ago

Seems like a good article, thank you! It's like a long sequel to the performance tips section of the Julia documentation. Only read a bit so far, found this error: > 0.001321 seconds > about 13 miliseconds Uh no, that's about 1.3 milliseconds.

jakobnissen 5y ago

Hey all, author here. That link is to an older version of the post - an updated version can be found here: https://viralinstruction.com/posts/hardware/ https://viralinstruction.com/posts/hardware/

junon 5y ago
@dang can it be replaced?
rolph 5y ago
i can read >>that link<< -but, the updated version is inaccessable to me, you may want to fix that if you want everybody to be able to read what you have published.
branko_d 5y ago
Excellent write up, thank you! It starts kind of slow, but then somehow morphs into effortlessly explaining advanced topics. I knew bits and pieces of this before, but I found information that was genuinely new to me, like this gem: ”Going back to the original example, that is why the perfectly predicted copy_odds_branches! performs better than code_odds_branchless!. Even though the latter has fewer instructions, it has a memory dependency: The index of dst where the odd number gets stored to depends on the last loop iteration. So fewer instructions can be executed at a time compared to the former function, where the branch predictor predicts several iterations ahead and allow for the parallel computation of multiple iterations.”
jacobolus 5y ago
CSS/layout note: the floating table of contents to the right covers part of the main body column (in Safari, Firefox, and Chrome here), making this document illegible unless I use a narrow enough browser window that the table of contents goes away entirely. Also, this document doesn’t load whatsoever when Javascript is disabled.
- jakobnissen 5y ago
  Thanks for the heads-up. I've filed a bug report for the table of content issue. Wrt. Javascript, I think the presentation style of the notebook is worth requiring Javascript.
BlueTemplar 5y ago
Curious, looks like you're assuming that most scientists are using laptops rather than desktops ? Or are the researchers that I know atypical ? Such a great and condensed write up... and then at the last moment it looks like one is finally going to learn about (dedicated) GPU programming, and... nope ! :P
- jakobnissen 5y ago
  Thanks! Well every one of my colleagues use laptops - but I think that detail should not matter for the content of the blog.

belval 5y ago

This is a lot of words to essentially repeat the "Latency Numbers every developer should know" by Peter Norvig: https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832 EDIT: That was an uncharitable take and I apologize to the author.

jakobnissen 5y ago
What about the parts about SIMD, struct of arrays, the heap vs the stack, multithreading, the short introduction to assembly code, inlining and unrolling, and the actual explanations of what all those phrases in the doc you linked means? You know, 80% of the content of my blog post?
- zdyn5 5y ago
  Great article. Ignore the (if I may, arrogant) lossy compression attempted above.
  - aeroman 5y ago
    I agree - this post is really useful, thanks for writing it!
- deleted 5y ago
  [deleted]
ChrisRackauckas 5y ago
What? That post has none of the information in this post. Did you even read it?

vlovich123 5y ago

> Both your operating system and your programming language will make use of caches, however, sometimes it is necessary to manually buffer your files. Hmmm? This part stands out. It sounds like the author is here. Can you help me understand this advice? It feels like it should have a million caveats and not be general advice. Often times memory mapping the file is going to be better. Unless you’re an expert you’re probably not going to do better than the page cache.

jandrewrogers 5y ago
Using mmap() is popular because it is easy despite its many sharp edges, but outside a limited number of access patterns and hardware setups it is often quite slow because the page cache behavior is poor for many use cases. Manually buffering is often simpler, and with better performance, than trying to make the page cache perform well. Furthermore, mmap() doesn't work for large storage volumes generally, so you might find yourself doing your own buffering regardless. If you have a relatively simple access pattern, and most applications do, then you don't need to be a database kernel engineer to design a trivial manual buffering system that can significantly outperform mmap(). It is a handful of lines of code in most languages.
- vlovich123 5y ago
  Hmmm… this does not match my experience at all. Some user space buffering helps but that’s just because it’s easier to write your application logic (ie read 4 bytes, read 17 bytes, read 46kb). I have not heard the advice to layer another layer of buffer on top of if that. The reason is that it lets you write such code efficiently without paying the syscall overhead. These buffers are very stupid and io_uring or mmap (with a read ahead option) will perform better for larger files because there’s either no extra memory copy or the cost is hidden better. I’ve seen people struggle and fail to outperform the page cache. The only time I’ve beaten it is when it was caching the wrong thing (it was caching compressed and uncompressed pages via fuse so telling it to forget about the compressed pages after decompression eased memory pressure).
  - 5y ago

henrydark 5y ago

Good piece, goes into such depth that probably not all scientists must know all of this :) That said, my own experience of noticing what people's machines are wasting time on around me is very different. For scientists at firms (outside of academia) a major bottleneck is downloading, decompressing and shuffling data around. Unfortunately the lessons in the piece are in fact detrimental to this workload: 1. decompression is usually not faster than 500MB/s, which is much slower than SSD or network, but I still see compression everywhere with the idea that "cpu is faster". 2. Standard tools for downloading data, like awscli, are crazy slow compared to the actual ability of the network cards that are attached to the machines we (in big firms) use in the cloud. Specifically awscli caps at about 250MB/s no matter the configuration or network bandwidth, and people think it's normal because "network io is slow". (this of course doesn't apply to scientists who work out of the cloud, with much lower download speeds) None of this contradicts anything in the piece. It's that the units of work for different sectors are different. Whereas a geoscientist working on modeling the ocean has bottlenecks in computation, scientists at firms often have them in data.

ReleaseCandidat 5y ago
> 1. decompression is usually not faster than 500MB/s, which is much slower than SSD or network, There are some, but you have to select the right one. https://github.com/inikep/lzbench#benchmarks https://github.com/inikep/lzbench#benchmarks
throwaway984393 5y ago
I think the distinction is in systems programming versus system programming. Systems programming involves multiple systems with many components. It's not uncommon to find you simply can't make a solution work any better than it's designed, and you need to develop brand new solutions to problems (which makes the system even bigger). System programming is simpler because it's one system and one or two codebases. It can still be quite complex, but you can often just optimize the one solution rather than having to come up with new ones.
criticaltinker 5y ago
If you’re working with time series data and using a vectorized algorithm implemented with SIMD instructions you can achieve decompression speeds approaching 15 GB/s [1]. Anecdotally when working with biological signals at my day job, compression is a massive win and an absolute no brainer tradeoff when shuffling data across the network or even just storing in memory. That said, I think your first point is still reasonable for certain types of data and compression algorithms. [1] https://github.com/lemire/SIMDCompressionAndIntersection https://github.com/lemire/SIMDCompressionAndIntersection

spekcular 5y ago

If I'm reading the table under "fast instruction, slow instruction" correctly, float multiplication has half the reciprocal throughput of integer multiplication, which means float multiplication has twice the throughput. So, if I want to multiply a bunch of integers together, does this mean it's faster for me to represent them as floats (even though I don't need the additional precision)?

NohatCoder 5y ago
It depends on the processor, but yes in some cases using floats would be faster for this reason. Please note that floats don't have "additional" precision, they have a different structure that means they can express a different set of numbers. Integers should in general always be preferred when the data you deal with is of integral nature. Even if your CPU can do some instructions faster with floats it is generally not the kind of optimization you would want to make. You would end up making things more complicated for little practical benefit. Funny thing by the way, integer arithmetic units are generally cheaper to make than float units, but most multiply/division heavy code is floating point, so CPU makers prioritise the floating point units.
- spekcular 5y ago
  Thank you for the detailed explanation.

dang 3y ago

That site now directs to malware, but the article is readable here: http://web.archive.org/web/20221208105716/https://biojulia.net/post/hardware/ http://web.archive.org/web/20221208105716/https://biojulia.n...