5 ms·

A cache miss isn't noticeable by a human. Code that cache misses a lot runs 10-100x slower than code that takes into consideration that it's running on a physi

A cache miss isn't noticeable by a human. Code that cache misses a lot runs 10-100x slower than code that takes into consideration that it's running on a physical machine and not an abstraction. That is very noticeable by humans. Even when your data structures and algos are designed with a nice O(logN), it's very noticeable when one program bogs down with 1/6 the data compared to another. I work in games, so the story I tell new kids is: The PlayStation2 ran at 300Mhz and a cache miss would cost you 50 cycles. The PlayStation3 ran at 3200Mhz and a cache miss would cost you 500 cycles. So, if you are cache missing a lot, your PS3 game will run as if it were on a PS2. In other words, not paying attention to cache make your computer run like it's 10 years older than it is. You paid for a modern machine, but you are getting the results of a Craigslist junker. This is true outside of games. It's the reason 4x2Ghz cellphones struggle to run seemingly simple apps. It's a big part of the reason people struggle to orchestrate armies of servers (distributed computing is easier when it's 90% less distributed). Is it really harder to work with the cache system instead of ignoring it? Yeah, it requires a tiny bit of study an a little bit of planning. In contrast, the theme I see a lot online is to completely dismiss physical reality in favor or theory. And, the theme I see almost universally in the students (and many senior engineers) I interview is complete ignorance of the very existence of cache and it's effects on the code they write. It's very concerning...

qwertyuiop924 10y ago
No, I don't deny it's important to know that cache misses exist, and what they can do: to the contrary, it's vital. However, in 90% of applications, it's not going to matter, because those applications are spending hundreds of cycles waiting anyways: Disk or network IO, user input, all that stuff is way slower than a cache miss. If you're writing a video game, or a database, or other software with very high soft-realtime speed requirements, or heavy data access, by all means, optimize to avoid cache miss. But if you're writing a company-internal Rails app, nobody's going to notice, even if you're getting cache miss after cache miss. Which you probably won't. Actually, if your language isn't compiled, a cache miss is the least of your worries, perf-wise. And now I've got to see if I can optimize my code to avoid cache misses. But the code's in Scheme, so unless the initial access cost is amortized, I'm already doomed...
- yxhuvud 10y ago
  Considering the Ruby object and memory models, you probably will get cache miss after cache miss. You just won't have the tools to do anything about them.
  - qwertyuiop924 10y ago
    You're probably right. I was just thinking, "It can't possibly be THAT bad," especially seeing as there are so many objects that are referenced frequently.
- Koromix 10y ago
  You're looking at this problem backward. For example, you mention user input. Users may need a second to click or touch a button, but when they do the software should react instantly, and that does not leave you that many cycles. My smartphone's lock screen is my go-to example: most times it fails to follow my finger, and I barely have anything running on it. Most of the dynamic languages are data and instruction cache-miss machines. They chase objects and pointers all around the memory.
  - qwertyuiop924 10y ago
    >My smartphone's lock screen is my go-to example: most times it fails to follow my finger, and I barely have anything running on it. ...That doesn't sound like a cache miss. Knowing Android, A cache miss is probably the least of your worries. >Users may need a second to click or touch a button, but when they do the software should react instantly, and that does not leave you that many cycles. You raise a good point... >Most of the dynamic languages are data and instruction cache-miss machines. They chase objects and pointers all around the memory. ...and this is part of my point. If you look at a problem and think, "A high-level language is fast enough," then you are implicitly saying that the latency of a cache miss is acceptable. And IME, in most cases that's true. I mean, heck, I'm using Scheme, so while I may have pointer chases like the Amazon has trees, but I CAN optimize them into array lookups, and my code is compiled: not great, but better than most HLLs. It's the same argument as always: perf vs. development speed. You can be in the C and FP loop, or the Lisp and JS loop.

qwertyuiop924 10y ago

No, I don't deny it's important to know that cache misses exist, and what they can do: to the contrary, it's vital. However, in 90% of applications, it's not going to matter, because those applications are spending hundreds of cycles waiting anyways: Disk or network IO, user input, all that stuff is way slower than a cache miss. If you're writing a video game, or a database, or other software with very high soft-realtime speed requirements, or heavy data access, by all means, optimize to avoid cache miss. But if you're writing a company-internal Rails app, nobody's going to notice, even if you're getting cache miss after cache miss. Which you probably won't. Actually, if your language isn't compiled, a cache miss is the least of your worries, perf-wise. And now I've got to see if I can optimize my code to avoid cache misses. But the code's in Scheme, so unless the initial access cost is amortized, I'm already doomed...

yxhuvud 10y ago
Considering the Ruby object and memory models, you probably will get cache miss after cache miss. You just won't have the tools to do anything about them.
- qwertyuiop924 10y ago
  You're probably right. I was just thinking, "It can't possibly be THAT bad," especially seeing as there are so many objects that are referenced frequently.
Koromix 10y ago
You're looking at this problem backward. For example, you mention user input. Users may need a second to click or touch a button, but when they do the software should react instantly, and that does not leave you that many cycles. My smartphone's lock screen is my go-to example: most times it fails to follow my finger, and I barely have anything running on it. Most of the dynamic languages are data and instruction cache-miss machines. They chase objects and pointers all around the memory.
- qwertyuiop924 10y ago
  >My smartphone's lock screen is my go-to example: most times it fails to follow my finger, and I barely have anything running on it. ...That doesn't sound like a cache miss. Knowing Android, A cache miss is probably the least of your worries. >Users may need a second to click or touch a button, but when they do the software should react instantly, and that does not leave you that many cycles. You raise a good point... >Most of the dynamic languages are data and instruction cache-miss machines. They chase objects and pointers all around the memory. ...and this is part of my point. If you look at a problem and think, "A high-level language is fast enough," then you are implicitly saying that the latency of a cache miss is acceptable. And IME, in most cases that's true. I mean, heck, I'm using Scheme, so while I may have pointer chases like the Amazon has trees, but I CAN optimize them into array lookups, and my code is compiled: not great, but better than most HLLs. It's the same argument as always: perf vs. development speed. You can be in the C and FP loop, or the Lisp and JS loop.