10 ms·

Optimizations can have unexpectedly large effects when combined with caches

282 points by zdw 4y ago

mjb 4y ago
I really love this point. For most applications the optimal size of a (time-based) cache is around the point that keeping an item around in the cache is equal to the cost of accessing it again. This is a point that Jim Gray made in the classic "The 5 Minute Rule for Trading Memory for Disc Accesses" (http://www.hpl.hp.com/techreports/tandem/TR-86.1.pdf http://www.hpl.hp.com/techreports/tandem/TR-86.1.pdf) back in 1986. A side effect of that is that systems that run faster, and therefore access each cached it more often, get more benefit out of a cache by pushing that set point out further. Whether that happens in any given system depends a lot on the access patterns, and especially temporal locality within those access patterns, but the effect is real in a lot of different kinds of systems. > It's common wisdom that systems with caches can fail badly when the caches are cold. When every request misses the cache, the system is overloaded, and never recovers. What I observed was the mirror image. The bottom line is that both locality and memoization are huge contributor to the performance of systems, both positive and negative. When folks (including me) point out the risks of caches, we're not implying that caches are always bad. Instead, it's pointing out a risk that sometimes they are too good, leading to a system that can't function when the normal level of locality and memoization are not available.
- bombcar 4y ago
  Reminds me of: There are only two hard things in Computer Science: cache invalidation, and naming things, and off-by-one errors. What people don't realize is caches are much harder than just invalidation. I'm a fan of continually testing (and running) things without cache at all just to verify what they do, caching can often cover a multitude of sins that you really don't need to be doing.
  - hinkley 4y ago
    Caches are global shared state. Correction: lossy global shared state. 'Naming things' when they are in a global namespace is really hard, to the point where I'm starting to thing that cache invalidation and naming things are not two problems but the same problem in two manifestations.
    - kwhitefoot 4y ago
      > Caches are global shared state Not necessarily. I designed an object oriented simulation system which had a lot of calculated properties where the calculation was a substantial amount of work but the result would be queried many times before the input the function changed. Caching the results inside the object provided a substantial speed up (orders of magnitude) while keeping the cache completely local and hidden from the client code and keeping the program very modular. Keeping the cache inside the objects allowed us to keep the parts of the program neatly separated and minimized the amount of knowledge that client code, and more importantly client coders, needed to have about the objects they were using.
      - hinkley 4y ago
        What happens when the system of record changes the answer halfway through the transaction? By caching answers to calculations you've moved the source of truth from the function to the cache. But the cache is also acting as a proxy for the system of record, which is a conflict of interests. That's substantially part of the problem with cache invalidation. ETA: There's also a common category error here that I see with people who grossly misunderstand DP as 'caching'. First that they think that caching and memoization are the same thing, which they emphatically are not, and second that DP is about memoization, which means they listened for the first ten minutes and then fell asleep. Tabulation is much more powerful than memoization, so equating DP with memoization is impoverishing it and yourself. But importantly here, the distinction between memoization and caching is that memoization is scoped while caching is global. With memoization a single call tree gets the same answers every time, which means the mental process of Local Reasoning is maintained. If you know the call graph for this operation you know everything that interacts with it. You know if 'eviction' is even a thing. You also know that your unit tests are testing the code as it will behave under load, not some fictional happy path. tl;dr: If your calculation is important it should show up in your data flow, not arrive out of band from a magic oracle.
- hinkley 4y ago
  Caching is a weapon of last resort. In most cases it is the last major performance improvement you will see from your code, even if other options were available, and any information architecture problems you have are set in stone or will get worse. Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods. This turns your cache into global shared state, and once that happens you can't turn the cache off again. There are other problems with this arrangement as well. The easiest to understand (and yet I've failed on at least one project) is that if the working set for a problem ever becomes proportional to the size of the cache (>1 for single task systems, > 1/n for systems with n concurrent tasks), then you can end up evicting data in the middle of a transaction. The obvious consequence is that it causes performance to jump off a cliff. Particularly bad behavior with LRUs but switching to a different statistical model simply reduces the height of the cliff. There is still a cliff. The less obvious consequence is now you have concurrency bugs, because the code assumes that decisions made about the data will go the same way on each call, but due to eviction, two halves of the transaction may make conflicting decisions with undefined behavior. But speaking as the guy who fixes the perf issues other people gave up on: Once you introduce caching you poison the performance well. Flame charts lie when cache is used, and after people rely on the cache it is lying in the other direction when the cache is off, because you're calling that path 5 times now and it's swamping other bottlenecks. This is what I really mean when I say it's a weapon of last resort. Because once you use it, it attempts to take any other options off the table. Caching will be most of the last order of magnitude in scalability that your system ever sees. Every 'win' from there on will take more time and energy than most people are willing to invest.
  - brightball 4y ago
    I like applying the Circuit Breaker approach to caching. Rather than applying a cache everywhere, setup a cache wrapper that only triggers if the accessed resource is slower than a certain threshold. The goal is for the cache to go unused most of the time, but during times of unexpected system stress the caches will kick in, provide relief to the system and reduce impact to users.
    - 4y ago
- gav 4y ago
  > Instead, it's pointing out a risk that sometimes they are too good, leading to a system that can't function when the normal level of locality and memoization are not available. And at that point cache management starts to be a huge problem. Clearing caches becomes a death spiral because your system can't actually support your normal load. For example I worked with a large retailer, who had a giant monolithic enterprise Java ecommerce system. It was so slow that product detail pages took upwards of 30 seconds to render. There was a cache in front of that system (Ehcache), then a page-level cache (Varnish) in front of that, then finally Akamai took 98% of the load. Rollouts were terrifying, you cleared the cache and the site might come back up. Changing Akamai configuration was also similarly risky. It's critical that systems don't become cache dependent for load, they should be a performance boost, not a required component.
RicoElectrico 4y ago
What kind of batch process could that be?
- mananaysiempre 4y ago
  Judging by the off-hand mentions in the article, something along the lines of ingesting the cargo manifest of a large ship?
- boruto 4y ago
  could be thousands of RPC calls ;)
PaulKeeble 4y ago
Multiple times I have found the impact of cache performance to be dramatic. I find that the usual optimisations in three aspects are usually obvious to achieve and can bring surprisingly big improvements. The first is that RAM accessed randomly or in the wrong direction performs very poorly compared to sequential access in order. The same is true of cache because its much easier to predict and prefetched data when its accessed serially. The effect is so dramatic that its sometimes worth "accessing" data you don't need to get access to this hardware optimisation to maintain serial access. At times for example I have found Treemaps and listmaps perform better than hashmaps as the cheap search serially and in memory order surprisingly out does the O(log(n)) or O(n) algorithms and the cost of the hash. If you have a data structure that is larger than optimal and overflows cache lines it can have big impacts on cache utilisation and performance if the algorithm doesn't need all the fields. Object orientation impacts quite heavily on this and certain designs can make very large objects and these often get poor cache usage. Unnecessary allocation and deallocation of memory in algorithms often results in the cache getting thrashed and avoiding this and using the same memory can result in better hot cache usage and reducing garbage collection usage. Most profilers can give you a good idea of the allocations but the other two are often harder to find and mostly just show up in cache misses.
dathinab 4y ago
Artickle: speed up lead to avoiding cache misses which lead to further speed up which lead to further less cache misses This kind of interaction is important to keep in mind. It applies to all kinds of caches in all kinds of situations. To some degree even the instruction and data caches of modern cpus. But most interesting is that in complex systems in (very very) rare cases the "whole system effect" can go the other way around too (due to a now faster components trashing the cache for other components in a way which makes the system as a whole slower).
hinkley 4y ago
About 15 years ago I was on a team that was having so many problems with caching that I began to see it for the liability that it is. It started with a 2-4 week delay each release cycle while they fixed caching bugs the testers had found, but once I joined the project I saw how deep the problem went. A number of their features invited a full table scan, which is fundamentally incompatible with an LRU cache, and in particular has disastrous stair step functions as the working set grows. Note that this is also a couple of years before anyone had heard of memcached so these were DIY implementations with multiple potential sources of bugs. About the time I found my feet on the project someone wanted to add caching to yet another workflow that was taking 30 seconds to run. It was an admin workflow so there was a little tolerance of slow response times, but not 30 seconds. I volunteered for the story because I thought we could do better than more caches. Through conversation, step debugging, and profiling I found the spot where they had wanted the caching. What the feature was doing was two relatively expensive lookups of the same data, filtering it on two very different dimensions, and returning the intersection of the two. In typical ORM fashion, both of the concerns were taking it upon themselves to fetch their own data instead of having it handed to them. Since they were both called from the same function, I hoisted the lookup out of the callee into the caller, sending the same list to both. Then I changed the code again to feed the result of call 1 to call 2, to make it work more like consecutive filters. The end result was that the page now loaded in 3 seconds, and I was able to write better unit tests for this code. One might expect that most of that improvement came from running the second filter on a much smaller list, but that's where you'd be wrong. The majority of that improvement came from the hoisting, not the chaining. Due to the amount of work in the filters, the initial lookup shouldn't have accounted for more than a third of the page load time, but just sharing the same data dropped the response time to around 6 seconds. Chaining the filters ended up only saving 3 seconds. The problem with large data sets is that CPUs have caches too. And in a garbage collected programming language, there are additional costs to carrying around duplicated data, particularly if it is in tree form like this was. Most likely we were smashing the hell out of CPU caches and introducing extra GC pressure to boot. By passing in the same object tree the second call followed warm and/or hot paths through heap memory. I'd had a similar but less dramatic experiences a few years before that, where I identified that a function, which accounted for 10% of the runtime, was being invoked roughly twice as often as the problem required. By that point 5% looked like low hanging fruit, so I eliminated the duplicate calls and reran the benchmarks. But the results didn't make sense so I tweaked and prodded and fiddled. The new time was 20% lower than the old time, for what should have been 5%. This is already way too long so I won't go into the reasons here, but in addition to the scenarios I mentioned for the previous issue, there's also problems of timing accuracy in profilers, and cyclic use patterns that can misattribute the expense of a call to siblings or the parent. A function that creates many objects may never reach the GC threshold, but a large allocation later in the sequence diagram may. Or collide with cache lines that are needed again later in the call graph.
lob_it 4y ago
Caching db queries and media assets already got a big boost a decade ago going from spinning rust to ssd to even faster NVME. Optimizing DB queries on spinning rust was an exciting experience though. Chasing milliseconds with caching and warming caches up still brings back good memories :) Scale and architecture makes new designs much easier. 1Mb of code... A 100Mb database and 1Gb of media assets is becoming a relic like "big iron". Blank slate canvases should skip a lot of it. Technology debt only gets harder to manage over time.
nly 4y ago
Caching is easy, cache invalidation is hard.
- latchkey 4y ago
  "There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors." https://twitter.com/codinghorror/status/506010907021828096 https://twitter.com/codinghorror/status/506010907021828096
  - withinboredom 4y ago
    I have no idea why you’re quoting that tweet. This saying is much older than 2014.
nerpderp82 4y ago
We really don't know what causes perf differences, layout has more of an effect than -O1 to -O3. "Performance Matters" by Emery Berger https://www.youtube.com/watch?v=r-TLSBdHe1A https://www.youtube.com/watch?v=r-TLSBdHe1A
- tmtvl 4y ago
  I wonder if he mentions running from a different directory and running as a different user to highlight how annoying the effect from the environment is, as USER and PWD are part of the environment.
- josephg 4y ago
  If thats true, it seems like you could make a great optimization pass which chooses a code layout in order to make the program run faster. I wonder how predictable code layout is on performance across different CPUs. Sounds like an interesting problem!
  - nerpderp82 4y ago
    Totally agree, area is ripe for a lot of experimentation and data visualization. I think it would be fun to see what is the minimal amount of insertions or shifts in layout to get the most pathological decrease in performance. Could be a great vehicle to challenge performance assumptions we all have.
ISL 4y ago
This happens in life, too. When your counterparts lose continuity with your work, they turn to other work and the context-switch to return to working with you becomes increasingly hard. Helps to be quick.
- TeMPOraL 4y ago
  Very much true. I recently went through flat remodeling which took a month longer than it should and resulted in lower quality, because at the very beginning, I couldn't supply enough work for the contractors (we had to rush into it because ${life reasons}, so not everything was specced out when the work started), so the initial team of workers ended up taking better jobs abroad, and our general contractor had to find people to fill in.
joosters 4y ago
"Almost all programming can be viewed as an exercise in caching."
musingsole 4y ago
There's a similarity in the performance of caches and ML algorithms -- they can have some theoretical absolute performance (or worst case scenario) but whether or not that falls into harmony or discord with the real world use is another question altogether. Statistics may give you reasons for your faith, but you have to leap on to the next workload regardless.
mannyv 4y ago
I worked on a system once where the majority of time seemed to be in “ObjectCacheMiss.” I’m looking at you, MagicCap! Needless to say a a cache that slows you down isn’t really a cache.
drivebycomment 4y ago
When thinking about cache, instead of thinking about cache misses directly and thinking about time based optimization trying to improve the temporal locality, it is often easier and more fruitful to consider "working set size" - how much memory the code / loop / function / algorithm is touching at a time.
pierrebai 4y ago
So.... they could have mostly fixed it by increasing the cache timeout? No code changes, just increase the timeout. The real lesson is to really know your system and how it behaves. They ran a profiler, but in the call stack they did not see the computation was triggered by a cache miss...
tyingq 4y ago
Similar for anything that improves a large swath of batch jobs where they suddenly stop running on top of each other.

282 points by zdw 4y ago

mjb 4y ago

I really love this point. For most applications the optimal size of a (time-based) cache is around the point that keeping an item around in the cache is equal to the cost of accessing it again. This is a point that Jim Gray made in the classic "The 5 Minute Rule for Trading Memory for Disc Accesses" (http://www.hpl.hp.com/techreports/tandem/TR-86.1.pdf http://www.hpl.hp.com/techreports/tandem/TR-86.1.pdf) back in 1986. A side effect of that is that systems that run faster, and therefore access each cached it more often, get more benefit out of a cache by pushing that set point out further. Whether that happens in any given system depends a lot on the access patterns, and especially temporal locality within those access patterns, but the effect is real in a lot of different kinds of systems. > It's common wisdom that systems with caches can fail badly when the caches are cold. When every request misses the cache, the system is overloaded, and never recovers. What I observed was the mirror image. The bottom line is that both locality and memoization are huge contributor to the performance of systems, both positive and negative. When folks (including me) point out the risks of caches, we're not implying that caches are always bad. Instead, it's pointing out a risk that sometimes they are too good, leading to a system that can't function when the normal level of locality and memoization are not available.

bombcar 4y ago
Reminds me of: There are only two hard things in Computer Science: cache invalidation, and naming things, and off-by-one errors. What people don't realize is caches are much harder than just invalidation. I'm a fan of continually testing (and running) things without cache at all just to verify what they do, caching can often cover a multitude of sins that you really don't need to be doing.
- hinkley 4y ago
  Caches are global shared state. Correction: lossy global shared state. 'Naming things' when they are in a global namespace is really hard, to the point where I'm starting to thing that cache invalidation and naming things are not two problems but the same problem in two manifestations.
  - kwhitefoot 4y ago
    > Caches are global shared state Not necessarily. I designed an object oriented simulation system which had a lot of calculated properties where the calculation was a substantial amount of work but the result would be queried many times before the input the function changed. Caching the results inside the object provided a substantial speed up (orders of magnitude) while keeping the cache completely local and hidden from the client code and keeping the program very modular. Keeping the cache inside the objects allowed us to keep the parts of the program neatly separated and minimized the amount of knowledge that client code, and more importantly client coders, needed to have about the objects they were using.
    - hinkley 4y ago
      What happens when the system of record changes the answer halfway through the transaction? By caching answers to calculations you've moved the source of truth from the function to the cache. But the cache is also acting as a proxy for the system of record, which is a conflict of interests. That's substantially part of the problem with cache invalidation. ETA: There's also a common category error here that I see with people who grossly misunderstand DP as 'caching'. First that they think that caching and memoization are the same thing, which they emphatically are not, and second that DP is about memoization, which means they listened for the first ten minutes and then fell asleep. Tabulation is much more powerful than memoization, so equating DP with memoization is impoverishing it and yourself. But importantly here, the distinction between memoization and caching is that memoization is scoped while caching is global. With memoization a single call tree gets the same answers every time, which means the mental process of Local Reasoning is maintained. If you know the call graph for this operation you know everything that interacts with it. You know if 'eviction' is even a thing. You also know that your unit tests are testing the code as it will behave under load, not some fictional happy path. tl;dr: If your calculation is important it should show up in your data flow, not arrive out of band from a magic oracle.
hinkley 4y ago
Caching is a weapon of last resort. In most cases it is the last major performance improvement you will see from your code, even if other options were available, and any information architecture problems you have are set in stone or will get worse. Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods. This turns your cache into global shared state, and once that happens you can't turn the cache off again. There are other problems with this arrangement as well. The easiest to understand (and yet I've failed on at least one project) is that if the working set for a problem ever becomes proportional to the size of the cache (>1 for single task systems, > 1/n for systems with n concurrent tasks), then you can end up evicting data in the middle of a transaction. The obvious consequence is that it causes performance to jump off a cliff. Particularly bad behavior with LRUs but switching to a different statistical model simply reduces the height of the cliff. There is still a cliff. The less obvious consequence is now you have concurrency bugs, because the code assumes that decisions made about the data will go the same way on each call, but due to eviction, two halves of the transaction may make conflicting decisions with undefined behavior. But speaking as the guy who fixes the perf issues other people gave up on: Once you introduce caching you poison the performance well. Flame charts lie when cache is used, and after people rely on the cache it is lying in the other direction when the cache is off, because you're calling that path 5 times now and it's swamping other bottlenecks. This is what I really mean when I say it's a weapon of last resort. Because once you use it, it attempts to take any other options off the table. Caching will be most of the last order of magnitude in scalability that your system ever sees. Every 'win' from there on will take more time and energy than most people are willing to invest.
- brightball 4y ago
  I like applying the Circuit Breaker approach to caching. Rather than applying a cache everywhere, setup a cache wrapper that only triggers if the accessed resource is slower than a certain threshold. The goal is for the cache to go unused most of the time, but during times of unexpected system stress the caches will kick in, provide relief to the system and reduce impact to users.
  - 4y ago
gav 4y ago
> Instead, it's pointing out a risk that sometimes they are too good, leading to a system that can't function when the normal level of locality and memoization are not available. And at that point cache management starts to be a huge problem. Clearing caches becomes a death spiral because your system can't actually support your normal load. For example I worked with a large retailer, who had a giant monolithic enterprise Java ecommerce system. It was so slow that product detail pages took upwards of 30 seconds to render. There was a cache in front of that system (Ehcache), then a page-level cache (Varnish) in front of that, then finally Akamai took 98% of the load. Rollouts were terrifying, you cleared the cache and the site might come back up. Changing Akamai configuration was also similarly risky. It's critical that systems don't become cache dependent for load, they should be a performance boost, not a required component.

RicoElectrico 4y ago

What kind of batch process could that be?

mananaysiempre 4y ago
Judging by the off-hand mentions in the article, something along the lines of ingesting the cargo manifest of a large ship?
boruto 4y ago
could be thousands of RPC calls ;)

PaulKeeble 4y ago

Multiple times I have found the impact of cache performance to be dramatic. I find that the usual optimisations in three aspects are usually obvious to achieve and can bring surprisingly big improvements. The first is that RAM accessed randomly or in the wrong direction performs very poorly compared to sequential access in order. The same is true of cache because its much easier to predict and prefetched data when its accessed serially. The effect is so dramatic that its sometimes worth "accessing" data you don't need to get access to this hardware optimisation to maintain serial access. At times for example I have found Treemaps and listmaps perform better than hashmaps as the cheap search serially and in memory order surprisingly out does the O(log(n)) or O(n) algorithms and the cost of the hash. If you have a data structure that is larger than optimal and overflows cache lines it can have big impacts on cache utilisation and performance if the algorithm doesn't need all the fields. Object orientation impacts quite heavily on this and certain designs can make very large objects and these often get poor cache usage. Unnecessary allocation and deallocation of memory in algorithms often results in the cache getting thrashed and avoiding this and using the same memory can result in better hot cache usage and reducing garbage collection usage. Most profilers can give you a good idea of the allocations but the other two are often harder to find and mostly just show up in cache misses.

dathinab 4y ago

Artickle: speed up lead to avoiding cache misses which lead to further speed up which lead to further less cache misses This kind of interaction is important to keep in mind. It applies to all kinds of caches in all kinds of situations. To some degree even the instruction and data caches of modern cpus. But most interesting is that in complex systems in (very very) rare cases the "whole system effect" can go the other way around too (due to a now faster components trashing the cache for other components in a way which makes the system as a whole slower).

hinkley 4y ago

About 15 years ago I was on a team that was having so many problems with caching that I began to see it for the liability that it is. It started with a 2-4 week delay each release cycle while they fixed caching bugs the testers had found, but once I joined the project I saw how deep the problem went. A number of their features invited a full table scan, which is fundamentally incompatible with an LRU cache, and in particular has disastrous stair step functions as the working set grows. Note that this is also a couple of years before anyone had heard of memcached so these were DIY implementations with multiple potential sources of bugs. About the time I found my feet on the project someone wanted to add caching to yet another workflow that was taking 30 seconds to run. It was an admin workflow so there was a little tolerance of slow response times, but not 30 seconds. I volunteered for the story because I thought we could do better than more caches. Through conversation, step debugging, and profiling I found the spot where they had wanted the caching. What the feature was doing was two relatively expensive lookups of the same data, filtering it on two very different dimensions, and returning the intersection of the two. In typical ORM fashion, both of the concerns were taking it upon themselves to fetch their own data instead of having it handed to them. Since they were both called from the same function, I hoisted the lookup out of the callee into the caller, sending the same list to both. Then I changed the code again to feed the result of call 1 to call 2, to make it work more like consecutive filters. The end result was that the page now loaded in 3 seconds, and I was able to write better unit tests for this code. One might expect that most of that improvement came from running the second filter on a much smaller list, but that's where you'd be wrong. The majority of that improvement came from the hoisting, not the chaining. Due to the amount of work in the filters, the initial lookup shouldn't have accounted for more than a third of the page load time, but just sharing the same data dropped the response time to around 6 seconds. Chaining the filters ended up only saving 3 seconds. The problem with large data sets is that CPUs have caches too. And in a garbage collected programming language, there are additional costs to carrying around duplicated data, particularly if it is in tree form like this was. Most likely we were smashing the hell out of CPU caches and introducing extra GC pressure to boot. By passing in the same object tree the second call followed warm and/or hot paths through heap memory. I'd had a similar but less dramatic experiences a few years before that, where I identified that a function, which accounted for 10% of the runtime, was being invoked roughly twice as often as the problem required. By that point 5% looked like low hanging fruit, so I eliminated the duplicate calls and reran the benchmarks. But the results didn't make sense so I tweaked and prodded and fiddled. The new time was 20% lower than the old time, for what should have been 5%. This is already way too long so I won't go into the reasons here, but in addition to the scenarios I mentioned for the previous issue, there's also problems of timing accuracy in profilers, and cyclic use patterns that can misattribute the expense of a call to siblings or the parent. A function that creates many objects may never reach the GC threshold, but a large allocation later in the sequence diagram may. Or collide with cache lines that are needed again later in the call graph.

lob_it 4y ago

Caching db queries and media assets already got a big boost a decade ago going from spinning rust to ssd to even faster NVME. Optimizing DB queries on spinning rust was an exciting experience though. Chasing milliseconds with caching and warming caches up still brings back good memories :) Scale and architecture makes new designs much easier. 1Mb of code... A 100Mb database and 1Gb of media assets is becoming a relic like "big iron". Blank slate canvases should skip a lot of it. Technology debt only gets harder to manage over time.

nly 4y ago

Caching is easy, cache invalidation is hard.

latchkey 4y ago
"There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors." https://twitter.com/codinghorror/status/506010907021828096 https://twitter.com/codinghorror/status/506010907021828096
- withinboredom 4y ago
  I have no idea why you’re quoting that tweet. This saying is much older than 2014.

nerpderp82 4y ago

We really don't know what causes perf differences, layout has more of an effect than -O1 to -O3. "Performance Matters" by Emery Berger https://www.youtube.com/watch?v=r-TLSBdHe1A https://www.youtube.com/watch?v=r-TLSBdHe1A

tmtvl 4y ago
I wonder if he mentions running from a different directory and running as a different user to highlight how annoying the effect from the environment is, as USER and PWD are part of the environment.
josephg 4y ago
If thats true, it seems like you could make a great optimization pass which chooses a code layout in order to make the program run faster. I wonder how predictable code layout is on performance across different CPUs. Sounds like an interesting problem!
- nerpderp82 4y ago
  Totally agree, area is ripe for a lot of experimentation and data visualization. I think it would be fun to see what is the minimal amount of insertions or shifts in layout to get the most pathological decrease in performance. Could be a great vehicle to challenge performance assumptions we all have.

ISL 4y ago

This happens in life, too. When your counterparts lose continuity with your work, they turn to other work and the context-switch to return to working with you becomes increasingly hard. Helps to be quick.

TeMPOraL 4y ago
Very much true. I recently went through flat remodeling which took a month longer than it should and resulted in lower quality, because at the very beginning, I couldn't supply enough work for the contractors (we had to rush into it because ${life reasons}, so not everything was specced out when the work started), so the initial team of workers ended up taking better jobs abroad, and our general contractor had to find people to fill in.

joosters 4y ago

"Almost all programming can be viewed as an exercise in caching."

musingsole 4y ago

There's a similarity in the performance of caches and ML algorithms -- they can have some theoretical absolute performance (or worst case scenario) but whether or not that falls into harmony or discord with the real world use is another question altogether. Statistics may give you reasons for your faith, but you have to leap on to the next workload regardless.

mannyv 4y ago

I worked on a system once where the majority of time seemed to be in “ObjectCacheMiss.” I’m looking at you, MagicCap! Needless to say a a cache that slows you down isn’t really a cache.

drivebycomment 4y ago

When thinking about cache, instead of thinking about cache misses directly and thinking about time based optimization trying to improve the temporal locality, it is often easier and more fruitful to consider "working set size" - how much memory the code / loop / function / algorithm is touching at a time.

pierrebai 4y ago

So.... they could have mostly fixed it by increasing the cache timeout? No code changes, just increase the timeout. The real lesson is to really know your system and how it behaves. They ran a profiler, but in the call stack they did not see the computation was triggered by a cache miss...

tyingq 4y ago

Similar for anything that improves a large swath of batch jobs where they suddenly stop running on top of each other.