9 ms·

In my extensive experience of optimising stuff... > Caching is a weapon of last resort. In most cases it is the last major performance improvement ...is wrong

In my extensive experience of optimising stuff... > Caching is a weapon of last resort. In most cases it is the last major performance improvement ...is wrong. It is one of the very first things I reach for because it is the most effective and simplest, once you've eliminated O(n^2) behaviour etc. It's not a magic wand but nothing is.

hinkley 4y ago
That's not a very convincing argument. "I use it because it works" without refuting any reasons I presented as to why it "works" but breaks your other options. How do you teach people to effectively use a profiler on code with pernicious caching? Part of why I'm the "Cleaner" is because I've worked with any number of people who will present "evidence" that everything that can be done has been done. Here are the metrics from a year ago when we turned on the cache. Here are the flame charts. See? Nothing to see here. They get really uncomfortable six months later when I'm reporting 2-3x in code that can't be optimized further. And in each case everything was made more difficult by the people ahead of me chumming the waters. They were partially correct. There was in fact nothing left for them to see, so nothing that they knew to do. But "there's nothing to see here" and "there's nothing here" are two very, very different statements.
- cogman10 4y ago
  The issue with caches is they are easy to apply. You can easily drop a cache in any place you load a resource without significant code changes. I tend to think they are ok weapons of sometimes first resort (rarely changing resource with heavy system impact). But you'll definitely get better results simply by rethinking and rearchitecting stuff. (Did you really need to load the database just to lookup a single value?)
  - hinkley 4y ago
    > You can easily drop a cache in any place you load a resource without significant code changes. Only when the cache is brand new. The problem with people with cache addiction is that they think they're free. So once they know a cache is there, they begin to lean on it, and lean on it. I've seen it with many different teams in a number of verticals. Sometimes while I'm there, sometimes before I arrived. It's the people not the company. This illusion that you can always turn it off later is dangerous. That's only true if you have an impeccable information architecture when you start. And if you have that, you've already done a lot of heavy lifting before caching becomes your only option.
- zasdffaa 4y ago
  It's dificult to refute your argument because I don't understand your points: > Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods what does that even mean? What does 'data dependencies' mean specicially? > then you can end up evicting data in the middle of a transaction So in the middle of a transaction, instead of looking up data from a key - very cheap - you should do ... what? Recalculate it instead? > The less obvious consequence is now you have concurrency bugs, because the code assumes that decisions made about the data will go the same way on each call, but due to eviction, two halves of the transaction may make conflicting decisions with undefined behavior Assuming I even understand this, you are saying where there's concurrencty there's a risk of pulling stale data out of a cache. Well, true. That's just something you have to be careful of. Add a Tx reference to the cache key. > Once you introduce caching you poison the performance well. Not my experience at all. I can't understand why you're saying this. > Flame charts lie when cache is used How so? > and after people rely on the cache it is lying in the other direction when the cache is off, because you're calling that path 5 times now and it's swamping other bottlenecks I've no idea what this means > This is what I really mean when I say it's a weapon of last resort. Because once you use it, it attempts to take any other options off the table Heh! Absolutely not! Other optimisations are orthogonal to caching IME: Better algo - orthogonal Go lower level - orthogonal Go multicore - orthogonal Noobish SQL query is begging for a rewrite - orthogonal Use L1/2/3/ chip cache better, or RAM, by careful data layout - orthogonal Use a better library - orthogonal Use better data structure eg. than linked list - orthogonal
  - svieira 4y ago
    >> Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods > what does that even mean? What does 'data dependencies' mean specicially? It means that instead of having a method like `entryPoint(int)` that looks up a `Foo` by id and then passes the `Foo` around (`verify(Foo)`, `Foo update(Foo)`) you instead wind up with a system that passes around an `int` (`verify(int)`, `/* Unused */ Foo update(int)`) and each layer then looks up `Foo` in the `FooService` (because `FooService` has a cache and subsequent lookups should be essentially-free). Even worse when `FooService` is abstracted behind a thread-local like `ThreadLocal<Foo> currentFoo`, the implementation of which is `fooService.findById(currentFooId.get()) /* cheap, because fooService has a cache */`. My own take on this is that caches are global-by-default when you really want local-by-default (e. g. dynamic programming instead of a `static final Cache theCacheForTheEntireProgram`). Local-by-default caches are easy to make transactional (when the call stack returns the cache is gone) and don't grow in an unbounded manner in the general case.
- trgn 4y ago
  There's a somewhat tangentially related heuristic to your points imho. When writing code, when in doubt, task CPU over memory, it will generally yield leaner, better software. Easier to profile, debug, and optimize. Much easier to scale in production environments too. Code which optimizes for fast running time, tends to lean heavily into using memory-bloating data structures and caches. These are often poison for aggregate performance in real world environments, and like you said among other things, reduce visibility in the workings of system. There's some implications of this stance; e.g. optimizations to reduce memory footprint are generally never wasted, while optimizations to squeeze out a few ns of runtime may have many more unexpected consequences.
- ilyt 4y ago
  > How do you teach people to effectively use a profiler on code with pernicious caching? Profile the function that is called if object is not found in cache ? Have pluggable cache with plugin that just doesn't cache? That is not hard problem to solve. > They get really uncomfortable six months later when I'm reporting 2-3x in code that can't be optimized further. And in each case everything was made more difficult by the people ahead of me chumming the waters Well, they delivered (I'm guessing) 10x improvement in a month by adding cache. Caching is entirely reasonable first step and it can be good enough to be last one. And frankly ability to use profiler and knowing some architectural optimizations seems to be pretty rare skill.
  - zasdffaa 4y ago
    Exactly! People here seem to be making a complex thing out of a simple thing.
    - hinkley 4y ago
      What makes caching one of the hardest parts of computing is this assertion that it's easy. Everything is easy when you oversimplify it. Some things explode in complexity when you try to force them to be simple.
      - andrekandre 4y ago
        > Some things explode in complexity when you try to force them to be simple. do you mean to say iow "simple in the small but when assembled (in the large) is more complex" whereas what we should be aiming for is perhaps "a little more complex in the small but with the benefit that it enables a simpler (and probably more scalable) architecture once assembled (in the large)"??
  - TeMPOraL 4y ago
    > Profile the function that is called if object is not found in cache ? Have pluggable cache with plugin that just doesn't cache? That is not hard problem to solve. The more the dynamic behavior of the profiled system and the real system deviate, the harder it is to isolate what's causing the slowdowns. > Well, they delivered (I'm guessing) 10x improvement in a month by adding cache. Yes, but by doing this they also tremendously increased the complexity of the project, and (unless the team has someone like GP in it) prevented further optimizations. It may still be a good trade-off, but the overall sorry state of pretty much all software everywhere suggests it may not be a good one usually. > And frankly ability to use profiler and knowing some architectural optimizations seems to be pretty rare skill. That's kind of reinforcing the parent's point, though.
- Aeolun 4y ago
  I kinda feel that your convincing counterargument is just ‘I have a different experience’. I’ve never seen people reach for caches first, and then discard all further optimizations. Or rather, I’ve seen them do that, but not out of some misplaced feeling that it was the only solution, just the one that was easiest to accomplish in very little time.
- throwaway74829 4y ago
  Interested in any resources that you've found vital in your journey. I'm angling to go deeper into the "Cleaner" path, and it's a very slow "figure it out as you go" experience.
PaulHoule 4y ago
Caching can turn O(n²) algorithms into much faster algorithms. I would point to how transformative tabling is in Prolog, not to mention memoization rescuing the calculation of fibonacci numbers by recursion, which otherwise would be gross malpractice.
- sicp-enjoyer 4y ago
  Well, usually memoization gets you polynomial time from exponential time with the space tradeoff. It's unusual to start with `O(n^2)` and get a better DP solution.
- samus 4y ago
  These are very well-behaved instances of caching. They are so well-behaved that they can be fully described with a little bit of analysis. Once I/O is involved or the load isn't well-known beforehand, it becomes much more difficult. Lazy I/O in Haskell is a very divisive topic for a reason.
  - hinkley 4y ago
    Those aren't caching. They're dynamic programming. There's a profound difference and it's about scope and reachability. Edit: and about who's driving.
    - wahern 4y ago
      Memoization is caching and a form of dynamic programming; thus caching is sometimes an important tool in dynamic programming. But the fact that caching can be useful, even essential if your scenario demands such a clear-cut space-time tradeoff as in memoization, seems beside your point.
- fwip 4y ago
  Caching is fine is there's no state involved. e.g: pure functions like the fibonacci sequence. As soon as your cache depends on a mutable state, things get really hairy. Cache invalidation is one of the 2 hard problems in computer science, after all.
notinfuriated 4y ago
In my extensive experience of optimizing stuff it is right. I cannot even count how many issues were created downstream of the decision to start caching X, Y, or Z. At a point, every other software engineer gets the bright idea to cache their special little function or request or database call. Soon enough, there are N layers of cache responsible for delivering some special piece of data. The engineers build tooling to clear those caches on updates, either manually or automatically. Both suffer from issues, the latter occasionally failing to perform and just causing general confusion about why stale data is returning, and the former being far worse, recruiting dozens of business stakeholders into the problem of knowing-how-to-expire-cache. They start making their own internal wiki pages with steps or various cache tools to clear, often incorrect steps because they got lucky one day by clearing cache X and not cache Y that they did not know about. Overall, in my experience, allowing software engineers to do caching as they wish without any real guardrails has caused massive problems most likely on the order of N!. I cannot roll my eyes far enough into the back of my head when I hear someone suggest caching as a solution any longer, at least not on an application that has more than a dozen business stakeholders managing data and expecting it to be updateable in any reasonable way so that customers see the most recent data when the business wants it.
- hinkley 4y ago
  I have one particularly baroque data path that I think passes through three layers of cache. The same people worked on it the whole time, and I'm 90% sure that at least one of those layers is doing absolutely nothing, except making the code harder to read. Once they're added nobody takes them out. Like all global state it's problematic to be absolutely sure it's not being used anywhere.

hinkley 4y ago

That's not a very convincing argument. "I use it because it works" without refuting any reasons I presented as to why it "works" but breaks your other options. How do you teach people to effectively use a profiler on code with pernicious caching? Part of why I'm the "Cleaner" is because I've worked with any number of people who will present "evidence" that everything that can be done has been done. Here are the metrics from a year ago when we turned on the cache. Here are the flame charts. See? Nothing to see here. They get really uncomfortable six months later when I'm reporting 2-3x in code that can't be optimized further. And in each case everything was made more difficult by the people ahead of me chumming the waters. They were partially correct. There was in fact nothing left for them to see, so nothing that they knew to do. But "there's nothing to see here" and "there's nothing here" are two very, very different statements.

cogman10 4y ago
The issue with caches is they are easy to apply. You can easily drop a cache in any place you load a resource without significant code changes. I tend to think they are ok weapons of sometimes first resort (rarely changing resource with heavy system impact). But you'll definitely get better results simply by rethinking and rearchitecting stuff. (Did you really need to load the database just to lookup a single value?)
- hinkley 4y ago
  > You can easily drop a cache in any place you load a resource without significant code changes. Only when the cache is brand new. The problem with people with cache addiction is that they think they're free. So once they know a cache is there, they begin to lean on it, and lean on it. I've seen it with many different teams in a number of verticals. Sometimes while I'm there, sometimes before I arrived. It's the people not the company. This illusion that you can always turn it off later is dangerous. That's only true if you have an impeccable information architecture when you start. And if you have that, you've already done a lot of heavy lifting before caching becomes your only option.
zasdffaa 4y ago
It's dificult to refute your argument because I don't understand your points: > Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods what does that even mean? What does 'data dependencies' mean specicially? > then you can end up evicting data in the middle of a transaction So in the middle of a transaction, instead of looking up data from a key - very cheap - you should do ... what? Recalculate it instead? > The less obvious consequence is now you have concurrency bugs, because the code assumes that decisions made about the data will go the same way on each call, but due to eviction, two halves of the transaction may make conflicting decisions with undefined behavior Assuming I even understand this, you are saying where there's concurrencty there's a risk of pulling stale data out of a cache. Well, true. That's just something you have to be careful of. Add a Tx reference to the cache key. > Once you introduce caching you poison the performance well. Not my experience at all. I can't understand why you're saying this. > Flame charts lie when cache is used How so? > and after people rely on the cache it is lying in the other direction when the cache is off, because you're calling that path 5 times now and it's swamping other bottlenecks I've no idea what this means > This is what I really mean when I say it's a weapon of last resort. Because once you use it, it attempts to take any other options off the table Heh! Absolutely not! Other optimisations are orthogonal to caching IME: Better algo - orthogonal Go lower level - orthogonal Go multicore - orthogonal Noobish SQL query is begging for a rewrite - orthogonal Use L1/2/3/ chip cache better, or RAM, by careful data layout - orthogonal Use a better library - orthogonal Use better data structure eg. than linked list - orthogonal
- svieira 4y ago
  >> Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods > what does that even mean? What does 'data dependencies' mean specicially? It means that instead of having a method like `entryPoint(int)` that looks up a `Foo` by id and then passes the `Foo` around (`verify(Foo)`, `Foo update(Foo)`) you instead wind up with a system that passes around an `int` (`verify(int)`, `/* Unused */ Foo update(int)`) and each layer then looks up `Foo` in the `FooService` (because `FooService` has a cache and subsequent lookups should be essentially-free). Even worse when `FooService` is abstracted behind a thread-local like `ThreadLocal<Foo> currentFoo`, the implementation of which is `fooService.findById(currentFooId.get()) /* cheap, because fooService has a cache */`. My own take on this is that caches are global-by-default when you really want local-by-default (e. g. dynamic programming instead of a `static final Cache theCacheForTheEntireProgram`). Local-by-default caches are easy to make transactional (when the call stack returns the cache is gone) and don't grow in an unbounded manner in the general case.
trgn 4y ago
There's a somewhat tangentially related heuristic to your points imho. When writing code, when in doubt, task CPU over memory, it will generally yield leaner, better software. Easier to profile, debug, and optimize. Much easier to scale in production environments too. Code which optimizes for fast running time, tends to lean heavily into using memory-bloating data structures and caches. These are often poison for aggregate performance in real world environments, and like you said among other things, reduce visibility in the workings of system. There's some implications of this stance; e.g. optimizations to reduce memory footprint are generally never wasted, while optimizations to squeeze out a few ns of runtime may have many more unexpected consequences.
ilyt 4y ago
> How do you teach people to effectively use a profiler on code with pernicious caching? Profile the function that is called if object is not found in cache ? Have pluggable cache with plugin that just doesn't cache? That is not hard problem to solve. > They get really uncomfortable six months later when I'm reporting 2-3x in code that can't be optimized further. And in each case everything was made more difficult by the people ahead of me chumming the waters Well, they delivered (I'm guessing) 10x improvement in a month by adding cache. Caching is entirely reasonable first step and it can be good enough to be last one. And frankly ability to use profiler and knowing some architectural optimizations seems to be pretty rare skill.
- zasdffaa 4y ago
  Exactly! People here seem to be making a complex thing out of a simple thing.
  - hinkley 4y ago
    What makes caching one of the hardest parts of computing is this assertion that it's easy. Everything is easy when you oversimplify it. Some things explode in complexity when you try to force them to be simple.
    - andrekandre 4y ago
      > Some things explode in complexity when you try to force them to be simple. do you mean to say iow "simple in the small but when assembled (in the large) is more complex" whereas what we should be aiming for is perhaps "a little more complex in the small but with the benefit that it enables a simpler (and probably more scalable) architecture once assembled (in the large)"??
- TeMPOraL 4y ago
  > Profile the function that is called if object is not found in cache ? Have pluggable cache with plugin that just doesn't cache? That is not hard problem to solve. The more the dynamic behavior of the profiled system and the real system deviate, the harder it is to isolate what's causing the slowdowns. > Well, they delivered (I'm guessing) 10x improvement in a month by adding cache. Yes, but by doing this they also tremendously increased the complexity of the project, and (unless the team has someone like GP in it) prevented further optimizations. It may still be a good trade-off, but the overall sorry state of pretty much all software everywhere suggests it may not be a good one usually. > And frankly ability to use profiler and knowing some architectural optimizations seems to be pretty rare skill. That's kind of reinforcing the parent's point, though.
Aeolun 4y ago
I kinda feel that your convincing counterargument is just ‘I have a different experience’. I’ve never seen people reach for caches first, and then discard all further optimizations. Or rather, I’ve seen them do that, but not out of some misplaced feeling that it was the only solution, just the one that was easiest to accomplish in very little time.
throwaway74829 4y ago
Interested in any resources that you've found vital in your journey. I'm angling to go deeper into the "Cleaner" path, and it's a very slow "figure it out as you go" experience.

PaulHoule 4y ago

Caching can turn O(n²) algorithms into much faster algorithms. I would point to how transformative tabling is in Prolog, not to mention memoization rescuing the calculation of fibonacci numbers by recursion, which otherwise would be gross malpractice.

sicp-enjoyer 4y ago
Well, usually memoization gets you polynomial time from exponential time with the space tradeoff. It's unusual to start with `O(n^2)` and get a better DP solution.
samus 4y ago
These are very well-behaved instances of caching. They are so well-behaved that they can be fully described with a little bit of analysis. Once I/O is involved or the load isn't well-known beforehand, it becomes much more difficult. Lazy I/O in Haskell is a very divisive topic for a reason.
- hinkley 4y ago
  Those aren't caching. They're dynamic programming. There's a profound difference and it's about scope and reachability. Edit: and about who's driving.
  - wahern 4y ago
    Memoization is caching and a form of dynamic programming; thus caching is sometimes an important tool in dynamic programming. But the fact that caching can be useful, even essential if your scenario demands such a clear-cut space-time tradeoff as in memoization, seems beside your point.
fwip 4y ago
Caching is fine is there's no state involved. e.g: pure functions like the fibonacci sequence. As soon as your cache depends on a mutable state, things get really hairy. Cache invalidation is one of the 2 hard problems in computer science, after all.

notinfuriated 4y ago

In my extensive experience of optimizing stuff it is right. I cannot even count how many issues were created downstream of the decision to start caching X, Y, or Z. At a point, every other software engineer gets the bright idea to cache their special little function or request or database call. Soon enough, there are N layers of cache responsible for delivering some special piece of data. The engineers build tooling to clear those caches on updates, either manually or automatically. Both suffer from issues, the latter occasionally failing to perform and just causing general confusion about why stale data is returning, and the former being far worse, recruiting dozens of business stakeholders into the problem of knowing-how-to-expire-cache. They start making their own internal wiki pages with steps or various cache tools to clear, often incorrect steps because they got lucky one day by clearing cache X and not cache Y that they did not know about. Overall, in my experience, allowing software engineers to do caching as they wish without any real guardrails has caused massive problems most likely on the order of N!. I cannot roll my eyes far enough into the back of my head when I hear someone suggest caching as a solution any longer, at least not on an application that has more than a dozen business stakeholders managing data and expecting it to be updateable in any reasonable way so that customers see the most recent data when the business wants it.

hinkley 4y ago
I have one particularly baroque data path that I think passes through three layers of cache. The same people worked on it the whole time, and I'm 90% sure that at least one of those layers is doing absolutely nothing, except making the code harder to read. Once they're added nobody takes them out. Like all global state it's problematic to be absolutely sure it's not being used anywhere.