5 ms·
Caching is a weapon of last resort. In most cases it is the last major performance improvement you will see from your code, even if other options were available
by hinkley 4y ago
Caching is a weapon of last resort. In most cases it is the last major performance improvement you will see from your code, even if other options were available, and any information architecture problems you have are set in stone or will get worse.
Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods. This turns your cache into global shared state, and once that happens you can't turn the cache off again. There are other problems with this arrangement as well.
The easiest to understand (and yet I've failed on at least one project) is that if the working set for a problem ever becomes proportional to the size of the cache (>1 for single task systems, > 1/n for systems with n concurrent tasks), then you can end up evicting data in the middle of a transaction. The obvious consequence is that it causes performance to jump off a cliff. Particularly bad behavior with LRUs but switching to a different statistical model simply reduces the height of the cliff. There is still a cliff. The less obvious consequence is now you have concurrency bugs, because the code assumes that decisions made about the data will go the same way on each call, but due to eviction, two halves of the transaction may make conflicting decisions with undefined behavior.
But speaking as the guy who fixes the perf issues other people gave up on: Once you introduce caching you poison the performance well. Flame charts lie when cache is used, and after people rely on the cache it is lying in the other direction when the cache is off, because you're calling that path 5 times now and it's swamping other bottlenecks. This is what I really mean when I say it's a weapon of last resort. Because once you use it, it attempts to take any other options off the table. Caching will be most of the last order of magnitude in scalability that your system ever sees. Every 'win' from there on will take more time and energy than most people are willing to invest.
- brightball 4y agoI like applying the Circuit Breaker approach to caching. Rather than applying a cache everywhere, setup a cache wrapper that only triggers if the accessed resource is slower than a certain threshold. The goal is for the cache to go unused most of the time, but during times of unexpected system stress the caches will kick in, provide relief to the system and reduce impact to users.
- bawolff 4y agoHaving a different code path that only triggers during high load feels pretty risky to me.
- bell-cot 4y agoIn theory, the best way to manage this situation may be with a grayzone - if all is well, then there's only a (say) 5% chance of going to cache. (To at least catch "the cache is broken" errors sooner.) As things get worse, that chance goes up as you cross-fade to mostly using the cache. Important Warning: If you do not have a very good understanding of your overall system, or neglect to carefully dampen the cross-fade functions - then the whole thing can demonstrate fast and unpredictable swings in behavior. Note that being subpoenaed to testify in a "Why did the Crater City chemical plant blow up?" investigation is usually bad for one's career.
- hinkley 4y agoThat's better but composition is always the devil in the details. Which might be what you mean when you say: > or neglect to carefully dampen the cross-fade functions People with a 'caching problem' don't have one cache, they have a bunch of them. So that 1 in 20 chance of getting grey zoned on one cache, is 1 in 400 chance of getting grey zoned on two caches, and a 1 in 8000 chance to get grey zoned on 3. That is a very small fraction but when you're dealing with the Law of Large Numbers that's something that's happening at least once a second for many services. But if you're using this you probably have a lot more than 3 caches, and then we have to talk about the odds that any request gets grey zoned at least once. which is going to be 1 - (19/20)^n. At six caches you're already over 1 in 4, which is going to potentially change your mean response time quite a bit.
- bell-cot 4y ago> "...which is going to potentially change your mean response time quite a bit." My experience is that the changing mean response time isn't nearly as bad as the changing standard deviation. The really big problems tend to arise when you have unsuspected or under-appreciated positive feedback loops - whether in the software, or in associated physical systems and human behaviors. Or resonance effects. Those situations are where good dampening gets really important.
- zasdffaa 4y agoIn my extensive experience of optimising stuff... > Caching is a weapon of last resort. In most cases it is the last major performance improvement ...is wrong. It is one of the very first things I reach for because it is the most effective and simplest, once you've eliminated O(n^2) behaviour etc. It's not a magic wand but nothing is.
- hinkley 4y agoThat's not a very convincing argument. "I use it because it works" without refuting any reasons I presented as to why it "works" but breaks your other options. How do you teach people to effectively use a profiler on code with pernicious caching? Part of why I'm the "Cleaner" is because I've worked with any number of people who will present "evidence" that everything that can be done has been done. Here are the metrics from a year ago when we turned on the cache. Here are the flame charts. See? Nothing to see here. They get really uncomfortable six months later when I'm reporting 2-3x in code that can't be optimized further. And in each case everything was made more difficult by the people ahead of me chumming the waters. They were partially correct. There was in fact nothing left for them to see, so nothing that they knew to do. But "there's nothing to see here" and "there's nothing here" are two very, very different statements.
- cogman10 4y agoThe issue with caches is they are easy to apply. You can easily drop a cache in any place you load a resource without significant code changes. I tend to think they are ok weapons of sometimes first resort (rarely changing resource with heavy system impact). But you'll definitely get better results simply by rethinking and rearchitecting stuff. (Did you really need to load the database just to lookup a single value?)
- hinkley 4y ago> You can easily drop a cache in any place you load a resource without significant code changes. Only when the cache is brand new. The problem with people with cache addiction is that they think they're free. So once they know a cache is there, they begin to lean on it, and lean on it. I've seen it with many different teams in a number of verticals. Sometimes while I'm there, sometimes before I arrived. It's the people not the company. This illusion that you can always turn it off later is dangerous. That's only true if you have an impeccable information architecture when you start. And if you have that, you've already done a lot of heavy lifting before caching becomes your only option.
- dist1ll 4y agoIf I had to pick the two worst sources of bugs I can imagine, it would be (1) concurrency and (2) caches. Those functions often require extensive performance and load testing and have tons of non-deterministic failure modes.
- dmoy 4y agoNo way, the worst two sources of bugs I can imagine are definitely (1) conc(2) cachesurrency and (3) caches
- hinkley 4y agoThis hurt to read. Like, in my heart.
- eurasiantiger 4y agoTry CNN subtitles on YouTube. They are almost always data race gibberish.
- MayeulC 4y agoThe classic joke is to make (3) off-by-one errors
- dmoy 4y agoIndeed I was trying to make the duplicate entry an off by one error without explicitly saying so, but I suppose I should have for clarity
- jiggawatts 4y agoMy first impulse was to disagree with you, but I realised I gave the exact same advice as a consultant to a group managing one of the bigger government websites that had some performance issues. They wanted to paper over the issues by slapping a CDN in front of the site. Their users are all local, so the global aspect of the CDN wasn't the point, the benefit was that it would cache the content and provide a performance boost. I told them that, yes, a CDN might help, but they should optimise their site first, and only then add a CDN on top if they really need to. Otherwise they'll just mask their issues. It's too easy to end up with queries that take 1 minute but are cached for, say, 1000 separate users. Most people most of the time will have a good experience, but you'll end up with a hideous 99th percentile that will be really hard to detect and fix.
- youngtaff 4y agoI’d often take the opposite approach (and have done with consulting clients) Slap the CDN on the site and then chase down the slow server responses It delivers an improved experience for the people using the site sooner than optimizing the site first The challenge of course is you then have to persuade people to go chase down the slow responses when they may response “there’s so few why should I care?”
- hinkley 4y agoFrom a web standpoint I reminded them that the browser already has a cache so you would be better off getting your cache semantics clear first, debug it at the client and then infill later if and as necessary.
- YZF 4y agoI'm not sure everyone here is talking about the same thing when they talk about caching but if you took caching out of the modern Internet it would collapse immediately. There's caches in your CPU, there's caches in your disk drives, there's caches in Linux, there's caches in every layer of every major cloud service, there's caches in your network (CDN). If your CPU had to fetch every instruction and every bit of data from main memory it would run x100 slower. If your disks didn't have caching they'd generally run slower. If Linux didn't cache your filesystem it'd need a lot more disk accesses. etc. etc. Ofcourse a properly designed cache has to consider things like what are the access patterns, how to figure out what to keep in the cache and what to toss away, how large is the dataset we're caching, invalidation etc. But there's no performance optimization that's going to get that NetFlix movie to 100M subscribers directly from the disk it's stored on, through the network. Feels like some of the discussion is about something like storing f(x) in a map instead of computing f(x). I wouldn't use this terminology (caching) in this case. This is more like your standard space to time algorithmic tradeoffs.
- hinkley 4y agoAnd then there’s Spectre and Meltdown. Here we are talking about building your own cache data store, not the ones built into your processor.
- ehaliewicz2 4y agoDo cpu caches poison the performance well?
- hinkley 4y agoEverything nondeterministic adds difficulty to the Scientific Method. Maybe we'll have better ways to filter the noise in the future, I certainly hope so, but that's not where we are now. But this is a little whataboutism. Caches in the CPU or anywhere else are a leaky abstraction, and sometimes you learn more about a problem by having the abstraction, sometimes you learn more by not having it. Part of being on a growing project is that you get more opportunities to see both sides of a transition and learn from it. When you have coworkers who clearly have only learned from one half of a timeline and have a great deal to learn about the other, wanting to put the brakes on and have them learn some discipline first is not a crime. Or rather it is, but it shouldn't be. It's like a weird version of Fake it Til You Make it that is verging on fraud. We'll keep up the fakery until someone notices and then we'll get a new job and reset the clock.
- ehaliewicz2 4y agoI agree, it is quite refreshing to work on an architecture without any caching, although quite slow as well :)