7 ms·

How Google taught me to cache and cash-in

83 points by helwr 17y ago

atlbeer 17y ago
When coding sites I always follow the design pattern of, how do I return this HTTP request as fast as possible. That's my number one priority, get the HTTP request back to the browser before doing anything else. All other work to be done because of that request, that can be done in the background, will be done in the background.
jasonkester 17y ago
One thing I'd add to that list to everybody who's site is not Google: - Don't add caching yet. There is a lot of low-hanging fruit that you can improve in the "fewer queries", "more efficient queries" and "more efficient rendering" areas that you'll never get a chance to realize if your first step when load increases is to cache the piss out of everything. My preferred order of doing things: 1. launch your site 2. wait until it starts to bog under load 3. optimize until it's fast again 4. repeat 2&3 as many times as possible 5. add caching Do it that way and you'll buy yourself a lot more time before running into (and possibly avoid altogether) the dreaded step nine: 9. buy more servers
- ejs 17y ago
  I read this a lot how people say caching should be a last resort, but it really depends on what you are doing. Many times caching can be the easiest thing to implement on certain things. I have done sites that have a signup page that have a lot of dynamic statistics on that page... I could have spent plenty of time optimizing the calls, but it is far easier to just cache the whole thing, and it will be faster. Clearing the cache is usually not nearly that bad either.
  - jcapote 17y ago
    Definitely, I think cacheing should be first resort.
  - justinchen 17y ago
    You just have to be smart about when to apply caching. People can always cite competing caching rules and examples that support them, but only you know your app and whether or not a particular slowness can benefit from more optimizing or more caching.
  - jasonkester 17y ago
    You're right that caching is the easiest thing. That's why people do it. The problem is that they do it right away, and then eventually the things slows down anyway, and then they're hosed. In the example you describe, the hosing would be relatively minor, since you could hand-optimize the queries and simply plug them back in. In the more common case of caching your 50-query homepage, you might end up needing to buy more servers without ever realizing that you could have gotten it down to one query. So yeah, if you're confident that your signup page is never going to get the sort of load that'll slow it down despite your caching then you're right, you're all set.
    - natrius 17y ago
      You're not hosed when things slow down. It's easy to disable caching on a development machine and identify the slowdowns.
- richardw 17y ago
  Basically, don't optimize prematurely. Especially since doing so increases the amount of moving parts and, therefore, cost of changes.
- smharris65 17y ago
  Don't punish your users by launching your site without load testing. It's easy to simulate load. Do it before launching.
  - btilly 17y ago
    It is surprisingly hard to simulate load in a way that realistically matches what load will do with various caching layers.
- ww520 17y ago
  In general I agree not to optimize prematurely; however, the system needs to design upfront to have the proper layering and choke points, otherwise retrofitting the code to add caching in will be difficult.
- anatoly 17y ago
  I don't understand. Why not start with caching, then do all those things you mentioned when the site bogs down again? It'll be after all the same queries, the same rendering etc. post-cache; the same technique to improve them will work in the presence of cache. So why not start with caching, especially if it buys you a large chunk of time to prepare the other stuff?
  - jerf 17y ago
    An uncached system is easy to understand; things happen when the code says they happen. A cached system is much more opaque when it comes to understanding performance. You want to opaquify your system after it is broadly correct, not before. "Spend" your caching dollars on the good system, not the bad one. IMHO, this is not a fundamental truth, but an accidental one. If a framework or codebase was designed from day one to avoid this effect, it might be possible to dodge this problem. But I haven't seen one really get this right yet, and I think you might have to go strongly functional to get there.
    - stonemetal 17y ago
      But don't you end up doing unneeded optimizations? Optimizing the heck out of something that is in the cache 99% of the time seems like a waste.
      - anamax 17y ago
        Let's do some arithmetic. Suppose that the unoptimized computation takes 200 units, the optimized takes 100 units and a cache access takes 1 unit. With a 99% hit rate, 100 accesses will take 100 units plus the cost of 1 computation. In the unoptimized case, that's 300 units, or an average of 3 units per access. In the optimized case, that's 200 units, or an average of 2 units per case. This ignores the user-experience benefit of serving that 1 access significantly faster.
- laut 17y ago
  You don't have to "be Google" to benefit from caching. An e-commerce site that is more responsive sells more. One reason for using caching in a website is having less load so that you can serve more visitors. Another is simply to give each visitor an experience with a faster website. If you can cache entire pages and bring a response down from say 300ms to 30ms that makes a big difference.
  - nostrademons 17y ago
    I think the point is that premature caching can sweep algorithmic problems under the rug. You will eventually get around to caching things to improve performance. But if if you cache first, you might never get around to improving the algorithmic bottlenecks, since they're obscured by the cache. A day with a profiler can often result in many-fold speedups. It's not unusual for certain naively-written subsystems to speed up by 1000-fold once they've been profiled and optimized. (Often as a result of local caching - that's the "layered caches" the article was talking about.) You want to speed up the subsystems first, then cache them, not cache them and try to speed them up.
    - peterwwillis 17y ago
      I think the point should be, if you're caching really really well, algorithmic problems will not matter. Why? I help to admin a site which is algorithmically shitty yet performs big feats of performance and makes bank. Making bank is the goal. If you can accomplish it even with a shitty algorithm via caching, do that. And honestly, for most horizontally-scaled systems i've seen it's not difficult to "turn down the cache" and watch for load issues. And the best way to debug problems (such as algorithm limitations) is by knowing the hard limits of every facet of your system and comparing against current use metrics. Time-consuming but not very complex.
      - laut 17y ago
        I agree. If algorithms are slow, it doesn't matter as long as they are fast enough. You should not spend time prematurely optimizing for performance that you don't need, if you can more easily use caching to give you real, experienced performance. If you want to optimize uncached performance you can turn off cache in a development/staging environment. If you run a website that is in business do not give your visitors bad performance (uncached) on purpose.
- cellis 17y ago
  I know that you're thinking: "Premature optimization is the root of all evil". But when I use caching, I don't consider it premature at all. It's just a best practice, like not writing O(n^2) loops. Why wait until your site starts to "bog under load"?
  - jrockway 17y ago
    Is it easier to add the cache or not add the cache? If it's the second one and you do it anyway, it's premature optimization. Like the OP says, this may obscure other problems that you can fix with less effort. You want a simple system that is good, not a complex system that is "good enough". You can add complexity later, but it's very hard to remove. So be careful about that. If all of your sites follow some similar pattern, then you aren't doing the optimization, your framework is. And that's fine; presumably you already tested the framework and didn't introduce any unnecessary complexity.
- hartror 17y ago
  My preferred way: 1. launch your site 2. wait until it starts to bog under load 3. add caching in the bogged down areas so you can still make money 4. use the time you just gave yourself to optimize 5. Repeat 2-4 Adding caching should be as quick as finding the bottle neck. Sure optimizing might take you the same time but it might be a critical flaw that you're better off caching around until you have time for a refactor. Either way your customers are happy while you toil away in the code.
jordanlev 17y ago
Can anyone explain what "don't try to enforce partial ordering simultaneously with your updates" means?
callmeed 17y ago
Can someone explain #14? Is it really the golden rule?

83 points by helwr 17y ago

When coding sites I always follow the design pattern of, how do I return this HTTP request as fast as possible. That's my number one priority, get the HTTP request back to the browser before doing anything else. All other work to be done because of that request, that can be done in the background, will be done in the background.

jasonkester 17y ago

One thing I'd add to that list to everybody who's site is not Google: - Don't add caching yet. There is a lot of low-hanging fruit that you can improve in the "fewer queries", "more efficient queries" and "more efficient rendering" areas that you'll never get a chance to realize if your first step when load increases is to cache the piss out of everything. My preferred order of doing things: 1. launch your site 2. wait until it starts to bog under load 3. optimize until it's fast again 4. repeat 2&3 as many times as possible 5. add caching Do it that way and you'll buy yourself a lot more time before running into (and possibly avoid altogether) the dreaded step nine: 9. buy more servers

ejs 17y ago
I read this a lot how people say caching should be a last resort, but it really depends on what you are doing. Many times caching can be the easiest thing to implement on certain things. I have done sites that have a signup page that have a lot of dynamic statistics on that page... I could have spent plenty of time optimizing the calls, but it is far easier to just cache the whole thing, and it will be faster. Clearing the cache is usually not nearly that bad either.
- jcapote 17y ago
  Definitely, I think cacheing should be first resort.
- justinchen 17y ago
  You just have to be smart about when to apply caching. People can always cite competing caching rules and examples that support them, but only you know your app and whether or not a particular slowness can benefit from more optimizing or more caching.
- jasonkester 17y ago
  You're right that caching is the easiest thing. That's why people do it. The problem is that they do it right away, and then eventually the things slows down anyway, and then they're hosed. In the example you describe, the hosing would be relatively minor, since you could hand-optimize the queries and simply plug them back in. In the more common case of caching your 50-query homepage, you might end up needing to buy more servers without ever realizing that you could have gotten it down to one query. So yeah, if you're confident that your signup page is never going to get the sort of load that'll slow it down despite your caching then you're right, you're all set.
  - natrius 17y ago
    You're not hosed when things slow down. It's easy to disable caching on a development machine and identify the slowdowns.
richardw 17y ago
Basically, don't optimize prematurely. Especially since doing so increases the amount of moving parts and, therefore, cost of changes.
smharris65 17y ago
Don't punish your users by launching your site without load testing. It's easy to simulate load. Do it before launching.
- btilly 17y ago
  It is surprisingly hard to simulate load in a way that realistically matches what load will do with various caching layers.
ww520 17y ago
In general I agree not to optimize prematurely; however, the system needs to design upfront to have the proper layering and choke points, otherwise retrofitting the code to add caching in will be difficult.
anatoly 17y ago
I don't understand. Why not start with caching, then do all those things you mentioned when the site bogs down again? It'll be after all the same queries, the same rendering etc. post-cache; the same technique to improve them will work in the presence of cache. So why not start with caching, especially if it buys you a large chunk of time to prepare the other stuff?
- jerf 17y ago
  An uncached system is easy to understand; things happen when the code says they happen. A cached system is much more opaque when it comes to understanding performance. You want to opaquify your system after it is broadly correct, not before. "Spend" your caching dollars on the good system, not the bad one. IMHO, this is not a fundamental truth, but an accidental one. If a framework or codebase was designed from day one to avoid this effect, it might be possible to dodge this problem. But I haven't seen one really get this right yet, and I think you might have to go strongly functional to get there.
  - stonemetal 17y ago
    But don't you end up doing unneeded optimizations? Optimizing the heck out of something that is in the cache 99% of the time seems like a waste.
    - anamax 17y ago
      Let's do some arithmetic. Suppose that the unoptimized computation takes 200 units, the optimized takes 100 units and a cache access takes 1 unit. With a 99% hit rate, 100 accesses will take 100 units plus the cost of 1 computation. In the unoptimized case, that's 300 units, or an average of 3 units per access. In the optimized case, that's 200 units, or an average of 2 units per case. This ignores the user-experience benefit of serving that 1 access significantly faster.
laut 17y ago
You don't have to "be Google" to benefit from caching. An e-commerce site that is more responsive sells more. One reason for using caching in a website is having less load so that you can serve more visitors. Another is simply to give each visitor an experience with a faster website. If you can cache entire pages and bring a response down from say 300ms to 30ms that makes a big difference.
- nostrademons 17y ago
  I think the point is that premature caching can sweep algorithmic problems under the rug. You will eventually get around to caching things to improve performance. But if if you cache first, you might never get around to improving the algorithmic bottlenecks, since they're obscured by the cache. A day with a profiler can often result in many-fold speedups. It's not unusual for certain naively-written subsystems to speed up by 1000-fold once they've been profiled and optimized. (Often as a result of local caching - that's the "layered caches" the article was talking about.) You want to speed up the subsystems first, then cache them, not cache them and try to speed them up.
  - peterwwillis 17y ago
    I think the point should be, if you're caching really really well, algorithmic problems will not matter. Why? I help to admin a site which is algorithmically shitty yet performs big feats of performance and makes bank. Making bank is the goal. If you can accomplish it even with a shitty algorithm via caching, do that. And honestly, for most horizontally-scaled systems i've seen it's not difficult to "turn down the cache" and watch for load issues. And the best way to debug problems (such as algorithm limitations) is by knowing the hard limits of every facet of your system and comparing against current use metrics. Time-consuming but not very complex.
    - laut 17y ago
      I agree. If algorithms are slow, it doesn't matter as long as they are fast enough. You should not spend time prematurely optimizing for performance that you don't need, if you can more easily use caching to give you real, experienced performance. If you want to optimize uncached performance you can turn off cache in a development/staging environment. If you run a website that is in business do not give your visitors bad performance (uncached) on purpose.
cellis 17y ago
I know that you're thinking: "Premature optimization is the root of all evil". But when I use caching, I don't consider it premature at all. It's just a best practice, like not writing O(n^2) loops. Why wait until your site starts to "bog under load"?
- jrockway 17y ago
  Is it easier to add the cache or not add the cache? If it's the second one and you do it anyway, it's premature optimization. Like the OP says, this may obscure other problems that you can fix with less effort. You want a simple system that is good, not a complex system that is "good enough". You can add complexity later, but it's very hard to remove. So be careful about that. If all of your sites follow some similar pattern, then you aren't doing the optimization, your framework is. And that's fine; presumably you already tested the framework and didn't introduce any unnecessary complexity.
hartror 17y ago
My preferred way: 1. launch your site 2. wait until it starts to bog under load 3. add caching in the bogged down areas so you can still make money 4. use the time you just gave yourself to optimize 5. Repeat 2-4 Adding caching should be as quick as finding the bottle neck. Sure optimizing might take you the same time but it might be a critical flaw that you're better off caching around until you have time for a refactor. Either way your customers are happy while you toil away in the code.

jordanlev 17y ago

Can anyone explain what "don't try to enforce partial ordering simultaneously with your updates" means?

callmeed 17y ago

Can someone explain #14? Is it really the golden rule?