3 ms·
Having a different code path that only triggers during high load feels pretty risky to me.
by bawolff 4y ago
Having a different code path that only triggers during high load feels pretty risky to me.
- bell-cot 4y agoIn theory, the best way to manage this situation may be with a grayzone - if all is well, then there's only a (say) 5% chance of going to cache. (To at least catch "the cache is broken" errors sooner.) As things get worse, that chance goes up as you cross-fade to mostly using the cache. Important Warning: If you do not have a very good understanding of your overall system, or neglect to carefully dampen the cross-fade functions - then the whole thing can demonstrate fast and unpredictable swings in behavior. Note that being subpoenaed to testify in a "Why did the Crater City chemical plant blow up?" investigation is usually bad for one's career.
- hinkley 4y agoThat's better but composition is always the devil in the details. Which might be what you mean when you say: > or neglect to carefully dampen the cross-fade functions People with a 'caching problem' don't have one cache, they have a bunch of them. So that 1 in 20 chance of getting grey zoned on one cache, is 1 in 400 chance of getting grey zoned on two caches, and a 1 in 8000 chance to get grey zoned on 3. That is a very small fraction but when you're dealing with the Law of Large Numbers that's something that's happening at least once a second for many services. But if you're using this you probably have a lot more than 3 caches, and then we have to talk about the odds that any request gets grey zoned at least once. which is going to be 1 - (19/20)^n. At six caches you're already over 1 in 4, which is going to potentially change your mean response time quite a bit.
- bell-cot 4y ago> "...which is going to potentially change your mean response time quite a bit." My experience is that the changing mean response time isn't nearly as bad as the changing standard deviation. The really big problems tend to arise when you have unsuspected or under-appreciated positive feedback loops - whether in the software, or in associated physical systems and human behaviors. Or resonance effects. Those situations are where good dampening gets really important.