3 ms·

How to Reliably Scale Your Data Platform for High Volumes

61 points by oedmarap 6y ago

0xbadcafebee 6y ago
These are really good tips, but a lot of them could get their own blog post just about why they're so important. Just the "every alert should be actionable by somebody at 3AM" requires a lot of work, because the implications of a single alert and how to respond to it can become very deep. And you probably won't meet it until you actually get the call and realize you missed something. I prefer to imagine alerts as if they're going to your Grandmother. She can turn her computer on and surf the web and make phone calls. But she doesn't have tribal knowledge about your product, or very specific technical knowledge. If she can't figure out what to do from reading the runbook in the alert, it's likely your on-call people can't either and will just end up waking up other people, and at that point you're just delaying resolving the alert and waking up too many people.
jugg1es 6y ago
I would love a post that focused on alerting - how to structure a scalable alerting system that avoids alert fatigue. How to handle errors that only matter if they happen often in a small window but also could matter even if they happen infrequently.
- deleted 6y ago
  [deleted]
vinay_ys 6y ago
I have been in similar situation of handling huge spike in traffic due to festive sale season. After building huge amount of cheap datacenter capacity and optimizing our hardware/software stack, only reasonable thing we could do was to build lots of 'feature degradation flags' into our stack – things we could turn off to reduce stress on the system when under heavy traffic. For example, we could reduce the number of product cards shown per page, reduce the number of recommendation widgets per page etc. These could be turned on/off instantly by flipping a dynamic config flag (no deployments or restarts necessary). Our organic search traffic used to generate huge amount of load on our backend services like pricing/promotions and logistics-promise services. We would isolate capacity of these backend services for serving such expensive upstream services and also put rate-limiting circuit breakers to avoid them from killing those critical backend services. Obviously, we had similar feature reduction/degradation mechanisms built into our data platforms. Capacity was reallocated for critical data pipelines and all experimental/training pipelines were stopped. Drill-downs or adhoc query interfaces were locked down. And, it goes without saying that during these few sale days, the proactive monitoring/alerting and oncall rigor was on high-alert to detect and react quickly in case systems didn't behave as expected.
siliconc0w 6y ago
One thing I don't see a lot of are SREing working with the business to set product direction - i.e differentiating features or tiers of service that reflect different operational, security, or infrastructure costs. Reliability, security, and performance are usually seen as binary in non-SRE land when they're just as much about tradeoffs as anything else and different customers can have wildly different requirements in these areas.

61 points by oedmarap 6y ago

These are really good tips, but a lot of them could get their own blog post just about why they're so important. Just the "every alert should be actionable by somebody at 3AM" requires a lot of work, because the implications of a single alert and how to respond to it can become very deep. And you probably won't meet it until you actually get the call and realize you missed something. I prefer to imagine alerts as if they're going to your Grandmother. She can turn her computer on and surf the web and make phone calls. But she doesn't have tribal knowledge about your product, or very specific technical knowledge. If she can't figure out what to do from reading the runbook in the alert, it's likely your on-call people can't either and will just end up waking up other people, and at that point you're just delaying resolving the alert and waking up too many people.

jugg1es 6y ago

I would love a post that focused on alerting - how to structure a scalable alerting system that avoids alert fatigue. How to handle errors that only matter if they happen often in a small window but also could matter even if they happen infrequently.

deleted 6y ago
[deleted]

vinay_ys 6y ago

I have been in similar situation of handling huge spike in traffic due to festive sale season. After building huge amount of cheap datacenter capacity and optimizing our hardware/software stack, only reasonable thing we could do was to build lots of 'feature degradation flags' into our stack – things we could turn off to reduce stress on the system when under heavy traffic. For example, we could reduce the number of product cards shown per page, reduce the number of recommendation widgets per page etc. These could be turned on/off instantly by flipping a dynamic config flag (no deployments or restarts necessary). Our organic search traffic used to generate huge amount of load on our backend services like pricing/promotions and logistics-promise services. We would isolate capacity of these backend services for serving such expensive upstream services and also put rate-limiting circuit breakers to avoid them from killing those critical backend services. Obviously, we had similar feature reduction/degradation mechanisms built into our data platforms. Capacity was reallocated for critical data pipelines and all experimental/training pipelines were stopped. Drill-downs or adhoc query interfaces were locked down. And, it goes without saying that during these few sale days, the proactive monitoring/alerting and oncall rigor was on high-alert to detect and react quickly in case systems didn't behave as expected.

siliconc0w 6y ago

One thing I don't see a lot of are SREing working with the business to set product direction - i.e differentiating features or tiers of service that reflect different operational, security, or infrastructure costs. Reliability, security, and performance are usually seen as binary in non-SRE land when they're just as much about tradeoffs as anything else and different customers can have wildly different requirements in these areas.