5 ms·

If you haven't, you should read James Hamilton's paper on "On Designing and Deploying Internet-Scale Services". http://static.usenix.org/event/lisa07/tech/full

If you haven't, you should read James Hamilton's paper on "On Designing and Deploying Internet-Scale Services". http://static.usenix.org/event/lisa07/tech/full_papers/hamilton/hamilton_html/ http://static.usenix.org/event/lisa07/tech/full_papers/hamil...

zerop 14y ago
Thanks a lot for sharing this great resource. A very simple & covering large range of concerns on designing highly scalable & available services.
peterwwillis 14y ago
This is a fantastic guide. The 'big red button' is important when handling peak traffic but needs to be expanded to 'little red buttons' as well. You might use the fail whale, or split up your site features in different URIs and disable/fail-whale those individually, or be able to dynamically increase logging (another great point in the guide) to find high-latency pieces of code and big-red-button them. Perhaps you find one host is being slammed while others are relatively idle; being able to gracefully stop incoming requests to the slammed host and allow them to reconnect to the idle hosts can help keep the fire at bay. (Of course there are features built into some cluster protocols that are supposed to do this automatically, but these don't always work as expected in the real world. It can be better to rely on humans to control the flow during emergencies instead of automated rules) Meter admission is perhaps the second most important factor in handling peak traffic. Random jitter doesn't help if you're at 100% capacity. You need to be able to dial back everything - database queries, disk logging, app server requests, frontend server connections, etc - so you can handle the maximum number of traffic without falling over. The hard part is making this totally dynamic so you don't have to keep restarting services to find the sweet spot. Of course, this also requires extremely low overhead mostly-real-time metrics to have an accurate view to tune your knobs by.
- donavanm 14y ago
  Disagree on the "dynamically increase" logging. Log everything all the time. When you enable logging/debug you've just changed your resource profile. Presumably your already in a precarious/high risk situation, don't add more change. Storage is cheap to save your bits. If you need to save money record different granularities, discard old records, or use (good) sampling. A system to dynamically/automatically capture something like oprofile/systemtap/dtrace can be handy. But on the converse I've seen it make a bad instance dead.
  - peterwwillis 14y ago
    The problem there is the sheer amount of logs puts strain on the network when logging remotely, and will destroy the box from i/o load locally. The more hardware you have, the cheaper you'll want to be. And unfortunately the engineers don't get to make the budgets ;-) If you have your meter admission up to par, you can throttle back on requests to the box while enabling specific logging. You can employ tricks like enable extra logging for only specific sessions, URIs, etc or only for a specific time period. It's not like you're going to profile your whole application while the site's almost down, but usually there may be a query or a subset of users or something that's abnormally slow.
Estragon 14y ago
It would be great to read some war stories about small companies which started with a "just ship it" mentality and ended up adopting these practices. Designing with the practices in mind would be the best approach, but that's a lot of extra design goals, and likely to slow development down.

zerop 14y ago

Thanks a lot for sharing this great resource. A very simple & covering large range of concerns on designing highly scalable & available services.

peterwwillis 14y ago

This is a fantastic guide. The 'big red button' is important when handling peak traffic but needs to be expanded to 'little red buttons' as well. You might use the fail whale, or split up your site features in different URIs and disable/fail-whale those individually, or be able to dynamically increase logging (another great point in the guide) to find high-latency pieces of code and big-red-button them. Perhaps you find one host is being slammed while others are relatively idle; being able to gracefully stop incoming requests to the slammed host and allow them to reconnect to the idle hosts can help keep the fire at bay. (Of course there are features built into some cluster protocols that are supposed to do this automatically, but these don't always work as expected in the real world. It can be better to rely on humans to control the flow during emergencies instead of automated rules) Meter admission is perhaps the second most important factor in handling peak traffic. Random jitter doesn't help if you're at 100% capacity. You need to be able to dial back everything - database queries, disk logging, app server requests, frontend server connections, etc - so you can handle the maximum number of traffic without falling over. The hard part is making this totally dynamic so you don't have to keep restarting services to find the sweet spot. Of course, this also requires extremely low overhead mostly-real-time metrics to have an accurate view to tune your knobs by.

donavanm 14y ago
Disagree on the "dynamically increase" logging. Log everything all the time. When you enable logging/debug you've just changed your resource profile. Presumably your already in a precarious/high risk situation, don't add more change. Storage is cheap to save your bits. If you need to save money record different granularities, discard old records, or use (good) sampling. A system to dynamically/automatically capture something like oprofile/systemtap/dtrace can be handy. But on the converse I've seen it make a bad instance dead.
- peterwwillis 14y ago
  The problem there is the sheer amount of logs puts strain on the network when logging remotely, and will destroy the box from i/o load locally. The more hardware you have, the cheaper you'll want to be. And unfortunately the engineers don't get to make the budgets ;-) If you have your meter admission up to par, you can throttle back on requests to the box while enabling specific logging. You can employ tricks like enable extra logging for only specific sessions, URIs, etc or only for a specific time period. It's not like you're going to profile your whole application while the site's almost down, but usually there may be a query or a subset of users or something that's abnormally slow.

Estragon 14y ago

It would be great to read some war stories about small companies which started with a "just ship it" mentality and ended up adopting these practices. Designing with the practices in mind would be the best approach, but that's a lot of extra design goals, and likely to slow development down.