5 ms·

Graceful Degradation (2014)

45 points by ColinWright 5y ago

ColinWright 5y ago
Prompted by the discussion[0] of "Suspicious Discontinuities"[1] I thought I'd share my experience. [0] https://news.ycombinator.com/item?id=28452926 https://news.ycombinator.com/item?id=28452926 [1] https://danluu.com/discontinuities/ https://danluu.com/discontinuities/
simpsond 5y ago
Back pressure is often useful. Slow down the pipeline from the bottleneck to the source. At some point it’s important to cause failures, informing the source. Those failures can also be intentionally delayed. These techniques are used by networking systems to keep services from falling over. Tuning the knobs for when to fail, when to delay, etc is a hard problem. That is probably why there are so many congestion control algorithms for TCP.
- kwhitefoot 5y ago
  That reminds me of a time, many years ago, when we upgraded a from hubs to switches in the design office. All of a sudden all the HP Unix workstations started to misbehave. It turned out that NFS needed the immediate back pressure caused by collisions on the Ethernet. The switches had a buffer for each port so there was no back pressure until too late and NFS would report that packets were lost. We put the hubs back and all was well,
- cratermoon 5y ago
  I've never taken the time to really dig into the heuristics around fail-fast vs graceful degradation. The two strategies seem to contradict each other, and I'd love to read some analysis of how to navigate the options.
  - ColinWright 5y ago
    It depends on the at-the-time-consequences of the failure. You ask yourself: What happens if/when it fails? If you're in a position of the developer trying to get something working reliably, absolutely you need things to fail early and hard. But when something is in the field and in the hands of the user, you don't want things to crash.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
code_duck 5y ago
Interesting. It reminds me of how traffic jams flow better when drivers simply drive a slow, steady pace rather than repeatedly accelerating to the car in front of them and stopping.
- eyelidlessness 5y ago
  I don’t know if it’s apocryphal, but I’ve heard of people studying this by purposefully spanning across a highway during stop and go rush hour, idling or barely accelerating to avoid braking. If true, this resulted in the flow of traffic behind them easing and gradually speeding up. When I drive (not often) it’s generally in a car with a manual transmission, and frequently coming to a complete stop is much more onerous than just lagging behind traffic a little. So I’ve tried this myself as my own one car experiment. First of all, I have definitely seen traffic behind me ease and flow better. Second of all, I’ve seen it ease around me as more aggressive drivers take advantage by merging in front of me. More importantly, it made driving in those conditions a hell of a lot less stressful. Getting used to “traffic isn’t moving much anyway, it’s really no big deal if people are passing me” gave me more patience and more time to look around me and see what’s going on with the swarm. And it gave me something intellectually engaging without taking my attention away from the task of driving. - - - I actually find the original topic very interesting but I wanted to share this in the analogy for any drivers who may get value from it.
  - Zircom 5y ago
    I've also tried that method personally and have had numerous road-rage incidents directed at me because of it, so just fair warning to anyone who's also thought of taking this up.
    - eyelidlessness 5y ago
      I’m sorry you’ve experienced road rage as a response. I should say, I feel relatively safe trying this here. But I’m in the PNW where people are notoriously passive aggressive. But they seem perfectly happy to just “skip ahead” of me. But yeah I definitely could imagine a more aggressive response where I first learned to drive (DC metro).
kjellsbells 5y ago
It feels like the diamond in the story is the insertion of randomness in the backoff/retry. Networking protocols have an awkward habit of not doing this and accidentally inducing thundering herds of retries. In the till situation, classic expo backoff would have been catastrophic: since all tills at Xmas are about equally busy, they'll all backoff and retry in the same way. So the randomness essentially simulates different levels of till busyness.

45 points by ColinWright 5y ago

Prompted by the discussion[0] of "Suspicious Discontinuities"[1] I thought I'd share my experience. [0] https://news.ycombinator.com/item?id=28452926 https://news.ycombinator.com/item?id=28452926 [1] https://danluu.com/discontinuities/ https://danluu.com/discontinuities/

simpsond 5y ago

Back pressure is often useful. Slow down the pipeline from the bottleneck to the source. At some point it’s important to cause failures, informing the source. Those failures can also be intentionally delayed. These techniques are used by networking systems to keep services from falling over. Tuning the knobs for when to fail, when to delay, etc is a hard problem. That is probably why there are so many congestion control algorithms for TCP.

kwhitefoot 5y ago
That reminds me of a time, many years ago, when we upgraded a from hubs to switches in the design office. All of a sudden all the HP Unix workstations started to misbehave. It turned out that NFS needed the immediate back pressure caused by collisions on the Ethernet. The switches had a buffer for each port so there was no back pressure until too late and NFS would report that packets were lost. We put the hubs back and all was well,
cratermoon 5y ago
I've never taken the time to really dig into the heuristics around fail-fast vs graceful degradation. The two strategies seem to contradict each other, and I'd love to read some analysis of how to navigate the options.
- ColinWright 5y ago
  It depends on the at-the-time-consequences of the failure. You ask yourself: What happens if/when it fails? If you're in a position of the developer trying to get something working reliably, absolutely you need things to fail early and hard. But when something is in the field and in the hands of the user, you don't want things to crash.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

code_duck 5y ago

Interesting. It reminds me of how traffic jams flow better when drivers simply drive a slow, steady pace rather than repeatedly accelerating to the car in front of them and stopping.

eyelidlessness 5y ago
I don’t know if it’s apocryphal, but I’ve heard of people studying this by purposefully spanning across a highway during stop and go rush hour, idling or barely accelerating to avoid braking. If true, this resulted in the flow of traffic behind them easing and gradually speeding up. When I drive (not often) it’s generally in a car with a manual transmission, and frequently coming to a complete stop is much more onerous than just lagging behind traffic a little. So I’ve tried this myself as my own one car experiment. First of all, I have definitely seen traffic behind me ease and flow better. Second of all, I’ve seen it ease around me as more aggressive drivers take advantage by merging in front of me. More importantly, it made driving in those conditions a hell of a lot less stressful. Getting used to “traffic isn’t moving much anyway, it’s really no big deal if people are passing me” gave me more patience and more time to look around me and see what’s going on with the swarm. And it gave me something intellectually engaging without taking my attention away from the task of driving. - - - I actually find the original topic very interesting but I wanted to share this in the analogy for any drivers who may get value from it.
- Zircom 5y ago
  I've also tried that method personally and have had numerous road-rage incidents directed at me because of it, so just fair warning to anyone who's also thought of taking this up.
  - eyelidlessness 5y ago
    I’m sorry you’ve experienced road rage as a response. I should say, I feel relatively safe trying this here. But I’m in the PNW where people are notoriously passive aggressive. But they seem perfectly happy to just “skip ahead” of me. But yeah I definitely could imagine a more aggressive response where I first learned to drive (DC metro).

kjellsbells 5y ago

It feels like the diamond in the story is the insertion of randomness in the backoff/retry. Networking protocols have an awkward habit of not doing this and accidentally inducing thundering herds of retries. In the till situation, classic expo backoff would have been catastrophic: since all tills at Xmas are about equally busy, they'll all backoff and retry in the same way. So the randomness essentially simulates different levels of till busyness.