4 ms·

I once lost an entire Christmas vacation to fixing up the damage caused when an Elasticsearch cluster running in AWS responded poorly to a network partition eve

I once lost an entire Christmas vacation to fixing up the damage caused when an Elasticsearch cluster running in AWS responded poorly to a network partition event and started producing results that ruined our users' day (and business records) in a "costing millions of dollars" kind of way. It was a very old version of ES, and the specific behavior that led to the problem has been fixed for a long time now. But still, the fact that something like this can happen in a cloud deployment demonstrates that this article's advice rests on an egregiously simplistic perspective on the possible failure modes of distributed systems. In particular, the major premise that intermittent connectivity is only a problem on internetworks is just plain wrong. Hubs and switches flake out. Loose wires get jiggled. Subnetworks get congested. And if you're on the cloud, nobody even tries to pretend that they'll tell you when server and equipment maintenance is going to happen.

hinkley 2y ago
Twice I’ve had to take an Ethernet cable from an IT lead and throw it away. “It’s still good!” No it fucking isn’t. One of them compromised by pulling it out of the trash and cutting the ends off. When there’s maintenance going on in the server room and a machine they promised not to touch starts having intermittent networking problems, it’s probably a shitty cable getting jostled. There is an entire generation of devs now that have had physical hardware abstracted away and aren’t learning the lessons. Though my favorite stories are the cubicle neighbor who taps their foot leading to packet loss.
- deleted 2y ago
  [deleted]
- kstrauser 2y ago
  I watched a colleague diagnosis a flaky cable. Then he immediately pulled out a pocket knife and sliced the cable in 2. He said he didn't ever even want to be tempted to trust it again. That's been my model since then. I've never worked with a cable so expensive or irreplaceable that I'd want to take a chance on it a second time. I'm sure they exist. I haven't been around one.
  - hot_gril 2y ago
    Lightning cable. Gotta mess with it until the iPhone charges. I'm not replacing it until it's tied into a blackwall hitch and still isn't charging.
- devin 2y ago
  It's been awhile since I've been in a data center, but fiber cables were usually the ones that seemed to have more staying power even when they started to flake. Maybe because some of them were long runs and they weren't cheap. You'd see some absolutely wrenched cable going into rack at a horrifying angle from the tray on 30ft run with greater regularity than I'd care to admit.

hinkley 2y ago

Twice I’ve had to take an Ethernet cable from an IT lead and throw it away. “It’s still good!” No it fucking isn’t. One of them compromised by pulling it out of the trash and cutting the ends off. When there’s maintenance going on in the server room and a machine they promised not to touch starts having intermittent networking problems, it’s probably a shitty cable getting jostled. There is an entire generation of devs now that have had physical hardware abstracted away and aren’t learning the lessons. Though my favorite stories are the cubicle neighbor who taps their foot leading to packet loss.

deleted 2y ago
[deleted]
kstrauser 2y ago
I watched a colleague diagnosis a flaky cable. Then he immediately pulled out a pocket knife and sliced the cable in 2. He said he didn't ever even want to be tempted to trust it again. That's been my model since then. I've never worked with a cable so expensive or irreplaceable that I'd want to take a chance on it a second time. I'm sure they exist. I haven't been around one.
- hot_gril 2y ago
  Lightning cable. Gotta mess with it until the iPhone charges. I'm not replacing it until it's tied into a blackwall hitch and still isn't charging.
devin 2y ago
It's been awhile since I've been in a data center, but fiber cables were usually the ones that seemed to have more staying power even when they started to flake. Maybe because some of them were long runs and they weren't cheap. You'd see some absolutely wrenched cable going into rack at a horrifying angle from the tray on 30ft run with greater regularity than I'd care to admit.