4 ms·

in my experience, which is also a fair bit more than two years, that is deeply not-great advice for what the author's talking about, which is available systems.

in my experience, which is also a fair bit more than two years, that is deeply not-great advice for what the author's talking about, which is available systems. Systems with one-true-deciders need to serialize through the decider, which tends to lower availability and limits scalability. And, the decider can die or make wrong decisions, or get partitioned outside of a perfectly functioning set of shards or whatever. It is, no lie, far harder to make a system that can stay available and does the right thing during the partition. Even though the research has been around for decades, we're still in baby steps in actual functioning systems. Even Riak, the poster child for this sort of thing, was straight up last-write-wins for years while they were making fun of everyone else. Hard problems!

notacoward 11y ago
I think you're confused about what "one part of the system" means. I don't mean one process running on one piece of hardware. I mean one software module with many instances - itself a distributed system, with one purpose in life (determine liveness) so it can be simple and verifiable. There's no serialization, and it does not lower availability or scalability as you claim. In fact, it preserves those properties far better than "everyone doing their own failure detection a hundred different ways" does.
- felixgallo 11y ago
  oh, not confused at all! I use erlang, like the OP. Right now I have several erlang nodes that are doing a few hundred thousand http connections concurrently. Looking at the logs, it looks like about 10-20 of those concurrent processes are dying every second for various reasons (upstream crappiness, network errors, parse errors, sunspots for all I know right now), and are getting resurrected by their supervisors. Every blue moon, an entire network target will go out and the restart frequency will be too high, and the supervisors themselves will commit suicide and be restarted by their supervisors. Sometimes a machine will go down and come back up. Looks like the one I'm on has an uptime of about 90 days, which seems low, but doesn't really matter. If too many of them go down at the same time, I'm notified. Haven't ever been notified. You don't have to do 'failure detection a hundred different ways' (what does that even mean?) if you just crash, and build your systems to go with the flow, independently, like ants.
  - notacoward 11y ago
    Well, somebody has to detect the failures, and restart failed processes, etc. so you can just "go with the flow". If you can "just crash" that's great, but it doesn't lead to much helpful advice for people who actually design the underlying systems.
    - felixgallo 11y ago
      Sorry I wasn't clear. That's exactly the advice. Design systems so that they can crash harmlessly in the small, and availability and scalability come to you quite naturally. I agree the (distant) second best alternative is to do as you suggest: to try to rely on a paxos implementation that keeps a stable leader and tries to keep track of liveness. That's pretty hard too! Besides all of aphyr's jepsen test series, check this shit out: https://github.com/coreos/fleet/issues/1289 https://github.com/coreos/fleet/issues/1289 . Tough! Tough even when you think you know what you're doing. And of course no help against freak occurrences where you can't elect a stable leader any more, or you flap leaders, or the election gets stuck. Currently the only known-good implementation of raft is in Idris. Do you know Idris? I sure as hell don't. Fortunately I know ants. Ants can die in great numbers and still move the leaf. In practice, in production, thousands of terabytes, even on AWS, my ants move and never bother me. Let it crash.
      - YuriNiyazov 11y ago
        Could you go into greater detail about how the only known-good implementation of raft is in Idris? Never heard of raft before you mentioned it (i don't deal with distributed systems day-to-day), but I am reading the paper now. and what's ants?
        deleted 11y ago
        [deleted]
        liveoneggs 11y ago
        ants (the bug) is an analogy for erlang workers
curun1r 11y ago
> Even Riak, the poster child for this sort of thing, was straight up last-write-wins for years while they were making fun of everyone else. Disclaimer: I've only used Riak for toy projects, never on a real production app, but... I think that's being overly simplistic. They've had vector clocks from basically the beginning (though I gather they've been superseded now) and have favored the strategy of offloading conflict resolution to the code that's reading the value. Without implementing that conflict resolution, then it might default to last-write-wins, but that's not the same thing as "straight up last-write-wins" since they give you the tools to implement something more intelligent.
- felixgallo 11y ago
  I was referring to the shipped default. You could always change to client-resolution, but then, how's the client supposed to know? I would have looked forward to their increasingly-good-looking CRDT ideas if that company hadn't undergone uncommanded descent into terrain.

notacoward 11y ago

I think you're confused about what "one part of the system" means. I don't mean one process running on one piece of hardware. I mean one software module with many instances - itself a distributed system, with one purpose in life (determine liveness) so it can be simple and verifiable. There's no serialization, and it does not lower availability or scalability as you claim. In fact, it preserves those properties far better than "everyone doing their own failure detection a hundred different ways" does.

felixgallo 11y ago
oh, not confused at all! I use erlang, like the OP. Right now I have several erlang nodes that are doing a few hundred thousand http connections concurrently. Looking at the logs, it looks like about 10-20 of those concurrent processes are dying every second for various reasons (upstream crappiness, network errors, parse errors, sunspots for all I know right now), and are getting resurrected by their supervisors. Every blue moon, an entire network target will go out and the restart frequency will be too high, and the supervisors themselves will commit suicide and be restarted by their supervisors. Sometimes a machine will go down and come back up. Looks like the one I'm on has an uptime of about 90 days, which seems low, but doesn't really matter. If too many of them go down at the same time, I'm notified. Haven't ever been notified. You don't have to do 'failure detection a hundred different ways' (what does that even mean?) if you just crash, and build your systems to go with the flow, independently, like ants.
- notacoward 11y ago
  Well, somebody has to detect the failures, and restart failed processes, etc. so you can just "go with the flow". If you can "just crash" that's great, but it doesn't lead to much helpful advice for people who actually design the underlying systems.
  - felixgallo 11y ago
    Sorry I wasn't clear. That's exactly the advice. Design systems so that they can crash harmlessly in the small, and availability and scalability come to you quite naturally. I agree the (distant) second best alternative is to do as you suggest: to try to rely on a paxos implementation that keeps a stable leader and tries to keep track of liveness. That's pretty hard too! Besides all of aphyr's jepsen test series, check this shit out: https://github.com/coreos/fleet/issues/1289 https://github.com/coreos/fleet/issues/1289 . Tough! Tough even when you think you know what you're doing. And of course no help against freak occurrences where you can't elect a stable leader any more, or you flap leaders, or the election gets stuck. Currently the only known-good implementation of raft is in Idris. Do you know Idris? I sure as hell don't. Fortunately I know ants. Ants can die in great numbers and still move the leaf. In practice, in production, thousands of terabytes, even on AWS, my ants move and never bother me. Let it crash.
    - YuriNiyazov 11y ago
      Could you go into greater detail about how the only known-good implementation of raft is in Idris? Never heard of raft before you mentioned it (i don't deal with distributed systems day-to-day), but I am reading the paper now. and what's ants?
      - deleted 11y ago
        [deleted]
      - liveoneggs 11y ago
        ants (the bug) is an analogy for erlang workers

curun1r 11y ago

> Even Riak, the poster child for this sort of thing, was straight up last-write-wins for years while they were making fun of everyone else. Disclaimer: I've only used Riak for toy projects, never on a real production app, but... I think that's being overly simplistic. They've had vector clocks from basically the beginning (though I gather they've been superseded now) and have favored the strategy of offloading conflict resolution to the code that's reading the value. Without implementing that conflict resolution, then it might default to last-write-wins, but that's not the same thing as "straight up last-write-wins" since they give you the tools to implement something more intelligent.

felixgallo 11y ago
I was referring to the shipped default. You could always change to client-resolution, but then, how's the client supposed to know? I would have looked forward to their increasingly-good-looking CRDT ideas if that company hadn't undergone uncommanded descent into terrain.