4 ms·

Dynamo is not exactly a performant or efficient model. It's the equivalent of pulling all the distributed systems guts out and handing them to the user to deal

Dynamo is not exactly a performant or efficient model. It's the equivalent of pulling all the distributed systems guts out and handing them to the user to deal with. And the resulting toll is quantifiable: http://damienkatz.net/2013/05/dynamo_sure_works_hard.html http://damienkatz.net/2013/05/dynamo_sure_works_hard.html

kstrauser 9y ago
Damien's a very smart guy, but I don't think I agree with him here: > Within a datacenter, the Mean Time To Failure (MTTF) for a network switch is one to two orders of magnitude higher than servers, depending on the quality of the switch. A switch is highly unlikely to fail. They seem to be bulletproof. But having worked with a datacenter (on the engineering team of an early AWS competitor), switch _misconfiguration_ was all too common. Maybe a tech accidentally plugs in the wrong ethernet cable and forms a switching loop. Maybe someone fat-fingers a tag and a broken VLAN gets automatically deployed to 10,000 nodes. Either way, the _switch_ is alive, well, and pushing packets - but they're the _wrong_ packets and the result is indistinguishable from hardware failure to the end user. At datacenter scales, these things happen... not infrequently. If you engineer your database to expect that netsplits are rare, you're going to have a bad time.
- elcritch 9y ago
  Good points. Weren't the last couple of AWS outages partly due to misconfigured network configs? Depending on your problem the replicated reads makes sense given those kinda of outages. Though I'm new-ish to riak's "core" design, when you get the APL with the servers, isn't it feasible to create a design similar to what Damien's proposing using a preferred master for a given vnode?
- kyledrake 9y ago
  VLANs were the bane of my existence when I had to figure out how to deal with them. I don't envy anyone whose job is to manage them on switches for a lot of servers.

kstrauser 9y ago

Damien's a very smart guy, but I don't think I agree with him here: > Within a datacenter, the Mean Time To Failure (MTTF) for a network switch is one to two orders of magnitude higher than servers, depending on the quality of the switch. A switch is highly unlikely to fail. They seem to be bulletproof. But having worked with a datacenter (on the engineering team of an early AWS competitor), switch _misconfiguration_ was all too common. Maybe a tech accidentally plugs in the wrong ethernet cable and forms a switching loop. Maybe someone fat-fingers a tag and a broken VLAN gets automatically deployed to 10,000 nodes. Either way, the _switch_ is alive, well, and pushing packets - but they're the _wrong_ packets and the result is indistinguishable from hardware failure to the end user. At datacenter scales, these things happen... not infrequently. If you engineer your database to expect that netsplits are rare, you're going to have a bad time.

elcritch 9y ago
Good points. Weren't the last couple of AWS outages partly due to misconfigured network configs? Depending on your problem the replicated reads makes sense given those kinda of outages. Though I'm new-ish to riak's "core" design, when you get the APL with the servers, isn't it feasible to create a design similar to what Damien's proposing using a preferred master for a given vnode?
kyledrake 9y ago
VLANs were the bane of my existence when I had to figure out how to deal with them. I don't envy anyone whose job is to manage them on switches for a lot of servers.