4 ms·

Dynomite – Making Non-Distributed Databases Distributed

185 points by stefanwild 12y ago

jedberg 12y ago
When I saw this internally a few months ago, I said that I haven't been more excited about a new project in a long time. I personally think this will solve a lot of scalability problems people have while letting them keep their favorite database.
- stefanwild 12y ago
  It looks very promising. I like the fact that it doesn't only work for AWS environments.
evantahler 12y ago
Are there any deployment notes/examples?
- raxen 12y ago
  not really, the best you are going to find so far: https://github.com/Netflix/dynomite https://github.com/Netflix/dynomite
- nrbafna 12y ago
  Setup and Installation Wiki - https://github.com/Netflix/dynomite/wiki/Getting+Started#setup-and-installation https://github.com/Netflix/dynomite/wiki/Getting+Started#set...
void_star 12y ago
It's a little unfortunate that consistency is only mentioned as a footnote. If this is being used internally at Netflix, how are they dealing with consistency issues in the face of failed or conflicting writes?
- antirez 12y ago
  This currently looks like the Dynamo-alike sharding without the read-repair part making values eventually convergent in all the replicas. This latter feature is much needed, but they are already planning it apparently.
  - void_star 12y ago
    Except no mention of vector clocks, so it's unclear how one would detect or repair a conflict. Maybe that is an implementation detail that is intended to be ironed out later...
    - jedberg 12y ago
      With a replication factor of 3 or more you don't need a vector clock, you can just choose majority wins. That being said, I believe it is using a vector clock, but I'm not sure.
      - grogers 12y ago
        You still need ordering amongst writes. Without that you will get into a situation where there is no majority, all three nodes have different data.
        Xorlev 12y ago
        You need quorum writes and reads. Like Cassandra you can use timestamps even though vector clocks would be much preferred.
        antirez 12y ago
        Converging to a single value (which is a liveness property) and what the values will contain (which are usually safety properties) are different stories... when the partition heals, if the nodes detect they have a different version for a given piece of data, they can just remove it, and the system is convergent, which means it is eventually consistent (a liveness property). What happens in the merge function instead provides additional safety properties. For example in the case of a CRDT that models a Set type, with the merge function that is "union of all the values in the nodes", you can state the safety property that if a Set element X was added into a node, at no time will happen that the nodes will be able to talk again, and the system will converge into a Set which excludes such value X.
fizx 12y ago
For projects with similar goals, see also Twitter's Gizzard (abandoned) and LinkedIn's Voldemort.
- wowmsi 12y ago
  Also Riak. However, one interesting aspect of Dynomite seems to be offer default support for clustering across multiple data centers with pluggable stores.
  - darkr 12y ago
    Riak (enterprise version) does that, and has pluggable stores (albeit limited to in-memory, bitcask + eleveldb)
- anonfunction 12y ago
  Add Mashape's Astronode to the list: https://github.com/Mashape/Astronode-Broadcaster https://github.com/Mashape/Astronode-Broadcaster
jlafon 12y ago
It's interesting that Netflix decided to write this rather than using Amazon's own DynamoDB. I wonder specifically if DynamoDB was too expensive (as I have found), or was there some other reason?
- jetblackio 12y ago
  This is completely different than DynamoDB. Essentially this is a framework for sharding or horizontally scaling multiple datastores (MySQL, Memcache, Redis, etc).
  - jlafon 12y ago
    That makes sense. I had only seen the Memcache & Redis datastores being used (and both being comparable to DynamoDB).
  - amelius 12y ago
    Why would you use multiple datastores? Isn't it much more difficult to atomically store data this way? Assume a transaction succeeds in store 1, and fails in store 2, you'd need to roll back the transaction in store 1 after it has committed (!)
    - earless1 12y ago
      Because there is no such thing as a one size fits all datasotre. Some data may make way more sense stored in a relational database, while some data makes more sense as key:value pairs. Some data may be infrequently accessed and stored to disk while others are stored in memory for quick access. Trying to make a single datastore work for many use cases will cause more pain than it is worth.
    - michaelmior 12y ago
      You wouldn't likely use multiple data stores for the same data. That is, you probably wouldn't have a single system which stores data in both Redis and MySQL. However, you can imagine different use cases for each of these stores and you can potentially increase operational efficiency by having a common layer which handles the distribution of data for each store.
deleted 12y ago
[deleted]
ppearcy 12y ago
Any ideas if this supports Postgres? Also, I noticed comments about an architecture document here: https://github.com/Netflix/dynomite https://github.com/Netflix/dynomite Anyone know where the doc they are referencing is located?
- jedberg 12y ago
  It doesn't support Postgres yet but that is on the (longer term) roadmap. However, since it is open source, contributions are welcome! :)
halayli 12y ago
nitpick, the code check for NULLs at a low level but misses checking NULLs at higher levels like here: https://github.com/Netflix/dynomite/blob/master/src/dyn_dnode_msg.c#L601 https://github.com/Netflix/dynomite/blob/master/src/dyn_dnod...
personZ 12y ago
Netflix always makes for an interesting case, but what always strikes me is how incredibly "trivial" their application really is: It's a simple application at a very large scale. This doesn't dismiss their accomplishments, of course, but often it does seem like Netflix spends an enormous engineering effort on fairly low value aspects of the service.
- brown9-2 12y ago
  this seems like a pretty common fallacy for people to think about an organization from the outside
  - personZ 12y ago
    How is it a 'fallacy'? To be clear, this has nothing to do with the actual streaming (which does not happen from EC2) but instead are the basics like viewing history and categories. It absolutely is the simple part of the application, albeit, as mentioned, at a very large scale. Given that someone else mentioned it, does Netflix even bother with the whole recommendation algorithm thing anymore? My personal experience has been that it gives the most superficial of recommendations. And if we're being honest, the Netflix app layer is kind of terrible. I could enumerate the problems, discoverability being king, but it is not the compelling part of the service.
    - MichaelGG 12y ago
      The recommendations even contradict their own ratings. Netflix often recommends complete crap to me, and even says I'm likely to rate it one star. It even offers me things I've rated as not interesting or one star. Meanwhile, app level features like restricting certain types of content isn't available. I want to be able to delete crap from the Netflix catalog when my kids log in.. I don't care about nudity, language, etc. but more about just how dumb some of the cartoons are.
- jrallison 12y ago
  You're discounting the "very large scale" portion of your statement. What may seem fairly low value from the outside or consumer perspective is likely of huge value internally to ensure you can always watch Netflix without even thinking about everything that has to happen to deliver as much video as they do to their customer's browsers.
nteon 12y ago
Netflix is a Java-heavy shop (the client talked up in the blog post is the Java client). I was pleasantly surprised to see that dynomite is implemented in C with an eye towards both BSD + Linux. I'm curious if this was because a GC'ed language like Java or Go was tried and proven not up to the task, or what.
- viksit 12y ago
  My guess would be that the ecosystem in their (seed) target set of databases and protocols is written in C. In one case, they're using Twitters twemproxy to provide connection counts for redis/memcached.
  - zerobfd 12y ago
    Not just connection counts; when I first looked at the github I was surprised at how close the code organization was to twemproxy, and when I looked further a huge amount of the code is built on top of existing twemproxy code. Netflix probably saw an opportunity to built the Dynamo layer on top of what Twitter had already written, and decided to just keep building on in C instead of rewriting a fairly large codebase.
- mumrah 12y ago
  My guess would be that they went with C to make it easier to hook into existing data systems like MySQL or Redis.
- padho 12y ago
  My guess is that this has a performance reason
lobster_johnson 12y ago
Funnily enough Dynomite was also the name of an Erlang implementation (https://github.com/moonpolysoft/dynomite https://github.com/moonpolysoft/dynomite) of Dynamo that seemed to get off to a very promising start before it was abandoned a few years ago.
diltonm 12y ago
I don't know if it was intentional but the name reminds me of the character JJ in the TV series Good Times. Awesome show that was. It sounds solid overall but one thing caught my eye, "A client can connect to any node to read the data. Similar to writes, a node serves the read request if it owns the data, otherwise it forwards the read request to the data owning node in the same rack.". I wonder if that could lead to an owning node arbitration storm when more than one node in a rack owns the data?
NKCSS 12y ago
Cool, it's something that Microsoft SQL Server (Enterprise edition) offers out of the box (Peer-2-Peer replication). The only problem with P2P replication is that, at large scale, requires massive amounts of bandwidth to keep everything synced, and it gets worse the more nodes you add.
- diptanu 12y ago
  Depends how things are replicated. In Dynamo style replication everything isn't replicated everywhere. Based on consistent hashing, and how vnodes etc are configured, a key lands itself in more than one nodes so that things are generally available. So scale-out in a dynamo-like system happens more gracefully.
RealCasually 12y ago
Perhaps I missed it, but is there any discussion on how their sharding mechanism works? How do they define a shard and where is ownership of various shards managed so a "non-owning node can redirect to an owning node"?
- marktangotango 12y ago
  I questioned tbat as well, could be a property of the underlying store (memcache hash). How would mysql sharding work without being application managed?

185 points by stefanwild 12y ago

When I saw this internally a few months ago, I said that I haven't been more excited about a new project in a long time. I personally think this will solve a lot of scalability problems people have while letting them keep their favorite database.

stefanwild 12y ago
It looks very promising. I like the fact that it doesn't only work for AWS environments.

evantahler 12y ago

Are there any deployment notes/examples?

raxen 12y ago
not really, the best you are going to find so far: https://github.com/Netflix/dynomite https://github.com/Netflix/dynomite
nrbafna 12y ago
Setup and Installation Wiki - https://github.com/Netflix/dynomite/wiki/Getting+Started#setup-and-installation https://github.com/Netflix/dynomite/wiki/Getting+Started#set...

void_star 12y ago

It's a little unfortunate that consistency is only mentioned as a footnote. If this is being used internally at Netflix, how are they dealing with consistency issues in the face of failed or conflicting writes?

antirez 12y ago
This currently looks like the Dynamo-alike sharding without the read-repair part making values eventually convergent in all the replicas. This latter feature is much needed, but they are already planning it apparently.
- void_star 12y ago
  Except no mention of vector clocks, so it's unclear how one would detect or repair a conflict. Maybe that is an implementation detail that is intended to be ironed out later...
  - jedberg 12y ago
    With a replication factor of 3 or more you don't need a vector clock, you can just choose majority wins. That being said, I believe it is using a vector clock, but I'm not sure.
    - grogers 12y ago
      You still need ordering amongst writes. Without that you will get into a situation where there is no majority, all three nodes have different data.
      - Xorlev 12y ago
        You need quorum writes and reads. Like Cassandra you can use timestamps even though vector clocks would be much preferred.
      - antirez 12y ago
        Converging to a single value (which is a liveness property) and what the values will contain (which are usually safety properties) are different stories... when the partition heals, if the nodes detect they have a different version for a given piece of data, they can just remove it, and the system is convergent, which means it is eventually consistent (a liveness property). What happens in the merge function instead provides additional safety properties. For example in the case of a CRDT that models a Set type, with the merge function that is "union of all the values in the nodes", you can state the safety property that if a Set element X was added into a node, at no time will happen that the nodes will be able to talk again, and the system will converge into a Set which excludes such value X.

fizx 12y ago

For projects with similar goals, see also Twitter's Gizzard (abandoned) and LinkedIn's Voldemort.

wowmsi 12y ago
Also Riak. However, one interesting aspect of Dynomite seems to be offer default support for clustering across multiple data centers with pluggable stores.
- darkr 12y ago
  Riak (enterprise version) does that, and has pluggable stores (albeit limited to in-memory, bitcask + eleveldb)
anonfunction 12y ago
Add Mashape's Astronode to the list: https://github.com/Mashape/Astronode-Broadcaster https://github.com/Mashape/Astronode-Broadcaster

jlafon 12y ago

It's interesting that Netflix decided to write this rather than using Amazon's own DynamoDB. I wonder specifically if DynamoDB was too expensive (as I have found), or was there some other reason?

jetblackio 12y ago
This is completely different than DynamoDB. Essentially this is a framework for sharding or horizontally scaling multiple datastores (MySQL, Memcache, Redis, etc).
- jlafon 12y ago
  That makes sense. I had only seen the Memcache & Redis datastores being used (and both being comparable to DynamoDB).
- amelius 12y ago
  Why would you use multiple datastores? Isn't it much more difficult to atomically store data this way? Assume a transaction succeeds in store 1, and fails in store 2, you'd need to roll back the transaction in store 1 after it has committed (!)
  - earless1 12y ago
    Because there is no such thing as a one size fits all datasotre. Some data may make way more sense stored in a relational database, while some data makes more sense as key:value pairs. Some data may be infrequently accessed and stored to disk while others are stored in memory for quick access. Trying to make a single datastore work for many use cases will cause more pain than it is worth.
  - michaelmior 12y ago
    You wouldn't likely use multiple data stores for the same data. That is, you probably wouldn't have a single system which stores data in both Redis and MySQL. However, you can imagine different use cases for each of these stores and you can potentially increase operational efficiency by having a common layer which handles the distribution of data for each store.

deleted 12y ago

[deleted]

ppearcy 12y ago

Any ideas if this supports Postgres? Also, I noticed comments about an architecture document here: https://github.com/Netflix/dynomite https://github.com/Netflix/dynomite Anyone know where the doc they are referencing is located?

jedberg 12y ago
It doesn't support Postgres yet but that is on the (longer term) roadmap. However, since it is open source, contributions are welcome! :)

halayli 12y ago

nitpick, the code check for NULLs at a low level but misses checking NULLs at higher levels like here: https://github.com/Netflix/dynomite/blob/master/src/dyn_dnode_msg.c#L601 https://github.com/Netflix/dynomite/blob/master/src/dyn_dnod...

personZ 12y ago

Netflix always makes for an interesting case, but what always strikes me is how incredibly "trivial" their application really is: It's a simple application at a very large scale. This doesn't dismiss their accomplishments, of course, but often it does seem like Netflix spends an enormous engineering effort on fairly low value aspects of the service.

brown9-2 12y ago
this seems like a pretty common fallacy for people to think about an organization from the outside
- personZ 12y ago
  How is it a 'fallacy'? To be clear, this has nothing to do with the actual streaming (which does not happen from EC2) but instead are the basics like viewing history and categories. It absolutely is the simple part of the application, albeit, as mentioned, at a very large scale. Given that someone else mentioned it, does Netflix even bother with the whole recommendation algorithm thing anymore? My personal experience has been that it gives the most superficial of recommendations. And if we're being honest, the Netflix app layer is kind of terrible. I could enumerate the problems, discoverability being king, but it is not the compelling part of the service.
  - MichaelGG 12y ago
    The recommendations even contradict their own ratings. Netflix often recommends complete crap to me, and even says I'm likely to rate it one star. It even offers me things I've rated as not interesting or one star. Meanwhile, app level features like restricting certain types of content isn't available. I want to be able to delete crap from the Netflix catalog when my kids log in.. I don't care about nudity, language, etc. but more about just how dumb some of the cartoons are.
jrallison 12y ago
You're discounting the "very large scale" portion of your statement. What may seem fairly low value from the outside or consumer perspective is likely of huge value internally to ensure you can always watch Netflix without even thinking about everything that has to happen to deliver as much video as they do to their customer's browsers.

nteon 12y ago

Netflix is a Java-heavy shop (the client talked up in the blog post is the Java client). I was pleasantly surprised to see that dynomite is implemented in C with an eye towards both BSD + Linux. I'm curious if this was because a GC'ed language like Java or Go was tried and proven not up to the task, or what.

viksit 12y ago
My guess would be that the ecosystem in their (seed) target set of databases and protocols is written in C. In one case, they're using Twitters twemproxy to provide connection counts for redis/memcached.
- zerobfd 12y ago
  Not just connection counts; when I first looked at the github I was surprised at how close the code organization was to twemproxy, and when I looked further a huge amount of the code is built on top of existing twemproxy code. Netflix probably saw an opportunity to built the Dynamo layer on top of what Twitter had already written, and decided to just keep building on in C instead of rewriting a fairly large codebase.
mumrah 12y ago
My guess would be that they went with C to make it easier to hook into existing data systems like MySQL or Redis.
padho 12y ago
My guess is that this has a performance reason

lobster_johnson 12y ago

Funnily enough Dynomite was also the name of an Erlang implementation (https://github.com/moonpolysoft/dynomite https://github.com/moonpolysoft/dynomite) of Dynamo that seemed to get off to a very promising start before it was abandoned a few years ago.

diltonm 12y ago

I don't know if it was intentional but the name reminds me of the character JJ in the TV series Good Times. Awesome show that was. It sounds solid overall but one thing caught my eye, "A client can connect to any node to read the data. Similar to writes, a node serves the read request if it owns the data, otherwise it forwards the read request to the data owning node in the same rack.". I wonder if that could lead to an owning node arbitration storm when more than one node in a rack owns the data?

NKCSS 12y ago

Cool, it's something that Microsoft SQL Server (Enterprise edition) offers out of the box (Peer-2-Peer replication). The only problem with P2P replication is that, at large scale, requires massive amounts of bandwidth to keep everything synced, and it gets worse the more nodes you add.

diptanu 12y ago
Depends how things are replicated. In Dynamo style replication everything isn't replicated everywhere. Based on consistent hashing, and how vnodes etc are configured, a key lands itself in more than one nodes so that things are generally available. So scale-out in a dynamo-like system happens more gracefully.

RealCasually 12y ago

Perhaps I missed it, but is there any discussion on how their sharding mechanism works? How do they define a shard and where is ownership of various shards managed so a "non-owning node can redirect to an owning node"?

marktangotango 12y ago
I questioned tbat as well, could be a property of the underlying store (memcache hash). How would mysql sharding work without being application managed?