3 ms·

I know nothing about storage, so I'm a bit confused about why Twitter needed: 1. Hadoop 2. Graph 3. Redis/Memcache 4. Blobstore 5. SQL variants (and a few

by tabeth 10y ago

I know nothing about storage, so I'm a bit confused about why Twitter needed: 1. Hadoop 2. Graph 3. Redis/Memcache 4. Blobstore 5. SQL variants (and a few others). I do see that the post has a short snippet briefly describing what they're storing, but I'd be curious to know why (speed, cost, latency, space tradeoffs/constraints). Also, if any more experienced folks want to chime in: Elixir/Erlang is "built for concurrency" as they say. I'd love to hear people's opinions on their one sentence simplifications of what kind of situation Hadoop/SQL/Redis/etc should be used for (similar to how Erlang is best used for situations where concurrency and fault tolerance is desired). In particular, is there a "Code Complete" type book for storage?

redler 10y ago
You might try "Seven Databases in Seven Weeks". It's a few years old now but still solid.
timClicks 10y ago
Erlang was behind the NoSQL database that started the NoSQL hype train (CouchDB), a document database. The Erlang distribution contains an amazing distributed database called Mnesia. For books, read the AOSA Chapter on Riak (http://www.aosabook.org/en/riak.html http://www.aosabook.org/en/riak.html) and then read through the tutorials on Riak Core. Oh, and Erlang in Anger https://s3.amazonaws.com/erlang-in-anger/text.v1.0.3.pdf https://s3.amazonaws.com/erlang-in-anger/text.v1.0.3.pdf
- rozap 10y ago
  Mnesia is a joy to use, but keep mind it was written before the CAP theorem existed, so it doesn't behave that well in the face of network partitions.
marcinzm 10y ago
That seems a pretty standard stack, I don't know about twitter but I can say why I'd use them in my experience: 5. SQL variants: This is where you keep your source of truth data needed to serve request. It's flexible, reliable but moderately fast. The backbone of your infrastructure and you can consider everything else as enhancements to it to overcome specific limitations. 4. Blobstore: Your SQL database is not going to be good at storing large binary blobs such as files or images. 3. Redis/Memcache: Your SQL database may be too slow for some requests and/or it's data model may require too much work in the API layer to align with the final presentation model. So you want a blazing fast store that you can use for that final presentation view when requests don't differ from each other. 2. Graph: SQL databases aren't the best for graph queries or data (I mean you can hack it but it's going to be slower and uglier code wise) so if you need to look at data as a graph something dedicated makes sense. 1. Hadoop: You're getting lot's of log data from users, what they do, when they do it and so on. You want to analyze this data, build models (who to recommend for people to follow for example) and use it to probably serve ads (gotta pay the bills somehow). You could put this in SQL but it's a lot of data and you'll be doing a lot of read requests which will end up being slow and messy in SQL. Hadoop, on the other hand, is well suited for this type of data and workload. Large slow read heavy queries on data that is appended to but not modified in place.
- dajohnson89 10y ago
  FTFA: >Graph: Our legacy Gizzard/MySQL based sharded cluster for storing our graphs. Flock, our social graph, can handle peaks over 9 million QPS, averaging our MySQL servers to 30k - 45k QPS. Isn't storing a GraphDB in MySql a Bad Idea? Graph queries are extremely poorly suited for relational databases.
  - thinkingfish 10y ago
    I don't work on GraphDB, but basically the situation is this: 1) yes, it is not a great use case for MySQL, but that's how it started, partly because Twitter needed join on its graphs; 2) legacy systems die hard, especially at scale- Twitter is working on a better solution, but for now Flock is still what's running in production.
    - dajohnson89 10y ago
      Thanks for explaining. I had a similar situation -- storing a graph db in postgres. The performance sucked, but we didn't have much of a choice.

by tabeth 10y ago

redler 10y ago

You might try "Seven Databases in Seven Weeks". It's a few years old now but still solid.

timClicks 10y ago

Erlang was behind the NoSQL database that started the NoSQL hype train (CouchDB), a document database. The Erlang distribution contains an amazing distributed database called Mnesia. For books, read the AOSA Chapter on Riak (http://www.aosabook.org/en/riak.html http://www.aosabook.org/en/riak.html) and then read through the tutorials on Riak Core. Oh, and Erlang in Anger https://s3.amazonaws.com/erlang-in-anger/text.v1.0.3.pdf https://s3.amazonaws.com/erlang-in-anger/text.v1.0.3.pdf

rozap 10y ago
Mnesia is a joy to use, but keep mind it was written before the CAP theorem existed, so it doesn't behave that well in the face of network partitions.

marcinzm 10y ago

That seems a pretty standard stack, I don't know about twitter but I can say why I'd use them in my experience: 5. SQL variants: This is where you keep your source of truth data needed to serve request. It's flexible, reliable but moderately fast. The backbone of your infrastructure and you can consider everything else as enhancements to it to overcome specific limitations. 4. Blobstore: Your SQL database is not going to be good at storing large binary blobs such as files or images. 3. Redis/Memcache: Your SQL database may be too slow for some requests and/or it's data model may require too much work in the API layer to align with the final presentation model. So you want a blazing fast store that you can use for that final presentation view when requests don't differ from each other. 2. Graph: SQL databases aren't the best for graph queries or data (I mean you can hack it but it's going to be slower and uglier code wise) so if you need to look at data as a graph something dedicated makes sense. 1. Hadoop: You're getting lot's of log data from users, what they do, when they do it and so on. You want to analyze this data, build models (who to recommend for people to follow for example) and use it to probably serve ads (gotta pay the bills somehow). You could put this in SQL but it's a lot of data and you'll be doing a lot of read requests which will end up being slow and messy in SQL. Hadoop, on the other hand, is well suited for this type of data and workload. Large slow read heavy queries on data that is appended to but not modified in place.

dajohnson89 10y ago
FTFA: >Graph: Our legacy Gizzard/MySQL based sharded cluster for storing our graphs. Flock, our social graph, can handle peaks over 9 million QPS, averaging our MySQL servers to 30k - 45k QPS. Isn't storing a GraphDB in MySql a Bad Idea? Graph queries are extremely poorly suited for relational databases.
- thinkingfish 10y ago
  I don't work on GraphDB, but basically the situation is this: 1) yes, it is not a great use case for MySQL, but that's how it started, partly because Twitter needed join on its graphs; 2) legacy systems die hard, especially at scale- Twitter is working on a better solution, but for now Flock is still what's running in production.
  - dajohnson89 10y ago
    Thanks for explaining. I had a similar situation -- storing a graph db in postgres. The performance sucked, but we didn't have much of a choice.