4 ms·

It's interesting to compare this with the Facebook stack, drawn by the same author: http://malteschwarzkopf.de/research/assets/facebook-stack.pdf http://maltes

by mrry 11y ago

It's interesting to compare this with the Facebook stack, drawn by the same author: http://malteschwarzkopf.de/research/assets/facebook-stack.pdf http://malteschwarzkopf.de/research/assets/facebook-stack.pd... …and it would be doubly interesting to see the "related work" arrows between the different companies' infrastructure platforms. We'd need a 3D visualization for that though.

aristus 11y ago
I'd take this chart with a grain of salt, given the FB paper. Peregrine is no longer in service, IIRC. It was a friendly rival to Scuba, and was eventually replaced by Presto, which is not even mentioned. Also not mentioned are several important things like the pub/sub ecosystem built on Scribe. Haystack is long dead, except for maybe some archived stuff. Lastly, PHP-to-C++ has not been a thing since early 2012.
- ms705 11y ago
  Original author here. This is interesting -- I tried to do my best to find out what's still in use and what is deprecated based on public information; happy to amend if anything is incorrect. (If you have publicly accessible written sources saying so, that'd be ideal!) Note that owing to its origin (as part of my PhD thesis), this chart only mentions systems about which scientific, peer-reviewed papers have been published. That's why Scribe and Presto are missing; I couldn't find any papers about them. For Scribe, the Github repo I found explicitly says that it is no longer being maintained, although maybe it's still used internally. Re Haystack: I'm surprised it's deprecated -- the f4 paper (2014) heavily implies that it is still used for "hot" blobs. Re HipHop: ok, I wasn't sure if it was still current, since I had heard somewhere that it's been superseded. Couldn't find anything definite saying so, though. If you have a pointer, that'd be great. BTW, one reason I posted these on Twitter was the hope to get exactly this kind of fact-checking going, so I'm pleased that feedback is coming :-)
  - kajecounterhack 11y ago
    Re Hiphop I think HHVM + Hack (Facebook's internal "improved PHP") has superseded it but while HHVM is open sourced Hack isn't public.
    - chrsm 11y ago
      Hack is just a piece of HHVM, it's open: http://hacklang.org/ http://hacklang.org/
      - kajecounterhack 11y ago
        Ah thanks, I didn't realize it was out. I read a separate article that said it hadn't been released yet -- it was probably outdated.
  - aristus 11y ago
    HipHop's replacement was pretty widely reported: http://www.wired.com/2013/06/facebook-hhvm-saga/ http://www.wired.com/2013/06/facebook-hhvm-saga/ This link has papers on pub/sub, HHVM, and so on: https://research.facebook.com/publications/ https://research.facebook.com/publications/ Re Haystack: possible I am misremembering, or the project to completely replace Haystack stalled since I left. If you want to gather a more complete picture of infrastructure at these companies I suggest, well, not imposing the strange limitation of only reading peer-reviewed papers. Almost none of the stuff I worked on ended up in conference proceedings.
    - ms705 11y ago
      Thanks, I've added HHVM and marked HipHop as superseded. I also added Wormhole, which I think is the pub/sub system you're referring to (published in NSDI 2015: https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-sharma.pdf https://www.usenix.org/system/files/conference/nsdi15/nsdi15...). Updated version at original URL: http://malteschwarzkopf.de/research/assets/facebook-stack.pdf http://malteschwarzkopf.de/research/assets/facebook-stack.pd... Regarding the focus on academic papers: I agree that this does not reflect the entirety of what goes on inside companies (or indeed how open they are; FB and Google also release lots of open-source software). Certainly, only reading the papers would be a bad idea. However, a peer-reviewed paper (unlike, say, a blog post) is a permanent, citable reference and is part of the scientific literature. This sets a quality bar (enforced through peer review, which deemed the description to be plausible and accurate), and allows the amount of information to remain manageable. The number of other sources of information makes them impractical to write up concisely, and it is hard to say what ought to be included and what should not when going beyond published papers. I don't think anyone should base their perception of how Google's or Facebook's stack works on these charts and bibliographies -- not least because they will quickly be out of date. However, I personally find them useful as a quick reference to comprehensive, high-quality descriptions of systems that are regularly mentioned in discussions :-)
- iribe 11y ago
  Presto is a copy of dremel. Putting it all together: gfs -> hdfs bigtable -> hbase google mapreduce -> hadoop dremel -> presto protocol buffers -> swift
  - low_battery 11y ago
    protocol buffers -> thrift
- nocarrier 11y ago
  Carlos, you are mistaken re: Haystack. I don't think it was ever planned to be replaced, it's always been in use as a hot blob storage engine since it was launched. There are three storage layers for blobs at Facebook: hot (Haystack @ 3.6x replication), warm (F4 @ 2.1x), cold (Cold Storage @ 1.4x). We have published papers describing each layer. Haystack still handles the initial writes and the burst of heavy read traffic when an object is fresh. It has a higher space overhead of 3.6x because it's optimized for throughput and latency versus cost savings. The Haystack tier's size is now dwarfed by the size of the F4 and Cold Storage tiers, but it still handles almost all of the reads for user blobs on Facebook due to the age/demand exponential curve. After a haystack volume is locked and has cooled down enough, it moves to F4 to bump its replication factor way down for cheaper, longer term online storage. And then Cold Storage is used for the older stuff that gets barely any reads but that you still want to store online for perpetuity. That is why the team that works on media storage is called Everstore; they take storing people's photos and videos seriously and view it as a promise to keep your memories available forever. It feels really good to see photos from 5+ years ago and have them still work, and someday there will be 30 year old photos and videos on Everstore as well. Source: I built Haystack with a couple other people and founded the Everstore team. :-)
bra-ket 11y ago
related: https://www.quora.com/What-is-Facebooks-architecture https://www.quora.com/What-is-Facebooks-architecture

by mrry 11y ago

aristus 11y ago

I'd take this chart with a grain of salt, given the FB paper. Peregrine is no longer in service, IIRC. It was a friendly rival to Scuba, and was eventually replaced by Presto, which is not even mentioned. Also not mentioned are several important things like the pub/sub ecosystem built on Scribe. Haystack is long dead, except for maybe some archived stuff. Lastly, PHP-to-C++ has not been a thing since early 2012.

ms705 11y ago
Original author here. This is interesting -- I tried to do my best to find out what's still in use and what is deprecated based on public information; happy to amend if anything is incorrect. (If you have publicly accessible written sources saying so, that'd be ideal!) Note that owing to its origin (as part of my PhD thesis), this chart only mentions systems about which scientific, peer-reviewed papers have been published. That's why Scribe and Presto are missing; I couldn't find any papers about them. For Scribe, the Github repo I found explicitly says that it is no longer being maintained, although maybe it's still used internally. Re Haystack: I'm surprised it's deprecated -- the f4 paper (2014) heavily implies that it is still used for "hot" blobs. Re HipHop: ok, I wasn't sure if it was still current, since I had heard somewhere that it's been superseded. Couldn't find anything definite saying so, though. If you have a pointer, that'd be great. BTW, one reason I posted these on Twitter was the hope to get exactly this kind of fact-checking going, so I'm pleased that feedback is coming :-)
- kajecounterhack 11y ago
  Re Hiphop I think HHVM + Hack (Facebook's internal "improved PHP") has superseded it but while HHVM is open sourced Hack isn't public.
  - chrsm 11y ago
    Hack is just a piece of HHVM, it's open: http://hacklang.org/ http://hacklang.org/
    - kajecounterhack 11y ago
      Ah thanks, I didn't realize it was out. I read a separate article that said it hadn't been released yet -- it was probably outdated.
- aristus 11y ago
  HipHop's replacement was pretty widely reported: http://www.wired.com/2013/06/facebook-hhvm-saga/ http://www.wired.com/2013/06/facebook-hhvm-saga/ This link has papers on pub/sub, HHVM, and so on: https://research.facebook.com/publications/ https://research.facebook.com/publications/ Re Haystack: possible I am misremembering, or the project to completely replace Haystack stalled since I left. If you want to gather a more complete picture of infrastructure at these companies I suggest, well, not imposing the strange limitation of only reading peer-reviewed papers. Almost none of the stuff I worked on ended up in conference proceedings.
  - ms705 11y ago
    Thanks, I've added HHVM and marked HipHop as superseded. I also added Wormhole, which I think is the pub/sub system you're referring to (published in NSDI 2015: https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-sharma.pdf https://www.usenix.org/system/files/conference/nsdi15/nsdi15...). Updated version at original URL: http://malteschwarzkopf.de/research/assets/facebook-stack.pdf http://malteschwarzkopf.de/research/assets/facebook-stack.pd... Regarding the focus on academic papers: I agree that this does not reflect the entirety of what goes on inside companies (or indeed how open they are; FB and Google also release lots of open-source software). Certainly, only reading the papers would be a bad idea. However, a peer-reviewed paper (unlike, say, a blog post) is a permanent, citable reference and is part of the scientific literature. This sets a quality bar (enforced through peer review, which deemed the description to be plausible and accurate), and allows the amount of information to remain manageable. The number of other sources of information makes them impractical to write up concisely, and it is hard to say what ought to be included and what should not when going beyond published papers. I don't think anyone should base their perception of how Google's or Facebook's stack works on these charts and bibliographies -- not least because they will quickly be out of date. However, I personally find them useful as a quick reference to comprehensive, high-quality descriptions of systems that are regularly mentioned in discussions :-)
iribe 11y ago
Presto is a copy of dremel. Putting it all together: gfs -> hdfs bigtable -> hbase google mapreduce -> hadoop dremel -> presto protocol buffers -> swift
- low_battery 11y ago
  protocol buffers -> thrift
nocarrier 11y ago
Carlos, you are mistaken re: Haystack. I don't think it was ever planned to be replaced, it's always been in use as a hot blob storage engine since it was launched. There are three storage layers for blobs at Facebook: hot (Haystack @ 3.6x replication), warm (F4 @ 2.1x), cold (Cold Storage @ 1.4x). We have published papers describing each layer. Haystack still handles the initial writes and the burst of heavy read traffic when an object is fresh. It has a higher space overhead of 3.6x because it's optimized for throughput and latency versus cost savings. The Haystack tier's size is now dwarfed by the size of the F4 and Cold Storage tiers, but it still handles almost all of the reads for user blobs on Facebook due to the age/demand exponential curve. After a haystack volume is locked and has cooled down enough, it moves to F4 to bump its replication factor way down for cheaper, longer term online storage. And then Cold Storage is used for the older stuff that gets barely any reads but that you still want to store online for perpetuity. That is why the team that works on media storage is called Everstore; they take storing people's photos and videos seriously and view it as a promise to keep your memories available forever. It feels really good to see photos from 5+ years ago and have them still work, and someday there will be 30 year old photos and videos on Everstore as well. Source: I built Haystack with a couple other people and founded the Everstore team. :-)

bra-ket 11y ago

related: https://www.quora.com/What-is-Facebooks-architecture https://www.quora.com/What-is-Facebooks-architecture