8 ms·
The Google Stack [pdf]
- deleted 11y ago[deleted]
- wwweston 11y agoWhen I'm looking at this diagram I'm struck by how much it diverges from the typical entities invoked when we generally describe a tech stack. There's almost no discussion of language or framework or "patterns" in the software idiom sense. Instead it's largely a wide variety of data stores and access layers. Bad programmers worry about code, good programmers worry about data structures?
- duaneb 11y ago> Bad programmers worry about code, good programmers worry about data structures? Bingo. At this scale, all that matters are the bounds a service can guarantee. So long as it can comply to the bounds, the underlying algorithm (let along language, patterns, etc) doesn't matter. Facebook's development of the HHVM is a perfect example of this—they took a nearly universally despised language (from an engineering perspective) and tooled it to fit into the necessary constraints. While crucial for explaining their stack, it's not in any way necessary for any of the individual nodes.
- endtime 11y agoI think it's a matter of scale, rather than quality - we obviously use languages and frameworks and patterns at Google.
- wwweston 11y agoScale of operations at Google, or conceptual scale of the diagram? And if you feel like language and framework choice plays a significant enough role at Google that "stack" in the more traditional sense is something that's carefully considered by engineering, I'd love to hear about the details.
- reagency 11y agoThis diagram is about services that run o a machine and vend data to other machines, not the libraries or frameworks (GWT, etc) that implement the services. This diagram is more "zoomed out"
- endtime 11y agoThe scale of the diagram, though obviously a diagram of that scale can only be drawn for an organization with operations at that scale - so kind of both. I've mostly done backend/data stuff at Google, so I can't speak to traditional web dev decisions, but I've written design docs which discuss the tradeoffs between using Bigtable and Spanner, or Flume vs. Mapreduce, or one serving strategy vs. another, for some specific thing. Maybe those choices are vaguely analogous to choosing between Postgres or Mongo, nginx or Apache, etc. I imagine the guys who write webapps (not Search, obviously, but internal apps or things like our help pages) consider whether to use App Engine, Angular, Django, etc. on a per-app basis.
- asuffield 11y ago(Tedious disclaimer: my opinion, not my employer's. Not representing anybody else. I'm an SRE at Google, and use some of this stuff all the time.) In all honesty, this diagram is missing so much information that you won't learn very much from it about how we do things (and all the things you can't see are confidential). It's more like a map of the relationships between the few bits that have been published. However, I would tend to agree with your final suggestion that data structures are much more interesting than code.
- NicoJuicy 11y agoEveryone worries about code, but on top of code you have architecture. Worrying about code makes other programmers life easier, just like worrying about endpoints or architecture.
- vgt 11y agoInteresting is that much of this is externalized through Google Cloud in one way or another: - GFS/Colossus = GCS - Borg = Kubernetes - Dremel = BigQuery - BigTable = BigTable (naturally) - FlumeJava,MapReduce,MillWheel = Dataflow
- skj 11y agoKubernetes is not an externalized borg. They are completely separate products that do more-or-less similar things. Similarly, dataflow is completely new code that does similar things. As far as I know, GCS, bigquery, and bigtable actually are externalized offerings of their internal equivalents.
- vgt 11y agoRight, there are clearly levels of differences between Google-internal and external. Some more than others.
- exacube 11y agoGFS/Colossues are distributed file systems, which GCS doesn't really address (I think GCS is akin to AWS' S3?)
- ecnahc515 11y agoYep, but GCS is probably powered by GFS
- jrv 11y agoGFS was phased out many years ago - it has been replaced by Colossus.
- thrownaway2424 11y agoGFS is so dead few people even remember using it. But to say that something is "based on GFS" or "based on Colossus" is to say very little. If it stores data it is eventually "based on Colossus". You could say just as much if you said it was "based on ext4".
- mrry 11y agoIt's interesting to compare this with the Facebook stack, drawn by the same author: http://malteschwarzkopf.de/research/assets/facebook-stack.pdf http://malteschwarzkopf.de/research/assets/facebook-stack.pd... …and it would be doubly interesting to see the "related work" arrows between the different companies' infrastructure platforms. We'd need a 3D visualization for that though.
- aristus 11y agoI'd take this chart with a grain of salt, given the FB paper. Peregrine is no longer in service, IIRC. It was a friendly rival to Scuba, and was eventually replaced by Presto, which is not even mentioned. Also not mentioned are several important things like the pub/sub ecosystem built on Scribe. Haystack is long dead, except for maybe some archived stuff. Lastly, PHP-to-C++ has not been a thing since early 2012.
- ms705 11y agoOriginal author here. This is interesting -- I tried to do my best to find out what's still in use and what is deprecated based on public information; happy to amend if anything is incorrect. (If you have publicly accessible written sources saying so, that'd be ideal!) Note that owing to its origin (as part of my PhD thesis), this chart only mentions systems about which scientific, peer-reviewed papers have been published. That's why Scribe and Presto are missing; I couldn't find any papers about them. For Scribe, the Github repo I found explicitly says that it is no longer being maintained, although maybe it's still used internally. Re Haystack: I'm surprised it's deprecated -- the f4 paper (2014) heavily implies that it is still used for "hot" blobs. Re HipHop: ok, I wasn't sure if it was still current, since I had heard somewhere that it's been superseded. Couldn't find anything definite saying so, though. If you have a pointer, that'd be great. BTW, one reason I posted these on Twitter was the hope to get exactly this kind of fact-checking going, so I'm pleased that feedback is coming :-)
- kajecounterhack 11y agoRe Hiphop I think HHVM + Hack (Facebook's internal "improved PHP") has superseded it but while HHVM is open sourced Hack isn't public.
- amelius 11y agoGoogle has many services. This diagram means little if it does not specify what services use what parts of the stack.
- johannes1234321 11y agoAll services need to process and store large amounts of data. For that they need the building blocks firming the core stack which most projects (YouTube seems to be the big exception from what I hear) share. That is shown.
- thrownaway2424 11y agoAn interesting exercise in blind men describing the elephant.
- mrry 11y agoMalte Schwarzkopf is the first author of the Omega paper: http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/S... ...so not entirely blind.
- thrownaway2424 11y agoOne of the reasons Omega failed was because its authors went up in an ivory tower and ignored everything about what was actually happening in Google datacenters for years. That a diagram this full of WTF could be produced by one of the Omega authors does not surprise me.
- Xorlev 11y agoHow about Mesa?
- yukinon 11y agoIt looks like there might be some missing parts to this. I was under the impression Google has a layer of MariaDB databases somewhere.