3 ms·
Evolution of GitHub's data centers
- rdl 9y agoThis is pretty generic -- isn't that how virtually any decent footprint site with physical infrastructure operates (at least from ~2000 onward)? I'd be more interested in blog posts about either what they were doing before which was different/worse and got upgraded, or how they decided on specific things. Their in-house tool gPanel is pretty cool, though (https://githubengineering.com/githubs-metal-cloud/ https://githubengineering.com/githubs-metal-cloud/) -- I've seen a lot of people build their own of that in ways which are far less sophisticated, or used one of the ~10 other tools for it. How much effort an org puts into making sure firmware is all the same is probably a good indicator of how competent they are, IMO.
- gcvbvghb 9y agoDo you get a kick out of shitting on somebody else’s work? Idiot.
- jameskegel 9y agoI was let down that this wasn’t more of a visual experience.
- ramshanker 9y agoAgreed. Their past post with huge lot of pretty pictures have been awesome.
- svdr 9y agoHere are 256 photos from Stackoverflow hardware: https://m.imgur.com/r/cableporn/X1HoY https://m.imgur.com/r/cableporn/X1HoY
- sandGorgon 9y ago>In memtesting we boot a custom MemTest86 image and monitor it while it completes a full pass. Our custom version of MemTest86 changes the color of the failure message to red which allows us to detect trouble. We’ve hacked together a Ruby script that retrieves a console screenshot via IPMI and checks the color in the image to determine if we’ve hit a failure or not. Wow. Why is this done this way? Is there no other way to receive hardware memtest info? I would have thought that there was some way to revive machine serial output status in a datacenter.
- hueving 9y agoYeah, this seems pretty terrible compared to a modified memtrest that reports results.
- dom0 9y agoOver network? Needs a networking stack in memtest. Over serial? Physical serial is rare. SOL would work, but since it is connection-based you can't poll it. This seems like a simple and reliable method to me.
- sschueller 9y agoSince they take a screenshot of the console they could output a qr code and read that.
- fapjacks 9y agoThat's sort of what they're doing. It's just a two-bit QR code: Red, and Not Red.
- gumoro 9y agoSo, one bit ;)
- fapjacks 9y agoHah! Yes, totally. Haha I wonder why I said that?
- BraveNewCurency 9y agoThere is also a "not decided yet" state. Hence, you were right, it needs two bits. :)
- dom0 9y ago
- gm-conspiracy 9y agoAre they using Dell equipment?
- rdl 9y agoBased on reference to iDRAC, certainly seems that way. Presumably they 1) don't have a huge amount of hardware (in the sense a commodity hosting provider or cloud provider would) and 2) get a decent deal from Dell.
- rcarmo 9y agoTwo things that stuck out at me: - No mention of an European data center (data sovereingty and regulation might be something they don't care about, but I deal with it every day). - No mention whatsoever of virtualization (although there was an older post mentioning a move to Kubernetes for a part of their stack). I wonder what the economics of taking this to a public cloud provider would look like (full disclosure - I work on Azure), and how much they could do architecturally to benefit from that.
- SOLAR_FIELDS 9y agoAt a scale of Github’s it’s highly unlikely that this would be a cost effective maneuver - see Gitlab’s effort to move off of cloud providers as an example. There comes a point where you get so big that it no longer makes sense to use third party hosting. Dropbox is another example of this.
- manigandham 9y agoGitlab isn’t moving: https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-the-cloud/ https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-t... Dropbox was primarily about moving storage off S3 which makes sense given their business.
- lwf 9y agoDropbox has run compute* in its own datacenters since 2008 or so. *: That is, work that does not interact heavily with content
- codersbrew 9y agoWe moved from bare metal and AWS combo to a full dockerized DC/OS deploy and love life now.
- toast0 9y agoWhen you're using full racks of hardware for basically one product, what does a virtualization layer add? Virtualization is useful for doing many small tasks on a single machine, especially if those tasks are owned by different companies (and you are ok with the tenant separation you get), but if you're using full machines it adds overhead.