8 ms·
Google reveals details about its datacenters
- thrownaway2424 11y agoPreviously: https://news.ycombinator.com/item?id=9977414 https://news.ycombinator.com/item?id=9977414
- kijiki 11y agoShameless plug, but if you'd like to build a network like Google's, at any scale, let us know, we'd love to help: https://cumulusnetworks.com/ https://cumulusnetworks.com/ - nolan cofounder, cto cumulus networks
- stingraycharles 11y agoSince you add no value to the discussion, this is just spam -- imho pretty shameless indeed. Next time, at least share a story of something cool you have done, that would make your post much more appealing.
- epistasis 11y agoShameless plugs like this were at one time OK on HN. Also, it does add value to the discussion. At my company we've been using Cumulus Linux on white box switches to be able to build huge Clos networks at a fraction of the cost of traditional vendors like Juniper or Cisco. Such white box solutions are essential for building networks like this because traditional vendors are far too cost prohibitive. Also, if you still doubt the relevance of Cumulus, the top comment on HN's previous thread specifically mentioned Cumulus, and that comment was from someone involved in building the first three generations of network: https://news.ycombinator.com/item?id=9979035 https://news.ycombinator.com/item?id=9979035
- GauntletWizard 11y agoOne of the biggest things I've had to unlearn as an SRE leaving google is this: RPC traffic is not free, fast, and reliable (So long as you don't go cross-datacenter). For most companies it is expensive and slow. Facebook's networks are still designed for early-2000s era topologies and their newer topologies won't fix that; They've still got way too little top-of-rack bandwidth to the other racks nearby. Microsoft hasn't even caught on yet, and is still designing for bigger and bigger monolithic servers. I can't tell what Amazon is doing, but they seem to have the idea with ELBs at multiple layers.
- balakk 11y agoTry this link for what MS is doing : https://www.youtube.com/watch?v=RffHFIhg5Sc https://www.youtube.com/watch?v=RffHFIhg5Sc
- GauntletWizard 11y agoMicrosoft has ambitions, and then they hire bottom-feeders to mismanage it to hell. No, that's not at all what they're doing.
- richmarr 11y ago> ... then they hire bottom-feeders... I don't work for Microsoft, but I'm kinda disgusted by that comment. By all means criticise their work, but calling them derogatory names is deeply deeply uncool.
- GauntletWizard 11y agoI'm sorry, when I learn a non-derogatory phrase of criticism, I'll use it. Unfortunately, worthless hacks like yourself have have redefined every judgement as being offensive, so I'm left calling you a spineless fuckwad til you learn to tell the difference.
- DanBC 11y agoCome on. You know that's not welcome here.
- tux1968 11y ago> RPC traffic is expensive and slow, not free, fast, reliable If i've understood your point, you should have rather said this is what you've learned, not unlearned.
- GauntletWizard 11y ago
- ucaetano 11y agoI once saw the co-founder of Cloudera saying that Google exists in a time-warp 5-10 years in the future, and every now and then it gives the rest of us a glimpse of what the future looks like. Felt exaggerated at the time, but it often seems like the truth.
- achamayou 11y agoSome parts of Google definitely, but others not so much. If anything they were late on social networks and they still haven't really caught up.
- mtgx 11y agoI think he was referring to the technology Google is using to build products not the products per se.
- threeseed 11y agoI actually thought you were exaggerating but 5-10 years is actually very true in some areas. GFS only translated into HDFS after 5 years, same with MapReduce, BigTable etc.
- Xorlev 11y agoAnd still doesn't work nearly as well.
- east2west 11y agoI have always suspected so since Google still does not open source its MapReduce, but I haven't found any actual evidence directly comparing Hadoop MapReduce and Google's. There could also be technical issues preventing Google from open sourcing its implementation, although I don't know any. Now that we are moving on to more sophisticated models and implementation I am hoping more details can emerge.
- 11y ago
- mkj 11y agoSo every cluster machine has 40gbit ethernet (?) - does anywhere else do that? Looking at Table 2 http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p183.pdf http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p183....
- A010 11y agoPerhaps not related but I remember LHC's raw data rate is about PB/s and they use ASIC hardwares to filter it. https://i.imgur.com/4IaNr5a.png https://i.imgur.com/4IaNr5a.png
- peterwaller 11y agoThis is different to the sort of thing Google is doing in the data centre. Most of the PBs/sec don't really see the light of day since you may have a multi-megabyte "image" being captured at something like 40MHz, but practically much of the image is zeros for any given capture. So zero-suppression in the first instance already brings the data down to much more manageable numbers, before they hit off-the-shelf computing hardware. (At least, that was the case a few years ago. I don't know how much it has changed but I would be surprised if they had totally overhauled things).
- virtuallynathan 11y agoSome perhaps, but it is likely broken down into 4x10G. A 16x40G ASIC would get you 48x hosts and 4x40G uplinks.
- vonmoltke 11y agoIBM has had 4x DDR (20Gb) and QDR (40Gb) Infiniband as an option on their blades and frame nodes for about seven years. I used to work on a program that is using IBM blades with 4x DDR as an MPI processing cluster. The cluster was significantly smaller than what Google is discussing, though.
- matheweis 11y agoYes, this. Infiniband has scaled up significantly since then as well. I believe you can do 4x25 (100Gb) today. Having this background knowledge gives you a much better idea of why the interconnects is going to be such a big problem going forward.... When I was working with QDR IB 5 years ago, it didn't matter if they had faster cards - the PCI bus at the time couldn't support anything faster. So you were literally riding the edge of the I/O capability all the way through the stack.
- lorenzhs 11y agoThe key takeaway for algorithms research seems to be that "[w]e don’t know how to build big networks that deliver lots of bandwidth". This is exactly what S. Borkar argued in his IPDPS'13 keynote [1]. An exa-scale cluster can't be cost-efficient unless the bisection bandwidth is highly sublinearly in the cluster's computing power. We need new algorithms that - require communication volume and latency significantly sublinear in the local input size (ideally polylogarithmic) - don't depend on randomly distributed input data (most older work does) It's really too bad that many in the theoretical computer science community think that distributed algorithms were solved in the 90s. They weren't. [1] http://www.ipdps.org/ipdps2013/SBorkar_IPDPS_May_2013.pdf http://www.ipdps.org/ipdps2013/SBorkar_IPDPS_May_2013.pdf
- fauigerzigerk 11y agoIronically, if you look at the data center as a computer, this looks very much like scaling up, not scaling out. I wonder if one day we will find that sending all data to a data center for processing doesn't scale. I think that's already a given for some realtime'ish types of applications and it could become more important. Obviously, the success of decentralised computing depends a lot on the kinds of connected devices and whether or not data makes sense without combining it with data from other devices and users. With small mobile devices you always have battery issues. With cars, factory equipment or buildings, not so much. But management issues could still make everyone prefer centralisation.
- duaneb 11y ago> I wonder if one day we will find that sending all data to a data center for processing doesn't scale. Of course it scales. Some problems demand more scale. Through more datacenters at it. Tweak the algorithms to distribute data more conservatively.
- fauigerzigerk 11y agoWhat could happen is that both processing capacities and sensor capacities grow faster than networking capacities for years. That could make latencies untenable for large classes of applications.
- duaneb 11y agoLatency is bounded by the speed of energy transfer (i.e. speed of light under superconducting situations). Even if bandwidth IS a big problem—as it is—latency will always be a sticking point for distributed systems. As someone else pointed, we need to start analyzing distributed algorithms with a) geographical locality and b) complexity on round trips, which have a constant cost determined by latency.
- fauigerzigerk 11y agoSpeed of light determines the lower bound of roundtrip latency, but the sticking point is the upper bound, and that depends on how congested networks are. If sensors produce more data and data centers can process more data but network capacities can't keep up, then the analysis you are suggesting will show that we need more decentralised processing, not more data centers.
- kordless 11y ago> The amount of bandwidth that needs to be delivered to Google’s servers is outpacing Moore’s Law. Which means, roughly, that compute and storage continue to track with Moore's Law but bandwidth doesn't. I keep wondering if this isn't some sort of universal limitation on this reality that will force high decentralization.
- duaneb 11y ago> I keep wondering if this isn't some sort of universal limitation on this reality that will force high decentralization. To be fair, bandwidth has a high cap. Not all problems will need to be decentralized.
- kordless 11y ago> Not all problems will need to be decentralized. Our use of bandwidth is growing without bounds. That means most problems will need to be decentralized, which is akin to saying we're going to need to decentralize. :)
- duaneb 11y agoWell, I have full faith humans can adapt. We just haven't needed to, yet, except with e.g. slower memory latency reduction vs cpu speedup.
- Mettalknight 11y ago> The end of Moore’s Law means how programs are built is changing. I feel like this is just not true. Yes we have slowed down significantly over the past couple of years. BUT I do believe that once something big happens (My bet is the replacement of Silicon in chips or much stronger batteries). I feel like Moores law will pick up right where it left off.
- ska 11y agoBetting on the technological advances we haven't thought of yet to happen is pretty safe - but betting on them solving a particular current engineering challenge is a good way to fail.
- andrewstuart2 11y ago> The I/O gap is huge. Amin says it has to get solved, if it doesn’t then we’ll stop innovating. I can imagine you can solve the throughput problem with relative ease, but the speed of light limits latency at a fundamental level, so proximity will always win there. I tend to think that storage speed/density tech rather than networking is where the true innovations will eventually need to happen for datacenters. You can treat a datacenter as a computer, but you can't ignore the fact that light takes longer to travel from one end of a DC to another than it would from one end of a microchip.