9 ms·
220,000 cores and counting: largest ever public cloud job
- t3soro 10y agoHow does any of this math get used? http://www.lmfdb.org/Genus2Curve/Q/11664/a/11664/1 http://www.lmfdb.org/Genus2Curve/Q/11664/a/11664/1
- 64738 10y agoI'd like to know, too. Not intending to sound irreverent, but does any of that math matter? (I'm not rude, I'm ignorant:)
- Game_Ender 10y agoThere is always the existential "Does anything really matter, everybody dies in the end?". This could have direct application, or as with many math concepts, in 50-100 years we could find an application for it. Despite all of that it doesn't have to matter, the process of exploration can have value in and of itself.
- onuralp 10y agoI think it has to do with public key cryptography - in particular, elliptic curve cryptography[0], which was mentioned by Schneider to be one of the key topics on NSA's agenda[1]. [0] https://arstechnica.com/security/2013/10/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/ https://arstechnica.com/security/2013/10/a-relatively-easy-t... [1] https://www.schneier.com/blog/archives/2015/10/why_is_the_nsa_.html https://www.schneier.com/blog/archives/2015/10/why_is_the_ns...
- mathgenius 10y ago"The LMFDB makes visible the connections predicted by the Langlands program. " The Langlands program is mathematicians version of "string theory". And what this guy is doing with google cloud is comparable to the LHC: searching for new particles (of mathematics) in order to find the deeper unity therein.
- strictnein 10y agoSadly no note on how much that cost.
- boulos 10y ago220,000 cores of preemptible was about $2200/hr (I believe Drew is using n1-highcpu-32s). Disclosure: I work on Google Cloud, launched Preemptible (and approved Drew's quota requests!)
- gaddferreira 10y agoThat's a really nice info to share, i was thinking almost an order of magnitude higher, thank you.
- Theodores 10y agoThe fuel bill per hour for the 737 going over my house right now is easily double $2200. I can visualise what is going on with a plane burning through fuel/money but with 220000 CPU cores it is all imaginary. How big a house would be needed for such an array of 'cores'? How much electricity is required, and again can that be compared to aeroplanes, Teslas or even toasters?
- pjc50 10y agoLet's say you're using 2x 10 core CPUs per 1U PC, in a 42U rack. That's 840 cores per rack, 262 racks. You won't fit that in a house, you need a moderately large datacenter. At 100W per CPU that's 22kW, which feels low; about a tenth of a Tesla and a fraction of a plane. That doesn't account for the cooling you'll need! (You can probably do a lot better, but that's my piano tuners estimate)
- brianwawok 10y agoI would be shocked if Google wasn't getting 2x 32 core CPUs per 1U at a minimum, which brings down your rack count a bit.
- 10y ago
- IncRnd 10y agoDo you always disclose customer information?
- strictnein 10y agoThere was an article written about him, which he obviously had to approve. How is disclosing what product he used "customer information"?
- lightbyte 10y agoNot the mention the article already says the machine he started with was these n1-highcpu-32s, so it's a pretty safe bet that he continued using it for this.
- IanCal 10y agoFor those that don't know google's list of instance types, this is the largest high-cpu instance that's not in beta.
- boulos 10y agoI should clarify: one of the reasons Drew stuck with those instead of larger (I had recommended customs with 48s at one point) is so that he can use the same config in all regions. We have some older zones still with Ivy Bridges and such that won't do >32. I fully expect that Drew will add a simple override map to say "use 64 threads here, 48 here, 32 here and so on" but with a goal to minimize preemptions.
- IanCal 10y agoNone of that information is private. The number of cores is in the article, the type of machine he used to use is in the article and the hourly cost of those machines is public knowledge. At worst the revelation here is that someone at google thinks the customer might be using what they've publicly said they used before. And that instance type is the largest high-cpu version available that's not in beta.
- throwaway2016a 10y agoCan someone explain Google Preamble Instances vs Amazon Spot Instances? The author says Sutherland was dissuaded by having to specify a spot price upfront on AWS but I don't see how that is any different than what Google is doing.
- kyrra 10y agoLooks like there are a number of differences: https://www.quora.com/What-are-the-key-differences-between-AWS-Spot-Instances-and-Googles-Preemptive-Instances https://www.quora.com/What-are-the-key-differences-between-A...
- user5994461 10y agoAWS Spot Instances are under bid. The highest bidder takes the instances, the price changes all the time. Google Spot Instances (preemptibles) are 80% off and that's it. It's simple.
- throwaway2016a 10y agoThat does sound pretty nice.
- jfroma 10y agoWe run specific components of our application in spot instances across different regions and from what I have seen in the last 3 months is that the price is generally much less than 80%. I'd say is less than 20% most of the time, but some times there is a peak and it goes beyond 150% for 1 to 3 hours, the frequency of these events may vary but I'd say once a week. I think AWS might be cheaper in this regard but less predictable, so it is a tradeoff. Edit: I haven't looked yet into "Spot Fleets".
- joshuamorton 10y agoNot 80%, 80% off.
- 10y ago
- chubot 10y agoWow I'd be interested to see the code for this. Is it just a tiny amount of C running on a huge number of machines, or is it a big stack of mathematical libraries? I guess since it's discrete math it probably doesn't use Fortran/BLAS?
- massel 10y agoPossibly a dumb question, but if it's this embarrassingly parallel, wouldn't this be a workload more suited for calculation on a GPU? I'm assuming there's a good reason he's not using one, so could someone who understands this a little better explain it?
- lightbyte 10y agoGPU's are significantly more expensive on GCE, could just be a cost thing.
- scott_s 10y ago> if it's this embarrassingly parallel, wouldn't this be a workload more suited for calculation on a GPU? Maybe, but one does not necessarily follow from the other. Consider the task of compiling 1 million separate C++ projects. That is obviously embarrassingly parallel, but not well suited for a GPU. It's trivial to do many compilations at once, but compiling itself is not easy to parallelize. That example is obviously contrived, but I think it demonstrates the principle that it's the computational profile of the core problem that will determine if you can use a GPU. If the core problem requires 10s of GB of RAM, or it's excessively branchy code, it may not be well suited for a GPU.
- milcron 10y agoA great paper which delves into different approaches for parallel computing is "Three layer cake for shared-memory programming" [0]. They characterize parallel programming into three broad strategies: 1. SIMD (parallel lines) 2. Fork-Join (a directed acyclic graph of operations) 3. Message-Passing (a graph of operations) GPUs are great at SIMD, but bad at the other sorts of parallelism. [0] https://www.researchgate.net/publication/228683178_Three_layer_cake_for_shared-memory_programming https://www.researchgate.net/publication/228683178_Three_lay...
- scott_s 10y agoThis is a good paper, but not quite how I think about it. I use the terms data-parallel (for SIMD), task-parallel (for fork-join; kinda) and message passing. GPUs are basically data-parallel machines, but over the years, GPUs have been getting more and more capable, so I imagine some people out there are using them for task-parallel workloads.
- alexnewman 10y agoI've done bigger on aws
- bjd2385 10y agoThis is really cool!
- plantain 10y agoSo that's why my preemption rates are so high lately.
- boulos 10y agoNope. First, preemptible VMs don't preempt each other. So if you got shot it wasn't Drew directly. However, when we're full and someone needs to get shot that can be you or it could be someone else. Drew being there actually makes it more likely that he would take the heat. But Drew runs all out well off peak (weekend mornings like our docs encourage!) so unless you had a bad weekend it wasn't him :). Disclosure: I work on Google Cloud (and have gone back and forth with Drew over email).
- AdamJacobMuller 10y agoDo you get refunded if you get preempted?
- boulos 10y agoYou do if you get shot in the first 10 minutes. Otherwise, no. We happen to preferentially "shoot the young" to minimize work lost: https://cloud.google.com/compute/docs/instances/preemptible#preemption_selection https://cloud.google.com/compute/docs/instances/preemptible#...
- tomschlick 10y agoGoogle cloud uses per minute billing IIRC so I'd assume you just don't pay any more after preempted.
- wkoszek 10y agoHow much? :)
- snowwrestler 10y agoImagine a Beowulf cluster of these.
- GistNoesis 10y agoWe recently developed Sienna (http://sienna.gistnoesis.net/ http://sienna.gistnoesis.net/) as an experiment for easter (which went rather unnoticed by HN) to distribute computation. It allows to ML researchers to build cluster over peerjs rapidly by asking their friends to visit a webpage, so they can share their unused resources. We are not yet to 220,000 cores but we can keep dreaming :)
- boulos 10y agoCool! If you wanted you could accept donations and run on Preemptible to supplement spare desktops ;). I do love the SETI@Home vibe though. Especially since nothing beats free.
- ramshanker 10y agoI guess, It must have heat up the environment little bit in the respective data centre. All servers in all racks running at full CPU utilisation out to be probabilistically rare.
- venture_lol 10y agoWow, and folks used to calculate the trajectories that put men on the moon with just pencil and paper :)