13 ms·
Compute Engine machine types with up to 96 vCPUs and 624GB of memory
- djb_hackernews 9y agoInteresting that usually GCP is one upping AWS on some metric but in this case it isn't touching the current largest compute/memory instance on EC2, the X1 family with up to 128 vCPUs and 4TB of memory. Though the blog does allude to them testing such types in a closed beta, it still is a game of catch up still.
- RKearney 9y agoIt’s also unsurprising that this thread isn’t flooded with “disclosure: I work at google” posts. When GCP announces a feature that’s 1% better than AWS their employees flood the HN post, but when this story hits no one shows up.
- peatmoss 9y agoYikes, I'm a little out of the loop. I didn't realize you could get 4TB of RAM in a single machine on EC2. I've been seeing that medium data keeps getting bigger (i.e. the features of traditional RDBMS are eating away at the need for specialized / distributed stores for data analysis). But so too does it appear that small data is getting a lot bigger too—just load that dataset into memory for analysis. 4TB of memory allows for pretty big "small data." "I remember back when we used to do gradient descent to estimate linear models; back in the long ago when we didn't have 900 exabytes of memory attached to our NVidia Matrix Crusher 9000 linear algebra accelerator unit."
- sqldba 9y agoIs it possible that data isn't getting bigger - but that the people who work with it just want to process larger data sets than before? I mean before they'd train a model of 1,000 inputs and then test it against another 50 and call it a day. Now they want to train it against 1,000,000 inputs. Am I completely off base? It's not my area, though I work with databases, my observation is that developers always want to use the most data possible even when it doesn't really provide any benefit.
- adwf 9y agoThat is my experience recently. Developers storing 500GB on a database (pre-launch), with < 1GB of meaningful data. A bunch of json logs that they knew data science would want eventually, but couldn't be bothered to either pare down or put in a more sensible place. The thing is, it didn't really matter; Postgres still had a ton of performance left over even after the product went live. If you can still fit it in RAM, why waste $$$ of dev time over the $$ cost of a bigger instance.
- peatmoss 9y agoSorry, I was being a bit playful with language. What I mean is that, if you roughly define small, medium, and large data in terms of the strategies required to process, then the absolute size of the data that can be processed using simpler methods grows. And whether or not more data is needed or collectible varies by discipline. Astrophysics collects way more data than they used to because 1. they need it. 2. instrumentation allows it. Some kinds of data collection hasn't scaled up however. Surveying humans is expensive and labor intensive. And for many things that you might want to study about humans, you can't simply afix a sensor to them. So, what might have been only accomplished through big data, or medium data methods a few years ago can now be loaded into memory (i.e. small data strategies).
- samfisher83 9y agoWhat kind of computer is this? What motherboard supports this much memory?
- Cthulhu_ 9y agoIt's virtual, so it's up in the air. What motherboard supports 96 CPUs?
- mschuster91 9y agovCPU == cores. 2 Xeon Platinum 8180 get you 114 vCPUs.
- mschuster91 9y agoAnything that can fit e.g. the Xeon Platinum 8180M (https://ark.intel.com/products/120498/Intel-Xeon-Platinum-8180M-Processor-38_5M-Cache-2_50-GHz https://ark.intel.com/products/120498/Intel-Xeon-Platinum-81...), which supports up to 1.5TB (!) RAM - link two of these on one board and you get 3TB RAM support. As for the RAM itself - take 12 DDR4 128GB modules, for example https://www.heise.de/preisvergleich/crucial-lrdimm-kit-128gb-ct4k32g4lfq4213-a1137317.html https://www.heise.de/preisvergleich/crucial-lrdimm-kit-128gb... , and fit them into three channels of four modules each to get 1536 MB.
- dagw 9y ago24 slots x 64GB per slot is 1.5 TB. HP recently announced a workstation that supports up to 3 TB of RAM
- errantblaze 9y agoThe Super Micro SuperServer 8048B-TR4FT lists that it supports up to 12TB DDR4 ECC RAM (which could have 4xE7-8890v4 for 96 cores / 192 threads). And the 7088B-TR4FT lists that it supports up to 24TB DDR4 ETC RAM (with a corresponding 192 cores / 384 threads).
- thisoneforwork 9y agoAzure's M128 VM size has 128 vCPUs and 2048GB of memory already.
- packetslave 9y agosure, for $20/hour
- thisoneforwork 9y agoThis is not a SKU to play with. If $20/hr is indeed the price (I don't know), this is the hourly cost of a couple of waiters. You get to run SAP on someone's infra and someone to support it.
- mahdix 9y agoI wonder who will use these types of instances?
- malux85 9y agoscientific computing and simulation. I have had need for machines like this - for a project that was very compute heavy but embarrassingly parallel, but also was very "chatty" - I.e updated the data structures a lot during compute I can't be too specific about it, but if involved creating a very large tree structure and updating, pruning and transversing the tree a lot If the algorithm is updating and reading a large data structure a lot it's only practical from a speed point of view to hold the whole structure in RAM
- make3 9y agohow does the problem fair on gpus
- hyperbovine 9y agoIf it requires synchronizing across workers, poorly.
- mahdix 9y agoThat's interesting. What tools do you use to run your computations in parallel on a cluster? Hadoop?
- 9y ago
- deleted 9y ago[deleted]
- aneutron 9y agoI honestly wonder why did Google cite "SAP HANA" as the _reference_ workload for such setups. I never heard of that product. Noobie asking here, is it the _reference_ workload for such setups ? And are there more demanding workloads ?
- willyyr 9y agoI guess because SAP HANA is an in memory database which requires, well, a lot of memory. Other providers use HANA as a reference as well.
- codeduck 9y agoAWS do the same: https://aws.amazon.com/ec2/instance-types/x1/ https://aws.amazon.com/ec2/instance-types/x1/ Interestingly, AWS have X1 and X1E-class EC2 offerings, two of which are significantly larger than this new Google offering.
- Jdam 9y agoBut only Haswell, where GCP offers Skylake
- p0rkbelly 9y ago...but curious to see the performance gain on SAP (or similar) workloads with a new CPU architecture. RAM is the bottleneck typically.
- dom0 9y agoSo -80 % memory for 0-5 % more compute throughput?
- boulos 9y agoI think their latest instances were 4 socket Broadwell. Disclosure: I work on Google Cloud.
- Jdam 9y agoMaybe because SAP HANA requires vertical scaling. For horizontal scaling, I don't see a point in using such big instances.
- jijji 9y agoprice per second?
- baldeagle 9y agoThe dominant model was price per minute, rounded up. Price per second is better for groups that spin up and spin down instances often.
- roter 9y agoPricing is here [0]. The 96 vCPU, 360GB instance costs $4.9405/hr. If you don't mind if it gets killed, $1.0401/hr (pre-emptible [1]). [0] https://cloud.google.com/compute/pricing#machinetype https://cloud.google.com/compute/pricing#machinetype [1] https://cloud.google.com/compute/docs/instances/preemptible https://cloud.google.com/compute/docs/instances/preemptible
- claudius 9y agoDoes someone have experience using this (or other) cloud solutions in a scientific HPC context? The only obvious disadvantage I can think of is that every student running something incurs a certain cost whereas after one has bought the actual computers, running jobs is somewhat free.
- dagw 9y agoThe biggest downside (other than cost) I've found is that each vCPU core is quite a bit slower that what you'll get on equivalent real hardware. So any code that doesn't scale more or less linearly across an arbitrary number of cores will suffer.
- brutos 9y agoDepends heavily on the projected utilization. If you know your compute node is going to be computing for the next 3 years with at least medium utilization, then the self hosted metal is probably going to be quite a bit cheaper. Its amazing how much hardware you can pack into a single machine for 10k€. Last year our group bought two additional high-memory (768GB) nodes for around that price each (including support for a couple years from the vendor). A few years before we bought 40 nodes with 128GB RAM each, for a similar price to last years high-memory nodes (and a fast interconnect and a lot of storage). If you are at a larger research institution, you probably also have an IT department that can co-locate your hardware for next to nothing (compared to cloud). There you also will save a lot of ingress/egress, storage, backup, etc. costs. Regarding the per student costs, even with cloud instances I would consider running a traditional HPC job system (grid engine, lsf, torque, ...). The MIT had a nice solution with Starcluster [1] to easily deploy a SGE on AWS. It looks a bit dead now though. [1] http://star.mit.edu/cluster/ http://star.mit.edu/cluster/
- leifaffles 9y agoHoly moly!
- positivecomment 9y agoCould be useful if you wrote your neural net with Applescript, I guess.
- skj 9y agoI agree. Once we passed the 640k mark it all became an excuse for lazy tool development.
- tgtweak 9y agoOr if you use sap Hana, or Microsoft SQL server, or R, or Oracle, or anything that you pay "per instance" licensing on. There are many applications where a single addressable memory space are still required/preferred. Look into "tidalscale" if you want too see the logical extension for this.
- dx034 9y agoWhy would you use that instead of bare metal? You'll come out >$4k per month with sustainable use, for that price you get the same power in bare metal? Preemptible is a bit cheaper but are 600GB memory really worth it for short running applications? Until you loaded everything in memory your machine probably gets destroyed.. EDIT: Not sure about the exact CPU performance but should be quite close to what OVH offers here? With the same memory configuration and 2TB NVMe this still costs <$1,500/month (https://www.ovh.co.uk/dedicated_servers/hg/180bhg1.xml https://www.ovh.co.uk/dedicated_servers/hg/180bhg1.xml).
- DasIch 9y agoThere are quite a few problems where you need a lot of memory and CPU performance for just a relatively short amount of time like a few hours per day or even just a few hours per week. Forecasting or complex optimization problems for example. In these cases the amount of money you spend on hardware virtual or otherwise is negligible. Depending on what you do it might just as well be a rounding error.
- mpartel 9y agoThere's the high-cpu variant with only 86.4 GB of RAM
- dx034 9y agoBut that one costs 6.4x the 16 core variant for 6x the cores. Are there any applications where you're heavily dependent on having all cores in one machine?
- nl 9y agoYes. XGBoost will use all the cores you throw at it, and despite the recent work on GPU versions most of the time CPU cores are best. I'll commonly run a model for 48 hours on an i7. I'd love to be able to try more models.
- dagw 9y agoAnything where the workers have to synchronize their work with each other often. Having everybody talking to each other over the network quickly kills performance.
- mattparlane 9y agoCan you really spin one of these up on demand? That obviously means that they have machines of that size (or greater) sitting idle, waiting for someone to use them. That's mind-boggling.
- dx034 9y agoThey can still divide them up for the smaller sizes preemptible and destroy those once they get a request in. That way utilisation should be quite good. Yield probably not as high (since hardware costs will be higher than for multiple small machines) but the yield of renting out the highest variant sometimes will be worth it.
- misframer 9y agoGCP has live migration. Maybe they make space if it doesn't exist.
- ericfrederich 9y agoIdle, or running their own internal computations, or running lower preemptiable stuff.
- jsolson 9y agoPreemptible VMs exist in part to avoid leaving capacity sitting idle. We hate idle resources. :)
- kevin42 9y ago
- _joel 9y agoI couldn't see much info about how NUMA is handled
- cottonseed 9y agoDid they increase the per-instance network bandwidth caps? Previously instances had 2Gb/s/core of network bandwidth capped at 16Gb/s, which makes 8-core nodes the sweet spot for network bandwidth. I never quite understood why this is necessary if they're cutting up larger machines. Why should the total network cap matter if I have two 8-core instances or one 16-core instance on the same physical machine? edit: https://cloud.google.com/docs/compare/data-centers/networking https://cloud.google.com/docs/compare/data-centers/networkin... "The egress traffic from a given VM instance is subject to maximum network egress throughput caps. These caps are dependent on the number of cores that the VM instance has. Each core is subject to a 2 Gbps cap for peak performance. Each additional core increases the network cap, up to a theoretical maximum of 16 Gbps for each instance. The actual performance you experience will vary depending on your workload. All caps are meant as maximum possible performance, and not sustained performance."
- brianwawok 9y agoI doubt the average mysql server needs more than 16gb of bandwidth.
- sqldba 9y ago... let me show you the queries the developers are writing ^^
- brianwawok 9y agoSo you need to fix the developers not upgrade your network cards :)
- cottonseed 9y agoI don't even know how to reply to this. MySQL isn't the only cloud workload.
- tgtweak 9y agoPretty simple to understand why, they're "guaranteeing" that bandwidth. The hosts have a finite amount of network bandwidth available, they also have a finite amount of cores. Judging by the ratio it's likely something such as 2x 56gbe network ports on the host (=112Gbps), which has 56 vcpu (2x xeon 14c/28T)
- jackmott 9y agoIf a VM with 96 cpus is being offered, how many cpus does the bare metal behind the VM have? What is the hardware like here?
- iofiiiiiiiii 9y agoI imagine you get that 96 if you stick together 4x Xeon CPUs with 24 cores each onto one server. https://www.intel.com/content/www/us/en/products/processors/xeon/e7-processors/e7-8890-v4.html https://www.intel.com/content/www/us/en/products/processors/...
- alxv 9y agoYou only need 2 processors, since each cores gives you 2 vCPUs. "For the n1 series of machine types, a virtual CPU is implemented as a single hardware hyper-thread" --https://cloud.google.com/compute/docs/machine-types https://cloud.google.com/compute/docs/machine-types
- jackmott 9y agoso the bare metal might still have 4, such that it could host 2 of these VMs at once I guess.
- ewams 9y agoQuad socket 24 core would fit. Skylake has up to 28cores and 8 socket support.
- tgtweak 9y agoLikely quad socket 24 core with 768GB of memory to accommodate 2 regular instances or 1 memory + 1 compute instance.
- boulos 9y agoThere aren't quad socket Skylake yet :). You can also see that the largest public single package is 28 cores (56 threads). I'll sadly have to let you figure out how many because silly. Disclosure: I work on Google Cloud.
- qaq 9y agoCan we stop this whole vCpu nonsense
- qaq 9y agoDo people really enjoy not knowing what they are buying some providers provide some info on what vCPU is other don't. Many people think it's actual CPU core.
- amelius 9y agoWith that many vCPUs I guess it becomes important to talk about things like how the memory hierarchy is wired. I'm surprised no benchmarks are mentioned.
- em3rgent0rdr 9y agoIntel pages says these 28-core hyperthreaded processors work for 8+ sockets. Let's see...8 sockets * 56 virtual processors per socket = 448 virtual processors potentially in one VM. https://www.intel.com/content/www/us/en/products/processors/xeon/scalable/platinum-processors/platinum-8180.html https://www.intel.com/content/www/us/en/products/processors/...
- qhwudbebd 9y agoAnd still no IPv6. In 2017.