11 ms·
EC2 Bare Metal Instances with Direct Access to Hardware
- empath75 9y agoSeems like this would be better for container farms, depending on cost.
- jopsen 9y agoincluding hypercontainer which uses KVM for hardware isolation with low overhead...
- icebraining 9y agoSeems like a good way to build your own version of Joyent Triton, running containers without VMs.
- nextzuckerberg 9y agoWhat differentiates these from dedicated boxes in server rack? Is their dedicated "cloud" hardware somehow managing access to RAM/storage/etc? On another tangent - how do Google Cloud and EC2 attach GPUs to instances - given that you can choose CPU and RAM the GPUs must somehow be modularized away from a dedicated server?
- a012 9y agoNetwork and other managed services those are integrated into AWS.
- ianai 9y agoSo you could magic up a virtual machine that spans more hardware than would typically be in a single, enclosed chassis?
- wmf 9y agoNo, that isn't possible for VMs or bare metal.
- IanRobertsonGMX 9y agoActually, it is possible, but not on Amazon's cloud. Check out www.scalemp.com - aggregating a collection of nodes into a large, single VM.
- aliguori 9y agoYou can provision these servers just like any other instance. They work just like any other Amazon EC2 instance (same Nitro System platform as C5). Disclaimer: I work at AWS on the team responsible for the Nitro System including EC2 Bare Metal Instances.
- rrix2 9y agois there any information about nitro or ENA (assuming this is the "hardware accelerators" that are mentioned in tfa) that is publicly available? it seems like the most nifty little thing
- aliguori 9y agoThere is more coming at re:Invent. We have more talks queued up tomorrow on this too.
- rrix2 9y agoI'll keep an eye out, thanks
- uji 9y agohow many NICs can you attach to these?
- deleted 9y ago[deleted]
- _msw_ 9y ago15 Elastic Network Interfaces (ENIs) can be attached, just like i3.16xlarge.
- _msw_ 9y agoHere are videos from break out sessions yesterday at re:Invent https://www.youtube.com/watch?v=LabltEXk0VQ https://www.youtube.com/watch?v=LabltEXk0VQ https://www.youtube.com/watch?v=o9_4uGvbvnk https://www.youtube.com/watch?v=o9_4uGvbvnk
- comstock 9y agoI assume you can spin up new instances easily, and they’ll have per-hour billing. You don’t usually get that with dedicated servers. It sounds a bit like MAAS [1], which allows you to throw images onto, and manage real servers easily, very much like you might spin up VMs on AWS. [1] https://maas.io https://maas.io
- _msw_ 9y agoThese are billed the same as any other EC2 instance: per second.
- dummyfunnytoo 9y agoJust to point out, per second billing is incredibly recent for EC2: https://aws.amazon.com/about-aws/whats-new/2017/10/announcing-amazon-ec2-per-second-billing/ https://aws.amazon.com/about-aws/whats-new/2017/10/announcin...
- deleted 9y ago[deleted]
- rrix2 9y agoThey can exist in a VPC alongside virtualized resources, map EBS volumes (due to having the ENA onboard), are integrated with cloudwatch, ebs, etc
- brianwawok 9y ago> how do Google Cloud and EC2 attach GPUs to instances - given that you can choose CPU and RAM the GPUs must somehow be modularized away from a dedicated server? Rack A of servers has a base_server_x. Rack B of servers is base_server_x + GPU_Y. You ask for no GPU, you get a server from rack A. You ask for a GPU, you get a server from rack B. No magic monkey adding GPUs to instances ;)
- blasdel 9y agoOh there's magic monkeys too!: https://aws.amazon.com/ec2/elastic-gpus/ https://aws.amazon.com/ec2/elastic-gpus/
- a012 9y agoSo now you can rent a dedicated server on AWS, what is nice.
- thesandlord 9y agoAWS already had dedicated instances, but they still had a VM running on top. These are bare metal, which means you run directly on the hardware.
- CSDude 9y agoImpressive hardware but I wonder what will be the cost considering even the regular VMs of EC2 are generally more expensive than dedicated offerings of other providers.
- Johnny555 9y agoIt has similar specs as the i3.16xlarge, so will probably be priced similarly ($5/hour). i3.16xlarge: 64 vCPU 488GB 8 x 1900 NVMe SSD i3.metal: 72 hyperthreads 512GB 15 TB disk I wonder if this is the hardware for the host of the i3 series.
- shaklee3 9y agoIt's still the last generation of Xeons, but it's a beast.
- Radim 9y agoHead-to-head cost comparison of Amazon's AWS, IBM's Softlayer, Hetzner and Google's ComputeEngine on a machine learning benchmark: https://rare-technologies.com/machine-learning-hardware-benchmarks/ https://rare-technologies.com/machine-learning-hardware-benc...
- tyingq 9y agoInteresting, but the really big difference comes when you need to push data out to end users...network egress charges.
- Radim 9y agoTrue! Not with machine learning though. The target of this benchmark is hardcore number-crunching.
- Cacti 9y agoBut... that's not using GPUs? What use is a ML benchmark that is only measuring CPU usage?
- FBISurveillance 9y agoThis is really good news, happy to see this is an option now. And thanks for posting this here personally @jeffbarr.
- jeffbarr 9y agoAny time!
- oellegaard 9y agoI wonder what it will cost - as of now I don’t see it in the price list
- _msw_ 9y agoPricing will come with general availability. I suspect (and hope) everyone will be surprised and happy with the price.
- will_hughes 9y ago> Storage – 15.2 terabytes of local, SSD-based NVMe storage. That's probably the most interesting aspect for me. Does anyone know how that's provisioned? i.e 8x just under 2TB volumes, or something else?
- aliguori 9y agoIt's exactly the same as with the i3.16xlarge instance type. There are eight 1900 GB drives. In an i3.16xlarge, those eight drives are passed through to the instance with PCIe passthrough but for the i3.metal instance, you avoid going through a hypervisor and IOMMU and have direct access.
- will_hughes 9y agoThanks. I guess some other open questions: - If one of those drives fails, will Amazon hotswap them out, or do you need to migrate to a new instance (moving TBs of data to a new box without causing outages can be painful.) - Is there a hardware RAID controller for those drives, or is it software only? - Can anyone with access to one of these boxes produce some IO performance stats on them? Bonus points for stats on single drive vs concurrent across all drives (i.e is there any throttling). More points for RAID10 performance across the whole 8.
- _msw_ 9y agoThe local NVMe storage for i3.metal is the same as i3.16xlarge. There are 8 NVMe PCI devices. For i3.16xlarge those PCI devices are assigned to the instance running under the Xen hypervisor. When running i3.metal, there simply isn't a hypervisor and the PCI devices are accessed directly. - There is no hot swap for the NVMe storage. - The 8 NVMe devices are discrete, there is no hardware RAID controller - Anyone can get I/O performance stats on i3.16xlarge as a baseline. Intel VT-d can introduce some overhead from the handling (and caching) of DMA remapping requests in the IOMMU and interrupt delivery so I/O performance may be a bit higher on i3.metal, with a few microseconds lower latency.
- KenoFischer 9y agoI'm really, really, happy about this. I've been complaining about the lack of cloud servers with exposed performance counters to any cloud vendor that'll listen (though of course nothing ever came of that). Kudos AWS, this is really cool.
- aliguori 9y agoThanks! Would love to hear more about the counters that your interested in. We've exposed more in C5 than in previous instance types and we are trying to make more available over time in a safe way.
- KenoFischer 9y agoI have two use cases: - General performance analysis. For this more counters is generally incrementally better. - Running https://github.com/mozilla/rr https://github.com/mozilla/rr. This requires the retired-branch-counter to be available (and accurate - sometimes virtualization messes that up) The second one I actually care more about, because I've pretty much stopped trying to debug software when rr is not available, too painful ;). Feel free to email me (email is in my profile) for gory details.
- khuey 9y agoFor the benefit of anyone reading this, KVM and VMWare virtualization generally work. Xen has problems because of a stupid Xen workaround for a stupid Intel hardware bug from a decade ago. I can provide more details about that via email (in my profile) if desired.
- paulie_a 9y agoCan you please just post the info. Intel deserves to be shamed
- voidmain0001 9y agoIs this what khuey is referring to?: https://support.citrix.com/article/CTX136003 https://support.citrix.com/article/CTX136003
- LogicX 9y agoPacket.net, arguably the leader in api driven, on demand bare metal instances, recently blogged about this: https://www.packet.net/blog/why-we-cant-wait-for-aws-to-announce-its-bare-metal-product/ https://www.packet.net/blog/why-we-cant-wait-for-aws-to-anno... I am a customer of packet's, along with other virtual and dedicated hosting providers. I don't use aws ec2. I've been pleased with Packet, and their offerings are much more diverse than this initial offering from aws.
- dboreham 9y agoAlso a happy Packet customer. We use their small instances for things like service monitoring (where VM pauses cause false positives) and for routing infrastructure where bare metal is required to achieve VoIP-acceptable jitter. They’re also one of the few cloud hosting providers to support BGP.
- larrymcp 9y agoI just now took a look at Packet's web site and their data center locations. They categorize each location as either "core" or "edge", but I couldn't find anything to indicate what those terms mean in this context. Are you familiar with that distinction? The location nearest me is an "edge", not a "core". I wonder what I would be missing out on, if it's not "core".
- mcrae 9y agoEdge DCs only have one type of machine (1E) and no block storage, but I think are otherwise the same. Even in core DCs though the availability of different types of machines varies. Love packet.net btw -- the bgp stuff is really game changing.
- dantiberian 9y agomcrae covered this well, but to add to that, the Edge servers are targeted for low latency services close to users, think self driving cars, IOT, adtech, etc. You can see more details at https://www.packet.net/edge/ https://www.packet.net/edge/ and https://www.packet.net/blog/looking-over-the-edge/ https://www.packet.net/blog/looking-over-the-edge/.
- andy_ppp 9y agoFor all this progress the billing on AWS is so damn confusing to figure out if some machine is left on unused that I won’t use AWS again. GCE and Azure miles ahead here.
- _wmd 9y agoIt takes all of 3 clicks to figure this out using cost explorer
- andy_ppp 9y agoWhy did AWS support send me a 5329 word message on how to check every region and service then?
- _wmd 9y agoMaybe spend a few more minutes clicking through the apps billing section before firing off a support request next time :)
- syncsynchalt 9y agoCost Explorer takes [up to] 24 hours to set up, so it's not a good answer to support questions about billing.
- artursapek 9y agoAWS is incredibly complex. Are you complaining that their billing can get complex?
- SteveNuts 9y agoThis will expose virtualization? As in I can run my own virtualization stack on these instances (KVM, etc)?
- _msw_ 9y agoEC2 Bare Metal instances provide all the typical Intel processor features, including VT-x and VT-d. Yes you can use KVM.
- witten 9y agoAccounting question: Might this qualify as a capital expense? If you squint hard enough? For context, AWS is coy (at least publicly) about the existing dedicated instances and CapEx vs. OpEx.
- _msw_ 9y agoThe information found here should help your finance team or accountants determine how best to classify your expenses: https://aws.amazon.com/ec2/dedicated-hosts/faqs/#Should_I_Consider_a_Dedicated_Host_Reservation_a_Capital_or_Operational_Expense https://aws.amazon.com/ec2/dedicated-hosts/faqs/#Should_I_Co... Since EC2 Bare Metal instances will use the same pricing models as all other EC2 instances (on demand, reserved instances, dedicated host, spot), the same information is relevant.
- KaiserPro 9y agoFor the UK its always opex. As you never own the instance, you are hiring it as a service.
- k__ 9y agoOT: Job advice needed (because I think many back-end devs will be here :D) I'm thinking about going full-stack next year. I have a bit of experience building APIs besides being mainly a front-end developer. Is going "cloud only" a good idea? I thought about starting with AWS Lambda, S3, DynamoDB and the Serverless framework. Are the providers hugely different or is it a good idea to spread out and do some Azure and GCP too?
- dotancohen 9y agoThat's completely off topic. In fact, the question is so broad that I cannot think of anyplace other than the water cooler or Quora to ask it. Career advice: Never go "foobar-only". Make an effort to learn "foobar" but understand whatever is one layer below it in the stack. Want to go "cloud-only"? Learn OpenCloud, not AWS.
- k__ 9y agolol, while complaining you still gave me a decent answer, thanks :)
- dotancohen 9y agoOf course, we're all here to help each other! Good luck!
- gtsteve 9y agoIt's definitely worthwhile to learn Lambda, S3 and serverless apps but all that stuff can be learned on the job. S3 is especially easy to use for most use-cases and any decent programmer can learn to use it in an hour or two. However, I would definitely learn a SQL dialect and learn how RDBMSes such as Postgres work (especially what is meant by ACID) as most companies are based around a database. Don't believe the hype - SQL is not dead. Dynamo is a great technology but there are many problems it can't solve for you. Finally, I personally don't know Azure or GCP so well. Only knowing AWS in-depth hasn't held me back so far. I've used a few of Azure's services but I've never built a serious app on it. My recommendation is to not really worry about individual technologies and to focus on safely handling and working with data.
- cperciva 9y agoI'm looking forward to testing out FreeBSD on these... and also bhyve, for a fully BSD virtualization stack.
- nickpsecurity 9y agoWith them leaning bare-metal and low cost, I wonder if services like these could be used to bootstrap clouds in VAR form for niche OS's. Might be useful at the least for getting bugs out of the virtualization software using diverse workloads. If costs kept minimal, might even be profitable if the niche OS has enough users.
- brutopia 9y agoThanks Jeff!
- azinman2 9y agoSo if it’s truly bare, how does Amazon give and take control of the machine for provisioning? Don’t they still need done kind of hypervisor?
- IshKebab 9y agoMaybe they use Intel's Management Engine!
- RKearney 9y agoThe E5-2686 v4 doesn’t have IME.
- userbinator 9y agoThe CPU itself doesn't, but the chipset does. Amazon is a big enough customer that it wouldn't surprise me if they could get Intel to make special ME firmware with the features they want in it. ...which also means the ME exploits discussed recently here could lead to a whole lot more fun... ;-)
- KaiserPro 9y agoMost servers have some sort of "lights out" management, which gives KVM + remote imaging and bios control. With amazon, they have complete control over the network in and out, so cutting you off and re-imaging a server is pretty trivial. To be fair, its not that hard to do even if you're not amazon. Most of the big server vendor's out of band interfaces have an API, so telling a server to reboot from a network image is pretty trivial. Providing a netboot infrastructure to install images with a 'userdata' script is also not that difficult. you'll need a DHCP server, tftp to serve the boot image, and usuaally an NFS server to pull the rest of the image over. With some engineering work that could be made to use HTTP. https://wiki.centos.org/HowTos/NetworkInstallServer https://wiki.centos.org/HowTos/NetworkInstallServer
- detaro 9y agoIt's a bit harder if you host something like this for the general public to use (vs administrating machines in your private DC). Normal setups aren't really hardened against someone flashing firmware, messing with UEFI, ..., all of which mean you can't entirely trust a machine coming back from customer control. I wouldn't be surprised if Amazon took this seriously and invested effort in stopping such things. At their scale, they probably can customize the hardware enough.
- sdfjkl 9y agoInteresting. I expect we'll be seeing a lot more VPS providers running on AWS with these instance types.
- GrumpyNl 9y agoIsnt this just regular web hosting?
- dboreham 9y agoNot quite: this is cloud-provisioned so you can do things like supply your own image and it integrates with all the other AWS services like virtual machines do. Provisioning is automated and self-serve. Also per-second billing which you couldn't get in the olden days with hosting.
- xena 9y agoWhat is the price? I can't find it.
- syncsynchalt 9y agoFrom _msw_ in this thread: "Pricing will come with general availability. I suspect (and hope) everyone will be surprised and happy with the price."
- zsmith928 9y agoawesome stuff! Great to see AWS pushing things around the baremetal problem set.
- titzer 9y agoI think Amazon is exposing themselves to far greater security risks than they realize.
- aeorgnoieang 9y agoLike what?
- titzer 9y agoBlackhats, state actors, etc all trying to attack Amazon or colocated services. As an example (I don't know the extent of "bare metal" access, so I couldn't be sure) with the ability to run their own operating system, a client could potentially get all the way down to the NIC to form arbitrary network packets. With this they could potentially map and attack Amazon's internal network protocols (routers, etc). Any kind of vulnerability within Amazon's software stack on other servers now gets a whole lot worse. If the client did this at a very low rate, it would be difficult to detect. Firewalling off these servers only helps so much, since they could still attack colocated servers of other clients, or could potentially spoof the protocol of Amazon's own server management. I hope they have thought this through carefully, because it potentially exposes everyone on EC2 to more, potentially worse, attacks.
- sseveran 9y agoYou have the whole machine. There are no other colocated VMs.
- mugsie 9y agothat is the issue. You have full access to the machine, so you can update firmware / tinker with BIOS etc. Then let the machine go back into the pool, and wait for it dial home. There is some mitigation, but this is a major reason a lot of vendors do not do per second / minute bare metal.
- _msw_ 9y ago
- hartator 9y agoWhy fancy words when they are just offering regular dedicated servers?
- anonu 9y agoThese were my exact same thoughts. I suppose its almost like a step back from the framework of "virtualize everything"... what's old is new.. addon thoughts: nonetheless, the specs on the bare metal box are ridiculous. buying something like that will cost you $50k (someone correct me?) - then you need to find a place to host it... thats not easy to do.
- rrix2 9y agoBecause they're still virtualizing literally everything but the actual computer. You can attach NVMe backed EBS volumes, snapshot them as normal, etc. You can have this thing exist in a vpc next to your virtualized components, with 25gbps dedicated link. They're virtualizing the things you shouldn't need to care about, leaving you with a free Cpu and access to all the things that make aws aws
- wmf 9y agoRegular dedicated servers don't have VPC, EBS, etc.
- zrail 9y agoWill there be smaller instances available eventually? I'm interested in bare metal performance but I don't need an instance that huge for my current workload.
- _msw_ 9y agoOur goal is to for the majority of virtualized EC2 instances to be indistinguishable from bare metal (if not better). In most CPU and memory intensive benchmarks there is very little difference between an virtualized EC2 instance and bare metal, especially for smaller numbers of cores and memory sizes.