8 ms·
EC2 Instance Update – C5 Instances with Local NVMe Storage
- cthalupa 8y agoIt looks like i3.metal is also available - seeing them in the console
- jchw 8y agoThis is probably a big deal, if it's anything like the local SSD storage on GCE. The performance of local SSDs on Google Cloud is nearing absurd compared to anything else you can find right now. That being said, I think one of the less compelling parts of this is that it'll probably vary per instance type quite a bit, being limited to C5 to start, so if you have a workload needing way better disk I/O than CPU performance, you might have to waste. That's one thing GCP really does have on AWS, better granularity.
- runesoerensen 8y ago> being limited to C5 to start I3 and F1 instances also have NVMe SSD instance store volumes https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html#nvme-ssd-volumes https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-inst... Here's the full list of instances with instance storage: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Instance...
- jchw 8y agoNice, didn't know that.
- foobarbazetc 8y agoThe sequential write performance of local NVME on GCP is... not great.
- jchw 8y agoWould love some benchmarks, if you happen to have them. Frankly, I don't have any good benchmarks directly comparing GCP and AWS, but my current heuristics is that GCP "seems" to have better I/O and AWS "seems" to have higher frequency CPUs. Could be way off.
- jread 8y agoWith more instance types and storage classes, EC2 offers a much wider spectrum of performance capabilities for CPU, storage (ephemeral and durable), and networking.
- jchw 8y agoI don't see how this has to do with my question about benchmarks of sequential disk I/O on GCP instances with local NVMe SSDs. Did you respond to the wrong message? Either way, I use both GCP and AWS but I am finding that I'm spending less on GCP. I don't believe every workload will be cheaper in GCP, but it is clearly competitive at least.
- jread 8y agoI'm responding to your comment "GCE seems to have better I/O and AWS seems to have higher frequency CPUs" - this isn't true. You can get faster I/O, networking and CPU with EC2 if you select the correct instance/storage and are willing to pay for it.
- jread 8y agoFor seq read/write I've observed up to around 15000/2900 MB/s on i3.16xl and 2750/1400 MB/s on GCE w/4xlocal SSD
- jchw 8y agoThis is lacking in details: - What OS image(s) were used? This would aid in reproducibility. - What tool was used for benchmarking? Preferably with the configuration so I could reproduce it. - What GCE instance is used? I assume you will bottleneck on something other than I/O if comparing a very small GCE instance to a very large AWS instance. I will probably benchmark this on my own at some point now that my interest is piqued, but if you have already done the work to benchmark this I'd love to hear more. So far though it's too vague for me to use meaningfully. Is there a blog post or other material that is publicly accessible? I've Googled for such before but come up surprisingly empty handed.
- jread 8y agoGCE local SSD was the fastest thing around when first released in 2014, but hasn't improved and has since fallen behind. GCE local SSD 4k read IOPs max out at around 200k/volume and 800k/instance w/4 volumes. I haven't measured c5 local nvme yet, but on i3, local nvme volumes run at 100-400k/volume and up to 3.3 million IOPs on the largest instance type (i3.16xl).
- jchw 8y agoHey, you seem to know a lot about this. Can you explain why the docs seem to claim they max out at 680k/volume read IOPS and 360k/volume write IOPS? They seem to be talking about 4k blocks. https://cloud.google.com/compute/docs/disks/performance#type_comparison https://cloud.google.com/compute/docs/disks/performance#type... Is Google being disingenuous? I haven't benchmarked it, but I will say that anecdotally the GCP local NVMe SSDs feel very fast for things like building software compared to other setups I've done on other clouds (AWS, UpCloud.)
- Manozco 8y agoI'm not seeing the usage here. What would need a (very very) performant temporary storage that you could not achieve with io volumes ?
- plasma 8y agoNetwork disks really are terrible in comparison, even the super charged expensive ones (io volumes). Local SSD is just orders of magnitude faster if you need a fast place to put temporary (or replicated) data.
- pishpash 8y agoIt was a harebrained idea to not distinguish between local vs remote data needs. For a time Amazon pretended they could overcome physics.
- cobookman 8y agoAerospike
- Johnny555 8y agoio1 volumes are expensive and not nearly as performant as local nvme SSD's. io1's top out at 32K PIOPS per volume and a 32K PIOPS 225GB volume would cost around 2100/month. But an entire c5d.2xlarge with 225GB of local SSD will only cost around $283/month, and (based on i3 performance) will give you around 180K write IOPS.
- deleted 8y ago[deleted]
- Scarbutt 8y agocaches
- falsedan 8y agodocker layer caches jenkins workspaces
- soccerdave 8y agoThis is really exciting to see! I had assumed that they would be launching most new instance types with only EBS storage so this is awesome that it looks like this will be coming to even more instance types too. The bottom of the article mentions "PS – We will be adding local NVMe storage to other EC2 instance types in the months to come, so stay tuned!"
- plasma 8y agoWe use i3 instances for some workloads which are great, I just wish the CPU was as powerful as other instance types.
- tbrock 8y agoAmazing! I hope they adopt this for RDS. EBS volumes are great and all but not for database where the dataset is many multiples of the working set.
- deleted 8y ago[deleted]
- pjungwir 8y agoI've been intrigued by the idea of running databases on EC2 instance storage for a long time. (You couldn't use RDS though, at least not today.) Putting your db on something also called "ephemeral storage" seems risky, but maybe not much riskier than putting it on plain HDDs. The big issue to me is that most instances don't come with much space. If you need more you have to scale up the whole instance (not a separate dimension like with EBS), and if you're already on the biggest instance type you're just out of luck. I guess it could be worthwhile to use separate tablespaces so you could have some data on instance storage and some on EBS. But so far I've gotten acceptable perf by RAIDing over gp2 EBS volumes (12 TB in my case), following the approach here: https://news.ycombinator.com/item?id=13842044 https://news.ycombinator.com/item?id=13842044
- jedberg 8y agoAll of Reddit was Postgres on raided EBS up till I left in 2011 and I think still is today but I kinda hope not. It’s totally safe to use local storage if you build it right. But those raided EBSs caused a lot of problems. In short, when one gets slow the whole volume gets slow because software raid isn’t hardware raid. The main advantage of RDS is that they take care of the mundane redundancy for you.
- pwarner 8y agoNow they just need R5 instances for our RAM hungry apps. Keep the local nvme please.
- dis-sys 8y agoGreat news! I think Amazon should provide more specs on those NVME drivers. what kind of R/W latency/max bandwidth/low and high QD throughput I can expect?
- Elect2 8y agoIf I remember correct, all old ec2 instance support local storage. Then as the growing of EBS, they disabled local storage. Now it's back again..
- dwyerm 8y agoIt was less that local storage was disabled, and more that the newer hardware types didn't have any local storage to share.
- pishpash 8y agoSo back to instance storage we go?
- bostik 8y agoDepending on which kernel version you are, C5 (and M5) instances can be real sources of pain. The disk is exposed as a /dev/nvme* block device, and as such I/O goes through a separate driver. The earlier versions of the driver had a hard limit of 255 seconds before I/O operation times out. [0,1,2] When the timeout triggers, it is treated as a hard failure and the filesystem gets remounted read-only. Meaning: if you have anything that writes intensively to an attached volume, C5/M5 instances are dangerous. We experimented with them for our early prometheus nodes. Not a good idea. Having the alerts for an entire fleet start flapping due to a seemingly nonsensical "out of disk, write error" monitoring node failure is not fun.[ß] If you run stateless, in-memory only applications on them (preferably even without local logging), then you should be fine. 0: https://bugs.launchpad.net/ubuntu/bionic/+source/linux/+bug/1758466 https://bugs.launchpad.net/ubuntu/bionic/+source/linux/+bug/... 1: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729119 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729119 2: https://www.reddit.com/r/aws/comments/7s5gui/c5_instances_nvme_storage_stucks/ https://www.reddit.com/r/aws/comments/7s5gui/c5_instances_nv... ß: We handle nodes dying. The byzantine failure mode of nodes suddenly spewing wrong data is harder to deal with.
- otterley 8y agoInteresting - I’ve been using NVMe devices on Linux for a couple years now and never run into this problem. And an I/O timeout of 255 seconds seems really high to begin with. Is there frequently that much latency in the EBS storage backplane? (We also run c5.9xl instances and have not yet experienced the phenomenon you discuss.) In any event, the instance storage is unlikely to run afoul of any such timeouts since it’s more or less directly attached (albeit virtualized) and there’s no SAN involved.
- chillydawg 8y agoThe nvme disks are local, not remote EBS. Latency will be the PCI bus.
- cthalupa 8y agoEBS is exposed as nvme devices on c5 and m5 as well, which is what I assume otterley is talking about. Presumably the same applies to i3.metal as well, though I have not yet personally checked.
- debunn 8y agoI noticed the following comment in the article: > Encryption – Each local NVMe device is hardware encrypted using the XTS-AES-256 block cipher and a unique key. Each key is destroyed when the instance is stopped or terminated. Does anyone know if the existing i3 EC2 instances NVMe drives are also encrypted in this fashion? I can't find any articles stating this... Thanks!
- iconara 8y agoi3 and f1 also have encrypted disks. I have found some references on blogs for this, but the only place I've seen it mentioned by AWS directly is in this presentation from re:Invent 2017: https://youtu.be/o9_4uGvbvnk?t=30m20s https://youtu.be/o9_4uGvbvnk?t=30m20s (at 30:25 the presenter mentions that the nitro cards "protect the underlying flash device and customer data through encryption").
- _msw_ 8y agoHi, it's presenter in the video here. The documentation at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html#nvme-ssd-volumes https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-inst... will be updated soon with the same information.
- iconara 8y agoExcellent, thank you. It would be very good to have that information in the official documentation. I've wanted to refer to something like that for compliance reasons, for example.