7 ms·

I've also benchmarked GCP vs. AWS [0], and, for the tests that I ran, found that GCP outperformed AWS by a factor of 3:1. Specifically, a GCP instance n1-highcp

I've also benchmarked GCP vs. AWS [0], and, for the tests that I ran, found that GCP outperformed AWS by a factor of 3:1. Specifically, a GCP instance n1-highcpu-8 with a 256GB pd-ssd disk, clocked in at 11,728 IOPS vs an AWS c4.xlarge with a 256GB gp2 disk, clocking in at 3,634 IOPS. To put that in context of the blog post, it means your setup can drastically affect your results. Using local NVMe disk, for example, yields excellent results at the expense of increased risk. Also, AWS's io1 disk is very expensive—after my first io1 bill from AWS, I never used that disk type again. [0] http://engineering.pivotal.io/post/gobonniego_results/ http://engineering.pivotal.io/post/gobonniego_results/

deleted 8y ago
[deleted]
virtuallynathan 8y ago
I would compare against the C5 instance type; it uses the newer "Nitro" hypervisor.
- peferron 8y ago
  Just looking at the report's second graph, c5d.4xlarge has twice the throughput of i3.4xlarge, which is incredible. I wish AWS would release a new generation of "i" instances, with the same large amount of local storage as i3, but built on the same platform (including Nitro and processor choices) as c5/m5.
  - malisper 8y ago
    To be fair, an i3.4xl has 10x the disk and 4x the memory as a c5d.4xl at 1.5x the cost. If you are optimizing for GB/cost, i3s are still a reasonable choice. There's also the i3.metal which has no hypervisor. I'm curious how the performance of one of those compares to a traditional i3.
  - derefr 8y ago
    > I wish AWS would release a new generation of "i" instances, with the same large amount of local storage as i3 I figure they can't. [Economically, I mean.] I always figured the "local per-instance storage" on most instance-types was actually not literally local, but rather a set of disks allocated from a per-rack iSCSI disk server. (The host "wastage" if this wasn't true would be quite large.) The "i" instance-type hosts—especially the ones for the large instance-types—were likely just making a claim for the entire disk-server of their rack (leaving the rest of the rack to be schedulable only by instances that need no instance disks.) Since iSCSI and "bare metal" don't go together, on Nitro, you'd actually need to build instances with real physical reserved disk pools that the hardware can see. That may even mean putting the disks inside the computer (shock horror!) and thus needing complex 2Us that need to be "recycled" (= having the still-good stuff fished out of them when they die) and may need to be opened up by ops folks for more than one reason, rather than just having "throwaway" compute blades + equally "throwaway" hotswap disk pools. In such a 1990s-reminiscent setup, i3.metal seems like a sensible upper bound for how big such an instance could get, economically. Maybe you don't need the 2Us; maybe you can build the disk pool as something like a disk server (i.e. a dedicated PCIe backplane leading to RAID-controller daughterboards? something something Thunderbolt?) That'd lower the TCO of these host machines a bit, but it'd still be questionable how much usage such instances would get—and when they're unscheduled, despite the disks being a separate physical box, those disks would still be unavailable for any other instance to use, even ones scheduled to machines in the same rack. (Or maybe they could get really fancy, and have a disk-server rack that can present itself as a RAID controller over PCIe, such that, most of the time, it can just be a disk server, but when it gets reserved by a Nitro-i-instance, it can switch off its Infiniband cards and switch on its PCIe client interface cards, and moonlight as a RAID array. If AWS does get bigger i-type instances, I'd wager that this is what they would have built to achieve that. That or custom RAID controller cards that present Infiniband-rDMA targets as if they were local NVMe devices, and don't allow host configuration, only BMC configuration.)
    - altmind 8y ago
      you can still have remote disks(same or next rack, 2m runway) with bare metal if you use HBA with external sas cabling. this works reliably, native speed, not too expensive and is well tested.
adrr 8y ago
If you need raw speed for storage why mess around with EBS or any other network storage device. Your case with videos. You could shove the videos for permanent storage on S3 and then use I2 instance and it's ephemeral storage to serve videos. I think I2 NVME drives push 250,000 IOPS per drive and you can get up to 8 drives per instance.
- peferron 8y ago
  If you expect your read/writes to be mostly sequential, and about throughput more than about IOPS, then local HDDs (d2 or h1) may even do the job for a lower $/GB. (You may even be able to serve straight from S3 or CloudFront but that's veering off the topic.)
dpedu 8y ago
I don't think the AWS side of your test is a good example. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html#EBSVolumeTypes_gp2 https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolum... You will never get more than ~3,000 iops on a 256GB gp2 disk because the IOP cap for gp2 type disks of this size is 768 or 3,000 iops. Yes, "or". If you ran the test above for longer, you'd find that the iops will eventually drop to the baseline of 768 (which is the size of your disk multiplied times 3). Or, if you test a much larger gp2 volume you'll see higher numbers as well. Check out the description of the "burst bucket" in the link above.
- halbritt 8y ago
  This is why I tend to like GCP SSD persistent disk. For a 333GB disk I get ~10k IOPS consistently. It's $.17/GB vs. $.10/GB for EBS GP2, but still way cheaper than IO1.
sam0x17 8y ago
Yeah I think the instance types are different enough that this doesn't measure anything generally useful. Now comparing something like Lambda vs GCF, or GCS vs S3, that would be at least telling on a platform-wide scale.

deleted 8y ago

[deleted]

virtuallynathan 8y ago

I would compare against the C5 instance type; it uses the newer "Nitro" hypervisor.

peferron 8y ago
Just looking at the report's second graph, c5d.4xlarge has twice the throughput of i3.4xlarge, which is incredible. I wish AWS would release a new generation of "i" instances, with the same large amount of local storage as i3, but built on the same platform (including Nitro and processor choices) as c5/m5.
- malisper 8y ago
  To be fair, an i3.4xl has 10x the disk and 4x the memory as a c5d.4xl at 1.5x the cost. If you are optimizing for GB/cost, i3s are still a reasonable choice. There's also the i3.metal which has no hypervisor. I'm curious how the performance of one of those compares to a traditional i3.
- derefr 8y ago
  > I wish AWS would release a new generation of "i" instances, with the same large amount of local storage as i3 I figure they can't. [Economically, I mean.] I always figured the "local per-instance storage" on most instance-types was actually not literally local, but rather a set of disks allocated from a per-rack iSCSI disk server. (The host "wastage" if this wasn't true would be quite large.) The "i" instance-type hosts—especially the ones for the large instance-types—were likely just making a claim for the entire disk-server of their rack (leaving the rest of the rack to be schedulable only by instances that need no instance disks.) Since iSCSI and "bare metal" don't go together, on Nitro, you'd actually need to build instances with real physical reserved disk pools that the hardware can see. That may even mean putting the disks inside the computer (shock horror!) and thus needing complex 2Us that need to be "recycled" (= having the still-good stuff fished out of them when they die) and may need to be opened up by ops folks for more than one reason, rather than just having "throwaway" compute blades + equally "throwaway" hotswap disk pools. In such a 1990s-reminiscent setup, i3.metal seems like a sensible upper bound for how big such an instance could get, economically. Maybe you don't need the 2Us; maybe you can build the disk pool as something like a disk server (i.e. a dedicated PCIe backplane leading to RAID-controller daughterboards? something something Thunderbolt?) That'd lower the TCO of these host machines a bit, but it'd still be questionable how much usage such instances would get—and when they're unscheduled, despite the disks being a separate physical box, those disks would still be unavailable for any other instance to use, even ones scheduled to machines in the same rack. (Or maybe they could get really fancy, and have a disk-server rack that can present itself as a RAID controller over PCIe, such that, most of the time, it can just be a disk server, but when it gets reserved by a Nitro-i-instance, it can switch off its Infiniband cards and switch on its PCIe client interface cards, and moonlight as a RAID array. If AWS does get bigger i-type instances, I'd wager that this is what they would have built to achieve that. That or custom RAID controller cards that present Infiniband-rDMA targets as if they were local NVMe devices, and don't allow host configuration, only BMC configuration.)
  - altmind 8y ago
    you can still have remote disks(same or next rack, 2m runway) with bare metal if you use HBA with external sas cabling. this works reliably, native speed, not too expensive and is well tested.

adrr 8y ago

If you need raw speed for storage why mess around with EBS or any other network storage device. Your case with videos. You could shove the videos for permanent storage on S3 and then use I2 instance and it's ephemeral storage to serve videos. I think I2 NVME drives push 250,000 IOPS per drive and you can get up to 8 drives per instance.

peferron 8y ago
If you expect your read/writes to be mostly sequential, and about throughput more than about IOPS, then local HDDs (d2 or h1) may even do the job for a lower $/GB. (You may even be able to serve straight from S3 or CloudFront but that's veering off the topic.)

dpedu 8y ago

I don't think the AWS side of your test is a good example. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html#EBSVolumeTypes_gp2 https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolum... You will never get more than ~3,000 iops on a 256GB gp2 disk because the IOP cap for gp2 type disks of this size is 768 or 3,000 iops. Yes, "or". If you ran the test above for longer, you'd find that the iops will eventually drop to the baseline of 768 (which is the size of your disk multiplied times 3). Or, if you test a much larger gp2 volume you'll see higher numbers as well. Check out the description of the "burst bucket" in the link above.

halbritt 8y ago
This is why I tend to like GCP SSD persistent disk. For a 333GB disk I get ~10k IOPS consistently. It's $.17/GB vs. $.10/GB for EBS GP2, but still way cheaper than IO1.

sam0x17 8y ago

Yeah I think the instance types are different enough that this doesn't measure anything generally useful. Now comparing something like Lambda vs GCF, or GCS vs S3, that would be at least telling on a platform-wide scale.