3 ms·

With more instance types and storage classes, EC2 offers a much wider spectrum of performance capabilities for CPU, storage (ephemeral and durable), and networ

by jread 8y ago

With more instance types and storage classes, EC2 offers a much wider spectrum of performance capabilities for CPU, storage (ephemeral and durable), and networking.

jchw 8y ago
I don't see how this has to do with my question about benchmarks of sequential disk I/O on GCP instances with local NVMe SSDs. Did you respond to the wrong message? Either way, I use both GCP and AWS but I am finding that I'm spending less on GCP. I don't believe every workload will be cheaper in GCP, but it is clearly competitive at least.
- jread 8y ago
  I'm responding to your comment "GCE seems to have better I/O and AWS seems to have higher frequency CPUs" - this isn't true. You can get faster I/O, networking and CPU with EC2 if you select the correct instance/storage and are willing to pay for it.
- jread 8y ago
  For seq read/write I've observed up to around 15000/2900 MB/s on i3.16xl and 2750/1400 MB/s on GCE w/4xlocal SSD
  - jchw 8y ago
    This is lacking in details: - What OS image(s) were used? This would aid in reproducibility. - What tool was used for benchmarking? Preferably with the configuration so I could reproduce it. - What GCE instance is used? I assume you will bottleneck on something other than I/O if comparing a very small GCE instance to a very large AWS instance. I will probably benchmark this on my own at some point now that my interest is piqued, but if you have already done the work to benchmark this I'd love to hear more. So far though it's too vague for me to use meaningfully. Is there a blog post or other material that is publicly accessible? I've Googled for such before but come up surprisingly empty handed.
    - jchw 8y ago
      I've just tried to benchmark, but I'm not sure how. I tried using dd to measure sequential speed and this was giving pretty good results on Google Cloud: > dd if=/dev/zero of=/dev/disk/by-id/google-local-ssd-0 bs=8192 > 63356186624 bytes (63 GB, 59 GiB) copied, 36.6496 s, 1.7 GB/s > dd of=/dev/zero if=/dev/disk/by-id/google-local-ssd-0 bs=8192 > 63356186624 bytes (63 GB, 59 GiB) copied, 17.5434 s, 3.6 GB/s ...But I'm perplexed by why it cuts off at 64 GB. On Amazon I couldn't figure out how to get decent speeds at all. Comparable to my Google Cloud box, I set up an i3.xlarge, which had 4 CPUs and 1 SSD. While the dd operation was working fine, it slowed down to a paltry 430 MB/s quickly and never recovered. I doubt this is the true sequential performance of the drive, so I gave up. I would've attempted to do multiple SSD configurations, but those seemed even harder. mdadm-based RAIDs were clearly bottlenecking somewhere other than hardware, because they were slower than writing or reading directly. tl;dr: It turns out these new-fangled NVMe instance store setups are too complicated for me to understand.
      - _msw_ 8y ago
        You'll need to apply some SSD specific methodology to correctly characterize their performance. Writing zeros to some SSDs will trigger special logic in their flash translation layer (FTL) that will prevent the writes from hitting the flash media. They simply note in the metadata that you wrote some zeros, and reading them back will also be impossibly quick. This isn't the case only on new-fangled NVMe devices...

by jread 8y ago

With more instance types and storage classes, EC2 offers a much wider spectrum of performance capabilities for CPU, storage (ephemeral and durable), and networking.

jchw 8y ago

I don't see how this has to do with my question about benchmarks of sequential disk I/O on GCP instances with local NVMe SSDs. Did you respond to the wrong message? Either way, I use both GCP and AWS but I am finding that I'm spending less on GCP. I don't believe every workload will be cheaper in GCP, but it is clearly competitive at least.

jread 8y ago
I'm responding to your comment "GCE seems to have better I/O and AWS seems to have higher frequency CPUs" - this isn't true. You can get faster I/O, networking and CPU with EC2 if you select the correct instance/storage and are willing to pay for it.
jread 8y ago
For seq read/write I've observed up to around 15000/2900 MB/s on i3.16xl and 2750/1400 MB/s on GCE w/4xlocal SSD
- jchw 8y ago
  This is lacking in details: - What OS image(s) were used? This would aid in reproducibility. - What tool was used for benchmarking? Preferably with the configuration so I could reproduce it. - What GCE instance is used? I assume you will bottleneck on something other than I/O if comparing a very small GCE instance to a very large AWS instance. I will probably benchmark this on my own at some point now that my interest is piqued, but if you have already done the work to benchmark this I'd love to hear more. So far though it's too vague for me to use meaningfully. Is there a blog post or other material that is publicly accessible? I've Googled for such before but come up surprisingly empty handed.
  - jchw 8y ago
    I've just tried to benchmark, but I'm not sure how. I tried using dd to measure sequential speed and this was giving pretty good results on Google Cloud: > dd if=/dev/zero of=/dev/disk/by-id/google-local-ssd-0 bs=8192 > 63356186624 bytes (63 GB, 59 GiB) copied, 36.6496 s, 1.7 GB/s > dd of=/dev/zero if=/dev/disk/by-id/google-local-ssd-0 bs=8192 > 63356186624 bytes (63 GB, 59 GiB) copied, 17.5434 s, 3.6 GB/s ...But I'm perplexed by why it cuts off at 64 GB. On Amazon I couldn't figure out how to get decent speeds at all. Comparable to my Google Cloud box, I set up an i3.xlarge, which had 4 CPUs and 1 SSD. While the dd operation was working fine, it slowed down to a paltry 430 MB/s quickly and never recovered. I doubt this is the true sequential performance of the drive, so I gave up. I would've attempted to do multiple SSD configurations, but those seemed even harder. mdadm-based RAIDs were clearly bottlenecking somewhere other than hardware, because they were slower than writing or reading directly. tl;dr: It turns out these new-fangled NVMe instance store setups are too complicated for me to understand.
    - _msw_ 8y ago
      You'll need to apply some SSD specific methodology to correctly characterize their performance. Writing zeros to some SSDs will trigger special logic in their flash translation layer (FTL) that will prevent the writes from hitting the flash media. They simply note in the metadata that you wrote some zeros, and reading them back will also be impossibly quick. This isn't the case only on new-fangled NVMe devices...