5 ms·

I am not sure about the technical merit of this link. Best of show: "Google probably has the best networking technology on the planet." How do we quantify thi

I am not sure about the technical merit of this link. Best of show: "Google probably has the best networking technology on the planet." How do we quantify this? "This is important for several reasons. On EC2, if a node has a hardware problem, it will likely mean that you'll need to restart your virtual machine." I would much rather create a service that can tolerate single node outages than relying on "live migrations". I am not sure what he meant by the SSD comparison, Amazon EBS that can be SSD but still it is a network mounted storage. "Most of GCP's technology was developed internally and has high standards of reliability and performance." Guess what AWS was developed for. I like hand-wavy, articles as much as any other guy, but it seems to me they picked GCP and wrote an article to justify it, an cooked up some numbers with single dimension comparisons to make it look like scientific. I wish I was working on single dimension problems in real life, but it is always more complex than that. I am more interested in worst case scenarios and SLAs than micro-benchmark results when comparing cloud vendors. Discarding Azure was purely arbitrary, in fact, Azure is more than happy running Linux or other non-Windows operating systems, I am not sure where he got the idea of " Linux-second cloud". https://azure.microsoft.com/en-us/blog/running-freebsd-in-azure/ https://azure.microsoft.com/en-us/blog/running-freebsd-in-az...

mikecb 11y ago
If you're interested in datacenter networking, can't miss ONS. Amin Vahdat, the guy in charge of networking at Google, has keynoted the past two years, with hardware in 2015[1] and software in 2014[2]. Mark Russinovich also spoke about Azure's SDN, but it doesn't come close to Amin's presentation. Every once in a while you'll hear some random spec from one of these companies, and it's always pretty surprising, but Amin's team has achieved 5 Petabit/s in bisectional bandwidth. It's more than surprising. [1] https://www.youtube.com/watch?v=FaAZAII2x0w https://www.youtube.com/watch?v=FaAZAII2x0w [2] https://www.youtube.com/watch?v=n4gOZrUwWmc https://www.youtube.com/watch?v=n4gOZrUwWmc [3] https://www.youtube.com/watch?v=RffHFIhg5Sc https://www.youtube.com/watch?v=RffHFIhg5Sc
- StreamBright 11y ago
  Thanks for the links! This is a pretty interesting topic and I am going to watch these videos.
gricardo99 11y ago
Anyone know where one can go to get a more quantitative, price/performance comparison of the various cloud services? I know, lots of dimensions for that comparison, but probably picking a few dimensions, or letting users select a few, could give a reasonable ranking of services and prices.
- gricardo99 11y ago
  answering my own question, this looks pretty cool (no affiliation): https://www.cloudorado.com/cloud_server_comparison.jsp https://www.cloudorado.com/cloud_server_comparison.jsp
thesandlord 11y ago
Disclaimer: I work for Google Cloud > "Google probably has the best networking technology on the planet." How do we quantify this? In the article they did a bunch of tests. Quote: GCP does roughly 7x better for the comparison of 4-core machines, but for the largest machine sizes networking performance is roughly equivalent. There is also https://github.com/GoogleCloudPlatform/PerfKitBenchmarker https://github.com/GoogleCloudPlatform/PerfKitBenchmarker if you want to benchmark things yourself. Seriously, try it yourself. I think you will be pleasantly surprised. > I would much rather create a service that can tolerate single node outages than relying on "live migrations". Services should tolerate node failure even on GCP, live migration does not really help with that. It's more about reducing ops. With AWS, you have to manually reboot your machines when a infra upgrade happens. With GCP it is automatic. > I am not sure what he meant by the SSD comparison, Amazon EBS that can be SSD but still it is a network mounted storage. I'm not too sure what your question is? > Discarding Azure was purely arbitrary Agreed, would love to know more about why they didn't consider Azure
- StreamBright 11y ago
  Disclaimer: I used to work for Amazon, does not own any AMZN anymore 1. AWS does explicitly tells you up front that smaller instance sizes come with smaller network throughput. This is well known and well communicated even when you browse the instance offerings. Doing 7x better for a 4 core instance is hardly relevant (depending on the actual CPU type though), being able to saturate your pipe would probably consume much of your CPU time and you could hardly do anything else on the box. You can prove me wrong on this one. Synthetic benchmarks are not really relevant for production use cases. A good read in the subject: http://www.brendangregg.com/activebenchmarking.html http://www.brendangregg.com/activebenchmarking.html 2. On reducing OPS. You are implying that these OPSy things are not automated. You should ask your SRE co-workers about this one. For running a website this scale, you absolutely need to automate cases when the server is rebooted. Meaning, on shut down it needs to remove itself from the load-balancer or from the resource pool, and when it comes back it has to put itself back. Worst case scenario you can just terminate the instance and let auto-scaling do its job. All of these are completely human attention free operations in most cases, but I do understand that some smaller customers are not so advanced with automation and GCP might be optimizing for those clients. 3. I do not have any question, as I pointed out that in the article the author is talking about EBS while it might appear to the reader that he is talking about some sort of local SSD. 4. Great! I would like to know it too! We should petition together. :)
obulpathi 11y ago
We did a performance analysis of Google Cloud vs AWS. The results are in line with what is published in the post. The biggest thing that we can not quantify is "ease of use". Google Cloud is a pleasure to work with. AWS feel so clunky compared to Google Cloud. Don't take my word for it. Create a VM, login into it on AWS and Google Cloud, you will change your opinion about what a good cloud is.
- autotune 11y ago
  If you're using the GUI to manage your resources rather than going the Infrastructure As Code route, you're probably doing it wrong. You should be using a tool like Terraform, which lets you use multiple cloud providers (https://www.terraform.io/docs/providers/ https://www.terraform.io/docs/providers/), and can actually tell you if there are any immediate errors before attempting to launch a resource, so is friendly with Jenkins or any other CI tool you prefer to use as a result as well.
  - obulpathi 11y ago
    We don't use GUI to manage our resources. We use CloudFormation for AWS and Deployment Manager for Google. Let me tell you a couple of things about those services. In AWS some resources are zonal, some regional and some are global. It's a mess to work with. For example, same AMI image has different ids in different regions. You need to create maps and stuff to make your code work across regions. Come to Google Cloud, no more zonal/regional/global fuss. An image is a global resource. It's available by the same id in all regions. Your infrastructure template looks much cleaner. Combine the power of Jinja, you can create far powerful templates and evaluate them on the fly. AWS has "three" queuing systems, "two" storage solutions with different API's and different quirks. Google just has one and its nails the use cases for queuing and storage. AWS micro-instance go poof, without any notice. Their NATS are known for being unreliable. Load balancers can't scale. Every service that I looked into, Google is way better than AWS.
    - matwood 11y ago
      AWS definitely feels like a product that has grown organically and needs some house cleaning. GC was able to look at AWS, take those lessons, and improve from the start.
pbreit 11y ago
Author provided a lot of information. I wouldn't call it a "hand wavy" article at all.
- StreamBright 11y ago
  Well, providing lots of information vs. providing meaningful in-depth analysis are very different. I see your point though.
eloff 11y ago
Best networking is a dubious claim. On the bigger AWS instances you can bypass the hypervisor with SR-IOV. AFAIK you still can't do that with GCP. So if you really need maximum network performance AWS will likely win, especially on latency.
- zbjornson 11y ago
  On the contrary, this is one of the factors where GCP wins hands-down. In addition to the benchmark shown in the OP, check out one I posted on slide 14 of [1]. GCP achieves Gbit speeds on almost all instance types, and has higher speeds than AWS's biggest machines. [1] https://docs.google.com/presentation/d/1B1jvWWh0ACaDv4ryEzLlH-KropXUNf6y4IGGjIL5uBo/mobilepresent#slide=id.g109eca9a3a_0_18 https://docs.google.com/presentation/d/1B1jvWWh0ACaDv4ryEzLl...
  - eloff 11y ago
    I stand corrected. It also appears from those charts that the bigger GCP machines have more than 10gbps connection. It looks like 2x10gbps. That would explain why instances of all sizes are able to push more network traffic. Mind you the benchmarks are for bulk transfer with 9001 MTU. With more jittery workloads with lots of small packets, like a webserver has to deal with, then you see the benefit of SR-IOV. So AWS may still have the advantage on some workloads, and some measures (maybe latency, maybe CPU usage per packet.) However, it's clear if Google can support SR-IOV in the future they will mop the floor with AWS on networking because their network infrastructure is obviously superior.
frostmatthew 11y ago
> Guess what AWS was developed for. EC2 being developed for internal use is more myth than fact. The original idea was for internal use, but didn't exist much beyond a "short paper"[1] until it was green-lighted (by Bezos) as an external/sellable service. [1] http://blog.b3k.us/2009/01/25/ec2-origins.html http://blog.b3k.us/2009/01/25/ec2-origins.html

mikecb 11y ago

If you're interested in datacenter networking, can't miss ONS. Amin Vahdat, the guy in charge of networking at Google, has keynoted the past two years, with hardware in 2015[1] and software in 2014[2]. Mark Russinovich also spoke about Azure's SDN, but it doesn't come close to Amin's presentation. Every once in a while you'll hear some random spec from one of these companies, and it's always pretty surprising, but Amin's team has achieved 5 Petabit/s in bisectional bandwidth. It's more than surprising. [1] https://www.youtube.com/watch?v=FaAZAII2x0w https://www.youtube.com/watch?v=FaAZAII2x0w [2] https://www.youtube.com/watch?v=n4gOZrUwWmc https://www.youtube.com/watch?v=n4gOZrUwWmc [3] https://www.youtube.com/watch?v=RffHFIhg5Sc https://www.youtube.com/watch?v=RffHFIhg5Sc

StreamBright 11y ago
Thanks for the links! This is a pretty interesting topic and I am going to watch these videos.

gricardo99 11y ago

Anyone know where one can go to get a more quantitative, price/performance comparison of the various cloud services? I know, lots of dimensions for that comparison, but probably picking a few dimensions, or letting users select a few, could give a reasonable ranking of services and prices.

gricardo99 11y ago
answering my own question, this looks pretty cool (no affiliation): https://www.cloudorado.com/cloud_server_comparison.jsp https://www.cloudorado.com/cloud_server_comparison.jsp

thesandlord 11y ago

Disclaimer: I work for Google Cloud > "Google probably has the best networking technology on the planet." How do we quantify this? In the article they did a bunch of tests. Quote: GCP does roughly 7x better for the comparison of 4-core machines, but for the largest machine sizes networking performance is roughly equivalent. There is also https://github.com/GoogleCloudPlatform/PerfKitBenchmarker https://github.com/GoogleCloudPlatform/PerfKitBenchmarker if you want to benchmark things yourself. Seriously, try it yourself. I think you will be pleasantly surprised. > I would much rather create a service that can tolerate single node outages than relying on "live migrations". Services should tolerate node failure even on GCP, live migration does not really help with that. It's more about reducing ops. With AWS, you have to manually reboot your machines when a infra upgrade happens. With GCP it is automatic. > I am not sure what he meant by the SSD comparison, Amazon EBS that can be SSD but still it is a network mounted storage. I'm not too sure what your question is? > Discarding Azure was purely arbitrary Agreed, would love to know more about why they didn't consider Azure

StreamBright 11y ago
Disclaimer: I used to work for Amazon, does not own any AMZN anymore 1. AWS does explicitly tells you up front that smaller instance sizes come with smaller network throughput. This is well known and well communicated even when you browse the instance offerings. Doing 7x better for a 4 core instance is hardly relevant (depending on the actual CPU type though), being able to saturate your pipe would probably consume much of your CPU time and you could hardly do anything else on the box. You can prove me wrong on this one. Synthetic benchmarks are not really relevant for production use cases. A good read in the subject: http://www.brendangregg.com/activebenchmarking.html http://www.brendangregg.com/activebenchmarking.html 2. On reducing OPS. You are implying that these OPSy things are not automated. You should ask your SRE co-workers about this one. For running a website this scale, you absolutely need to automate cases when the server is rebooted. Meaning, on shut down it needs to remove itself from the load-balancer or from the resource pool, and when it comes back it has to put itself back. Worst case scenario you can just terminate the instance and let auto-scaling do its job. All of these are completely human attention free operations in most cases, but I do understand that some smaller customers are not so advanced with automation and GCP might be optimizing for those clients. 3. I do not have any question, as I pointed out that in the article the author is talking about EBS while it might appear to the reader that he is talking about some sort of local SSD. 4. Great! I would like to know it too! We should petition together. :)

obulpathi 11y ago

We did a performance analysis of Google Cloud vs AWS. The results are in line with what is published in the post. The biggest thing that we can not quantify is "ease of use". Google Cloud is a pleasure to work with. AWS feel so clunky compared to Google Cloud. Don't take my word for it. Create a VM, login into it on AWS and Google Cloud, you will change your opinion about what a good cloud is.

autotune 11y ago
If you're using the GUI to manage your resources rather than going the Infrastructure As Code route, you're probably doing it wrong. You should be using a tool like Terraform, which lets you use multiple cloud providers (https://www.terraform.io/docs/providers/ https://www.terraform.io/docs/providers/), and can actually tell you if there are any immediate errors before attempting to launch a resource, so is friendly with Jenkins or any other CI tool you prefer to use as a result as well.
- obulpathi 11y ago
  We don't use GUI to manage our resources. We use CloudFormation for AWS and Deployment Manager for Google. Let me tell you a couple of things about those services. In AWS some resources are zonal, some regional and some are global. It's a mess to work with. For example, same AMI image has different ids in different regions. You need to create maps and stuff to make your code work across regions. Come to Google Cloud, no more zonal/regional/global fuss. An image is a global resource. It's available by the same id in all regions. Your infrastructure template looks much cleaner. Combine the power of Jinja, you can create far powerful templates and evaluate them on the fly. AWS has "three" queuing systems, "two" storage solutions with different API's and different quirks. Google just has one and its nails the use cases for queuing and storage. AWS micro-instance go poof, without any notice. Their NATS are known for being unreliable. Load balancers can't scale. Every service that I looked into, Google is way better than AWS.
  - matwood 11y ago
    AWS definitely feels like a product that has grown organically and needs some house cleaning. GC was able to look at AWS, take those lessons, and improve from the start.

pbreit 11y ago

Author provided a lot of information. I wouldn't call it a "hand wavy" article at all.

StreamBright 11y ago
Well, providing lots of information vs. providing meaningful in-depth analysis are very different. I see your point though.

eloff 11y ago

Best networking is a dubious claim. On the bigger AWS instances you can bypass the hypervisor with SR-IOV. AFAIK you still can't do that with GCP. So if you really need maximum network performance AWS will likely win, especially on latency.

zbjornson 11y ago
On the contrary, this is one of the factors where GCP wins hands-down. In addition to the benchmark shown in the OP, check out one I posted on slide 14 of [1]. GCP achieves Gbit speeds on almost all instance types, and has higher speeds than AWS's biggest machines. [1] https://docs.google.com/presentation/d/1B1jvWWh0ACaDv4ryEzLlH-KropXUNf6y4IGGjIL5uBo/mobilepresent#slide=id.g109eca9a3a_0_18 https://docs.google.com/presentation/d/1B1jvWWh0ACaDv4ryEzLl...
- eloff 11y ago
  I stand corrected. It also appears from those charts that the bigger GCP machines have more than 10gbps connection. It looks like 2x10gbps. That would explain why instances of all sizes are able to push more network traffic. Mind you the benchmarks are for bulk transfer with 9001 MTU. With more jittery workloads with lots of small packets, like a webserver has to deal with, then you see the benefit of SR-IOV. So AWS may still have the advantage on some workloads, and some measures (maybe latency, maybe CPU usage per packet.) However, it's clear if Google can support SR-IOV in the future they will mop the floor with AWS on networking because their network infrastructure is obviously superior.

frostmatthew 11y ago

> Guess what AWS was developed for. EC2 being developed for internal use is more myth than fact. The original idea was for internal use, but didn't exist much beyond a "short paper"[1] until it was green-lighted (by Bezos) as an external/sellable service. [1] http://blog.b3k.us/2009/01/25/ec2-origins.html http://blog.b3k.us/2009/01/25/ec2-origins.html