16 ms·

GCP Outpaces Azure, AWS in the 2021 Cloud Report

266 points by mohangk 6y ago

polote 6y ago
Adding 20px of padding on all the slides seems to be the worst idea they got this year. Slides that you can't read is always better than slides you can read /s Direct access to pdf download : https://content.cdntwrk.com/files/aT0xMzI3NDk4JnY9NCZpc3N1ZU5hbWU9MjAyMS1jbG91ZC1yZXBvcnQtY29ja3JvYWNoLWxhYnMmY21kPWQmc2lnPWIyMGNlOWI4YzA2MzgwNjc4MGU4NDJiMTQzYmUzMWE0 https://content.cdntwrk.com/files/aT0xMzI3NDk4JnY9NCZpc3N1ZU...
- jarym 6y ago
  UX curse strikes again
- ericpauley 6y ago
  In the authors' defense, these don't appear to be slides at all, but rather a PDF version of what could be a printed document. However, whether the document would look good in print is also highly debatable.
deleted 6y ago
[deleted]
latch 6y ago
We're happy CockroachDB users. Can't put my finger on why, but this report comes off as almost pure marketing and not very substantive (say compared to the Backblaze reports). Maybe it's because I had to give an email (mailinator) with no option to opt-out of marketing emails to read it. Maybe it's because it seemed to try to paint all three as winners. We run CRDB on baremetal. I'd love to see how that stacks up - but I guess their managed offering is a major money maker. It's a shame because there's clearly a lot of effort put into it and I love the work they're doing (and how they do it). I will say that, as a non-cloud-believer, I'm much happier dealing with Google Cloud than AWS. It's straightforward and doesn't require nearly as much vendor-specific knowledge. The console is more user-friendly, and things are usually cheaper and faster (but still so much more expensive and slower than just using a dedicated host).
- derefr 6y ago
  > Maybe it’s because it seemed to try to paint all three as winners. Compared to the rest of the market, AWS + Azure + GCP are all winners. They’re all huge, all growing, and all outpacing the growth of traditional non-“cloud” hosting providers by a long shot. They’re also all greatly beating out any other cloud providers who aren’t them, e.g. IBM Cloud (nee Softlayer.) They’re essentially dividing up the hosting market together, like any good cabal. Compared to the growth all three of the big cloud providers are experiencing, the relative growth margins they use to claim that one of them is “the biggest” are basically noise. To put it another way: I’d much rather invest in all three of them, than in just one of them.
  - latch 6y ago
    Take the telecom companies (AT&T, Telefonica, Tata, China Unicom, and on and on and on and on (these companies have hundreds of data centers each)). Then take the wholesalers (Equinix, Digital Realty, etc) - some of who count some cloud vendors as customers. Then take tens of thousand of collocation and dedicated providers that own their own data centers (PhoenixNap, HE, OVH, Hetzner, Softlayer, ...). Then take the VPS providers (Linode, DO, Vultr, ....). Then take the shared hosting (GoDaddy, ..). Then take the government agencies and companies that have their own private data centers (e.g, banks). Cloud vendors are growing, but it's still a very small part of the market. What they're really good at is sales and marketing (and making much better margins.)
- autoditype 6y ago
  I agree, the results aren't substantive and don't directly correspond to such a one-sided title, as the results are much more nuanced and varied
- systemvoltage 6y ago
  I wouldn't touch this thing purely based on the name - CockroachDB. Yes, its unfair but they're absolutely asking for it. I am going to be dealing with this all day, I don't want to develop some kind of a Pavlovian conditioning with the name where everytime I think about databases, I think about cockroaches and all the disgusting things they do. What a monumentally stupid name.
jeffbee 6y ago
How much of this is affected just by "cloud weather"? It seems like network latency and some of these other measures would be influenced by adjacent workloads that happen to be running in your region, zone, facility, rack, or machine.
- jacques_chester 6y ago
  They're definitely thin on explaining the sample sizes. They say 54 configurations over "nearly 1,000", which suggest 17 tests (918 runs) or 18 tests(972 runs) per configuration. They run 4 different benchmarks (CPU, network, I/O, TPC-C), suggesting an average of around 4.25 or 4.5 per bechmark per configuration. If instead they ran 16 per configuration, that would be a nice round 4 per benchmark per configuration, but total runs would drop to 864, somewhat less than "nearly 1000". Assuming my figures are sound, we're looking at 4 to 5 samples per combination. Without some information about the within-group variation, though, it's difficult to distinguish what variation was due to "weather" and what was due to the platform. I do however think that the effect size of some results is enough to make them useful (eg, network throughput). But all of the close results (eg single-core difference between AWS and Azure) are not very reliable, in my view.
manigandham 6y ago
GCP is also far easier to use than the others. Everything from the organization/project hierarchy, g-suite user IAM permissions, simple primitives that can be assembled to your specifications, and web-based console access to everything makes it much simpler to deal with. The performance is a nice bonus.
izacus 6y ago
Wait, didn't HN pronounce GCP as dead in pretty much every content thread about clouds? What's going on?
- tyingq 6y ago
  They are using "outpaces" to talk about performance, not market share. AWS remains a gorilla there. Especially if you net out Gsuite and O365.
- isbvhodnvemrwvn 6y ago
  Most of these threads mention excellent technology, but piss poor product management.
eecks 6y ago
Google, like AWS and Azure, is 'only pay for what you use'. Can anyone tell me if there is a way to put limits in? Or to choose a 20/50/100 dollar per month plan?
- isbvhodnvemrwvn 6y ago
  Not quite. Please note that there are various things you pay for, let's take a look at several of those: - provisioned resources (instances, including the ones under relational databases or other stuff like that) - usage of on-demand resources like AWS Lambda and API Gateway - infrastructure such as load balancers (kind of mix of provisioned and pay-per-use) - persistent storage (which is all over the place in terms of payment methods) While you could shut down the first three to save some money, would you like to remove your data permanently? So far all mechanisms that are available are mostly reactive (billing alarms etc) rather than proactive (service quotas although they are meant to shield from poor design rather than a typo in terraform). There is clearly incentive for cloud providers for the former, but it's not an easy problem anyway (they might mark your accounts for "training" or something like that - I think this would be reasonable).
- corty 6y ago
  In GCP you can configure budgets for projects (groups of resources). Budgets can issue alerts when reaching certain percentages and reconfigure or shutdown resources when exceeded. The budget howto docs have an example to stop everything that incurs a cost on budget overrun: https://cloud.google.com/billing/docs/how-to/budgets https://cloud.google.com/billing/docs/how-to/budgets But as the warning says, that might delete data in storage or other resources you may want to keep paying for. For that case, you can execute a program, that shuts down everything you can get rid of, but the cost for storage and everything you forgot will continue to be billed.
- akh 6y ago
  I've seen many companies user "terminator" scripts that shutdown/delete things that aren't tagged as "keep" or something like that, though only in non-production accounts. The budget alerts from the cloud providers can be useful too (AWS recently released cost anomaly alerts). We're trying to tackle this problem with https://github.com/infracost/infracost https://github.com/infracost/infracost from another angle for people who use Terraform: show a cost estimate in pull requests so the user understands what costs money, and roughly how much it costs. I hope that helps clarify "only pay for what you use" without trawling through cloud pricing pages.
potency 6y ago
Are there any good alternatives to the big three? I'm looking to build out a platform with as little dependence on Google/MSFT/Amazon as possible.
- bastawhiz 6y ago
  IBM, if you're feeling lucky
- thiscatis 6y ago
  Oracle Cloud if you want to relive the nineties.
  - shiftpgdn 6y ago
    Oracle Cloud has live migration (still not available on EC2, afaik) and had cloud console before AWS.
    - deleted 6y ago
      [deleted]
- detaro 6y ago
  Depends on what you need. They offer a giant bag of services and abilities, so it really comes down to which subset of that you are looking for.
- dharmab 6y ago
  If you just need Linux VM and a couple of other basic capabilities, DigitalOcean.
  - maccard 6y ago
    Any recommendations for a DO-likd experience that supports windows? As much as I'd like to use DO, it's a show stopper for people who want the simplicity of DO but have a requirement on windows (hello, MSVC)
jrsdav 6y ago
I've done pretty extensive work in all three major cloud providers. If you were to ask me which one I'd use for a net new project, it would be GCP -- no question. Nearly all of their services I've used have been great with a feeling that they were purposefully engineered (BigQuery, GKE, GCE, Cloud Build, Cloud Run, Firebase, GCR, Dataflow, PubSub, Data Proc, Cloud SQL, goes on and on...). Not to mention almost every service has a Cloud API, which really goes a long way towards eliminating the firewall and helps you embrace the Zero Trust/BeyondCorp model. And BigQuery. I can't express enough how amazing BigQuery is. If you're not using GCP, it's worth going multi-cloud for BigQuery alone. But there is something to be said of AWS. Their SDKs are complete and predictable, their APIs are very fast and consistent, and AWS IAM, while having a steep learning curve, never leaves you guessing around what your principals have access to. For me, the real challenge with AWS has been introducing multiple AWS accounts. Governance just flat out sucks when you begin to scale past a handful of accounts (but it is getting better). Azure on the other hand, has terrible consistency issues between their APIs, their SDKs are awful, and it just feels like the entire product is an extension of the MCP System Administrator persona of old, where it's expected that someone's job will be sitting in front of a UI and clicking around to get things done (the whole blade thing with their portal has to be one of the worst user experiences I've ever seen). However, I do like their Logic Apps, and Azure Policy with auto remediation (when it works as advertised -- ref API consistency and how long it takes for things to propagate through their system) has tons of potential. But they still have a ways to go before I'd consider it for my workloads.
- dmitriid 6y ago
  I can kinda agree on GCP with one exception: Dataflow. I have no idea what the future holds for it. It is a managed Apache Beam service and is very useful for certain scenarios (like "hey, we have a million incoming PubSub messages that we need to transform into a dozen different branching streams of data"). It looks like even BigQuery actually transforms SQL statements into a bunch of Dataflow jobs. But... But... - Minor version updates to Google Dataflow SDK once every couple of months while deprecating most other minor versions? Check. - No visible contributions to Apache BEAM itself? Check. In 2021 I still don't know if I can use any Java versions beyond Java 8 to develop for and run in Dataflow. And Google is arguably one of the biggest users of Apache BEAm, and definitely a user with the largest pile of money to throw at the problem. - They've recently sent out a questionnaire about Dataflow to some of their customers that feels like a "hey, we're definitely considering deprecating this, we're gauging the potential impact"
ashtonkem 6y ago
The big problem for me is trust. I don’t care what the feature set or performance is; I don’t trust Google enough to bet a business on it. And I’m not even worried about Google being malicious; I’m worried about them being mercurial and changing/removing things I need without warning.
- bnt 6y ago
  Or some ML crob job algo going haywire and closing your account - with no human to contact to get it resolved.
  - SteveNuts 6y ago
    Google's services are so perfect they don't even need to invest in customer service! If you have a problem with anything, it's clearly your fault and you're doing it wrong. /s
    - ashtonkem 6y ago
      Sarcasm aside, at $COMPANY we have AWS reps integrated into our Slack, and they work hand in hand with our engineers quite often. Even for things as low down as “why is this query so slow on AuroraDB?” They even hop into war rooms for big events in case we need immediate assistance during high visibility outages. It’s hard to overstate how important this level of close support is for an enterprise that is literally betting the farm on a cloud provider. The idea of counting on Google’s historical level of support is an absolute non-starter for us.
      - theshrike79 6y ago
        Agree, we have an integrated AWS contact too and they're really invaluable for getting insight on what the actual limits are on their services.
      - 0xEFF 6y ago
        The role you describe is a technical account manager and Google has hired many. I’m a Google cloud partner and every project I’ve worked on has had multiple TAMS doing exactly what you describe, working hand in hand with customers, connecting the customer to product engineers, to support, navigating Black Friday and Christmas, etc...
me551ah 6y ago
I evaluated AWS and GCP for my startup and found GCP to be more expensive. The horror stories I've read about Google's lack of customer support put me off too.
- philshem 6y ago
  Many bootstrapped startups go with the cloud provider that gives the them largest startup bonus, which can be up to $100k to use in 12 months (AFAIK).
  - PedroBatista 6y ago
    and then they spend the next year migrating to a "more realistic" deployment or burn half their money on cloud computing.
- deadmutex 6y ago
  Out of curiosity, was there one cost that was significantly more? or was it across multiple services? Do you mind sharing your setup a bit more?
marcinzm 6y ago
The main reason I'm weary of recommending GCP is the support horror stories that keep coming up. I'm using it at work now since our massive Google Ad spend protects us from that. It's got some really good technologies although there's various rough edges. One thing that really irks me is GCP requiring me to talk to sales people (not support, sales) to have a relatively small quota increase. Why would they make it harder for me to give them money?
frabjoused 6y ago
This article screams bias, beginning with the title. I don't know how to give this credit.
StreamBright 6y ago
>> GCP Outpaces Azure, AWS in the 2021 Cloud Report (cockroachlabs.com) >> AWS network latencies are unbeatable Seems weird to see these sentences on the same site.
- lawrjone 6y ago
  To be fair, network latency across all the providers is reasonably similar. The network throughput was not even close though, with GCP winning by a huge margin. I'd say both statements are fair and not mutually exclusive.
  - arulajmani 6y ago
    One of the engineers who helped benchmark and compile the Cloud Report. You're right in noting that the statements aren't mutually exclusive. Our overall takeaway was informed by each Cloud's performance on the individual benchmarks as shown on page 4 of the report. The results were quite close though, as each of the Clouds had specific benchmarks they did excellent in.
deleted 6y ago
[deleted]
herdcall 6y ago
CockroachDB was founded by ex-Google employees and is partly funded by Google Ventures, so I'd take this report with a pinch of salt. IMO GCP is good for PoC/personal projects due to their liberal free tier quotas, but I don't know about going big. Anyone with large scale experience on GCP?
- phillipcarter 6y ago
  Yeah, there needs to be a disclaimer in these reports about this kind of stuff. Not that I doubt the technical accuracy of the claims, but it's just good form to make these kinds of notices.
- orangechairs 6y ago
  Report author here. We have no bias towards nor stake in any of the three cloud providers. We partnered with all three clouds to develop the testing methodology and benchmark set. Our bias is towards providing as much information as possible to our own customers as they select their clouds and machines. Reproduction are available on github: https://github.com/cockroachlabs/cloud-report-2021 https://github.com/cockroachlabs/cloud-report-2021
etxm 6y ago
GCP wins hands down when it comes to cloud governance and network design. I think the two biggest weaknesses are: - IAM - some resources have awkward relationships with IAM; although the GSuite integration is nice - CloudSQL (vs RDS) - for businesses that need relational data stores, but aren’t at the Cloud Spanner scale, RDS blows CloudSQL away in features
- mixmastamyk 6y ago
  Governance? What are the necessary features cloudsql doesn’t support?
  - etxm 6y ago
    Project organization, folders, orgs, and it’s integration with IAM / GSuite. I haven’t dug in on Cloud SQL in about a year, what I recall: - no support for logical replication in PG - there was some VPC funkiness - fewer number of extensions: ~60 extensions in RDS vs ~45 for CloudSQL - CSQL Only supports pl/pgsql vs perl, v8, tcl, and pgsql
rebelos 6y ago
I don't quite understand why, but much of the tech industry seems to be sleeping on Cloud Spanner. Google quietly completely revolutionized managed+consistent+available+scalable RDBMS and very few people seem to have caught on yet. Maybe it's too much of a threat to job security?
- jahewson 6y ago
  It’s way too expensive. Plus I wouldn’t want to deal with the limitations and latency that its consistency model brings unless I had truly Google-scale data, which I don’t.
- blaisio 6y ago
  I agree spanner and cockroach are the future. Most people don't need a database that can scale that well, and spanner is too expensive to use unless you really truly need it. Also, google and cockroach have not done enough marketing. Look at all the marketing mongodb did - they actually managed to convince people to use a database that would regularly lose data.
  - rebelos 6y ago
    You can get 3 nodes for about $27k a year and that'll handle 30k read QPS and in the neighborhood 1-2k write QPS iirc. It's a fraction of the cost of even a single dedicated engineer. And you'd probably need several engineers to achieve the same perf with open source alternatives and keep it stable/upright. There's a large class of businesses for which this choice is a no-brainer.
    - manigandham 6y ago
      Is Cloud Spanner the only managed option? Why would you compare to a dedicated engineer? AWS RDS or GCP Cloud SQL or Azure Managed SQL or IBM Compose or Aiven or any number of other vendors offer managed databases with more features, much higher performance, and far less cost. Even CRDB has its own cloud offering that's cheaper and more flexible than Spanner.
alfl 6y ago
I have substantial concerns running my core infra on Google products: deprecation, inhuman support, the allegations of anticompetitive behaviour in the states’ antitrust lawsuit. Might be good tech. Business risk seems high.
sknat 6y ago
I'm a bit dubious about the networking results they present. I did some quite extensive network performence testing last winter on those three CSP, and even if single queue TCP+gso performence can behave like this, I find the claim 'GCP is 3x faster than AWS' a bit bold. It's definitely possible to get 50G of TCP traffic in AWS, and a lot of things are in the balance (MTU, number of queues, drivers...) that make this claim a bit weird to me.
- arulajmani 6y ago
  One of the engineers who helped run benchmarks and compile the report here. It’s worth noting that for the majority of the machines we benchmarked on AWS, their tested bandwidth met the published AWS expectations. You may have noticed that some of the “network optimized” machines fell short of the published expectations though, and there’s an explanation in the report about how we tried to validate our findings. As you point out, there are a variety of variables that could be tuned to eek out better performance here, and they could bring the two clouds closer. Our claim, of course, only applies to the benchmark configuration we tested with. That being said, with the size of machine we were restricting our testing to (16 vCPUS), no AWS machine claimed to offer more than 25G of throughput.
  - talawahtech 6y ago
    All the 16 vCPU "n" instances (m5n, c5n, r5n, etc) are capable of hitting the 25 Gbps limit easily. In your report, all of the AWS results are limited to either 5Gbps or 10Gbps, but this is because of a very specific test condition. From my understanding of the test scenario, you are using a single TCP connection to run the throughput test, and hitting AWS' documented[1] throughput limit for a single flow: 10 Gbps if the two instances are in the same placement group and 5 Gbps otherwise. The reason some of the network-optimized instances were "slower" than the non-optimized ones is most likely a random draw of whether both instances in the test were physically close to each other (basically whether or not they are accidentally in placement groups). To show the true throughput you would need to use multiple connections/flows, 5-10 would probably suffice. If the single flow test case was important then maybe you should have mentioned that AWS has a specific limitation around this. Personally I don't think a single flow test case is particularly realistic for a throughput test. Either way, how it is presented is pretty misleading. 1. "Single TCP flow is limited to 10 Gbps for instances in the same placement group and 5 Gbps between instances anywhere else." https://docs.aws.amazon.com/whitepapers/latest/ec2-networking-for-telecom/overall-instance-bandwidth-limitations.html https://docs.aws.amazon.com/whitepapers/latest/ec2-networkin...
gundmc 6y ago
The network throughout is eye opening given how close most of the other benchmarks are. GCP's lowest performer is >50% higher than AWS's top performer and more than double Azure's best.
- talawahtech 6y ago
  The network throughput benchmark is pretty misleading. https://news.ycombinator.com/item?id=25816596 https://news.ycombinator.com/item?id=25816596
einszwei 6y ago
As someone who has worked on large-scale deployments on both AWS and GCP, I would always prefer AWS over GCP. While GCP products are IMO superior to similar AWS offerings their support (even premium tier) is total garbage compared to AWS.
znpy 6y ago
the website requires providing personal data including an email address to access the full report. I recommend using an email @cockroachlabs.com so that they can get spammed by their own marketing bs (besides the report). You will be directed to the download page anyway.
anthony_barker 6y ago
I have done projects on all except Azure (mostly due to a Microsoft aversion). I hate their special names for everything. Reminds me of Starbucks. Here is my take: GCP tools are better PubSub, CloudSQL. However they don't support email and their docs are not as up to date and helpful as AWS. I think the main reason to select the big three is a) security (network, instance, user management) b) you don't get fired for selecting the big guys c) some specialized tools (SES, s3, CDNs, github) I always feel that the time you invest to learn all the details of AWS you could have invested into Ansible, Docker, Wireguard, Iptables, zfs and linux, and deploy a much more cost effective solution on Heztner, (which I prefer over upcloud, do, vultr). But you need to know what you are doing. Many companies prefer to trust a vendor instead of their employees.
talawahtech 6y ago
There are number of issues with this report. The AWS networking section is particularly problematic and in need of extensive disclaimers or changes to the test methodology. On the throughput side, all this test does is demonstrate the documented[1] throughput limit for a single TCP connection. 10 Gbps if the two instances are in the same placement group and 5 Gbps otherwise. The reason some of the network-optimized instances were "slower" than the non-optimized ones was because it was simply a random draw of whether both instances in the test were physically close to each other. If they wanted to do a proper throughput test they would have used placement groups and multiple connections/flows. If they felt like the single flow test case was important they should have mentioned that AWS has a specific limitation around this. Personally I don't think a single flow test case is particularly realistic. The fact that the obvious discrepancy between their results and the documented (multi-flow) limits didn't cause them to dig deeper is enough to make me very skeptical of the purpose of this paper. The latency results are also basically a random spread. It is essentially distribution of all the different latencies you might randomly get between two instances if you don't use placement groups. It says absolutely nothing about the networking capabilities of different instances used in each test. 1. "Single TCP flow is limited to 10 Gbps for instances in the same placement group and 5 Gbps between instances anywhere else." https://docs.aws.amazon.com/whitepapers/latest/ec2-networking-for-telecom/overall-instance-bandwidth-limitations.html https://docs.aws.amazon.com/whitepapers/latest/ec2-networkin...
xiangy 6y ago
I think enabling InfiniBand in Azure brings much better Network performances (but not for all kinds of VMs) https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/hpc/enable-infiniband https://docs.microsoft.com/en-us/azure/virtual-machines/work...

266 points by mohangk 6y ago

Adding 20px of padding on all the slides seems to be the worst idea they got this year. Slides that you can't read is always better than slides you can read /s Direct access to pdf download : https://content.cdntwrk.com/files/aT0xMzI3NDk4JnY9NCZpc3N1ZU5hbWU9MjAyMS1jbG91ZC1yZXBvcnQtY29ja3JvYWNoLWxhYnMmY21kPWQmc2lnPWIyMGNlOWI4YzA2MzgwNjc4MGU4NDJiMTQzYmUzMWE0 https://content.cdntwrk.com/files/aT0xMzI3NDk4JnY9NCZpc3N1ZU...

jarym 6y ago
UX curse strikes again
ericpauley 6y ago
In the authors' defense, these don't appear to be slides at all, but rather a PDF version of what could be a printed document. However, whether the document would look good in print is also highly debatable.

deleted 6y ago

[deleted]

latch 6y ago

We're happy CockroachDB users. Can't put my finger on why, but this report comes off as almost pure marketing and not very substantive (say compared to the Backblaze reports). Maybe it's because I had to give an email (mailinator) with no option to opt-out of marketing emails to read it. Maybe it's because it seemed to try to paint all three as winners. We run CRDB on baremetal. I'd love to see how that stacks up - but I guess their managed offering is a major money maker. It's a shame because there's clearly a lot of effort put into it and I love the work they're doing (and how they do it). I will say that, as a non-cloud-believer, I'm much happier dealing with Google Cloud than AWS. It's straightforward and doesn't require nearly as much vendor-specific knowledge. The console is more user-friendly, and things are usually cheaper and faster (but still so much more expensive and slower than just using a dedicated host).

derefr 6y ago
> Maybe it’s because it seemed to try to paint all three as winners. Compared to the rest of the market, AWS + Azure + GCP are all winners. They’re all huge, all growing, and all outpacing the growth of traditional non-“cloud” hosting providers by a long shot. They’re also all greatly beating out any other cloud providers who aren’t them, e.g. IBM Cloud (nee Softlayer.) They’re essentially dividing up the hosting market together, like any good cabal. Compared to the growth all three of the big cloud providers are experiencing, the relative growth margins they use to claim that one of them is “the biggest” are basically noise. To put it another way: I’d much rather invest in all three of them, than in just one of them.
- latch 6y ago
  Take the telecom companies (AT&T, Telefonica, Tata, China Unicom, and on and on and on and on (these companies have hundreds of data centers each)). Then take the wholesalers (Equinix, Digital Realty, etc) - some of who count some cloud vendors as customers. Then take tens of thousand of collocation and dedicated providers that own their own data centers (PhoenixNap, HE, OVH, Hetzner, Softlayer, ...). Then take the VPS providers (Linode, DO, Vultr, ....). Then take the shared hosting (GoDaddy, ..). Then take the government agencies and companies that have their own private data centers (e.g, banks). Cloud vendors are growing, but it's still a very small part of the market. What they're really good at is sales and marketing (and making much better margins.)
autoditype 6y ago
I agree, the results aren't substantive and don't directly correspond to such a one-sided title, as the results are much more nuanced and varied
systemvoltage 6y ago
I wouldn't touch this thing purely based on the name - CockroachDB. Yes, its unfair but they're absolutely asking for it. I am going to be dealing with this all day, I don't want to develop some kind of a Pavlovian conditioning with the name where everytime I think about databases, I think about cockroaches and all the disgusting things they do. What a monumentally stupid name.

jeffbee 6y ago

How much of this is affected just by "cloud weather"? It seems like network latency and some of these other measures would be influenced by adjacent workloads that happen to be running in your region, zone, facility, rack, or machine.

jacques_chester 6y ago
They're definitely thin on explaining the sample sizes. They say 54 configurations over "nearly 1,000", which suggest 17 tests (918 runs) or 18 tests(972 runs) per configuration. They run 4 different benchmarks (CPU, network, I/O, TPC-C), suggesting an average of around 4.25 or 4.5 per bechmark per configuration. If instead they ran 16 per configuration, that would be a nice round 4 per benchmark per configuration, but total runs would drop to 864, somewhat less than "nearly 1000". Assuming my figures are sound, we're looking at 4 to 5 samples per combination. Without some information about the within-group variation, though, it's difficult to distinguish what variation was due to "weather" and what was due to the platform. I do however think that the effect size of some results is enough to make them useful (eg, network throughput). But all of the close results (eg single-core difference between AWS and Azure) are not very reliable, in my view.

manigandham 6y ago

GCP is also far easier to use than the others. Everything from the organization/project hierarchy, g-suite user IAM permissions, simple primitives that can be assembled to your specifications, and web-based console access to everything makes it much simpler to deal with. The performance is a nice bonus.

izacus 6y ago

Wait, didn't HN pronounce GCP as dead in pretty much every content thread about clouds? What's going on?

tyingq 6y ago
They are using "outpaces" to talk about performance, not market share. AWS remains a gorilla there. Especially if you net out Gsuite and O365.
isbvhodnvemrwvn 6y ago
Most of these threads mention excellent technology, but piss poor product management.

eecks 6y ago

Google, like AWS and Azure, is 'only pay for what you use'. Can anyone tell me if there is a way to put limits in? Or to choose a 20/50/100 dollar per month plan?

isbvhodnvemrwvn 6y ago
Not quite. Please note that there are various things you pay for, let's take a look at several of those: - provisioned resources (instances, including the ones under relational databases or other stuff like that) - usage of on-demand resources like AWS Lambda and API Gateway - infrastructure such as load balancers (kind of mix of provisioned and pay-per-use) - persistent storage (which is all over the place in terms of payment methods) While you could shut down the first three to save some money, would you like to remove your data permanently? So far all mechanisms that are available are mostly reactive (billing alarms etc) rather than proactive (service quotas although they are meant to shield from poor design rather than a typo in terraform). There is clearly incentive for cloud providers for the former, but it's not an easy problem anyway (they might mark your accounts for "training" or something like that - I think this would be reasonable).
corty 6y ago
In GCP you can configure budgets for projects (groups of resources). Budgets can issue alerts when reaching certain percentages and reconfigure or shutdown resources when exceeded. The budget howto docs have an example to stop everything that incurs a cost on budget overrun: https://cloud.google.com/billing/docs/how-to/budgets https://cloud.google.com/billing/docs/how-to/budgets But as the warning says, that might delete data in storage or other resources you may want to keep paying for. For that case, you can execute a program, that shuts down everything you can get rid of, but the cost for storage and everything you forgot will continue to be billed.
akh 6y ago
I've seen many companies user "terminator" scripts that shutdown/delete things that aren't tagged as "keep" or something like that, though only in non-production accounts. The budget alerts from the cloud providers can be useful too (AWS recently released cost anomaly alerts). We're trying to tackle this problem with https://github.com/infracost/infracost https://github.com/infracost/infracost from another angle for people who use Terraform: show a cost estimate in pull requests so the user understands what costs money, and roughly how much it costs. I hope that helps clarify "only pay for what you use" without trawling through cloud pricing pages.

potency 6y ago

Are there any good alternatives to the big three? I'm looking to build out a platform with as little dependence on Google/MSFT/Amazon as possible.

bastawhiz 6y ago
IBM, if you're feeling lucky
thiscatis 6y ago
Oracle Cloud if you want to relive the nineties.
- shiftpgdn 6y ago
  Oracle Cloud has live migration (still not available on EC2, afaik) and had cloud console before AWS.
  - deleted 6y ago
    [deleted]
detaro 6y ago
Depends on what you need. They offer a giant bag of services and abilities, so it really comes down to which subset of that you are looking for.
dharmab 6y ago
If you just need Linux VM and a couple of other basic capabilities, DigitalOcean.
- maccard 6y ago
  Any recommendations for a DO-likd experience that supports windows? As much as I'd like to use DO, it's a show stopper for people who want the simplicity of DO but have a requirement on windows (hello, MSVC)

jrsdav 6y ago

I've done pretty extensive work in all three major cloud providers. If you were to ask me which one I'd use for a net new project, it would be GCP -- no question. Nearly all of their services I've used have been great with a feeling that they were purposefully engineered (BigQuery, GKE, GCE, Cloud Build, Cloud Run, Firebase, GCR, Dataflow, PubSub, Data Proc, Cloud SQL, goes on and on...). Not to mention almost every service has a Cloud API, which really goes a long way towards eliminating the firewall and helps you embrace the Zero Trust/BeyondCorp model. And BigQuery. I can't express enough how amazing BigQuery is. If you're not using GCP, it's worth going multi-cloud for BigQuery alone. But there is something to be said of AWS. Their SDKs are complete and predictable, their APIs are very fast and consistent, and AWS IAM, while having a steep learning curve, never leaves you guessing around what your principals have access to. For me, the real challenge with AWS has been introducing multiple AWS accounts. Governance just flat out sucks when you begin to scale past a handful of accounts (but it is getting better). Azure on the other hand, has terrible consistency issues between their APIs, their SDKs are awful, and it just feels like the entire product is an extension of the MCP System Administrator persona of old, where it's expected that someone's job will be sitting in front of a UI and clicking around to get things done (the whole blade thing with their portal has to be one of the worst user experiences I've ever seen). However, I do like their Logic Apps, and Azure Policy with auto remediation (when it works as advertised -- ref API consistency and how long it takes for things to propagate through their system) has tons of potential. But they still have a ways to go before I'd consider it for my workloads.

dmitriid 6y ago
I can kinda agree on GCP with one exception: Dataflow. I have no idea what the future holds for it. It is a managed Apache Beam service and is very useful for certain scenarios (like "hey, we have a million incoming PubSub messages that we need to transform into a dozen different branching streams of data"). It looks like even BigQuery actually transforms SQL statements into a bunch of Dataflow jobs. But... But... - Minor version updates to Google Dataflow SDK once every couple of months while deprecating most other minor versions? Check. - No visible contributions to Apache BEAM itself? Check. In 2021 I still don't know if I can use any Java versions beyond Java 8 to develop for and run in Dataflow. And Google is arguably one of the biggest users of Apache BEAm, and definitely a user with the largest pile of money to throw at the problem. - They've recently sent out a questionnaire about Dataflow to some of their customers that feels like a "hey, we're definitely considering deprecating this, we're gauging the potential impact"

ashtonkem 6y ago

The big problem for me is trust. I don’t care what the feature set or performance is; I don’t trust Google enough to bet a business on it. And I’m not even worried about Google being malicious; I’m worried about them being mercurial and changing/removing things I need without warning.

bnt 6y ago
Or some ML crob job algo going haywire and closing your account - with no human to contact to get it resolved.
- SteveNuts 6y ago
  Google's services are so perfect they don't even need to invest in customer service! If you have a problem with anything, it's clearly your fault and you're doing it wrong. /s
  - ashtonkem 6y ago
    Sarcasm aside, at $COMPANY we have AWS reps integrated into our Slack, and they work hand in hand with our engineers quite often. Even for things as low down as “why is this query so slow on AuroraDB?” They even hop into war rooms for big events in case we need immediate assistance during high visibility outages. It’s hard to overstate how important this level of close support is for an enterprise that is literally betting the farm on a cloud provider. The idea of counting on Google’s historical level of support is an absolute non-starter for us.
    - theshrike79 6y ago
      Agree, we have an integrated AWS contact too and they're really invaluable for getting insight on what the actual limits are on their services.
    - 0xEFF 6y ago
      The role you describe is a technical account manager and Google has hired many. I’m a Google cloud partner and every project I’ve worked on has had multiple TAMS doing exactly what you describe, working hand in hand with customers, connecting the customer to product engineers, to support, navigating Black Friday and Christmas, etc...

me551ah 6y ago

I evaluated AWS and GCP for my startup and found GCP to be more expensive. The horror stories I've read about Google's lack of customer support put me off too.

philshem 6y ago
Many bootstrapped startups go with the cloud provider that gives the them largest startup bonus, which can be up to $100k to use in 12 months (AFAIK).
- PedroBatista 6y ago
  and then they spend the next year migrating to a "more realistic" deployment or burn half their money on cloud computing.
deadmutex 6y ago
Out of curiosity, was there one cost that was significantly more? or was it across multiple services? Do you mind sharing your setup a bit more?

marcinzm 6y ago

The main reason I'm weary of recommending GCP is the support horror stories that keep coming up. I'm using it at work now since our massive Google Ad spend protects us from that. It's got some really good technologies although there's various rough edges. One thing that really irks me is GCP requiring me to talk to sales people (not support, sales) to have a relatively small quota increase. Why would they make it harder for me to give them money?

frabjoused 6y ago

This article screams bias, beginning with the title. I don't know how to give this credit.

StreamBright 6y ago

>> GCP Outpaces Azure, AWS in the 2021 Cloud Report (cockroachlabs.com) >> AWS network latencies are unbeatable Seems weird to see these sentences on the same site.

lawrjone 6y ago
To be fair, network latency across all the providers is reasonably similar. The network throughput was not even close though, with GCP winning by a huge margin. I'd say both statements are fair and not mutually exclusive.
- arulajmani 6y ago
  One of the engineers who helped benchmark and compile the Cloud Report. You're right in noting that the statements aren't mutually exclusive. Our overall takeaway was informed by each Cloud's performance on the individual benchmarks as shown on page 4 of the report. The results were quite close though, as each of the Clouds had specific benchmarks they did excellent in.

deleted 6y ago

[deleted]

herdcall 6y ago

CockroachDB was founded by ex-Google employees and is partly funded by Google Ventures, so I'd take this report with a pinch of salt. IMO GCP is good for PoC/personal projects due to their liberal free tier quotas, but I don't know about going big. Anyone with large scale experience on GCP?

phillipcarter 6y ago
Yeah, there needs to be a disclaimer in these reports about this kind of stuff. Not that I doubt the technical accuracy of the claims, but it's just good form to make these kinds of notices.
orangechairs 6y ago
Report author here. We have no bias towards nor stake in any of the three cloud providers. We partnered with all three clouds to develop the testing methodology and benchmark set. Our bias is towards providing as much information as possible to our own customers as they select their clouds and machines. Reproduction are available on github: https://github.com/cockroachlabs/cloud-report-2021 https://github.com/cockroachlabs/cloud-report-2021

etxm 6y ago

GCP wins hands down when it comes to cloud governance and network design. I think the two biggest weaknesses are: - IAM - some resources have awkward relationships with IAM; although the GSuite integration is nice - CloudSQL (vs RDS) - for businesses that need relational data stores, but aren’t at the Cloud Spanner scale, RDS blows CloudSQL away in features

mixmastamyk 6y ago
Governance? What are the necessary features cloudsql doesn’t support?
- etxm 6y ago
  Project organization, folders, orgs, and it’s integration with IAM / GSuite. I haven’t dug in on Cloud SQL in about a year, what I recall: - no support for logical replication in PG - there was some VPC funkiness - fewer number of extensions: ~60 extensions in RDS vs ~45 for CloudSQL - CSQL Only supports pl/pgsql vs perl, v8, tcl, and pgsql

rebelos 6y ago

I don't quite understand why, but much of the tech industry seems to be sleeping on Cloud Spanner. Google quietly completely revolutionized managed+consistent+available+scalable RDBMS and very few people seem to have caught on yet. Maybe it's too much of a threat to job security?

jahewson 6y ago
It’s way too expensive. Plus I wouldn’t want to deal with the limitations and latency that its consistency model brings unless I had truly Google-scale data, which I don’t.
blaisio 6y ago
I agree spanner and cockroach are the future. Most people don't need a database that can scale that well, and spanner is too expensive to use unless you really truly need it. Also, google and cockroach have not done enough marketing. Look at all the marketing mongodb did - they actually managed to convince people to use a database that would regularly lose data.
- rebelos 6y ago
  You can get 3 nodes for about $27k a year and that'll handle 30k read QPS and in the neighborhood 1-2k write QPS iirc. It's a fraction of the cost of even a single dedicated engineer. And you'd probably need several engineers to achieve the same perf with open source alternatives and keep it stable/upright. There's a large class of businesses for which this choice is a no-brainer.
  - manigandham 6y ago
    Is Cloud Spanner the only managed option? Why would you compare to a dedicated engineer? AWS RDS or GCP Cloud SQL or Azure Managed SQL or IBM Compose or Aiven or any number of other vendors offer managed databases with more features, much higher performance, and far less cost. Even CRDB has its own cloud offering that's cheaper and more flexible than Spanner.

alfl 6y ago

I have substantial concerns running my core infra on Google products: deprecation, inhuman support, the allegations of anticompetitive behaviour in the states’ antitrust lawsuit. Might be good tech. Business risk seems high.

sknat 6y ago

I'm a bit dubious about the networking results they present. I did some quite extensive network performence testing last winter on those three CSP, and even if single queue TCP+gso performence can behave like this, I find the claim 'GCP is 3x faster than AWS' a bit bold. It's definitely possible to get 50G of TCP traffic in AWS, and a lot of things are in the balance (MTU, number of queues, drivers...) that make this claim a bit weird to me.

arulajmani 6y ago
One of the engineers who helped run benchmarks and compile the report here. It’s worth noting that for the majority of the machines we benchmarked on AWS, their tested bandwidth met the published AWS expectations. You may have noticed that some of the “network optimized” machines fell short of the published expectations though, and there’s an explanation in the report about how we tried to validate our findings. As you point out, there are a variety of variables that could be tuned to eek out better performance here, and they could bring the two clouds closer. Our claim, of course, only applies to the benchmark configuration we tested with. That being said, with the size of machine we were restricting our testing to (16 vCPUS), no AWS machine claimed to offer more than 25G of throughput.
- talawahtech 6y ago
  All the 16 vCPU "n" instances (m5n, c5n, r5n, etc) are capable of hitting the 25 Gbps limit easily. In your report, all of the AWS results are limited to either 5Gbps or 10Gbps, but this is because of a very specific test condition. From my understanding of the test scenario, you are using a single TCP connection to run the throughput test, and hitting AWS' documented[1] throughput limit for a single flow: 10 Gbps if the two instances are in the same placement group and 5 Gbps otherwise. The reason some of the network-optimized instances were "slower" than the non-optimized ones is most likely a random draw of whether both instances in the test were physically close to each other (basically whether or not they are accidentally in placement groups). To show the true throughput you would need to use multiple connections/flows, 5-10 would probably suffice. If the single flow test case was important then maybe you should have mentioned that AWS has a specific limitation around this. Personally I don't think a single flow test case is particularly realistic for a throughput test. Either way, how it is presented is pretty misleading. 1. "Single TCP flow is limited to 10 Gbps for instances in the same placement group and 5 Gbps between instances anywhere else." https://docs.aws.amazon.com/whitepapers/latest/ec2-networking-for-telecom/overall-instance-bandwidth-limitations.html https://docs.aws.amazon.com/whitepapers/latest/ec2-networkin...

gundmc 6y ago

The network throughout is eye opening given how close most of the other benchmarks are. GCP's lowest performer is >50% higher than AWS's top performer and more than double Azure's best.

talawahtech 6y ago
The network throughput benchmark is pretty misleading. https://news.ycombinator.com/item?id=25816596 https://news.ycombinator.com/item?id=25816596

einszwei 6y ago

As someone who has worked on large-scale deployments on both AWS and GCP, I would always prefer AWS over GCP. While GCP products are IMO superior to similar AWS offerings their support (even premium tier) is total garbage compared to AWS.

znpy 6y ago

the website requires providing personal data including an email address to access the full report. I recommend using an email @cockroachlabs.com so that they can get spammed by their own marketing bs (besides the report). You will be directed to the download page anyway.

anthony_barker 6y ago

I have done projects on all except Azure (mostly due to a Microsoft aversion). I hate their special names for everything. Reminds me of Starbucks. Here is my take: GCP tools are better PubSub, CloudSQL. However they don't support email and their docs are not as up to date and helpful as AWS. I think the main reason to select the big three is a) security (network, instance, user management) b) you don't get fired for selecting the big guys c) some specialized tools (SES, s3, CDNs, github) I always feel that the time you invest to learn all the details of AWS you could have invested into Ansible, Docker, Wireguard, Iptables, zfs and linux, and deploy a much more cost effective solution on Heztner, (which I prefer over upcloud, do, vultr). But you need to know what you are doing. Many companies prefer to trust a vendor instead of their employees.

talawahtech 6y ago

There are number of issues with this report. The AWS networking section is particularly problematic and in need of extensive disclaimers or changes to the test methodology. On the throughput side, all this test does is demonstrate the documented[1] throughput limit for a single TCP connection. 10 Gbps if the two instances are in the same placement group and 5 Gbps otherwise. The reason some of the network-optimized instances were "slower" than the non-optimized ones was because it was simply a random draw of whether both instances in the test were physically close to each other. If they wanted to do a proper throughput test they would have used placement groups and multiple connections/flows. If they felt like the single flow test case was important they should have mentioned that AWS has a specific limitation around this. Personally I don't think a single flow test case is particularly realistic. The fact that the obvious discrepancy between their results and the documented (multi-flow) limits didn't cause them to dig deeper is enough to make me very skeptical of the purpose of this paper. The latency results are also basically a random spread. It is essentially distribution of all the different latencies you might randomly get between two instances if you don't use placement groups. It says absolutely nothing about the networking capabilities of different instances used in each test. 1. "Single TCP flow is limited to 10 Gbps for instances in the same placement group and 5 Gbps between instances anywhere else." https://docs.aws.amazon.com/whitepapers/latest/ec2-networking-for-telecom/overall-instance-bandwidth-limitations.html https://docs.aws.amazon.com/whitepapers/latest/ec2-networkin...

xiangy 6y ago

I think enabling InfiniBand in Azure brings much better Network performances (but not for all kinds of VMs) https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/hpc/enable-infiniband https://docs.microsoft.com/en-us/azure/virtual-machines/work...