18 ms·

AWS vs. GCP reliability is wildly different

545 points by icyfox 4y ago

rwalle 4y ago
Looks like the author has never heard of the word "histogram" That graph is a pain to see.
- DangitBobby 4y ago
  A histogram would take away one of the dimensions, probably time, unless they resorted to some weird stacked layout. Without time, people would complain that they don't know if it was consistent across the tested period. The graph is fine.
s-xyz 4y ago
Would be interested to see a comparison of lambda functions vs google 2nd gen functions. I think that gcp is more serverless focused
duskwuff 4y ago
... why does the first graph show some instances as having a negative launch time? Is that meant to indicate errors, or has GCP started preemptively launching instances to anticipate requests?
- zaltekk 4y ago
  I don't know how that value (looks like -50?) was chosen, but it seems to correspond to the launch failures.
- tra3 4y ago
  The y axis here measures duration that it took to successfully spin up the box, where negative results were requests that timed out after 200 seconds. The results are pretty staggering
- staringback 4y ago
  Perhaps if you read the line directly about the graph you would see it was explained and would not have to ask this question
endisneigh 4y ago
this doesn't really seem like a fair comparison, nor is it a measure of "reliability".
- daneel_w 4y ago
  It seems entirely fair to me, but the term "reliability" has a few different angles. This time it's not about working or not working, but the ability to auto-scale by invoking resources on the spot, which can be a very real requirement.
  - endisneigh 4y ago
    unless you're willing to burn $150 a quarter doing this exact assessment, it tells you nothing other than the data center conditions at the time of running. it would be like doing this in us-central1 when us-central1 is down for one provider, and not another, resulting in increased latency, and saying how much faster one is than the other. unlike say a throughput test or similar, neither of these services promise particular cold-starts, and so the numbers here cannot be contexutalized against any metric given by either company and so are only useful in the sense that they can be compared, but since there are no guarantees the positions could switch anytime. that's why I like comparisons between serverless functions where there are pretty explicit SLAs and what not given by each company for you to compare against, as well as one another.
    - daneel_w 4y ago
      Given the stark contrast and that the pattern was identical every day over a two-week course, it tells me we're observing a fundamental systemic difference between GCP and AWS - and I think that's all the author really wanted to point out. I would not be surprised if the results are replicable three months from now.
AtNightWeCode 4y ago
This benchmark (too) is probably incorrect. It produces 409:s so there are errors in there that I doubt are caused by GCP.
playingalong 4y ago
This is great. I have always been feeling there is so little independent content on benchmarking the IaaS providers. There is so much you can measure in how they behave.
politelemon 4y ago
A few weeks ago I needed to change the volume type on an EC2 instance to gp3. Following the instructions, the change happened while the instance was running. I didn't need to reboot or stop the instance, it just changed the type. While the instance was running. I didn't understand how they were able to do this, I had thought volume types mapped to hardware clusters of some kind. And since I didn't understand, I wasn't able to distinguish it from magic.
- ArchOversight 4y ago
  Changing the volume type on AWS is somewhat magical. Seeing it happens on-line was amazing.
- xyzzyz 4y ago
  Dunno about AWS, but GCP uses live migration, and will migrate your VM across physical machines as necessary. The disk volumes are all connected over the network, nothing really depends on the actual physical machine your VM is ran on.
  - lbhdc 4y ago
    How does migrating a vm to another physical machine work?
    - rejectfinite 4y ago
      vsphere vmotion has been a thing for years lmao
    - roomey 4y ago
      VMware has been doing this for years, it's called vmotion and there is a lot of documentation about it if you are interested (eg https://www.thegeekpub.com/8407/how-vmotion-works/ https://www.thegeekpub.com/8407/how-vmotion-works/ ) Essential, memory state is copied to the new host, the VM is stunned for a millisecond and the cpu states is copied and resumed on the new host (you may see a dropped ping). All the networking and storage is virtual anyway so that is "moved" (it's not really moved) in the background.
PigiVinci83 4y ago
Thank you for this article, it confirms my direct experience. Never run a benchmarking test but I can see this every day.
MonkeyMalarky 4y ago
I would love to see the same for deploying things like a cloud/lambda function.
devxpy 4y ago
Is this testing for spot instances? In my limited experience, persistent (on-demand) GCP instances always boot up much faster than AWS EC2 instances.
- encryptluks2 4y ago
  I noticed that too and it does appear to be using spot instances. I have a feeling if it was ran without you may see much better startup times. Spot instances on GCP are hit and miss and you sort of have to build that into your workflow.
- marcinzm 4y ago
  In my experience GPU persistent instances often simply don't boot up on GCP due to lack of available GPUs. One reason I didn't choose GCP at my last company.
  - devxpy 4y ago
    Oh interesting. Which region and GPU type were you working with? (Asking so I can avoid in future)
    - marcinzm 4y ago
      I think it was us-east1 or us-east4. Had issues getting TPUs as well in us-central1. I know someone at a larger tech company who was told to only run certain workflows in a specific niche European region as that's the only one that had any A100 GPUs most of the time.
ajross 4y ago
Worth pointing out that the article is measuring provisioning latency and success rates (how quickly can you get a GPU box running and whether or not you get an error back from the API when you try), and not "reliability" as most readers would understand it (how likely they are to do what you want them to do without failure). Definitely seems like interesting info, though.
amaks 4y ago
The link is broken?
- lucb1e 4y ago
  Works for me using Firefox in Germany, although the article doesn't really match the title so maybe that's why you were confused? :p
danielmarkbruce 4y ago
It's meant to say "ephemeral"... right? It's hard to read after that.
- datalopers 4y ago
  ephemeral and ethereal are commonly confused words.
  - danielmarkbruce 4y ago
    I guess that's fair. It's sort of a smell when someone uses the wrong word (especially in writing) though. It suggests they aren't in industry, throwing ideas around with other folks. The word "ephemeral" is used extensively amongst software engineers.
  - dublin 4y ago
    Ephimerides really throws them. (And thank God for PyEphem, which makes all that otherwise quite fiddly stuff really easy...)
dark-star 4y ago
I wonder why someone would equate "instance launch time" with "reliability"... I won't go as far as calling it "clickbait" but wouldn't some other noun ("startup performance is wildly different") have made more sense?
- xmonkee 4y ago
  GCP also had 84 errors compared to 1 for AWS
  - danielmarkbruce 4y ago
    If not a 4xx, what should they return for instance not available?
    - eurasiantiger 4y ago
      503 service unavailable?
      - dheera 4y ago
        Using HTTP error codes for non-REST things is cringe. 503 would mean the IaaS API calls themselves are unavailable. Very different from the API working perfectly fine but the instances not being available.
        devmunchies 4y ago
        What? REST is just some API philosophy, its doesn't even have to be on top of HTTP. Why would you think HTTP status codes are made for REST? They are made for HTTP to describe the response of the resource you are requesting, and the AWS API uses HTTP so it makes sense to use HTTP status codes.
        eurasiantiger 4y ago
user- 4y ago
I wouldn't call this reliability, which already has a loaded definition in the cloud world, and instead something along time-to-start or latency or something.
- systemvoltage 4y ago
  It is though based on a specific definition. If X doesn't do Y based on Z metric with a large standard deviation and doesn't meet spec limits, it is not reliable as per the predefined tolerance T. X = Compute intances Y = Launch Z = Time to launch T = LSL (N/A), USL (10s), Std Dev (2s) Where LSL is lower spec limit, USL is upper spec limit. LSL is N/A since we don't care if the instance launches instantly (0 seconds). You can define T as per your requirements. Here we are ignoring the accuracy of the clock that measures time, assuming that the measurement device is infinitely accurate. If your criteria is to, say for example, define reliability as how fast it shuts down, then this article isn't relevant. Article is pretty narrow in testing reliability, they only care about launch time.
remus 4y ago
> The offerings between the two cloud vendors are also not the same, which might relate to their differing response times. GCP allows you to attach a GPU to an arbitrary VM as a hardware accelerator - you can separately configure quantity of the CPUs as needed. AWS only provisions defined VMs that have GPUs attached - the g4dn.x series of hardware here. Each of these instances are fixed in their CPU allocation, so if you want one particular varietal of GPU you are stuck with the associated CPU configuration. At a surface level, the above (from the article) seems like a pretty straightforward explanation? GCP gives you more flexibility in configuring GPU instances at the trade off of increased startup time variability.
- btgeekboy 4y ago
  I wouldn't be surprised if GCP has GPUs scattered throughout the datacenter. If you happen to want to attach one, it has to find one for you to use - potentially live migrating your instance or someone else's so that it can connect them. It'd explain the massive variability between launch times.
  - master_crab 4y ago
    Yeah that was my thought too when I first read the blurb. It’s neat…but like a lot of things in large scale operations, the devil is in the details. GPU-CPU communications is a low latency high bandwidth operation. Not something you can trivially do over standard TCP. GCP offering something like that without the ability to flawlessly migrate the VM or procure enough “local” GPUs means it’s just vaporware. As a side note, I’m surprised the author didn’t note the amount of ICE’s (insufficient capacity errors) AWS throws whenever you spin up a G type instance. AWS is notorious for offering very few G’s and P’s is certain AZs and regions.
    - my123 4y ago
      Fungible is selling a GPU decoupling solution via PCIe encapsulated over Ethernet today, so it can certainly be done. And NVIDIA's vGPU solutions do support live migration of GPUs to another host (in which case the vGPU gets moved too, to a GPU on that target).
kccqzy 4y ago
Heard from a Googler that the internal infrastructure (Borg) is simply not optimized for quick startup. Launching a new Borg job often takes multiple minutes before the job runs. Not surprising at all.
- epberry 4y ago
  Echoing this. The SRE book is also highly revealing about how Google request prioritization works. https://sre.google/sre-book/load-balancing-datacenter/ https://sre.google/sre-book/load-balancing-datacenter/ My personal opinion is that Google's resources are more tightly optimized than AWS and they may try to find the 99% best allocation versus the 95% best allocation on AWS.. and this leads to more rejected requests. Open to being wrong on this.
- valleyjo 4y ago
  As another comment points out, GPU resources are less common so it takes longer to create, which makes sense. In general, start up times are pretty quick on GCP as other comments also confirm.
- dekhn 4y ago
  booting VMs != starting a borg job.
  - kccqzy 4y ago
    The technology may be different but the culture carries over. People simply don't have the habit to optimize for startup time.
- readams 4y ago
  Borg is not used for gcp vms, though.
  - dilyevsky 4y ago
    It is used but borg scheduler does not manage vm startups
- 4y ago
londons_explore 4y ago
AWS normally has machines sitting idle just waiting for you to use. Thats why they can get you going in a couple of seconds. GCP on the other hand fills all machines with background jobs. When you want a machine, they need to terminate a background job to make room for you. That background job has a shutdown grace time. Usually thats 30 seconds. Sometimes, to prevent fragmentation, they actually need to shuffle around many other users to give you the perfect slot - and some of those jobs have start-new-before-stop-old semantics - that's why sometimes the delay is far higher too.
- dekhn 4y ago
  borg implements preemption but the delay to start VMs is not because they are waiting for a background task to clean up.
deleted 4y ago
[deleted]
zmmmmm 4y ago
> In total it scaled up about 3,000 T4 GPUs per platform > why I burned $150 on GPUs How do you rent 3000 GPUs over a period of weeks for $150? Were they literally requisitioning it and releasing it immediately? Seems like this is quite a unrealistic type of usage pattern and would depend a lot on whether the cloud provider optimises to hand you back the same warm instance you just relinquished. > GCP allows you to attach a GPU to an arbitrary VM as a hardware accelerator it's quite fascinating that GCP can do this. GPUs are physical things (!) do they provision every single instance type in the data center with GPUs? That would seem very expensive.
- bushbaba 4y ago
  Unlikely. More likely they put your VM on a host with GPU attached, and use live migration to move workloads around for better resource utilization. However, live-migration can cause impact to HPC workloads.
- ZiiS 4y ago
  GPUs are physical but VMs are not; I expect they just move them to a host with a GPU.
- NavinF 4y ago
  It probably live-migrates your VM to a physical machine that has a GPU available. ...if there are any GPUs available in the AZ that is. I had a hell of a time last year moving back and forth between regions to grab just 1 GPU to test something. The web UI didn't have a "any region" option for launching VMs so if you don't use the API you'll have to sit there for 20 minutes trying each AZ/region until you managed to grab one.
- geysersam 4y ago
  > $150 Was asking myself the same question. From the pricing information on gcp it seems minimum billing time is 1 minute, making 3000 GPUs cost $50 minimum. If this is the case then $150 is reasonable for the kind of usage pattern you describe.
dekhn 4y ago
What would you expect? AWS is an org dedicated to giving customers what they want and charging them for it, while GCP is an org dedicated to telling customers what they want and using the revenue to get slightly better cost margins on Intel servers.
- dilyevsky 4y ago
  I don’t believe this reasoning is used since at least Diane
  - dekhn 4y ago
    I haven't seen any real change from Google about how they approach cloud in the past decade (first as an employee and developer of cloud services there, and now as a customer). Their sales people have hollow eyes
    - dilyevsky 4y ago
      Didn’t say they were good at making the switch
- DangitBobby 4y ago
  They don't really tell us what we want, we just buy what we need. Might work for you.
cmcconomy 4y ago
I wish Azure was here to round it out!
charbull 4y ago
Can you put this in context of the problem/use case /need you are solving for ?
0xbadcafebee 4y ago
Reliability in general is measured on the basic principle of: does it function within our defined expectations? As long as it's launching, and it eventually responds within SLA/SLO limits, and on failure comes back within SLA/SLO limits, it is reliable. Even with GCP's multiple failures to launch, that may still be considered "reliable" within their SLA. If both AWS and GCP had the same SLA, and one did better than the other at starting up, you could say one is more performant than the other, but you couldn't say it's more reliable if they are both meeting the SLA. It's easy to look at something that never goes down and say "that is more reliable", but it might have been pure chance that it never went down. Always read the fine print, and don't expect anything better than what they guarantee.
1-6 4y ago
This is all about cloud GPUs, I was expecting something totally different from the title.
humanfromearth 4y ago
We have constant autoscaling issues because of this in GCP - glad someone plotted this - hope people in GCP will pay a bit more attention to this. Thanks to the OP!
outworlder 4y ago
Unclear what the article has to do with reliability. Yes, spinning up machines on GCP is incredibly fast and has always been. AWS is decent. Azure feels like I'm starting a Boeing 747 instead of a VM. However, there's one aspect where GCP is a clear winner on the reliability front. They auto-migrate instances transparently and with close to zero impact to workloads – I want to say zero impact but it's not technically zero. In comparison, in AWS you need to stop/start your instance yourself so that it will move to another hypervisor(depending on the actual issue AWS may do it for you). That definitely has impact on your workloads. We can sometimes architect around it but there's still something to worry about. Given the number of instances we run, we have multiple machines to deal with weekly. We get all these 'scheduled maintenance' events (which sometimes aren't really all that scheduled), with some instance IDs(they don't even bother sending the name tag), and we have to deal with that. I already thought stop/start was an improvement on tech at the time (Openstack, for example, or even VMWare) just because we don't have to think about hypervisors, we don't have to know, we don't care. We don't have to ask for migrations to be performed, hypervisors are pretty much stateless. However, on GCP? We had to stop/start instances exactly zero times, out of the thousands we run and have been running for years. We can see auto-migration events when we bother checking the logs. Otherwise, we don't even notice the migration happened. It's pretty old tech too: https://cloudplatform.googleblog.com/2015/03/Google-Compute-Engine-uses-Live-Migration-technology-to-service-infrastructure-without-application-downtime.html https://cloudplatform.googleblog.com/2015/03/Google-Compute-...
- willcipriano 4y ago
  I always wondered why you couldn't do that on AWS, mainly because I could do it at home with Hyper-V a decade ago. https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/microsoft-hyper-v-server-2008-R2/ee815293(v=ws.10) https://learn.microsoft.com/en-us/previous-versions/windows/...
- voidfunc 4y ago
  > Azure feels like I'm starting a Boeing 747 instead of a VM. Huh... interesting, this has not been my experience with Azure VM launch times. I'm usually surprised how quickly they pop up.
deleted 4y ago
[deleted]
lacker 4y ago
Anecdotally I tend to agree with the author. But this really isn't a great way of comparing cloud services. The fundamental problem with cloud reliability is that it depends on a lot of stuff that's out of your control, that you have no visibility into. I have had services running happily on AWS with no errors, and the next month without changing anything they fail all the time. Why? Well, we look into it and it turns out AWS changed something behind the scenes. There's a different underlying hardware behind the instance, or some resource started being in high demand because of some other customers. So, I completely believe that at the time of this test, this particular API was performing a lot better on AWS than on GCP. But I wouldn't count on it still performing this way a month later. Cloud services aren't like a piece of dedicated hardware where you test it one month, and then the next month it behaves roughly the same. They are changing a lot of stuff that you can't see.
- citizenpaul 4y ago
  That was my thoughts. People are probably pummeling GCP GPU free tier right now with stable diffusion image generators. Since it seems like all the free plug and play examples use the google python notebooks.
- RajT88 4y ago
  Instance types and regions make a big difference. Some regions and hardware generations are just busier than others. It may not be the same across cloud providers (although I suspect it is similar given the underlying market forces).
- ryukoposting 4y ago
  You've just perfectly characterized why on-site infrastructure will always have its place.
  - callalex 4y ago
    You can reserve capacity on both of these services as well.
herpderperator 4y ago
The author is using 'Quantile' which I hadn't heard of before, and when I did, it seems like it actually should be 'Percentile'. Percentiles are the percentages, which is what the author is referring to.
- mr_toad 4y ago
  Quantiles are a generic term for percentiles, deciles, quartiles etc. Percentiles would have been a more precise term.
rwiggins 4y ago
There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can have weird boundary cases, particularly because (1) it does cleanup after creation, which will have variable latency, (2) `int()` will truncate the fractional part of the second from `time()`, and (3) `time.time()` is not monotonic. I would not ask the author to spend money to test it again, but I think the 409s would probably disappear if you replaced `int(time())` with `uuid.uuid4()`. Disclosure: I work at Google - on Google Compute Engine. :-)
- Cthulhu_ 4y ago
  I've naively used millisecond precision things for a long time - not in anything critical I don't think - but I've only recently come to more of an awareness that a millisecond is a pretty long time. Recent example is that I used a timestamp to version a record in a database, but it's feasible that in a Go application, a record could feasilby be mutated multiple times a millisecond by different users / processes / requests. Unfortunately, millisecond-precise timestamps proved to be a bit tricky in combination with sqlite.
  - account42 4y ago
    To put this into perspective, a game (or anything) running at 60 FPS only as a bit over 16 milliseconds to render each frame. These days higher frame rates are common enough, often putting you down into single-digit milliseconds per frame. Not think about how many things there are simulated and rendered in each frame for common games. Way more than 16.
- sitharus 4y ago
  This is a very good point - AWS uses tags to give instances a friendly name, so the name does not have to be unique. The same logic would not fail on AWS.
kazinator 4y ago
> This is particularly true for GPUs, which are uniquely squeezed by COVID shutdowns, POW mining, and growing deep learning models Is the POW mining part true any more? Hasn't mining moved to dedicated hardware?
- CodesInChaos 4y ago
  Bitcoin mining has used dedicated hardware for a long time. But I believe Ethereum mining used GPUs before the very recent proof-of-stake update.
Animats 4y ago
> GCP allows you to attach a GPU to an arbitrary VM as a hardware accelerator - you can separately configure quantity of the CPUs as needed. That would seem to indicate that asking for a VM on GCP gets you a minimally configured VM on basic hardware, and then it gets migrated to something bigger if you ask for more resources. Is that correct? That could make sense if, much of the time, users get a VM and spend a lot of time loading and initializing stuff, then migrate to bigger hardware to crunch.
- zylent 4y ago
  This is not quite true - GPU's are limited to select VM types, and the number of GPU's you have influences the maximum number of cores you can get. In general they're only available on the n1 instances (except the a100's, but those are far less popular)
orf 4y ago
AWS has different pools of EC2 instances depending on the customer, the size of the account and any reservations you may have. Spawning a single GPU at varying times is nothing. Try spawning more than one, or using spot instances, and you’ll get a very different picture. We often run into capacity issues with GPU and even the new m6i instances at all times of the day. Very few realistic company size workloads need a single GPU. I would willingly wait 30 minutes for my instances to become available if it meant all of them where available at the same time.
Jamie9912 4y ago
Should probably change the title to "AWS vs GCP on-demand GPU launch time consistency"
- Terretta 4y ago
  Yep. Author colloquially meant, can I rely on a quick start.
mnutt 4y ago
It may or may not matter for various use cases, but the EC2 instances in the test use EBS and the AMIs are lazily loaded from S3 on boot. So it may be possible that the boot process touches few files and quickly gets to 'ready' state, but you may have crummy performance for a while in some cases. I haven't used GCP much, but maybe they load the image onto the node prior to launch, accounting for some of the launch time difference?
lmeyerov 4y ago
I'd say that's a weak test of capacity. Would love to see on Azure - T4s or an equiv aren't even really provided anymore! We find reliability a diff story. Eg, our main source of downtime on Azure is they restart (live migrate?) our reserved T4s every few weeks, causing 2-10min outages per GPU per month.
lomkju 4y ago
Having being a high scale AWS user with a bill of +$1M/month and now working since 2 years with a company which uses GCP. I would say AWS is superior and way ahead. ** NOTE: If you're a low scale company this won't matter to you ** 1. GKE When you cross a certain scale certain GKE components won't scale with you and SLOs on those components are crazy, it takes 15+ mins for us to update a GKE ingress controller backed Ingress. Cloud Logging hasn't been able to keep up with our scale, disabled since 2 years now. This last Q we got an email from them to enable it and try it again on our clusters, still have to confirm these claims as our scale is more higher now. Konnectivity agent release was really bad for us, it affected some components internally, total dev time we lost was more than 3 months debugging this issue. They had to disable konnectivity agent on our clusters, I had to collect TCP dumps and other evidences just to prove nothing was wrong on our end, fight with our TAM to get a meeting with the product team. After 4 months they agreed and reverted our clusters to SSH tunnels. Initially GCP support said they said they can't do this. Next Q Ill be updating the clusters hopefully they have fixed this by then. 2. Support. I think AWS support always were more pro active in debugging with us, GCP support agents most of the times lack the expertise or proactiveness to debug/solve things in simple cases. We pay for enterprise support and don't see getting much from them. At AWS we had reviews of the infra how we could better it every 2 Qs and we got new suggestion and was also the time when we shared what we would like to see in their roadmap. 3.Enterprisyness is missing with design A simple thing as cloudbuild doesn't have access to static IPs. We have to maintain a forward proxy just cause of this. L4 LBs were a mess you could only use specified ports in a (L4 LB) TCP proxy, For a tcp proxy based loadbalancer, the allowed set of ports are - [25, 43, 110, 143, 195, 443, 465, 587, 700, 993, 995, 1883, 3389, 5222, 5432, 5671, 5672, 5900, 5901, 6379, 8085, 8099, 9092, 9200, and 9300]. Today I see they have removed these restrictions. I don't know who came up with this idea to allow only a few ports on a L4 LB. I think such design decisions make it less Enterprisy.
jupp0r 4y ago
That's interesting but not what I expected when I read "reliability". I would have expected SLO metrics like uptime of the network or similar metrics that users would care about more. Usually when scaling a system that's built well you don't have hard short constraints on how fast an instance needs to be spun up. If you are unable to spin up any that can be problematic of course. Ideally this is all automated so nobody would care much about whether it takes a retry or 30s longer to create an instance. If this is important to you, you have other problems.
TheMagicHorsey 4y ago
This is not reliability. This is a measure of how much spare capacity AWS seems to be leaving idle for you to snatch on-demand. This is going to vary a lot based on the time of year. Why don't you try this same experiment at around some time when there's a lot of retail sales activity (Black Friday), and watch AWS suddenly have much less capacity to dole out on-demand. To me reliability is a measure of what a cloud does compared to what it says it will do. GCP is not promissing you on-demand instances instantaneously is it? If you want that ... reserve capacity.
runeks 4y ago
> These differences are so extreme they made me double check the process. Are the "states" of completion different between the two clouds? Is an AWS "Ready" premature compared to GCP? It anecdotally appears not; I was able to ssh into an instance right after AWS became ready, and it took as long as GCP indicated before I was able to login to one of theirs. This is a good point and should be part of the test: after launching, SSH into the machine and run a trivial task to confirm that the hardware works.
vienarr 4y ago
The article only talks about GPU start time, but the title is "CloudA vs CloudB reliability" bit of a stretch, right
DonHopkins 4y ago
Anybody know if on GCP the cheaper ephemeral spot instances are available on managed instance groups and cloud run, where it spins up more instances according to demand, and how well it deals with replacing spot instances that drop dead, if so? How about AWS?
jqpabc123 4y ago
Thanks for the report. It only confirms my judgment. The word "Google" attached to anything is a strong indicator that you should look for an alternative.
johndfsgdgdfg 4y ago
It's not surprising. Amazon is an amazing customer focus company. Google is a spyware company that only wants to make more by invading our privacy. Of course Amazon products will be better than Google.

545 points by icyfox 4y ago

rwalle 4y ago

Looks like the author has never heard of the word "histogram" That graph is a pain to see.

DangitBobby 4y ago
A histogram would take away one of the dimensions, probably time, unless they resorted to some weird stacked layout. Without time, people would complain that they don't know if it was consistent across the tested period. The graph is fine.

s-xyz 4y ago

Would be interested to see a comparison of lambda functions vs google 2nd gen functions. I think that gcp is more serverless focused

duskwuff 4y ago

... why does the first graph show some instances as having a negative launch time? Is that meant to indicate errors, or has GCP started preemptively launching instances to anticipate requests?

zaltekk 4y ago
I don't know how that value (looks like -50?) was chosen, but it seems to correspond to the launch failures.
tra3 4y ago
The y axis here measures duration that it took to successfully spin up the box, where negative results were requests that timed out after 200 seconds. The results are pretty staggering
staringback 4y ago
Perhaps if you read the line directly about the graph you would see it was explained and would not have to ask this question

endisneigh 4y ago

this doesn't really seem like a fair comparison, nor is it a measure of "reliability".

daneel_w 4y ago
It seems entirely fair to me, but the term "reliability" has a few different angles. This time it's not about working or not working, but the ability to auto-scale by invoking resources on the spot, which can be a very real requirement.
- endisneigh 4y ago
  unless you're willing to burn $150 a quarter doing this exact assessment, it tells you nothing other than the data center conditions at the time of running. it would be like doing this in us-central1 when us-central1 is down for one provider, and not another, resulting in increased latency, and saying how much faster one is than the other. unlike say a throughput test or similar, neither of these services promise particular cold-starts, and so the numbers here cannot be contexutalized against any metric given by either company and so are only useful in the sense that they can be compared, but since there are no guarantees the positions could switch anytime. that's why I like comparisons between serverless functions where there are pretty explicit SLAs and what not given by each company for you to compare against, as well as one another.
  - daneel_w 4y ago
    Given the stark contrast and that the pattern was identical every day over a two-week course, it tells me we're observing a fundamental systemic difference between GCP and AWS - and I think that's all the author really wanted to point out. I would not be surprised if the results are replicable three months from now.

AtNightWeCode 4y ago

This benchmark (too) is probably incorrect. It produces 409:s so there are errors in there that I doubt are caused by GCP.

playingalong 4y ago

This is great. I have always been feeling there is so little independent content on benchmarking the IaaS providers. There is so much you can measure in how they behave.

politelemon 4y ago

A few weeks ago I needed to change the volume type on an EC2 instance to gp3. Following the instructions, the change happened while the instance was running. I didn't need to reboot or stop the instance, it just changed the type. While the instance was running. I didn't understand how they were able to do this, I had thought volume types mapped to hardware clusters of some kind. And since I didn't understand, I wasn't able to distinguish it from magic.

ArchOversight 4y ago
Changing the volume type on AWS is somewhat magical. Seeing it happens on-line was amazing.
xyzzyz 4y ago
Dunno about AWS, but GCP uses live migration, and will migrate your VM across physical machines as necessary. The disk volumes are all connected over the network, nothing really depends on the actual physical machine your VM is ran on.
- lbhdc 4y ago
  How does migrating a vm to another physical machine work?
  - rejectfinite 4y ago
    vsphere vmotion has been a thing for years lmao
  - roomey 4y ago
    VMware has been doing this for years, it's called vmotion and there is a lot of documentation about it if you are interested (eg https://www.thegeekpub.com/8407/how-vmotion-works/ https://www.thegeekpub.com/8407/how-vmotion-works/ ) Essential, memory state is copied to the new host, the VM is stunned for a millisecond and the cpu states is copied and resumed on the new host (you may see a dropped ping). All the networking and storage is virtual anyway so that is "moved" (it's not really moved) in the background.

PigiVinci83 4y ago

Thank you for this article, it confirms my direct experience. Never run a benchmarking test but I can see this every day.

MonkeyMalarky 4y ago

I would love to see the same for deploying things like a cloud/lambda function.

devxpy 4y ago

Is this testing for spot instances? In my limited experience, persistent (on-demand) GCP instances always boot up much faster than AWS EC2 instances.

encryptluks2 4y ago
I noticed that too and it does appear to be using spot instances. I have a feeling if it was ran without you may see much better startup times. Spot instances on GCP are hit and miss and you sort of have to build that into your workflow.
marcinzm 4y ago
In my experience GPU persistent instances often simply don't boot up on GCP due to lack of available GPUs. One reason I didn't choose GCP at my last company.
- devxpy 4y ago
  Oh interesting. Which region and GPU type were you working with? (Asking so I can avoid in future)
  - marcinzm 4y ago
    I think it was us-east1 or us-east4. Had issues getting TPUs as well in us-central1. I know someone at a larger tech company who was told to only run certain workflows in a specific niche European region as that's the only one that had any A100 GPUs most of the time.

ajross 4y ago

Worth pointing out that the article is measuring provisioning latency and success rates (how quickly can you get a GPU box running and whether or not you get an error back from the API when you try), and not "reliability" as most readers would understand it (how likely they are to do what you want them to do without failure). Definitely seems like interesting info, though.

amaks 4y ago

The link is broken?

lucb1e 4y ago
Works for me using Firefox in Germany, although the article doesn't really match the title so maybe that's why you were confused? :p

danielmarkbruce 4y ago

It's meant to say "ephemeral"... right? It's hard to read after that.

datalopers 4y ago
ephemeral and ethereal are commonly confused words.
- danielmarkbruce 4y ago
  I guess that's fair. It's sort of a smell when someone uses the wrong word (especially in writing) though. It suggests they aren't in industry, throwing ideas around with other folks. The word "ephemeral" is used extensively amongst software engineers.
- dublin 4y ago
  Ephimerides really throws them. (And thank God for PyEphem, which makes all that otherwise quite fiddly stuff really easy...)

dark-star 4y ago

I wonder why someone would equate "instance launch time" with "reliability"... I won't go as far as calling it "clickbait" but wouldn't some other noun ("startup performance is wildly different") have made more sense?

xmonkee 4y ago
GCP also had 84 errors compared to 1 for AWS
- danielmarkbruce 4y ago
  If not a 4xx, what should they return for instance not available?
  - eurasiantiger 4y ago
    503 service unavailable?
    - dheera 4y ago
      Using HTTP error codes for non-REST things is cringe. 503 would mean the IaaS API calls themselves are unavailable. Very different from the API working perfectly fine but the instances not being available.
      - devmunchies 4y ago
        What? REST is just some API philosophy, its doesn't even have to be on top of HTTP. Why would you think HTTP status codes are made for REST? They are made for HTTP to describe the response of the resource you are requesting, and the AWS API uses HTTP so it makes sense to use HTTP status codes.
        eurasiantiger 4y ago

user- 4y ago

I wouldn't call this reliability, which already has a loaded definition in the cloud world, and instead something along time-to-start or latency or something.

systemvoltage 4y ago
It is though based on a specific definition. If X doesn't do Y based on Z metric with a large standard deviation and doesn't meet spec limits, it is not reliable as per the predefined tolerance T. X = Compute intances Y = Launch Z = Time to launch T = LSL (N/A), USL (10s), Std Dev (2s) Where LSL is lower spec limit, USL is upper spec limit. LSL is N/A since we don't care if the instance launches instantly (0 seconds). You can define T as per your requirements. Here we are ignoring the accuracy of the clock that measures time, assuming that the measurement device is infinitely accurate. If your criteria is to, say for example, define reliability as how fast it shuts down, then this article isn't relevant. Article is pretty narrow in testing reliability, they only care about launch time.

remus 4y ago

> The offerings between the two cloud vendors are also not the same, which might relate to their differing response times. GCP allows you to attach a GPU to an arbitrary VM as a hardware accelerator - you can separately configure quantity of the CPUs as needed. AWS only provisions defined VMs that have GPUs attached - the g4dn.x series of hardware here. Each of these instances are fixed in their CPU allocation, so if you want one particular varietal of GPU you are stuck with the associated CPU configuration. At a surface level, the above (from the article) seems like a pretty straightforward explanation? GCP gives you more flexibility in configuring GPU instances at the trade off of increased startup time variability.

btgeekboy 4y ago
I wouldn't be surprised if GCP has GPUs scattered throughout the datacenter. If you happen to want to attach one, it has to find one for you to use - potentially live migrating your instance or someone else's so that it can connect them. It'd explain the massive variability between launch times.
- master_crab 4y ago
  Yeah that was my thought too when I first read the blurb. It’s neat…but like a lot of things in large scale operations, the devil is in the details. GPU-CPU communications is a low latency high bandwidth operation. Not something you can trivially do over standard TCP. GCP offering something like that without the ability to flawlessly migrate the VM or procure enough “local” GPUs means it’s just vaporware. As a side note, I’m surprised the author didn’t note the amount of ICE’s (insufficient capacity errors) AWS throws whenever you spin up a G type instance. AWS is notorious for offering very few G’s and P’s is certain AZs and regions.
  - my123 4y ago
    Fungible is selling a GPU decoupling solution via PCIe encapsulated over Ethernet today, so it can certainly be done. And NVIDIA's vGPU solutions do support live migration of GPUs to another host (in which case the vGPU gets moved too, to a GPU on that target).

kccqzy 4y ago

Heard from a Googler that the internal infrastructure (Borg) is simply not optimized for quick startup. Launching a new Borg job often takes multiple minutes before the job runs. Not surprising at all.

epberry 4y ago
Echoing this. The SRE book is also highly revealing about how Google request prioritization works. https://sre.google/sre-book/load-balancing-datacenter/ https://sre.google/sre-book/load-balancing-datacenter/ My personal opinion is that Google's resources are more tightly optimized than AWS and they may try to find the 99% best allocation versus the 95% best allocation on AWS.. and this leads to more rejected requests. Open to being wrong on this.
valleyjo 4y ago
As another comment points out, GPU resources are less common so it takes longer to create, which makes sense. In general, start up times are pretty quick on GCP as other comments also confirm.
dekhn 4y ago
booting VMs != starting a borg job.
- kccqzy 4y ago
  The technology may be different but the culture carries over. People simply don't have the habit to optimize for startup time.
readams 4y ago
Borg is not used for gcp vms, though.
- dilyevsky 4y ago
  It is used but borg scheduler does not manage vm startups
4y ago

londons_explore 4y ago

AWS normally has machines sitting idle just waiting for you to use. Thats why they can get you going in a couple of seconds. GCP on the other hand fills all machines with background jobs. When you want a machine, they need to terminate a background job to make room for you. That background job has a shutdown grace time. Usually thats 30 seconds. Sometimes, to prevent fragmentation, they actually need to shuffle around many other users to give you the perfect slot - and some of those jobs have start-new-before-stop-old semantics - that's why sometimes the delay is far higher too.

dekhn 4y ago
borg implements preemption but the delay to start VMs is not because they are waiting for a background task to clean up.

deleted 4y ago

[deleted]

zmmmmm 4y ago

> In total it scaled up about 3,000 T4 GPUs per platform > why I burned $150 on GPUs How do you rent 3000 GPUs over a period of weeks for $150? Were they literally requisitioning it and releasing it immediately? Seems like this is quite a unrealistic type of usage pattern and would depend a lot on whether the cloud provider optimises to hand you back the same warm instance you just relinquished. > GCP allows you to attach a GPU to an arbitrary VM as a hardware accelerator it's quite fascinating that GCP can do this. GPUs are physical things (!) do they provision every single instance type in the data center with GPUs? That would seem very expensive.

bushbaba 4y ago
Unlikely. More likely they put your VM on a host with GPU attached, and use live migration to move workloads around for better resource utilization. However, live-migration can cause impact to HPC workloads.
ZiiS 4y ago
GPUs are physical but VMs are not; I expect they just move them to a host with a GPU.
NavinF 4y ago
It probably live-migrates your VM to a physical machine that has a GPU available. ...if there are any GPUs available in the AZ that is. I had a hell of a time last year moving back and forth between regions to grab just 1 GPU to test something. The web UI didn't have a "any region" option for launching VMs so if you don't use the API you'll have to sit there for 20 minutes trying each AZ/region until you managed to grab one.
geysersam 4y ago
> $150 Was asking myself the same question. From the pricing information on gcp it seems minimum billing time is 1 minute, making 3000 GPUs cost $50 minimum. If this is the case then $150 is reasonable for the kind of usage pattern you describe.

dekhn 4y ago

What would you expect? AWS is an org dedicated to giving customers what they want and charging them for it, while GCP is an org dedicated to telling customers what they want and using the revenue to get slightly better cost margins on Intel servers.

dilyevsky 4y ago
I don’t believe this reasoning is used since at least Diane
- dekhn 4y ago
  I haven't seen any real change from Google about how they approach cloud in the past decade (first as an employee and developer of cloud services there, and now as a customer). Their sales people have hollow eyes
  - dilyevsky 4y ago
    Didn’t say they were good at making the switch
DangitBobby 4y ago
They don't really tell us what we want, we just buy what we need. Might work for you.

cmcconomy 4y ago

I wish Azure was here to round it out!

charbull 4y ago

Can you put this in context of the problem/use case /need you are solving for ?

0xbadcafebee 4y ago

Reliability in general is measured on the basic principle of: does it function within our defined expectations? As long as it's launching, and it eventually responds within SLA/SLO limits, and on failure comes back within SLA/SLO limits, it is reliable. Even with GCP's multiple failures to launch, that may still be considered "reliable" within their SLA. If both AWS and GCP had the same SLA, and one did better than the other at starting up, you could say one is more performant than the other, but you couldn't say it's more reliable if they are both meeting the SLA. It's easy to look at something that never goes down and say "that is more reliable", but it might have been pure chance that it never went down. Always read the fine print, and don't expect anything better than what they guarantee.

1-6 4y ago

This is all about cloud GPUs, I was expecting something totally different from the title.

humanfromearth 4y ago

We have constant autoscaling issues because of this in GCP - glad someone plotted this - hope people in GCP will pay a bit more attention to this. Thanks to the OP!

outworlder 4y ago

Unclear what the article has to do with reliability. Yes, spinning up machines on GCP is incredibly fast and has always been. AWS is decent. Azure feels like I'm starting a Boeing 747 instead of a VM. However, there's one aspect where GCP is a clear winner on the reliability front. They auto-migrate instances transparently and with close to zero impact to workloads – I want to say zero impact but it's not technically zero. In comparison, in AWS you need to stop/start your instance yourself so that it will move to another hypervisor(depending on the actual issue AWS may do it for you). That definitely has impact on your workloads. We can sometimes architect around it but there's still something to worry about. Given the number of instances we run, we have multiple machines to deal with weekly. We get all these 'scheduled maintenance' events (which sometimes aren't really all that scheduled), with some instance IDs(they don't even bother sending the name tag), and we have to deal with that. I already thought stop/start was an improvement on tech at the time (Openstack, for example, or even VMWare) just because we don't have to think about hypervisors, we don't have to know, we don't care. We don't have to ask for migrations to be performed, hypervisors are pretty much stateless. However, on GCP? We had to stop/start instances exactly zero times, out of the thousands we run and have been running for years. We can see auto-migration events when we bother checking the logs. Otherwise, we don't even notice the migration happened. It's pretty old tech too: https://cloudplatform.googleblog.com/2015/03/Google-Compute-Engine-uses-Live-Migration-technology-to-service-infrastructure-without-application-downtime.html https://cloudplatform.googleblog.com/2015/03/Google-Compute-...

willcipriano 4y ago
I always wondered why you couldn't do that on AWS, mainly because I could do it at home with Hyper-V a decade ago. https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/microsoft-hyper-v-server-2008-R2/ee815293(v=ws.10) https://learn.microsoft.com/en-us/previous-versions/windows/...
voidfunc 4y ago
> Azure feels like I'm starting a Boeing 747 instead of a VM. Huh... interesting, this has not been my experience with Azure VM launch times. I'm usually surprised how quickly they pop up.

deleted 4y ago

[deleted]

lacker 4y ago

Anecdotally I tend to agree with the author. But this really isn't a great way of comparing cloud services. The fundamental problem with cloud reliability is that it depends on a lot of stuff that's out of your control, that you have no visibility into. I have had services running happily on AWS with no errors, and the next month without changing anything they fail all the time. Why? Well, we look into it and it turns out AWS changed something behind the scenes. There's a different underlying hardware behind the instance, or some resource started being in high demand because of some other customers. So, I completely believe that at the time of this test, this particular API was performing a lot better on AWS than on GCP. But I wouldn't count on it still performing this way a month later. Cloud services aren't like a piece of dedicated hardware where you test it one month, and then the next month it behaves roughly the same. They are changing a lot of stuff that you can't see.

citizenpaul 4y ago
That was my thoughts. People are probably pummeling GCP GPU free tier right now with stable diffusion image generators. Since it seems like all the free plug and play examples use the google python notebooks.
RajT88 4y ago
Instance types and regions make a big difference. Some regions and hardware generations are just busier than others. It may not be the same across cloud providers (although I suspect it is similar given the underlying market forces).
ryukoposting 4y ago
You've just perfectly characterized why on-site infrastructure will always have its place.
- callalex 4y ago
  You can reserve capacity on both of these services as well.

herpderperator 4y ago

The author is using 'Quantile' which I hadn't heard of before, and when I did, it seems like it actually should be 'Percentile'. Percentiles are the percentages, which is what the author is referring to.

mr_toad 4y ago
Quantiles are a generic term for percentiles, deciles, quartiles etc. Percentiles would have been a more precise term.

rwiggins 4y ago

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can have weird boundary cases, particularly because (1) it does cleanup after creation, which will have variable latency, (2) `int()` will truncate the fractional part of the second from `time()`, and (3) `time.time()` is not monotonic. I would not ask the author to spend money to test it again, but I think the 409s would probably disappear if you replaced `int(time())` with `uuid.uuid4()`. Disclosure: I work at Google - on Google Compute Engine. :-)

Cthulhu_ 4y ago
I've naively used millisecond precision things for a long time - not in anything critical I don't think - but I've only recently come to more of an awareness that a millisecond is a pretty long time. Recent example is that I used a timestamp to version a record in a database, but it's feasible that in a Go application, a record could feasilby be mutated multiple times a millisecond by different users / processes / requests. Unfortunately, millisecond-precise timestamps proved to be a bit tricky in combination with sqlite.
- account42 4y ago
  To put this into perspective, a game (or anything) running at 60 FPS only as a bit over 16 milliseconds to render each frame. These days higher frame rates are common enough, often putting you down into single-digit milliseconds per frame. Not think about how many things there are simulated and rendered in each frame for common games. Way more than 16.
sitharus 4y ago
This is a very good point - AWS uses tags to give instances a friendly name, so the name does not have to be unique. The same logic would not fail on AWS.

kazinator 4y ago

> This is particularly true for GPUs, which are uniquely squeezed by COVID shutdowns, POW mining, and growing deep learning models Is the POW mining part true any more? Hasn't mining moved to dedicated hardware?

CodesInChaos 4y ago
Bitcoin mining has used dedicated hardware for a long time. But I believe Ethereum mining used GPUs before the very recent proof-of-stake update.

Animats 4y ago

> GCP allows you to attach a GPU to an arbitrary VM as a hardware accelerator - you can separately configure quantity of the CPUs as needed. That would seem to indicate that asking for a VM on GCP gets you a minimally configured VM on basic hardware, and then it gets migrated to something bigger if you ask for more resources. Is that correct? That could make sense if, much of the time, users get a VM and spend a lot of time loading and initializing stuff, then migrate to bigger hardware to crunch.

zylent 4y ago
This is not quite true - GPU's are limited to select VM types, and the number of GPU's you have influences the maximum number of cores you can get. In general they're only available on the n1 instances (except the a100's, but those are far less popular)

orf 4y ago

AWS has different pools of EC2 instances depending on the customer, the size of the account and any reservations you may have. Spawning a single GPU at varying times is nothing. Try spawning more than one, or using spot instances, and you’ll get a very different picture. We often run into capacity issues with GPU and even the new m6i instances at all times of the day. Very few realistic company size workloads need a single GPU. I would willingly wait 30 minutes for my instances to become available if it meant all of them where available at the same time.

Jamie9912 4y ago

Should probably change the title to "AWS vs GCP on-demand GPU launch time consistency"

Terretta 4y ago
Yep. Author colloquially meant, can I rely on a quick start.

mnutt 4y ago

It may or may not matter for various use cases, but the EC2 instances in the test use EBS and the AMIs are lazily loaded from S3 on boot. So it may be possible that the boot process touches few files and quickly gets to 'ready' state, but you may have crummy performance for a while in some cases. I haven't used GCP much, but maybe they load the image onto the node prior to launch, accounting for some of the launch time difference?

lmeyerov 4y ago

I'd say that's a weak test of capacity. Would love to see on Azure - T4s or an equiv aren't even really provided anymore! We find reliability a diff story. Eg, our main source of downtime on Azure is they restart (live migrate?) our reserved T4s every few weeks, causing 2-10min outages per GPU per month.

lomkju 4y ago

Having being a high scale AWS user with a bill of +$1M/month and now working since 2 years with a company which uses GCP. I would say AWS is superior and way ahead. ** NOTE: If you're a low scale company this won't matter to you ** 1. GKE When you cross a certain scale certain GKE components won't scale with you and SLOs on those components are crazy, it takes 15+ mins for us to update a GKE ingress controller backed Ingress. Cloud Logging hasn't been able to keep up with our scale, disabled since 2 years now. This last Q we got an email from them to enable it and try it again on our clusters, still have to confirm these claims as our scale is more higher now. Konnectivity agent release was really bad for us, it affected some components internally, total dev time we lost was more than 3 months debugging this issue. They had to disable konnectivity agent on our clusters, I had to collect TCP dumps and other evidences just to prove nothing was wrong on our end, fight with our TAM to get a meeting with the product team. After 4 months they agreed and reverted our clusters to SSH tunnels. Initially GCP support said they said they can't do this. Next Q Ill be updating the clusters hopefully they have fixed this by then. 2. Support. I think AWS support always were more pro active in debugging with us, GCP support agents most of the times lack the expertise or proactiveness to debug/solve things in simple cases. We pay for enterprise support and don't see getting much from them. At AWS we had reviews of the infra how we could better it every 2 Qs and we got new suggestion and was also the time when we shared what we would like to see in their roadmap. 3.Enterprisyness is missing with design A simple thing as cloudbuild doesn't have access to static IPs. We have to maintain a forward proxy just cause of this. L4 LBs were a mess you could only use specified ports in a (L4 LB) TCP proxy, For a tcp proxy based loadbalancer, the allowed set of ports are - [25, 43, 110, 143, 195, 443, 465, 587, 700, 993, 995, 1883, 3389, 5222, 5432, 5671, 5672, 5900, 5901, 6379, 8085, 8099, 9092, 9200, and 9300]. Today I see they have removed these restrictions. I don't know who came up with this idea to allow only a few ports on a L4 LB. I think such design decisions make it less Enterprisy.

jupp0r 4y ago

That's interesting but not what I expected when I read "reliability". I would have expected SLO metrics like uptime of the network or similar metrics that users would care about more. Usually when scaling a system that's built well you don't have hard short constraints on how fast an instance needs to be spun up. If you are unable to spin up any that can be problematic of course. Ideally this is all automated so nobody would care much about whether it takes a retry or 30s longer to create an instance. If this is important to you, you have other problems.

TheMagicHorsey 4y ago

This is not reliability. This is a measure of how much spare capacity AWS seems to be leaving idle for you to snatch on-demand. This is going to vary a lot based on the time of year. Why don't you try this same experiment at around some time when there's a lot of retail sales activity (Black Friday), and watch AWS suddenly have much less capacity to dole out on-demand. To me reliability is a measure of what a cloud does compared to what it says it will do. GCP is not promissing you on-demand instances instantaneously is it? If you want that ... reserve capacity.

runeks 4y ago

> These differences are so extreme they made me double check the process. Are the "states" of completion different between the two clouds? Is an AWS "Ready" premature compared to GCP? It anecdotally appears not; I was able to ssh into an instance right after AWS became ready, and it took as long as GCP indicated before I was able to login to one of theirs. This is a good point and should be part of the test: after launching, SSH into the machine and run a trivial task to confirm that the hardware works.

vienarr 4y ago

The article only talks about GPU start time, but the title is "CloudA vs CloudB reliability" bit of a stretch, right

DonHopkins 4y ago

Anybody know if on GCP the cheaper ephemeral spot instances are available on managed instance groups and cloud run, where it spins up more instances according to demand, and how well it deals with replacing spot instances that drop dead, if so? How about AWS?

jqpabc123 4y ago

Thanks for the report. It only confirms my judgment. The word "Google" attached to anything is a strong indicator that you should look for an alternative.

johndfsgdgdfg 4y ago

It's not surprising. Amazon is an amazing customer focus company. Google is a spyware company that only wants to make more by invading our privacy. Of course Amazon products will be better than Google.