11 ms·
Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
- lemming 1y agoIf I understand this correctly, I should be able to stand up an API running arbitrary models (e.g. from Hugging Face), and it’s not quite charged by the token but should be very cheap if my usage is sporadic. Is that correct? Seems pretty huge if so, most of the providers I looked at required a monthly fee to run a custom model.
- BoredPositron 1y agoRunpod, vast, coreweave, replicate... just a bunch of alternatives that let you run serverless GPU inference.
- _zoltan_ 1y agoyou can't just sign up for coreweave, can you?
- BoredPositron 1y agoWe wrote them an email.
- lexandstuff 1y agoYes, that's basically correct. Except be warned that the cold start times can be huge (30-60 seconds). So scaling to 0 doesn't really work in practice, unless your users are happy to wait from time to time. Also, you also have to pay a small monthly fee for container storage (and a few other charges iirc).
- montebicyclelo 1y agoReason Cloud Run is so nice compared to other providers is that it has autoscaling, with scaling to 0. Meaning it can cost basically 0 if it's not being used. Also can set a cap on the scaling, e.g. 5 instances max, which caps the max cost of the service too. - Note, I only have experience with the CPU version of Cloud Run, (which is very reliable / easy).
- rvnx 1y agoEven regular Cloud Run can take a lot of time to boot (~3 to 30 seconds), so this can be a problem when scaling to 0
- lexandstuff 1y agoNot to mention, if it's an ML workload, you'll also have to factor in downloading the weights and loading them into memory, which can double that time or more.
- rvnx 1y agoAccording to the press release, "we achieved an impressive Time-to-First-Token of approximately 19 seconds for a gemma3:4b model" Imagine, you have a very small weak model, and you have to wait 20 seconds for your request.
- infecto 1y agoImagine running a production client facing api and not overprovisioning it.
- happyopossum 1y ago> Imagine, you have a very small weak model, and you have to wait 20 seconds for your request. For your first request, after having scaled to 0 while it wasn’t in use. For a lot of use cases, that sounds great.
- steren 1y agoAlso, a GPU instance needs 5s to start. The test depends on how large the model is. So a "very small weak model" can lead much faster than 20s
- mdhb 1y agoI’m looking at logs for a service I run on cloud run right now which scales to zero. Boot times are approximate 200ms for a Dart backend.
- isoprophlex 1y agoAll the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai
- kamranjon 1y agoI dunno, the scale to zero and pay per second features seemed super useful to me after forgetting to shut down some training instances with AWS. Also the fast startup ability, if it actually works as well as they say, would be amazing for a lot of the type of workloads that I have.
- isoprophlex 1y agoAgreed, but runpod or modal offer the same. Happy to use big cloud for a client if they pay the bills, but for personal quests... too scary.
- rikafurude21 1y agothats what billing limits are for
- aiiizzz 1y agoThose, on gcp, are just alerts, not hard limits, no?
- jsheard 1y agoYeah. I think you can hack together a function which pulls the plug automatically if a billing alert fires, but IIRC the alerts can take a few hours to respond, so extreme runaway usage could still result in a bad time.
- worldsayshi 1y agoI would rather not realize i had left a bug in that hack after the fact.
- gardnr 1y agoThe Nvidia L4 has 24GB of VRAM and consumes 72 watts, which is relatively low compared to other datacenter cards. It's not a monster GPU, but it should work OK for inference.
- m4r1k 1y agoI wrote about cloud run and inference w/ ollama on cloud run -> https://medium.com/google-cloud/ollama-on-cloud-run-with-gpu-in-less-than-20-seconds-d324abe33d80 https://medium.com/google-cloud/ollama-on-cloud-run-with-gpu... performance is okay, ada lovelace has cuda 8_9 support which brings native fp8 support. imo the best aspect is the speed of spinning up new containers and the overall easiness of this service. the live demo at google next 25 was quite something https://www.youtube.com/watch?v=PWPvX25R6dM&t=2140s https://www.youtube.com/watch?v=PWPvX25R6dM&t=2140s
- jbarrow 1y agoI’m personally a huge fan of Modal, and have been using their serverless scale-to-zero GPUs for a while. We’ve seen some nice cost reductions from using them, while also being able to scale WAY UP when needed. All with minimal development effort. Interesting to see a big provider entering this space. Originally swapped to Modal because big providers weren’t offering this (e.g. AWS lambdas can’t run on GPU instances). Assuming all providers are going to start moving towards offering this?
- chrishare 1y agoModal documentation is also very good.
- AndresSRG 1y agoI’m also a big fan. Modal has the fastest cold-start I’ve seen for 10GB+ models.
- scj13 1y agoModal is great, they even released a deep dive into their LP solver for how they're able to get GPUs so quickly (and cheaply). Coiled is another option worth looking at if you're a Python developer. Not nearly as fast on cold start as Modal, but similarly easy to use and great for spinning up GPU-backed VMs for bursty workloads. Everything runs in your cloud account. The built-in package sync is also pretty nice, it auto-installs CUDA drivers and Python dependencies from your local dev context. (Disclaimer: I work with Coiled, but genuinely think it's a good option for GPU serverless-ish workflows. )
- dr_kiszonka 1y agoThanks for sharing! They even support running HIPAA-compliant workloads, which I didn't anticipate.
- holografix 1y agoThe value in this really is running small custom models or the absolute latest open weight models. Why bother when you can get payg API access to popular open weights models like Llama on Vertex AI model garden or at the edge on Cloudflare?
- progbits 1y agoCustom models. We use this, pretty convenient and less hassle than managing our autoscaling GPU pools.
- mythz 1y agoThe pricing doesn't look that compelling, here are the hourly rate comparisons vs runpod.io vs vast.ai: 1x L4 24GB: google: $0.71; runpod.io: $0.43, spot: $0.22 4x L4 24GB: google: $4.00; runpod.io: $1.72, spot: $0.88 1x A100 80GB: google: $5.07; runpod.io: $1.64, spot: $0.82; vast.ai $0.880, spot: $0.501 1x H100 80GB: google: $11.06; runpod.io: $2.79, spot: $1.65; vast.ai $1.535, spot: $0.473 8x H200 141GB: google: $88.08; runpod.io: $31.92; vast.ai $15.470, spot: $14.563 Google's pricing also assumes you're running it 24/7 for an entire month, where as this is just the hourly price for runpod.io or vast.ai which both bill per second. Wasn't able to find Google's spot pricing for GPUs.
- counters 1y agoNothing but 1xL4 are even offered on Cloud Run GPUs, are they?
- ZiiS 1y agoI think the Google prices are billed per-second so under 20min you are better on Google?
- thousand_nights 1y agorunpod is billed by the minute
- bts4 1y agoTechnically we bill Pods by the millisecond. Pennies matter :)
- mythz 1y agoRunPod also charges per second [1], also this is Google's expected avg cost per hour after running it 24/7 for an entire month, I couldn't find an hourly cost for each GPU. When you need under <1hr than you can go with Runpod's Spot pricing which is ~4-7x cheaper than Google, where even 20min of Google would cost more than 1hr on RunPod. [1] https://docs.runpod.io/serverless/pricing https://docs.runpod.io/serverless/pricing
- jjuliano 1y agoI'm the developer of kdeps.com, and I really like Google Cloud Run, been using it since beta version. Kdeps outputs Dockerized full-stack AI agent apps that runs open-source LLMs locally, and my project works so well with GCR.
- ninetyninenine 1y agoIm tired of using AI in cloud services. I want user friendly locally owned AI hardware. Right now nothing is consumer friendly. I can’t get a packaged deal of some locally running ChatGPT quality UI or voice command system in an all in one package. Like what Macs did for PCs I want the same for AI.
- Hilift 1y agoOracle just announced they are spending $40 billion on GPU hardware. All cloud providers have an AI offering, and there are AI-specific cloud providers. I don't think retail is invited.
- Disposal8433 1y agoYour local computer is not powerful enough, and that's why you must welcome those brand new mainframes... I mean, "cloud services."
- pjmlp 1y agoIt is funny how using a Web IDE, and a cloud shell, is such a déjà vu from when I used to do development on a common UNIX server shared by the whole team.
- edoceo 1y agoTelnet from a Wyse terminal.
- pjmlp 1y agoMy first experience with such a setup was connecting to DG/UX, via the terminal application on Windows for Workgroups, or some thin client terminals in a mix of green or ambar phosphor, spread around the campus. The only time I used a Pascal compiler in ISO Pascal mode, it had the usual extensions inspired on UCSD, but we weren't allowed to use them on the assignments.
- ninetyninenine 1y ago
- Aeolun 1y agoThat’s 67ct / hour for a gpu enabled instance. That’s pretty good, but I have no idea how T4 GPU’s compare against others.
- wut42 1y agoL4 are pretty limited nowadays. It's usually rented at 40ct/hour on other providers.
- omneity 1y ago> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference) This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from the user’s perspective and that’s a best case scenario. If this is the best one of the most experienced teams in the world can do, with a small 4B model, then it feels like serverless is really restricted to non-interactive use cases.
- bravesoul2 1y agoLooks like GPU instances not "lambda", so presumable you would over-provision to compensate.
- diggan 1y agoThat has to be cold-start, and next N requests would surely be using the already started thing? It sounds bananas they'd even mention using something like that with 19 seconds latency for all requests in any context.
- agcat 1y agoThat's true. Traditional single-tier storage can not meet the throughput and latency demand. My cofounder wrote this piece on a three-tiered storage architecture optimized for both performance and cost - https://nilesh-agarwal.com/three-tier-storage-architecture-for-fast-llm-inference-in-the-cloud/ https://nilesh-agarwal.com/three-tier-storage-architecture-f...
- wut42 1y agoDefinitely -- and yet it's kinda a feat compared to other solutions: when i tried Runpod Serverless i could wait up to five minutes for a cold start to a even more smaller model than a 4B.
- deleted 1y ago[deleted]
- infecto 1y ago
- huksley 1y agoA small and independent EU GPU cloud provider, DataCrunch (I am not affiliated), offers VMs with Nvidia GPUs even cheaper than Run Pod, etc 1x A100 80Gb 1.37€/hour 1x H100 80Gb 2.19€/hour
- diggan 1y agoOr go P2P with Vast.ai, cheapest A100 right now is a setup with 2x A100 for $0.8/hour (so $0.4 per A100). Not affiliated with them, but mostly happy user. Be vary of network speeds though, some hosts are clearly on shared bandwidth and reported numbers don't always line up with reality, which kind of sucks when you're trying to shuffle around 100GB of data.
- triknomeister 1y agoYou really need NVL for some performance.
- diggan 1y agoOk, did you check the instance list? There is a bunch of 8x H200 NVL available?
- sigmoid10 1y agoThat's funny. You can get a 1x H100 80Gb VM at lambda.ai for $2.49/hour. At the current exchange rate, that's exactly 2.19€. Coincidence or is this actually some kind of ceiling?
- moeadham 1y agoif only they had some decent GPUs. L4s are pretty limited these days.
- gabe_monroy 1y agothis is something we are working on. stay tuned!
- steren 1y agoCloud Run PM lead here. Sign up for new GPU types at https://docs.google.com/forms/d/e/1FAIpQLSdZk5sCsDUjAoYQX-sqekQJ5UM9LqcTldgWAnaXRgdCrNS3RQ/viewform https://docs.google.com/forms/d/e/1FAIpQLSdZk5sCsDUjAoYQX-sq...
- ashishb 1y agoI love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a VM + GPU if the utilization of your service is even only for 30% of the day. 1 - https://ashishb.net/programming/free-deployment-of-side-projects/ https://ashishb.net/programming/free-deployment-of-side-proj...
- ivape 1y agoMaybe I just don't know, but I really don't think most people here can even point to a cloud GPU with 1000 concurrent users and not end up with a million dollar bill.
- icedchai 1y agoCloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)
- AChampaign 1y agoI think Lambda is more or less the AWS equivalent.
- ZeroCool2u 1y agoI think Cloud Run Functions would be the direct equivalent to Lambda.
- AChampaign 1y agoOh, you’re probably right.
- hn_throwaway_99 1y agoI agree, but in the GCP world, a lot of these things are merging. My understanding is that Cloud Run, Cloud Run Functions (previously known as Cloud Functions Gen2) and even App Engine Flexible all run in the same underlying cloud run infrastructure, so it's essentially just some interface differences that to me now seem more like historical legacy/backwards compatibility reasons than meaningful functionality differences (e.g. Functions can now handle multiple concurrent requests).
- ringeryless 1y agoi wonder what all this hype-driven overcapacity will be used for by future generations. once this bubble pops we are going to have some serious albeit high-latency hardware
- zaphar 1y agoCrunching really large amounts of numbers has always been useful. And that's all this really is. Running weather simulations, Advanced math problems, Complicated engineering simulations. The space of possible uses is incredibly wide.
- otherjason 1y agoThe last few generations of GPU architectures have been increasingly optimized for massive throughput of low-precision integer arithmetic operations, though, which are not useful for any of those other applications.
- esafak 1y agoWhat overcapacity? People are struggling to find affordable GPUs.
- happyopossum 1y ago> overcapacity I’m Not sure that word means what you think it means. There is a pretty severe shortage of GPU capacity in the industry right now.
- deleted 1y ago[deleted]
- ivape 1y agoDoes anyone actually run a modest sized app and can share numbers on what one gpu gets you? Assuming something like vllm for concurrent requests, what kind of throughput are you seeing? Serving an LLM just feels like a nightmare.
- Rachman91 1y ago[flagged]
- gabe_monroy 1y agoi'm the vp/gm responsible for cloud run and GKE. great to see the interest in this! happy to answer questions on this thread.
- treksis 1y agoEverything good except the price.
- albeebe1 1y agoOh this is great news. After a $1000 bill running a model on vertex.ai continuously for a little test i forgot to shut down, this will be my go to now. I've been using Cloud Run for years running production microservices, and little hobby projects and i've found it simple and cost effective.
- pier25 1y agoHow does this compare to Fly GPUs in terms of pricing?
- einpoklum 1y agoWhy is commercial advertising published as a content article here?
- m1 1y agoLove cloud run and this looks like a great addition. Only things I wish from cloud run is being able to run self hosted GitHub runners on it (last time I checked this wasn’t possible as it requires root), also the new worker pool feature seems great in practice but it looks like you have to write the scaler yourself rather than it being built in.
- aniruddhc 1y agoHi! I'm the Eng Manager responsible for Autoscaling for Serverless and Worker Pools. We're actively defining our roadmap, and understanding your use case would be incredibly valuable. If you're open to it, please email me at <my HN username>@google.com. I'd love to learn more about how you'd use worker pools and what kind of workloads you need to scale.
- deleted 1y ago[deleted]
- felix_tech 1y agoI've been using this for daily/weekly ETL tasks which saves quite a lot of money vs having an instance on all the time but it's been clunky. The main issue is despite there being a 60 minute timeout available the API will just straight up not return a response code if your request takes > ~5 minutes in most cases so you gotta make sure you can poll where the datas being stored and let the client time out
- covi 1y agoTake a look at SkyPilot. Good for running these batch workloads. You can use spot instances to save costs.