13 ms·
Alternative clouds are booming as companies seek cheaper access to GPUs
- freeqaz 3y agoHow are these companies even getting these GPUs? I would imagine NVIDIA would give them all to Microsoft and Google if they are in short supply.
- m3kw9 3y agoNvidia needs these alt cloud so when big co builds their own GPUs they have them to suck up supply. If they are all dead, Nvidia may have no back path
- rvba 3y agoWhy couldnt NVIDIA auction some to get best price? I mean locking customers long term is probably better, but a certain % of product still vould be sold for a spot price? This made me wonder what is the optimum split on that - obviously depends on your market placement, but isnt nvidia basically a monopoly now?
- zer00eyz 3y ago> Why couldnt NVIDIA auction some to get best price? Have you seen their pricing? They are robbing everyone and selling out quickly. I don't think "auctions" are going to add much to thee bottom line.
- autoexecbat 3y agoAt some point, nvidia does have to worry about amd getting their act together once they've annoyed all the current customers
- nothercastle 3y agoAmd will never figure out drivers and software stuff. That’s always been their weakness
- Palmik 3y agoMI250 and MI300 are competitive for inference and training. The problem is that there's no cloud with simple registration that offers these for reasonable on demand price. This means that basically no one in the open source AI community can use them, so most development and tooling is being tested and built for NVIDIA.
- latchkey 3y agoI am working on solving that issue with MI300x.
- margorczynski 3y agoI wonder what Lisa tells the shareholders & board when they ask why billions fly past their noses in valuation. Curious why ATI/AMD never could be on par to Nvidia regarding the software stack (drivers mainly), I would assume bad leadership and goals.
- latchkey 3y agoI'd call it a focus on very profitable ventures, like gaming. Nvidia effectively created this AI world by building tooling and software for something we didn't know existed up until not too long ago. AI was created on the shoulders of the work that was done. AMD wasn't taking the risk on the unknown, for whatever reason. Water under the bridge. The ship is certainly busy course correcting now. Give it some time.
- nothercastle 3y agoWell they didn’t have any money. They bet big on cpu r&d and they did great but there wasn’t enough money left over for other moon shoots. AMD was at risk of becoming a failed company not too long ago so they recovered quite well.
- oldpersonintx 3y ago[dead]
- mym1990 3y agoI think one of Jensen Huang's big interests is to enable a diverse number of markets that can utilize the technology in a short feedback loop. I don't believe that can be achieved if all of the hardware goes to Microsoft/Google.
- SllX 3y agoIt’s better to have a diversified customer-base rather than being completely beholden to a small number of big customers that might decide they don’t need you down the line.
- rjst01 3y agoIndeed, Google and Amazon have already shown an interest in designing their own silicon (TPU and Graviton). Probably Microsoft too, but I'm not aware of an example off the top of my head. Upstarts are not easily going to be able to pursue that, so NVidia has a strong interest in supporting them.
- SllX 3y agoMicrosoft too: https://www.theverge.com/2023/11/15/23960345/microsoft-cpu-gpu-ai-chips-azure-maia-cobalt-specifications-cloud-infrastructure https://www.theverge.com/2023/11/15/23960345/microsoft-cpu-g...
- yardie 3y agoNvidia is a shovel maker. If you only have 1-2 shovel buyers, buyers who've started on making their own shovels in-house, then you run the risk of being trapped in a monopsony. You lose 1 shovel buying client and then you might be insolvent before too long.
- jedberg 3y agoI head up Product at Lambda. We are an NVIDIA preferred partner (and keep winning the preferred partner award). I don't know what the allocation numbers are for other companies, but we get a lot. It's in NVIDIAs best interest to spread the love around and make sure all the GPUs don't go to the hyperscalers.
- leetharris 3y agoI head engineering at Rev and every time we talk to Nvidia about GPUs they tell us to go to you guys. Every time. They really like Lambda!
- jedberg 3y agoVery happy to hear this!
- tpurves 3y agoI anyone buying AMD MI300? I am curious if anyone is finding AMD, as an alternate GPU vendor, available or useful yet?
- latchkey 3y agoMy company, Hot Aisle, is.
- Palmik 3y agoAre there any plans to support your on demand cloud in the future? In the past few months there are basically no GPUs available. My understanding is that your reserved cloud starts at 64 GPUs (8 pods) and with quite a long reservation time frame.
- jedberg 3y agoWe have new product coming in just a couple weeks to get clusters as small as 16 GPUs per cluster and only a two week minimum reservation, with very short lead times (the first people to reserve will have almost no lead time!). Also we expect to have a bunch of capacity come online of the on-demand cloud this year. We're getting the GPUs as quickly as we can and racking them as quickly as possible, but we have to wait just like everyone else for the GPUs to roll off the fab. :)
- Workaccount2 3y agoGoogle doesn't use nVidia chips, at least for AI. They have their own in house solution.
- FrankPetrilli 3y agohttps://cloud.google.com/compute/docs/gpus#nvidia_gpus_for_compute_workloads https://cloud.google.com/compute/docs/gpus#nvidia_gpus_for_c...
- jeffbee 3y agoCoreWeave does not exactly have an arms-length relationship with nvidia.
- mym1990 3y agoIsn't the mid/late game of all of these alternative clouds going to be the same as the big cloud players...get customer lock in and then prices slowly creep up until its kind of a wash?
- lelanthran 3y ago> Isn't the mid/late game of all of these alternative clouds going to be the same as the big cloud players...get customer lock in and then prices slowly creep up until its kind of a wash? That might be the plan, but I doubt it's going to work for GPUs like it did for AWS. There's no moat here, other than the hardware. There's no value-add that the provider can add for free and lock you in.
- swores 3y ago> There's no value-add that the provider can add for free and lock you in. Couldn't you have said that about cloud services like AWS before they existed, that they can rent servers to people but there won't be any lock in? And equally, just because that's the case with GPU providers right now, is there a reason one or more of them wouldn't be able to develop software which runs on the GPUs to do some of the things people are currently renting GPUs for, or middle-ground software which make it easier to do other stuff on them than just renting plain GPU access, and turning it into optional services (with lock in) on top of the raw hardware rental? As an example idea: it's good that NVIDIA decided to let as many people use CUDA as possible (and I suspect if they hadn't then they wouldn't have seen nearly as much success), but if they or anyone else releases an equivalent to "CUDA v2" tomorrow, but instead of allowing anyone to download it instead put it behind a billing page with AWS-style pricing that covers both software and GPU, would it not succeed if the software did make thinks easier for people just like AWS does (in some ways)? edit: I just realised I ignored the "for free" bit of your comment - but it wasn't free for Amazon or Google to build their cloud software either.
- Gormo 3y agoThat doesn't seem to be the case for long-tail VPS providers. It doesn't seem to really be the case for the big cloud providers, either, as most of what they offer is fairly commoditized -- I've been using GCP for years to host Kubernetes clusters, Postgres databases, Redis instances, standard Linux VMs, etc., and while there'd be a cost associated with migrating out of GCP, I don't feel locked into it in any way, as the tools I'm using there are the same tools I'd be using regardless of where they were hosted. The market for IaaS and for SaaS are very different from each other, and the walled-garden approach doesn't seem to have significant traction in the IaaS world, thankfully.
- 015a 3y agoWe're absolutely seeing the same thing even in non-GPU traditional servers. Here is an estimated monthly cost breakdown for a new GCP instance type, the n4-standard-2, in us-central1: 2 vCPU + 8 GB memory: $69.18 10 GB Hyperdisk Balanced: $0.80 3060 provisioned IOPS: $15.30 155 MB/s provisioned throughput: $6.20 Total: $91.48 Like, you can tear apart that $69/mo for 2vCPU + 8gb of memory, no problem. That's utterly insane. Its Emerald Rapids, so you're paying a premium for new chips, whatever. You can also tear apart the network egress, obviously. But just look at the SSD pricing. That's a ten gigabyte SSD provisioned for 155MB/s, for $22/month. You can go just outright buy a 256gb NVME at significantly higher bandwidth on Amazon for like $25, flat. The n4 tier instances removed the ability to use their cheaper general-purpose SSDs; you have to use hyperdisks. I'd be surprised if we don't see the big cloud providers start struggling over the next ten years. I think they engineered planetary-scale systems that are just way too expensive and complex to justify the cost they're charging; ZIRP phenomena.
- carbocation 3y agoAlso on GCP, the vCPUs are usually hyperthreads (except for t2d- instance types, and perhaps a few others). So that machine you've described has 1 CPU core.
- williamstein 3y agoReference: https://cloud.google.com/compute/docs/machine-resource#recommendations_for_machine_types https://cloud.google.com/compute/docs/machine-resource#recom... The t2d, t2a and h3 instance types have vCPU = core, and all other instance types have vCPU = thread.
- ikiris 3y agoif you don't need hyperdisk, why are you on that type of server?
- 015a 3y agoIts the only instance class that's on Emerald Rapids. So if you want the best that Intel has to offer, you need to adopt hyperdisks. But, to be clear: We're not.
- mgdev 3y agoThis will be a boon for Oracle.
- apitman 3y agoOver the weekend a coworker was helping someone in our lab prepare data for an important conference talk. We accidentally ran up a rather large bill because while the EFS storage pricing was simple enough, the usage pricing bit us. It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. It feels weirdly similar to the US healthcare/insurance situation. More competition in this space can only be a good thing.
- matsz 3y agoThis is precisely why I stick to dedicated servers for my own personal projects. $40/month for a machine, doesn't get any more predictable than this.
- baobabKoodaa 3y agoIf you rent a dedicated server from AWS, you will be hit for various additional fees which will likely dwarf that $40/month. So the issue here is not really shared vs dedicated instances. The issue here is that a particular cloud provider (namely AWS) has set up an opaque fee structure.
- drdaeman 3y ago> If you rent a dedicated server from AWS Who, in their right mind, goes for a bare metal to AWS, when there are so many decent and time-tested options out there?
- Tollen 3y agoI wish there were more of these for artists who specifically need GPU rendering for their own animations and heavy duty production. It seems like all of this new infrastructure isn't going towards "classic" GPU use cases.
- lowlevelprog 3y agoI have switched to buying GPUs and I already see saved costs as compared to cloud renting. Also, networking is fun.
- latchkey 3y ago> Also, networking is fun. Let me know how your 400G deployment goes with your vendors VRF implementation.
- tehlike 3y agoSwitching to hetzner was one of the good things I did last year!
- iotapi322 3y agoCrypto has this covered with 4090's available on clore and several other platforms.
- Gbox4 3y agoA funny comment from that article: "On CoreWeave, renting an Nvidia A100 40GB — one popular choice for model training and inferencing — costs $2.39 per hour, which works out to $1,200 per month. On Azure, the same GPU costs $3.40 per hour, or $2,482 per month; on Google Cloud, it’s $3.67 per hour, or $2,682 per month." Am I missing something? I am sure I'm a bit rusty in math, but I can still handle a calculator. ~720 hours in a month (roughly), and that means CoreWeave would cost $1,720.80 per month, Azure is $2,448 per month, and Google Cloud is $2,642.40 per month. Why are all of these numbers reported in the article off? Some slightly--Azure and Google Cloud are close, but CoreWeave is off by about 30%. I won't go further into the numbers as to why the author came up with these results, but I'm just wondering if this article was written by AI, which would explain why basic multiplication is incorrect.
- spacebanana7 3y agoCould it be a discount for purchasing an entire month’s worth of capacity? Even if so, such costing plans should be explicit in the article
- fancyfredbot 3y agoThe whole thing is nuts. They have the wrong costs multiplied by the wrong time period to get the wrong answers. https://coreweave.com/gpu-cloud-pricing https://coreweave.com/gpu-cloud-pricing says an A100 40GB NVLink is $2.06 whereas the article says $2.39. That's $1483.20 a month, whereas the article says $1200 and should say $1720 if they'd got the maths right.
- programjames 3y agoMaybe an LLM helped with the math?
- Onawa 3y agoI think your guess of AI generation makes sense for the math discrepancy.
- 3y ago
- zackangelo 3y agoWould like to take a moment to recommend fly.io for GPU workloads. I've been building a prototype using them for the last couple of weeks and it's been great to use. I didn't have to jump through any hoops or apply for any quota adjustments to get started. And I especially appreciate how easy they make it to automatically scale your GPU instances to zero based on traffic.
- nextworddev 3y agoTaking a step back, was it necessary to roll your own LLM (or whatever FM) API as opposed to using an off the shelf API
- zackangelo 3y agoYes we’re running our own inference server because we do nonstandard things in the decoder loop (think logit masking, other types of sampling, etc).
- indigodaddy 3y agoWow that is quite decent pricing actually. This makes me want to try to deploy something like llamafile/ollama or similar to fly.io+gpu for my personal on-demand llama/llm whims. (@simonw I’m looking at you— I feel like if you haven’t already done this on fly.io, that you’re probably thinking about it :) ) Seems private enough.. could throw some basic auth on top of it for me and trusted friends/family so I don’t get crazy bills. But privacy-wise with fly.io I think it’s good enough. Only problem might be everytime it spins up to download the large model might be wasteful as far as getting charged for network/bw usage— wonder if it would be more cost-efficient to have persistent storage or just see how much time and bw it is to download on every cold start…
- Palmik 3y agoIt's horrible horrible pricing! Their on demand price for A100 is what gets you H100 sxm in other places.
- paxys 3y agoIt's funny that "We managed to get our hands on a handful of H100s/A100s" is an actual (and profitable) tech business model right now.
- latchkey 3y ago"Selling shovels in a gold rush" has always been a profitable business model.
- deleted 3y ago[deleted]
- ptero 3y agoIt seems to be that a large part of the AWS business model is the revenue received as a fledgling startup tries to grow (and ideally succeeds) into a large company. As a small tech startup, an easy button for compute needs is perfectly sensible, as it allows focusing on the product. If the startup grows switching becomes more expensive, so AWS gets its money as long as the amounts are not seen as the main cost driver. That stage, I think (with no hard data), is the AWS sweetspot. The company is paying a lot for AWS, but does not yet want to do a full analysis, hire dedicated cloud cost optimization staff and deal with friction of switching. If the startup grows stable and profitable it will likely do a proper cost analysis and make AWS bills saner, maybe with a mix of on-prem, AWS and non-AWS cloud services. But that requires a stable period, both in time and in functionality, which is not something that an unprofitable startup has. I think with the end of ZIRP and tighter access to VC funds the number of startups that can afford losing a lot of money to go through an explosive growth period will shrink, and so will the AWS profits.
- ctocoder 3y agohttps://console.crusoecloud.com/request https://console.crusoecloud.com/request They are providing H100s, A100s, L40s for very cheap. They also do not charge for the network usage. I Highly recommend them as the price per flop is unbeatable anyplace, and they have over 4000 gpus to use at a time.
- apitman 3y ago> Given hyperscaler dominance of the overall public cloud market, which demands vast investments in infrastructure and range of services that make little or no revenue, challengers like CoreWeave have an opportunity to succeed with a focus on premium AI services without the burden of hypercaler-level investments overall Interesting. GPU-only providers targeting the AI market only need to implement a fraction of the services that AWS does. They don't even need to be geographically distributed. What does it matter if your GPU cluster is on the other side of the planet?
- ReptileMan 3y agoTo have cheaper GPUs we need at least four companies to design them and four companies that are able to produce them. Before that you always have cartel like behavior.
- latchkey 3y agoMaybe on the consumer low end, but on the enterprise level... it'll always be expensive.
- rootedbox 3y ago@ Core-weave an A100 40GB NVLink is $2.06 ... the only way they are doing this is by burning investor money. At that price it running it 24/7 it will take way over a year to recoup hardware + electricity cost. So my suggestion.. dump all your work @ Core-weave.. It's cheaper than buying the hardware yourself let alone the cost of managing it.
- jerrygenser 3y agoDoes this assume retail price? Maybe they are getting a discount for buying in bulk
- latchkey 3y ago> the only way they are doing this is by burning investor money... it will take way over a year to recoup hardware They've been running for years already. They can also offer these lower end gpus at that price cause the higher end ones offset things.
- nextworddev 3y agoI have good reasons to believe that all this capex into GPUs will backfire. Metrics aren’t looking good for AI adoption, and growth rate is slowing down while inference and training costs are plummeting.
- latchkey 3y agoThe need for more powerful compute is never going to go away and it will continue to grow. Especially around compute that is difficult to obtain and deploy.
- DrNosferatu 3y agoBesides Alibaba Cloud and AWS, are there other FPGA cloud services?
- teitoklien 3y agoVMAccel is popular for FPGAs like Xillinx High End FPGAs, Achronix, etc. https://www.vmaccel.com/solutions https://www.vmaccel.com/solutions
- latchkey 3y agoI've run one in the past, we might do it again if there is enough demand. Let me know!
- maxchehab 3y agoBoth Core Weave & Lambda Labs have fairly predatory pricing making it impossible to rent GPUs without a yearly contract. This doesn't make sense for training models, where a training run is on the scale of days & weeks. I wished that the techcrunch article mentioned other companies, like sfcompute, which offer hourly compute instead of yearly contracts.
- thundergolfer 3y agoLambda Labs has on-demand GPUs. Just put in a credit card and you're able to launch. I launched an 1x H100 server just 10 minutes ago on Lambda Labs. The price is also $2.49/hr which does not seem predatory at all.
- matroid 3y agoI have never seen 1x H100 available on Lambda Labs. Don't know why though.
- williamstein 3y agoI've been checking about twice a week for the last 6 months, and they are very rare, but it does happen. I caught one on video 2 weeks ago! https://youtu.be/NkNx6tx3nu0?t=744 https://youtu.be/NkNx6tx3nu0?t=744
- latchkey 3y agoI don't think it is predatory, I think it just happened over time due to demand. It is a marketplace after all. https://news.ycombinator.com/item?id=40101377 https://news.ycombinator.com/item?id=40101377
- ec109685 3y agoThat would make the TechCrunch article highly misleading since AWS and the other clouds offer big savings for reserved instances.
- dmyriel 3y agoHardware is a vendor lock-in. I can’t imagine sinking millions into a massive project and getting stuck with a huge bill. Then, to migrate away you need all sorts of devops folks and the ability to deal with incompatibilities. Uncertainty about pricing and the hardware bottleneck is a real problem for our users. I just raised this point in our blog today. https://qdrant.tech/blog/are-you-vendor-locked/ https://qdrant.tech/blog/are-you-vendor-locked/
- Palmik 3y agoI really don't get this. Most of these low cost providers actually re-rent GPUs from Azure, AWS or GCP (*), yet they offer much better on-demand pricing (as low as $3.8/hr for H100 sxm and $2.5/hr for PCIe). And it's a fact that you can get even much better on-demand (not to mention reserved) pricing from the big clouds if you're a decent startup with connections. If one of these clouds offered fair pricing to SMBs, it could be a great bottoms up growth strategy. (*) Not LambdaLabs afaik, but they rarely have on demand capacity anyway, and you can only get reasonable price with 3 year reservation (which is, surprise surprise, more than the hardware cost).
- claytonjy 3y agoThe middlemen you're talking about do two things: buy lots of reserved compute on the hyperscalers, and then pit the hyperscalers against eachother to get better pricing. If you're reserving thousands of GPUs from the same hyperscaler, even if they're the only cloud you run on, you're not paying the price shown in the calculator. If you have other suppliers, you'll get an even better deal. Then you resell that reserved compute as on-demand compute, somewhere between your costs and what your customers would pay a hyperscaler directly.
- thundergolfer 3y agoIt's the opposite of what you say. Slim to none of the alternative cloud providers re-rent their GPUs. Coreweave, Fluidstack, Lambda Labs, Paperspace, Cudo Compute, Hydra, Datacrunch.io, Vultr, Crusoe Cloud, SF Compute. As far as I can tell none of these providers give you a GPU originating in Azure, AWS, or GCP.
- deleted 3y ago[deleted]
- Palmik 3y agoRunPod is one of the providers that has some of the best on demand pricing (there are places where you can get H100 PCIe for less, but they typically don't have any capacity) without contracts and doesn't own hardware (or didn't as of few months ago).
- edgoode 3y agoThis is a trend we noticed early last year, so we started building a single console for all these clouds at https://shadeform.ai https://shadeform.ai. It has been amazing to watch this industry explode, and we believe it is great for consumers. The same instances on Amazon versus these alternative providers are 3x more expensive. NVIDIA and many hardware providers are leaning into this trend. As clouds become more and more vertically integrated, AMD, NVIDIA, and others will benefit from spreading their hardware to more clouds. Knowing that these models will not be running in 3 easily controlled clouds may also benefit us in the long run as each provider will have different levels of comfort with models of varying capabilities.
- breakingcups 3y agoHow does Shadeform make money?
- ilaksh 3y agoIf you want a _really_ cheap alternative cloud, look at vast.ai. Not sure you can beat their prices, and they have 4090s and 3090s if you can use those. Something like RunPod might be second place for pricing.
- claytonjy 3y agoWhat are folk's experiences with alternative cloud GPUs for inference? If you're doing a lot of model training, buying GPUs or long-term reservations of GPUs is a no-brainer. But when it comes to inference, latency matters and it gets trickier talking between e.g. your AWS infra and your GPUs somewhere else. It seems lots of providers can give you enough to get by doing inference in a company's earliest stages. But what if I need hundreds or thousands of A100s during peak usage? Is anyone doing this successfully with a non-hyperscaler?
- htrp 3y agoJust go with an inference provider like fireworks/together/modal/baseten
- claytonjy 3y agoAn issue we've had when looking into some of these is that, they provide a layer of software abstraction we're not looking for. I don't want to use some providers bespoke library to wrap my model code; I just want to use NVIDIA Triton, either by providing an image or by providing a model repo. I only want the inference provider to handle the hardware. I understand that's exactly what those provider _don't_ want, because it means they can't lock us in. But particularly when comparing an inference provider to GCP, where we already run everything in Triton on GKE, I don't want to rewrite my code just to see what their hardware layer is like. Another complication is we often run multiple tightly-integrated models for a single application, where having them on the same GPU is critical. This is tricky or impossible in some inference-provider-frameworks. There's too many options for running the latest LLM, and far fewer for running a bespoke set of fine-tuned models on GPUs.
- bzmrgonz 3y agosaw one touting itself to be the airbnb for gpu's
- latchkey 3y agoThat was just a marketplace to resell compute that others are running. A super common practice in the industry.
- la64710 3y agoHow are people using these GPU clouds with their data residing in one of the big cloud providers like AWS or Azure? Are they paying for egress to get data onto the GPU clouds?
- user3939382 3y agoI’ve been using Wasabi for S3-interface cloud storage. Way cheaper and works great.
- d416 3y agoSkypilot is worth a mention here: https://github.com/skypilot-org/skypilot https://github.com/skypilot-org/skypilot Open source CLI to deploy multiple gpu vm’s on all major cloud providers, with an option to use spot pricing with 1 cheap vm used as a controller to always make sure you have the most inexpensive deployment available with failover and load balancing. It’s like beating the cloud providers at their own game I wouldn’t be surprised if they banned it.
- benced 3y agoThe San Francisco Compute Company has been plastering their name at the 4th and King Caltrain station with a similar thesis: https://sfcompute.com/ https://sfcompute.com/
- devops000 3y agoThey are booming in VC money, they are not talking about user usages.
- nektro 3y agothis is gonna crash big time and idk what's gonna happen on the other side