8 ms·
The State of Serverless GPUs
- ris 4y agoInferless provides Serverless GPU solutions
- pavlov 4y agoThe term “serverless GPU” somehow wrecks my brain. Logically the absence of a server suggests its opposite, and the opposite of a server is a client, and client GPUs are the default. But this means “server GPU that’s available on-demand for very short-lived jobs” I guess.
- touisteur 4y agoYeah, should be 'elastic' and 'very reduced runtime, possibly just inferencing', so exposing Triton to an API gateway and putting a custom load balancer and task queue facade ? Curious too. Also, GPUs do other things than inferencing, right?
- ralusek 4y agoI don't understand how people haven't gotten over this yet. When someone says serverless, I immediately understand that to mean "we've obfuscated the underlying server hardware from the consumer of this product." It means "you don't think about servers," not "there are no servers."
- pavlov 4y agoI’ve got used to the term, but when it’s in a new context like “serverless GPU”, its inherent absurdity resurfaces.
- eurekin 4y agoYup. My first intuition was "probably some server gpu (a100), but sold separately"
- gpm 4y agoBecause if I'm thinking about the gpu I'm fundamentally thinking about the hardware, the server. Serverless responding to http requests. Sure. I write some code. It gets fed data and returns data. I don't have to know how many cores the server has, or what microcode version the CPU is, or how many other things are running on the server, or if I'm writing an interpreted language (probably the case) even what architecture the CPU is. But... I need to know all of that if I'm writing GPU code today.
- rcme 4y agoAre you really thinking about the hardware when thinking about the GPU? For instance, if you use pytorch to write a NN, don't you kind of expect it to execute on the GPU without needing to get into the gritty details of it?
- seabass-labrax 4y agoThe specific version of Nvidia CUDA or AMD ROCm that the GPU supports is often very important; some software needs to be compiled from a specific branch or with specific settings to support a given version. Case in point: the official PyTorch website offers four distinct builds for various platforms, two of those being different versions of CUDA. I think it'll still be useful for plenty of people to choose which GPU runs their code, even if it's compatible with any GPU offered by the service. You might want to choose an older, cheaper GPU for basic parallel computation, but as supply catches up with demand for a newer and more energy-efficient model you'll then want to switch to that. There are only so many GeForce 4090s to go round :)
- 1letterunixname 4y agoIt comes from the magical thinking of having too many marketing people and not enough engineers who actually build shit.
- DonHopkins 4y agoSays the guy whose user name claims to have one letter, but actually contains one digit. ;)
- faeriechangling 4y agoI dunno, maybe it's because the jargon is stupid?
- mgraczyk 4y ago"Serverless" has been a standard term in industry for at least 7 years. Sometimes words don't map perfectly onto the subcomponents that form them. For example a "mailbox" isn't always a box, and doesn't always contain mail. You just have to learn to use the words and not worry so much about the etymology.
- pavlov 4y agoThe tortured etymology becomes apparent again when these words are combined in new ways. “Serverless GPU” might be something like “mailbox SSD” in your example. What would that mean? It’s not obvious at first sight. The metaphor loses its power when it’s attached to a physical descriptor which is not a metaphor.
- mgraczyk 4y agoNo, it's obvious to everyone working with GPUs who knows what "serverless" means. It's a very standard construction in English. "Serverless GPU" means "GPU" that is "serverless". If you know what both words mean in the jargon, you know what they mean together. It's ok to not know what they mean, but arguing that it's "tortured" rings to me as misguided obtuseness.
- Mike_12345 4y ago> It's a very standard construction in English. It's a new buzzword coined by the marketing team at Amazon in 2014. Somewhat confusing here as you are renting time on a GPU server described as "serverless".
- mgraczyk 4y agoWhen I said "standard construction", I was referring to the combination of "serverless" and "GPU" to make "serverless GPU". Also you're factually wrong. Amazon didn't invent this term and it was widely used before 2014. Here's a 2012 article using the term. https://readwrite.com/why-the-future-of-software-and-apps-is-serverless/ https://readwrite.com/why-the-future-of-software-and-apps-is... I truly don't understand the bizarre and conspiracy-theory laden opposition to terms like this. It's a word, everyone who matters knows that it means.
- 1letterunixname 4y agoIt's a stupid misnomer. "Cloud" is still another. Perhaps "distributed scheduling and execution" is too straightforward than marketing wank.
- TeMPOraL 4y agoI mean it's all because marketers didn't want people to think of DCOM or CORBA when dealing with this "new" technology. (I can kind of understand it in case of CORBA - if you put the word "broker" in the name of your product, no one else will want to associate with it.)
- thomastjeffery 4y agoSomething like "GPU for Serverless computing" would be much better. GPUs don't contain servers. Serverlesses contain GPUs.
- DonHopkins 4y agoThe term should be "Serverless GPUless" to be fair and consistent. What's good for the goose is good for the gander.
- dvh 4y agoServerless = pay per request Cloud computing = someone's else's computer
- ukuina 4y agoI, too, think of "classical serverless" rather than "neo-serverless": https://www.sqlite.org/serverless.html https://www.sqlite.org/serverless.html ------ Recently, folks have begun to use the word "serverless" to mean something subtly different from its intended meaning in this document. Here are two possible definitions of "serverless": Classic Serverless: The database engine runs within the same process, thread, and address space as the application. There is no message passing or network activity. Neo-Serverless: The database engine runs in a separate namespace from the application, probably on a separate machine, but the database is provided as a turn-key service by the hosting provider, requires no management or administration by the application owners, and is so easy to use that the developers can think of the database as being serverless even if it really does use a server under the covers.
- touisteur 4y agoWondering whether most major inference libraries support storage-direct or if the listed providers are cheaping out on storage latency. Several seconds to load a 100MB model when pcie 4 is 256Gb/s I'd have expected an order of magnitude less - at least from my experience with gpudirect and real-time processing of up to 200Gb/s network streams with datacenter gpus (not performing inference but I'm talking about actual streaming to gpu memory here). PCIe-5 and H100 (the only one in the line-up that has PCIe-5...) should also improve on that.
- touisteur 4y agoAdditional note, the L4 inferencing board is also still on PCIe-4 (like the L40, very sadly), so NVIDIA doesn't seem to foresee the need for more bandwidth on inferencing workloads in the next 3 years (until they maybe get a H30 out, with the binned not-fully-functional H100s? Maybe? Hopefully?) and/or will make you pay for a full H100.
- todd3834 4y agoThis is a great overview! I’ve been working on a project where I’ve tried a few of these and it’s definitely a space where this data is super valuable. I really wanted to like Banana.dev but runpod consistently outperforms them for my use case. I love the innovation in this space. Here is my wishlist: 1. Faster cold starts. If you’re building a consumer product that has a request and response using one of these services I’m seeing at least a 20 second delay before the server begins working. 2. Much cheaper GPUs. This is an unrealistic expectation right now because supply and demand have these services completely crowded with people happy to pay. I just wish I could afford to have a few of the faster GPUs prewarmed ready to go but that would be several thousand dollars a month. Doable eventually but for my bootstrapped side project that hasn’t found product market fit it’s a little rough. 3. Ability to create custom models from the service. I’m on an Intel Mac and so making a custom model requires me to ssh into a machine. If only there was a service that let me rent a high end GPU service by the second. /sarcasm. I wasn’t able to get access to docker or install it on runpod and support confirmed it isn’t something they support. For custom model building I found the prices and flexibility to do whatever you need on lambdalabs.com to be best. Also their prices are blowing all of these other services away. However no serverless option. The space is so crowded with consumers I’m almost afraid to even mention them because I worry I won’t have GPUs available for me. I’m seeing this mentality a lot.
- edunteman 4y agoHey! Banana cofounder here. Firstly, thank you so much for trying our service, we'll do our best to meet performance expectations and win you back! Re: #1 and #2, cold boots are the most vital thing for us to solve, because it fixes #1 directly and helps #2 indirectly. As we drive cold starts exponentially toward 0s (obv impossible but there's a near-zero asymptote to what's possible, limited only by disk read throughput), it makes it more viable to actually run serverless and scale from 0->1 for each user call. Our goal RN is to hit 1s, as that's generally the sweetspot for LLM app builders to stop feeling pain from the cold boot. We're getting close (2-5s) for most models with Turboboot (shill warning https://www.banana.dev/blog/turboboot https://www.banana.dev/blog/turboboot). Glorious future would be more like 100ms, but depending on where the size of models end up, 1s+ cold boots may just be the cost of doing business. Re: #3, if I understand, you're looking for interactive compute (IE you ssh in, mess around, do training runs, attach a jupyter notebook). For my personal ML training I use and suggest Lambda Labs or Brev.dev. Have heard great things of Coreweave, and users of Mosaic have seemed quite satisfied but I don't believe it's as interactive as you may want. Banana has no plans to support interactive GPU sessions, to conserve focus toward being best at cold boots. You're definitely not alone in this wishlist, so I validate you. If I were building applications on top of a provider, I'd expect the same things. Big gnarly challenge with all the tools being a few years old at best! Fastest route to dependable tools is intense focus, us on cold boots. AMA, if anyone is interested in digging
- chewbacha 4y agoServerless feels like a low-interest rate era technology.
- varelse 4y ago[dead]
- blandcoffee 4y agoIs this a sarcastic quip or are you able to expand on this? I use a lot of serverless daily, handling events (even ML inference), and it seems to work great, but would love to understand the alternatives and your perspective.
- chewbacha 4y agoThe overhead of abstracting away the servers is a luxury in many ways. This extra cost I believe was heavily funded by low-interest rates which flushed the VC world with dough. There’s been a lot less serverless talk since the fed started cranking the rates
- itisit 4y agoSorry, but this feels like a total non sequitir. Serverless or FaaS is pretty mature now. People get the concept, businesses understand the savings, and the services and tooling are stable. We don't talk about it because it's boring.
- d0100 4y ago> Serverless or FaaS is pretty mature now Maybe the backend is, but the frontend aspect is very bad Both GCP and AWS have terrible web UI's for their cloud functions offerings and every deploy is so slow I'm lucky that next monday is a holiday so I can rest from the stress of having to use GCP last friday (on a deadline) Codesandbox should offer their own serverless functions so I can actually have serverless for the whole development cycle
- crazysim 4y agoSome of these are a bit more host a server for you or others to run. I wish this comparison also compared billing models a bit and any other value-adds. The cool thing about replicate.com is that you can use someone else's public model and it's billed to the callee. For someone like me who is gluing models together for a hobbiest setting, it's been great. For some image identification tasks, it's been pretty neat for me to be able to call https://replicate.com/andreasjansson/blip-2 https://replicate.com/andreasjansson/blip-2 which is someone else's already deployed model which was already pre-warmed by somebody or level of activity, and get results back. For me, I've been captioning images and putting them into OpenAI prompts. I've also myself put out https://replicate.com/nelsonjchen/minigpt-4_vicuna-13b https://replicate.com/nelsonjchen/minigpt-4_vicuna-13b to see if maybe it's an improvement in captioning. Unfortunately, it takes like 15 minutes to spin up. That said, it's currently free for me to put up. If someone else wants to run it, they can wait/pay. And they only pay for the runtime and not setup. And if it were to get popular, it'll be naturally warm for everyone. For me, it was 6x the cost of https://replicate.com/andreasjansson/blip-2 https://replicate.com/andreasjansson/blip-2, and although my experiment did not produce something suitable or usable for me, maybe those caveats are appropriate for someone else's use case, and they can super-easily reuse my deployment on their dime without costing me any money. Not to mention that replicate also put out some pretty alright APIs or libraries to call their service. It's been consistent. All this ease of use does come with some caveats, replicate.com is pretty expensive for the raw calls.
- 0x445442 4y agoPardon my ignorance on this matter but if the processing unit isn’t displaying graphics shouldn’t it just be referred to as a cpu?
- mgraczyk 4y agoNo, "GPU" is a specific piece of hardware. https://en.wikipedia.org/wiki/Graphics_processing_unit https://en.wikipedia.org/wiki/Graphics_processing_unit
- 0x445442 4y agoRight, but what graphics are they processing in a serverless context or when it’s mining bitcoin?
- mgraczyk 4y agoNone, "GPU" is a word with a distinct meaning from "unit that processes graphics"
- pdntspa 4y agoThe math is similar so GPUs have been hijacked by AI nerds Gamers everywhere are pissed that GPUs are so expensive now.
- varelse 4y ago[dead]
- DonHopkins 4y agoThey're making the line go up when they're mining bitcoin. https://www.youtube.com/watch?v=YQ_xWvX1n9g https://www.youtube.com/watch?v=YQ_xWvX1n9g
- suprfsat 4y agoIf a CPU isn't "central" to much of anything shouldn't it just be referred to as a PU?
- ec109685 4y agoI don’t understand how these small companies can compete with the big clouds ofer time. They aren’t offering anything fundamentally different than each other or the big clouds themselves. At some point, Lambda will offer GPU’s on their instances and there are plenty of serverless offering right now that have GPU’s. Fly.io at least is differentiated with their “run your compute closer to the user” with zero hassle. Though even that doesn’t seem super sustainable in the long term.
- deleted 4y ago[deleted]
- arpowers 4y agoThe large clouds are intentionally hard to use platforms with a different target customer than the smaller players …
- jonfromsf 4y agoWow I always thought AWS was crazy complicated just like .. because stuff is hard. But you're right, if it's complex enough people define their career identity as an "AWS expert" and at that point you have lock-in for life.
- jiggawatts 4y agoI've found that a lot of the complexity of the cloud is self imposed, the other half is because they're running a huge range of mixed-quality software that needs a lot of scaffolding to even run properly. For example, if the clouds used IPv6, then 90% of the networking complexity would simply evaporate. But they can't, because despite two decades of warnings, nobody has working IPv6 LAN networks. Similarly, IPv4-only server software is still being written, today. In comparison IPv4 needs a ton of infrastructure to function. Stateful subnet and IP assignments. Methods for dealing with overlapping private networks. Split DNS, private endpoints, NAT, and on and on... I could list other examples, but you get the idea. The big clouds are pandering to their customers, and the customers can't modernise their end fast enough, so you end up with complex features to account for all the legacy software and networks.
- uvesten 4y agoThe ”Executive summary” isn’t, it’s an abstract. There is zero actionable information there, it’s just describing the article. I was hoping to get a recommendation for which provider to use for which kinds of workloads. I guess I have to read it all then :/
- cateye 4y agoAs an avid Runpod user, I have come across some information that is not entirely accurate. - Although the number of models is limited, the platform has a community feature where users can fork models. I'm not entirely sure of the intended meaning, but contradicts with being able to bring any container. However, if it pertains to the selection of pre-made template containers, it is true that the options may be limited. Nonetheless, there aren't a significant number of commonly-used open-source models available either... Or it is referring to the API endpoints. That is indeed correct but it is confusing what this review is exactly about. - Post-deployment, it can be confusing to understand how the platform works, which may result in users receiving a bill if they are not careful. Although documentation is sometimes lacking, it is very clear what it costs and there is not such thing as a bill... - There is no bot or instant support mechanism available. They have a live chat available, and their response time was good. They are also very active on their Discord channel, providing speedy support to users.
- rain1 4y agowhy is it called serverless when it is a server
- smw 4y agoIs it really useful to be pedantic about it? It's a useful term at this point because people in the community know what it means, no?
- Pulcinella 4y agoI literally don’t know what it means. Does serverless just mean doing it locally? Like an on premises cluster of GPUs in the IT closet? If you are doing it on someone else computer, and that computer is a big server farm, I find it odd to use the term “serverless.” And by odd I mean literally the exact opposite of serverless.
- throwaway70701 4y ago"Serverless" is a cloud-native software development paradigm. Of course there is a server - the machine, and there is a server running on the machine (that you don't control). The point is in the code you write - it allows you (the developer) to forget about servers as in software as well as servers as in machines. A serverless program (often called a function) is invoked by a server on-demand instead of running all the time, and it doesn't listen for anything on any port. The client request is passed to it by the server (e.g. on standard input) and it's supposed to pass the response back to the server (e.g. through standard output) - and the server sends it to the client. Then the program halts. Compare that to Django, Node.js or ASP.NET: A classic backend app exposes a HTTP server on a port and handles client connections itself - and thus it has to run all the time, it literally is a server (in the software sense). If you know PHP, that's the original serverless. As opposed to the Python/ASP.NET/Node.js backends, your page.php is invoked by the Apache daemon only when someone opens that page, and there's no "server.listen(3000)" in page.php. Serverless is cool because it allows the cloud provider to fully utilize a machine while the developer pays only for their portion of actual usage. You don't need to reserve a specific amount of compute resources and worry about up/down-scaling or about paying for unused hardware. Serverless GPUs are about bringing that concept to the GPU as a service space - ideally you'd have a function that uses the GPU. That function could be invoked by sending a request to the platform's server, at which point the server would load and execute it, pass the client request to it and pass the response back to the client once the function is done.
- CyberDildonics 4y agoA serverless GPU means a GPU on a server right?
- Animats 4y agoBy which they mean shared-hosting servers with GPUs.
- aledalgrande 4y agoDoes the summary at the end reek of autogeneration or is it just me? There is zero value in it.
- mountainriver 4y agoSurprised GKE autopilot isn’t mentioned as it now has some level of gpu virtualization, I’ve found it to be the most flexible fully featured way of approaching this problem