28 ms·
Fly.io has GPUs now
- Havoc 3y agoHow fast is the spin up/down on this scale to zero? If it is fast this could be pretty interesting
- amanda99 3y agoI think the bigger question is how long it takes to load any meaningful model onto the GPU.
- fideloper 3y agothat’s exactly right. gpu-friendly base images tend to be larger (1-3g+) so that takes time (30s - 2m range) to create a new Machine (vm). Then there’s “spin up time” of your software - downloading model files adds as long as it takes to download GB of model files. Models (and pip dependencies!) can generally be “cached” if you (re)use volumes. Attaching volumes to gpu machines dynamically created via the API takes a bit of management on your end (in that you’d need to keep track of your volumes, what region they’re in, and what to do if you need more volumes than you have)
- dathinab 3y agoI know it's not common in research and makes often little sense there. But at least in theory for deployments you should generate deployment images. I.e. no pip included in the image(!), all dependencies preloaded, unnecessary parts stripped, etc. Models likely might also be bundled, but not always. Still large images, but also depending on what they are for the same image might be reused often so it can be cached by the provider to some degree.
- iambateman 3y agoIt’s cool to see that they can handle scaling down to zero. Especially for working on experimental sites that don’t have the users to justify even modest server costs. I would love an example on how much time a request charges. Obviously it will vary, but is it 2 seconds or “minimum 60 seconds per spin up”?
- mrkurt 3y agoWe charge from the time you boot a machine until it stops. There's no enforced minimum, but in general it's difficult to get much out of a machine in less than 5 seconds. For GPU machines, depending on data size for whatever is going into GPU memory, it could need 30s of runtime to be useful.
- andes314 3y agoDo you offer some sort of keep_warm parameter that removes this latency (for a greater cost)?
- mrkurt 3y agoYou control machine lifecycles. To scale down, you just set the appropriate restart policy, then exit(0). You can also opt to let our proxy stop machines for you, but the most granular option is to just do it in code. So yes, kind of. You just wait before you exit.
- Aeolun 3y agoSo just to confirm, for these workloads, it’d start a machine when the request comes in, and then shut it down immediately after the request is finished (with some 30-60s in between I suppose)? Is there some way to keep it up if additional requests are in the queue? Edit: Found my answer elsewhere (yes).
- sodality2 3y agoHow long does model loading take? Loading 19GB into a machine can't be instantaneous (especially if the model is a network share).
- loloquwowndueo 3y agoThere are no “network shares”. The typical way to store model data would be in a volume, which is basically local nvme storage.
- holoduke 3y agoAnybody has experience with the performance. First glance is that they are quite expensive. Compared to for example Hetzner (cpu machines)
- impulser_ 3y agoI'm not sure about others, but you can get A100s with 90gb of RAM from DigitalOcean for $1.15 an hour. So about 1/3 the price. You can even get H100s for cheaper than these prices at $2.24 an hour. So these do seem a bit expensive, but this might be because there is high demand for them from customers and they don't have the supply.
- skrtskrt 3y agogetting supply is super hard right now, DigitalOcean just straight up bought Paperspace to get access to those GPUs. The whole reason Coreweave is on a fat growth trajectory right now is they used their VC money to buy a ton of GPUs at the right time
- deleted 3y ago[deleted]
- treesciencebot 3y agoJust to correct the record, both $1.15 per A100 and $2.24 per H100 require a 3-year-commitment. On-demand prices are 2.5X that.
- Aeolun 3y ago> $2.24/hour pricing is for a 3-year commitment. On-demand pricing for H100 is $5.95/hour under our special promo price.\$1.15/hour pricing is for a 3-year commitment. Wow, that’s some spectacularly false advertising.
- dathinab 3y agoCompany I work for had multiple times problems of not being able to allocate any gpus from some larger cloud providers (with the region restrictions we have, which still include all of EU as regions). (I'm not sure which of them it was, we are currently evaluating multiple providers and I'm not really involved in that process.)
- andes314 3y agoHas anyone who has used Beam.Cloud compare that service to this one?
- deleted 3y ago[deleted]
- nextworddev 3y agoSomehow cheaper than AWS?
- reactordev 3y agoAWS isn’t the cheapest so how is that a surprise? They are a business and know how to turn the right knobs to increase cash flow. GPUs for AI is one major knob right now.
- CGamesPlay 3y agoAWS is one of the most expensive infrastructure providers out there (especially anything beyond the "basic" services like EC2). And even though AWS still has some globally-notable uptime issues, "nobody ever got fired for picking AWS".
- dathinab 3y agoI mean from hearsay of people which had to work with AWS & Google Cloud & Microsoft Azure it seems to me that the other two are in practice worse to a point they would always pick AWS over them even through they hate the AWS UX. And if it's the best of the big 3 providers, then it can't that bad, right ..... right? /s
- andersa 3y agoIt would be absurd if it wasn't.
- seabrookmx 3y agoThey're a "real" cloud provider (with their own hardware) and not a reseller like Vercel and Netlify. So this isn't _that_ surprising. AWS economies of scale do allow them to make certain services cheap but only if they choose. A lot of time they choose to make money!
- patmorgan23 3y agoThey run their own data centers.
- nakovet 3y agoAbout Fly but not about the GPU announcement, I wish they had a S3 replacement, they suggest a GNU Affero project that is a dealbreaker for any business, needing to leave Fly to store user assets was a dealbreaker for us to use Fly on our next project, sad cause I love the simplicity, the value for money, the built in VPN.
- benatkin 3y agoThis looks promising https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs
- candiddevmike 3y agoSeaweed requires a separate coordination setup which may simplify the architecture but complicates the deployment.
- JoshTriplett 3y ago> I wish they had a S3 replacement, they suggest a GNU Affero project that is a dealbreaker for any business AGPL does not mean you have to share everything you've built atop a service, just everything you've linked to it and any changes you've made to it. If you're accessing an S3-like service using only an HTTPS API, that isn't going to make your code subject to the AGPL.
- bradfitz 3y agoRegardless, some companies have a blanket thou-shalt-not-use-AGPL-anything policy.
- anonzzzies 3y agoYep, our lawyers say to not use and we have to check components and libs we use too. People are really shooting themselves in the foot with that license.
- 3y ago
- LuciBb 3y ago[dead]
- hansonpeter 3y ago[dead]
- riquito 3y agoIs there any configuration to keep alive the machine for X seconds after a request has been served, instead of scaling down to zero immediately? I couldn't find it skimming the docs
- mrkurt 3y agoMachines are both dumber and more powerful than you'd think. Scaling down means just exit(0) if you have the right restart policy set. So you can implement any kind of keep-warm logic you want.
- Aeolun 3y agoOh! I hadn’t thought if it like that. That makes sense.
- kylemclaren 3y agoyou might also be looking for `kill_signal` and `kill_timeout`: https://fly.io/docs/reference/configuration/#runtime-options https://fly.io/docs/reference/configuration/#runtime-options
- xena 3y agoHi, author of the post and Fly.io devrel here in case anyone has any questions. GPUs went GA yesterday, you can experiment with them to your heart's content should the fraud algorithm machine god smile upon you. I'm mostly surprised my signal post about what the "GPUs" are didn't land well here: https://fly.io/blog/what-are-these-gpus-really/ https://fly.io/blog/what-are-these-gpus-really/ If anyone has any questions, fire away!
- qeternity 3y agoI posted further down before seeing your comment. First, congrats on the launch! But who is the target user of this service? Is this mostly just for existing fly.io customers who want to keep within the fly.io sandbox?
- subarctic 3y agoCommenters like this, for one thing: https://news.ycombinator.com/item?id=34242767 https://news.ycombinator.com/item?id=34242767
- xena 3y agoPart of it is for people that want to do GPU things on their fly.io networks. One of the big things I do personally is I made Arsène (https://arsene.fly.dev https://arsene.fly.dev) a while back as an exploration of the "dead internet" theory. Every 12 hours it pokes two GPUs on Fly.io to generate article prose and key art with Mixtral (via Ollama) and an anime-tuned Stable Diffusion XL model named Kohaku-XL. Frankly, I also see the other part of it as a way to ride the AI hype train to victory. Having powerful GPUs available to everyone makes it easy to experiment, which would open Fly.io as an option for more developers. I think "bring your own weights" is going to be a compelling story as things advance.
- deleted 3y ago[deleted]
- gooseyman 3y agohttps://en.m.wikipedia.org/wiki/Dead_Internet_theory https://en.m.wikipedia.org/wiki/Dead_Internet_theory What have you learned from the exploration?
- niz4ts 3y agoAs far as I know, Fly uses Firecracker for their VMs. I've been following Firecracker for a while now (even using it in a project), and they don't support GPUs out of the box (and have no plan to support it [1]). I'm curious to know how Fly figured their own GPU support with Firecracker. In the past they had some very detailed technical posts on how they achieved certain things, so I'm hoping we'll see one on their GPU support in the future! [1]: https://github.com/firecracker-microvm/firecracker/issues/1179#issuecomment-1846090947 https://github.com/firecracker-microvm/firecracker/issues/11...
- mrkurt 3y agoThe simple spoiler is that the GPU machines use Cloud Hypervisor, not Firecracker.
- niz4ts 3y agoWay simpler than what I was expecting! Any notes to share about Cloud Hypervisor vs Firecracker operationally? I'm assuming the bulkier Cloud Hypervisor doesn't matter much compared to the latency of most GPU workloads.
- tptacek 3y agoThey are operationally pretty much identical. In both cases, we drive them through a wrapper API server that's part of our orchestrator. Building the cloud-hypervisor wrapper took me all of about 2 hours.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- yencabulator 3y agoThere has been weirdly little discussion on HN about Cloud Hypervisor. I guess because it's such a horribly bland non-descriptive Enterprise Naming name? It looks pretty sweet. Rust & sharing libraries with Firecracker and ChromeOS's crosvm, with more emphasis on long-running stateful services than in Firecracker. https://github.com/cloud-hypervisor/cloud-hypervisor https://github.com/cloud-hypervisor/cloud-hypervisor https://github.com/rust-vmm https://github.com/rust-vmm
- qeternity 3y agoWho is the target market for this? Small/unproven apps that need to run some AI model, but won't/can't use hosted offerings by the literally dozens of race-to-zero startups offering OSS models? We run plenty of our own models and hardware, so I get wanting to have control over the metal. I'm just trying to figure out who this is targeted at.
- mrkurt 3y agoWe have some ideas but there's no clear answer yet. Probably people building hosting platforms. Maybe not obvious hosting platforms, but hosting platforms.
- KTibow 3y agoFly is an edge network - in theory, if your GPUs are next to your servers and your servers are next to your users, your app will be very fast, as highlighted in the article. In practice this might not matter much since inference takes a long time anyway.
- joshxyz 3y agothis is crazy, this move alone cements fly as an edge player for the next 3 / 5 / 10 years.
- tptacek 3y agoWe're really a couple things; the edge stuff was where we got started in 2020, but "fast booting VMs" is just as important to us now, and that's something that's useful whether or not you're doing edge stuff.
- 3kBPE 3y ago[dead]
- dathinab 3y agoTL;DR: (skip to last paragraph) - having the GPU compute in the same data center or at least from the same cloud provider can be a huge plus - it's not that rare for various providers we have tried out to run out of available A100 GPUs, even with large providers we had issues like that multiple times (less an issue if you aren't locked to specific regions) - not all providers provide a usable scale down to zero "on demand" model, idk. how well it works with fly long term but that could be another point - race-to-zero startups have the tendency to not last, it's kind by design from a 100 of them just a very few survive - if you are already on fly and write a non-public tech demo which just gets evaluated a few times their GPU offering can act like a default don't think much about it solution (through you using e.g. Huggingface services would be often more likely) - A lot of companies can't run their own hardware for various reasons, at best they can rent a rack in another Datacenter but for small use use-cases this isn't always worth it. Similar there are use cases which do might A100s but only run them rarely (e.g. on weekly analytics data). Potentially less then 1h/w in which case race-to-zero pricing might not look interesting at all To sum up I think there are many small reasons why some companies, not just startups, might have interest in fly GPUs, especially if they are already on fly. But there is no single "that's why" argument, especially if you are already deploying to another cloud.
- dcsan 3y agoCan fly run cog files like replicate uses? Would be nice to take those pre packaged models run them here with the same prediction API Maybe cos it's replicate they might be hesitant to adopt it but it does seem to make things a lot smoother Even with lambalabs' lambdastack I still hit cuda hell https://github.com/replicate/cog https://github.com/replicate/cog
- dcsan 3y agoupdate I guess cog files run fine on fly. https://news.ycombinator.com/item?id=39411748 https://news.ycombinator.com/item?id=39411748
- UncleOxidant 3y agoI don't want to deploy an app, I just want to play around with LLMs and don't want to go out and buy an expensive PC with a highend GPU just now. Is Fly.io a good way to go? What about alternatives?
- leourbina 3y agoPaperspace is a great way to go for this. You can start by just using their notebook product (similar to Collab), and you get to pick which type of machine/GPU it runs on. Once you have the code you want to run, you can rent machines on demand: https://www.paperspace.com/notebooks https://www.paperspace.com/notebooks
- janalsncm 3y agoI used paperspace for a while. Pretty cheap for mid tier gpu access (A6000 for example). There were a few things that annoyed me though. For one, I couldn’t access free GPUs with my team account. So I ended up quitting and buying a 4090 lol.
- mrkurt 3y agoYou might actually be better off building a gaming rig and using that. The datacenter GPUs are silly expensive, because this is how NVIDIA price discriminates. The consumer, game GPUs work really well and you can buy them for almost as cheap as you can lease datacenter ones.
- mrcwinn 3y agohttps://ollama.com/ https://ollama.com/ - Easy setup, run locally, free.
- UncleOxidant 3y agoYeah, but I've got an RTX1070 in my circa 2017 PC. How well is that going to work?
- 3y ago
- k8svet 3y agoDoes it have basic functioning other stuff? I am shocked at how our production usage of Fly has gone. Even basic stuff as support not being able to just... look up internal platform issues. Cryptic/non-existent error messages. I'm not impressed. It feels like it's compelling to those scared of or ignorant of Kubernetes. I thought I was over Kubernetes, but Fly makes me miss it.
- deleted 3y ago[deleted]
- chachra 3y agoBeen on it 7 months, 0 issues. Feel like you're alone on this potentially.
- deleted 3y ago[deleted]
- weird-eye-issue 3y agoAlone? Every thread about Fly has complaints about reliability and people complain about it on Twitter too
- nixgeek 3y agoThat hasn’t been my experience with Fly but I’m sorry to hear it seems to be others :(
- chachra 3y agook possibly not alone, maybe the issues happened before I started using them extensively. I've had ~no downtime that affects me in 7 months. I do wish they had some features I need, but their support and responses are top notch. And I've lost much less hair and time than I would going full-blown AWS or another cloud provider.
- jokethrowaway 3y agoTo be fair most hosting providers come with plenty of public complaints about downtime. The big ones do way better, the best one is AWS, then GC and last Azure. They cost stupid money though. Digital ocean has been terrible for me, some regions just go down every month and I lose thousands of requests, increasing my churn rate. Fly.io had tons of weird issues but it got better in the last months. It's still very incomplete in terms of functionality and figuring out how to deploy the first time is a massive pain. My plan is to add Hetzner and load balance with bunnycdn across DO and H
- m3kw9 3y agoNow having GPUs is news now?
- Williams77 3y ago[dead]
- Mikejames 3y agoanyone know if this is a PCI passthrough for a full a100? or some fancy clever vgpu thing?
- dvrp 3y agotoo expensive
- ec109685 3y agoThe recipe example or any any LLM use case seems like a very poor way of highlighting “inference at the edge” given the extra few hundred ms round trip won’t matter.
- manishsharan 3y agoThis. I cannot think of a business case for running LLMs on the edge. Is this a Pets.com moment for the AI industry?
- unraveller 3y agoThe better use case is obviously voice assistant at the edge. As in voice 2 text 2 search/GPT 2 voice generated response. That is where ms matter but it is also a high abuse angle no one wants to associate with just yet. My guess is they are going to do this in another post, and if so they should make their own perplexity style online-gpt. For now they just wanted to see what else people can think up by making the introduction of it boring.
- ec109685 3y agoThere’s three options for inference: 1) On device inference 2) Inference “on the edge” 3) Inference in a data center Given fly is deployed in equinox data centers just like everyone else, fundamentally there isn’t much difference between #2 and #3.
- bugbuddy 3y agoThis is amazing and it shows that Nvidia should be the most valuable stock in the world. Every company, country, city, town, village, large enterprise, medium and small business, AI bro, Crypto bro, gamer bro, big tech, small tech, old tech, new tech, and start up want Nvidia GPUs. Nvidia GPUs will become the new green oil of the 21st century. I am all in and nothing short of a margin call will change my mind.
- isoprophlex 3y agoAlmost twice as cheap as Modal! Very nice!
- pgt 3y agoI was an early adopter of Fly.io. It is not production-ready. They should fix their basic features before adding new ones.
- urduntupu 3y agoUnfortunately true. Also jumped the fly.io ship after initial high excitement for their offering. Moved back to DigitalOcean's app platform. A bit more config effort, significantly pricier, but we need stability on production. Can't have my customers call me b/c of service interruption.
- throwaway220033 3y ago+1 - It's the most unreliable hosting service I've ever used in my life with "nice looking" packaging. There were frequently multiple things broken at same time, status page would always be green while my meetings and weekends were ruined. Software can be broken but Fly handles incidents with unprofessional, immature attitude. Basically you pay 10x more money for an unreliable service that just looks "nice". I'm paying 4x less to much better hardware with Hetzner + Kamal; it works reliably, pricing is predictable, I don't pay 25% more for the same usage next month. https://news.ycombinator.com/item?id=36808296 https://news.ycombinator.com/item?id=36808296
- ecmascript 3y agoComments like these are just sad to see on HN. It is not constructive. What is these basic features that need fixing you're speaking about and what is the fixes required?
- cschmatzler 3y agoReliability and support. Having even “the entire node went down” tickets get an auto-response to “please go fuck off into the community forum” is insane. What is the community forum gonna do about your reliability issues? I can get a 4€/mo server at Hetzner and have actual people in the datacenter respond to my technical inquiries within minutes.
- DreamGen 3y agoGreat, more competition for the price-gouging platforms like Replicate and Modal is needed. As always with these, I would be curious about the cold-start time -- are you doing anything smart about being able to start (load models into VRAM) quickly? Most platforms that I tested are completely naive in their implementation, often downloading the docker image just-in-time instead of having it ready to be deployed on multiple machines.
- wslh 3y agoInteresting. We have this discussing this kind of services (offloading training) over the last several days [1] [2] [3]. Thinking on the opportunity to compete with top cloud services such as Google Cloud, AWS, and Azure. [1] https://news.ycombinator.com/item?id=39353663 https://news.ycombinator.com/item?id=39353663 [2] https://news.ycombinator.com/item?id=39329764 https://news.ycombinator.com/item?id=39329764 [3] https://news.ycombinator.com/item?id=39263422 https://news.ycombinator.com/item?id=39263422
- deleted 3y ago[deleted]
- unixhero 3y agoI use Fly.io free tier to run uptime monitoring with Uptime kuma. It works insanely well, and I'm a really happy camper.
- wilddavsafaris 3y ago[flagged]
- wilddavsafaris 3y ago[flagged]
- Donaldzibe 3y ago[flagged]
- faust201 3y ago> The speed of light is only so fast This is the title of one of the sections. Why? Think IT sector needs to stop using such titles.
- jimnotgym 3y agoIt is a bit of an odd thing that we still call GPUs GPUs when the main use for them seems to have little to do with Graphics!