3 ms·
Hi, author of the post and Fly.io devrel here in case anyone has any questions. GPUs went GA yesterday, you can experiment with them to your heart's content sho
by xena 3y ago
Hi, author of the post and Fly.io devrel here in case anyone has any questions. GPUs went GA yesterday, you can experiment with them to your heart's content should the fraud algorithm machine god smile upon you. I'm mostly surprised my signal post about what the "GPUs" are didn't land well here: https://fly.io/blog/what-are-these-gpus-really/ https://fly.io/blog/what-are-these-gpus-really/
If anyone has any questions, fire away!
- qeternity 3y agoI posted further down before seeing your comment. First, congrats on the launch! But who is the target user of this service? Is this mostly just for existing fly.io customers who want to keep within the fly.io sandbox?
- subarctic 3y agoCommenters like this, for one thing: https://news.ycombinator.com/item?id=34242767 https://news.ycombinator.com/item?id=34242767
- xena 3y agoPart of it is for people that want to do GPU things on their fly.io networks. One of the big things I do personally is I made Arsène (https://arsene.fly.dev https://arsene.fly.dev) a while back as an exploration of the "dead internet" theory. Every 12 hours it pokes two GPUs on Fly.io to generate article prose and key art with Mixtral (via Ollama) and an anime-tuned Stable Diffusion XL model named Kohaku-XL. Frankly, I also see the other part of it as a way to ride the AI hype train to victory. Having powerful GPUs available to everyone makes it easy to experiment, which would open Fly.io as an option for more developers. I think "bring your own weights" is going to be a compelling story as things advance.
- deleted 3y ago[deleted]
- gooseyman 3y agohttps://en.m.wikipedia.org/wiki/Dead_Internet_theory https://en.m.wikipedia.org/wiki/Dead_Internet_theory What have you learned from the exploration?
- xena 3y agoEnough that I'd probably need to write a blogpost about it and answer some questions that I have about it. The biggest one I want to do is a sentiment analysis of these horoscopes vs market results to see if they are "correct".
- cosmojg 3y agoInteresting setup! What's the monthly cost of running Arsène on fly.io?
- xena 3y agoBecause I have secret magical powers that you probably don't, it's basically free for me. Here's the breakdown though: The application server uses Deno and Fresh (https://fresh.deno.dev https://fresh.deno.dev) and requires a shared-1x CPU at 512 MB of ram. That's $3.19 per month as-is. It also uses 2GB of disk volume, which would cost $0.30 per month. As far as post generation goes: when I first set it up it used GPT-3.5 Turbo to generate prose. That cost me rounding error per month (maybe like $0.05?). At some point I upgraded it to GPT-4 Turbo for free-because-I-got-OpenAI-credits-on-the-drama-day reasons. The prose level increase wasn't significant. With the GPU it has now, a cold load of the model and prose generation run takes about 1.5 minutes. If I didn't have reasons to keep that machine pinned to a GPU (involving other ridiculous ventures), it would probably cost about 5 minutes per day (increased the time to make the math easier) of GPU time with a 40 GB volume (I now use Nous Hermes Mixtral at Q5_K_M precision, so about 32 GB of weights), so something like $6 per month for the volume and 2.5 hours of GPU time, or about $6.25 per month on an L40s. In total it's probably something like $15.75 per month. That's a fair bit on paper, but I have certain arrangements that make it significantly less cheap for me. I could re-architect Arsène to not have to be online 24/7, but it's frankly not worth it when the big cost is the GPU time and weights volume. I don't know of a way to make that better without sacrificing model quality more than I have to. For a shitpost though, I think it'd totally worth it to pay that much. It's kinda hilarious and I feel like it makes for a decent display of how bad things could get if we go full "AI replaces writers" like some people seem to want for some reason I can't even begin to understand. I still think it's funny that I have to explicitly tell people to not take financial advice from it, because if I didn't then they will.
- tptacek 3y agoThis isn't the target user, but the boy's been using it at the soil bacteria lab he works in to do basecalling for a FAST5 data from a nanopore sequencer.
- bl4kers 3y agoHow difficult world it be to set up Folding@home on these? https://foldingathome.org https://foldingathome.org
- xena 3y agoI'm not sure, the more it uses CUDA the easier I bet. I don't know if it would be fiscally worth it though.
- deleted 3y ago[deleted]
- yla92 3y agoNot a question but the link "Lovelace L40s are coming soon (pricing TBD)" is 404.
- xena 3y agoUhhhh that's not ideal. I'll go edit that after dinner. Thanks!
- thangngoc89 3y agoIf it's a link to nvidia.com then it's expected to be broken. Seriously, I've never seen a valid link to nvidia.com
- Nevin1901 3y agoHow fast are coldstarts, and how do you compare against other gpu providers (runpod modal etc)
- thangngoc89 3y agoThis is right on time. I'm evaluating "severless" GPU services for my upcoming project. I see on the announcement that pricing is per hours. Is scaling to zero priced based on minutes/seconds? For my workflow, medical image segmentation, one file takes about 5 minutes.
- LtdJorge 3y agoHow long does it take to process on CPU?
- thangngoc89 3y agoAbout 1-1.5 hours on a Core i9-13900K. Which is unacceptable for my use case.
- benreesman 3y agoI'd be fascinated to hear your thoughts on Apple hardware for inference in particular. I spend a lot of time tuning up inference to run locally for people with Apple Silicon on-prem or even on-desk, and I estimate a lot of headroom left even with all the work that's gone into e.g. GGUF. Do you think the process node advantage and SoC/HBM-first will hold up long enough for the software to catch up? High-end Metal gear looks expensive until you compare it to NVIDIA with 64Gb+ of reasonably high memory bandwidth attached to dedicated FP vector units :) One imagines that being able to move inference workloads on and off device with a platform like `fly.io` would represent a lot of degrees of freedom for edge-heavy applications.
- xena 3y agoWell, let me put it this way. I have a MacBook with 64 GB of vram so I can experiment with making an old-fashioned x.ai clone (the meeting scheduling one, not the "woke chatgpt" one) amongst other things now. I love how Apple Silicon makes things vroomy on my laptop. I do know that getting those working in a cloud provider setup is a "pain in the ass" (according to ex-AWS friends) so I don't personally have hope in seeing that happen in production. However, the premise makes me laugh so much, so who knows? :)
- LtdJorge 3y agoIf you want something similar to Apple silicon for AI, look for AMD Instinct MI300A. 24 cores, 256MB cache, 128GB of HBM3 and 1 petaflop of FP16.