5 ms·

I really liked playing around with fly gpus, but it's just too expensive for hobby-use. Same goes for the rest of fly.io honestly. The DX is great and I wish I

I really liked playing around with fly gpus, but it's just too expensive for hobby-use. Same goes for the rest of fly.io honestly. The DX is great and I wish I could move all of my homelab stuff and public websites to it, but it'd be way too expensive :(

mrkurt 2y ago
This is near and dear to me, because I want people to run stuff like homelabs and side projects. What part of the cost gets out of hand? Having to have a Machine for every process? Do you remember what napkin math pricing you were working with?
- johntash 2y ago
  Hmm, having a machine for every process is part of it but I actually like that kind of isolation. Storage and bandwidth also add up fast. For example, I could get a digitalocean vm with 2gb ram, 1vcpu, 50gb storage, 2tb bandwidth for $12/mo. For the same specs at fly.io, it'd be ~$22/mo not including any bandwidth. It could be less if it scales to zero/auto stops. I recently tried experimenting with two different projects at fly. One was an attic server to cache packages for NixOS. Only used by me and my own vms. Even with auto scaling to zero, I think it was still around $15-20/mo. The other was a fly gpu machine with Ollama on it. The cold start time + downloading a model each time was kind of painful, so I opted for just adding a 100gb volume. I don't actually remember what I was paying for that, but probably another 20/mo? I used it heavily for a few days to play around and then not so much later. I do remember doing the math and thinking it wouldn't be sustainable if I wanted to use it for stuff like home-assistant voice assistant or going through pdfs/etc with paperless. On their own, neither of these are super expensive. But if I want to run multiple home services, the cost is just going to skyrocket with every new app I run. If I can rent a decent dedicated server for $100-$200/mo, then I at least don't have to worry about the cost increasing on me if a machine never scales to zero due to a healthcheck I forgot about or something like that. Sorry if it's a bit rambly, happy to answer questions!
  - mrkurt 2y ago
    No problem at all! I would be curious how the Attic server would have gone with a Tigris bucket and local caching. Not sure how hard that is to pull off, but Tigris should be substantially cheaper than our NVMes and if you don't really NEED the io performance you're not getting anything for that money. Which is a long winded way of saying "we aren't great at block storage for anything but OLTP workloads and caches". One thing we've struggled to communicate is how _cheap_ autosuspend/autostop make things. If that Machine is alive for 8 hours per day you're probably below $8/mo for that config. And it's so fast that it's viable for it to start/stop 45 times per day. It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones? We are suboptimal for things that make more sense as a bunch of containers on one host.
    - johntash 2y ago
      Tbh I haven't looked at Tigris at all. I still have my attic server deployed (just disabled/not in use) so I might give it a shot just to compare pricing. I do remember a decent portion of the cost being storage-related, so it's a good idea. I'll have to look at autosuspend again too. I remember having autostop configured, but not autosuspend. I could see that helping with start times a lot for some stuff. It's not supported on GPU machines though, right? I thought I read that but don't see it in the docs at a quick glance. > It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones? Sorry, I did mean external healthchecks. Something like zabbix/uptimekuma. For something public facing, I'd want a health check just to make sure it's alive. With any type of serverless/functions, I'd probably want to reduce the healthcheck frequency to avoid the machine constantly running if it is normally low-traffic. > We are suboptimal for things that make more sense as a bunch of containers on one host. I think my ideal offering would be something where I could install a fly.io management/control plane on my own hardware for a small monthly fee and use that until it runs out of resources. I imagine it's a pretty niche case for enterprise unless you can get a bunch of customers with on-prem hardware, but homelabbers would probably be happy.
      - indigodaddy 2y ago
Lyngbakr 2y ago
Who do you use instead for hobby projects?
- BoorishBears 2y ago
  Not them, but Runpod and Vast are my gotos. Runpod cost slightly more, but is in turn more reliable, so for "hobby pro" I'd go with them, otherwise Vast. Salad Cloud is also very interesting if your models can fit on a consumer GPU, but it's a different model than typical GPU providers.
- johntash 2y ago
  I have a decent sized homelab in my basement that I use for most stuff, and then a couple cheap-ish dedicated servers for public-facing things. Nothing has GPUs though, so I don't have a good solution for llm/ai projects yet. I used to use cheap vms/vps from lowendtalk deals, but usually they're on over-subscribed hosts and can't do anything heavy. Actual host recommendations: I like Racknerd and Hivelocity currently. OVH too, but I've read a lot of horror stories so I guess ymmv.
Aeolun 2y ago
That sounds kinda weird to me. I use fly.io exactly because the pricing works out for hobby use. I can enable my machine for a few hours, run a bunch of inference, and turn it off again. The whole auto start/stop thing makes it seamless too.
- BoorishBears 2y ago
  Vast has a datacenter H200 for less than what their A100 goes for.
  - Aeolun 2y ago
    Sure, but vast feels like renting a GPU from a rando.
    - BoorishBears 2y ago
      Either you're familiar with GPU pricing and being willfully ignorant, or you're not familiar with the pricing in which case let someone who is point out: - "Datacenter" means it's comparable to Runpod's secure cloud pricing. - A spot instance of an H200 under someone's living room media console wouldn't go for A100 rates. $3.50 will also get you an H100 at a laundry list of providers people build real businesses on. Certainly all better track records than fly.io, especially on a post where they explain it's not working out for them as an offering and then promise they'll keep it shambling along.
      - johntash 2y ago
        You seem like you're familiar with vast. Have you used their autoscaler/serverless offering before? I haven't tried it yet, but it wasn't immediately obvious if I could have something like ollama running and scaled to zero instances when not in use.
- johntash 2y ago
  If you don't mind, what size models are you running and do you know around what you were paying? fly.io was the first provider I tried any gpu offerings at, I probably should give it another shot now that I've used a few others.

mrkurt 2y ago

This is near and dear to me, because I want people to run stuff like homelabs and side projects. What part of the cost gets out of hand? Having to have a Machine for every process? Do you remember what napkin math pricing you were working with?

johntash 2y ago
Hmm, having a machine for every process is part of it but I actually like that kind of isolation. Storage and bandwidth also add up fast. For example, I could get a digitalocean vm with 2gb ram, 1vcpu, 50gb storage, 2tb bandwidth for $12/mo. For the same specs at fly.io, it'd be ~$22/mo not including any bandwidth. It could be less if it scales to zero/auto stops. I recently tried experimenting with two different projects at fly. One was an attic server to cache packages for NixOS. Only used by me and my own vms. Even with auto scaling to zero, I think it was still around $15-20/mo. The other was a fly gpu machine with Ollama on it. The cold start time + downloading a model each time was kind of painful, so I opted for just adding a 100gb volume. I don't actually remember what I was paying for that, but probably another 20/mo? I used it heavily for a few days to play around and then not so much later. I do remember doing the math and thinking it wouldn't be sustainable if I wanted to use it for stuff like home-assistant voice assistant or going through pdfs/etc with paperless. On their own, neither of these are super expensive. But if I want to run multiple home services, the cost is just going to skyrocket with every new app I run. If I can rent a decent dedicated server for $100-$200/mo, then I at least don't have to worry about the cost increasing on me if a machine never scales to zero due to a healthcheck I forgot about or something like that. Sorry if it's a bit rambly, happy to answer questions!
- mrkurt 2y ago
  No problem at all! I would be curious how the Attic server would have gone with a Tigris bucket and local caching. Not sure how hard that is to pull off, but Tigris should be substantially cheaper than our NVMes and if you don't really NEED the io performance you're not getting anything for that money. Which is a long winded way of saying "we aren't great at block storage for anything but OLTP workloads and caches". One thing we've struggled to communicate is how _cheap_ autosuspend/autostop make things. If that Machine is alive for 8 hours per day you're probably below $8/mo for that config. And it's so fast that it's viable for it to start/stop 45 times per day. It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones? We are suboptimal for things that make more sense as a bunch of containers on one host.
  - johntash 2y ago
    Tbh I haven't looked at Tigris at all. I still have my attic server deployed (just disabled/not in use) so I might give it a shot just to compare pricing. I do remember a decent portion of the cost being storage-related, so it's a good idea. I'll have to look at autosuspend again too. I remember having autostop configured, but not autosuspend. I could see that helping with start times a lot for some stuff. It's not supported on GPU machines though, right? I thought I read that but don't see it in the docs at a quick glance. > It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones? Sorry, I did mean external healthchecks. Something like zabbix/uptimekuma. For something public facing, I'd want a health check just to make sure it's alive. With any type of serverless/functions, I'd probably want to reduce the healthcheck frequency to avoid the machine constantly running if it is normally low-traffic. > We are suboptimal for things that make more sense as a bunch of containers on one host. I think my ideal offering would be something where I could install a fly.io management/control plane on my own hardware for a small monthly fee and use that until it runs out of resources. I imagine it's a pretty niche case for enterprise unless you can get a bunch of customers with on-prem hardware, but homelabbers would probably be happy.
    - indigodaddy 2y ago

Lyngbakr 2y ago

Who do you use instead for hobby projects?

BoorishBears 2y ago
Not them, but Runpod and Vast are my gotos. Runpod cost slightly more, but is in turn more reliable, so for "hobby pro" I'd go with them, otherwise Vast. Salad Cloud is also very interesting if your models can fit on a consumer GPU, but it's a different model than typical GPU providers.
johntash 2y ago
I have a decent sized homelab in my basement that I use for most stuff, and then a couple cheap-ish dedicated servers for public-facing things. Nothing has GPUs though, so I don't have a good solution for llm/ai projects yet. I used to use cheap vms/vps from lowendtalk deals, but usually they're on over-subscribed hosts and can't do anything heavy. Actual host recommendations: I like Racknerd and Hivelocity currently. OVH too, but I've read a lot of horror stories so I guess ymmv.

Aeolun 2y ago

That sounds kinda weird to me. I use fly.io exactly because the pricing works out for hobby use. I can enable my machine for a few hours, run a bunch of inference, and turn it off again. The whole auto start/stop thing makes it seamless too.

BoorishBears 2y ago
Vast has a datacenter H200 for less than what their A100 goes for.
- Aeolun 2y ago
  Sure, but vast feels like renting a GPU from a rando.
  - BoorishBears 2y ago
    Either you're familiar with GPU pricing and being willfully ignorant, or you're not familiar with the pricing in which case let someone who is point out: - "Datacenter" means it's comparable to Runpod's secure cloud pricing. - A spot instance of an H200 under someone's living room media console wouldn't go for A100 rates. $3.50 will also get you an H100 at a laundry list of providers people build real businesses on. Certainly all better track records than fly.io, especially on a post where they explain it's not working out for them as an offering and then promise they'll keep it shambling along.
    - johntash 2y ago
      You seem like you're familiar with vast. Have you used their autoscaler/serverless offering before? I haven't tried it yet, but it wasn't immediately obvious if I could have something like ollama running and scaled to zero instances when not in use.
johntash 2y ago
If you don't mind, what size models are you running and do you know around what you were paying? fly.io was the first provider I tried any gpu offerings at, I probably should give it another shot now that I've used a few others.