4 ms·

Would like to take a moment to recommend fly.io for GPU workloads. I've been building a prototype using them for the last couple of weeks and it's been great t

Would like to take a moment to recommend fly.io for GPU workloads. I've been building a prototype using them for the last couple of weeks and it's been great to use. I didn't have to jump through any hoops or apply for any quota adjustments to get started. And I especially appreciate how easy they make it to automatically scale your GPU instances to zero based on traffic.

nextworddev 2y ago
Taking a step back, was it necessary to roll your own LLM (or whatever FM) API as opposed to using an off the shelf API
- zackangelo 2y ago
  Yes we’re running our own inference server because we do nonstandard things in the decoder loop (think logit masking, other types of sampling, etc).
indigodaddy 2y ago
Wow that is quite decent pricing actually. This makes me want to try to deploy something like llamafile/ollama or similar to fly.io+gpu for my personal on-demand llama/llm whims. (@simonw I’m looking at you— I feel like if you haven’t already done this on fly.io, that you’re probably thinking about it :) ) Seems private enough.. could throw some basic auth on top of it for me and trusted friends/family so I don’t get crazy bills. But privacy-wise with fly.io I think it’s good enough. Only problem might be everytime it spins up to download the large model might be wasteful as far as getting charged for network/bw usage— wonder if it would be more cost-efficient to have persistent storage or just see how much time and bw it is to download on every cold start…
- Palmik 2y ago
  It's horrible horrible pricing! Their on demand price for A100 is what gets you H100 sxm in other places.
  - dangrairo 2y ago
    I am curious what "other places" are you comparing it to.
    - indigodaddy 2y ago
      Also we’d have to do apples to apples right? One cannot complain about fly.io gpu pricing and then point to say vast.ai or lambda labs as “evidence” of that. They aren’t at all the same type of service..
- benwaffle 2y ago
  We've got a one-liner for spinning up your own ollama UI. See https://github.com/fly-apps/ollama-open-webui https://github.com/fly-apps/ollama-open-webui
  - indigodaddy 2y ago
    Oh snap thank you!
breakingcups 2y ago
Hn's opinion of Fly seems to fluctuate a lot. First it was an HN darling, especially with all the high-quality blog posts they were putting out, but on the latest threads there's been a lot of complaining about platform stability and features.

nextworddev 2y ago

Taking a step back, was it necessary to roll your own LLM (or whatever FM) API as opposed to using an off the shelf API

zackangelo 2y ago
Yes we’re running our own inference server because we do nonstandard things in the decoder loop (think logit masking, other types of sampling, etc).

indigodaddy 2y ago

Wow that is quite decent pricing actually. This makes me want to try to deploy something like llamafile/ollama or similar to fly.io+gpu for my personal on-demand llama/llm whims. (@simonw I’m looking at you— I feel like if you haven’t already done this on fly.io, that you’re probably thinking about it :) ) Seems private enough.. could throw some basic auth on top of it for me and trusted friends/family so I don’t get crazy bills. But privacy-wise with fly.io I think it’s good enough. Only problem might be everytime it spins up to download the large model might be wasteful as far as getting charged for network/bw usage— wonder if it would be more cost-efficient to have persistent storage or just see how much time and bw it is to download on every cold start…

Palmik 2y ago
It's horrible horrible pricing! Their on demand price for A100 is what gets you H100 sxm in other places.
- dangrairo 2y ago
  I am curious what "other places" are you comparing it to.
  - indigodaddy 2y ago
    Also we’d have to do apples to apples right? One cannot complain about fly.io gpu pricing and then point to say vast.ai or lambda labs as “evidence” of that. They aren’t at all the same type of service..
benwaffle 2y ago
We've got a one-liner for spinning up your own ollama UI. See https://github.com/fly-apps/ollama-open-webui https://github.com/fly-apps/ollama-open-webui
- indigodaddy 2y ago
  Oh snap thank you!

breakingcups 2y ago

Hn's opinion of Fly seems to fluctuate a lot. First it was an HN darling, especially with all the high-quality blog posts they were putting out, but on the latest threads there's been a lot of complaining about platform stability and features.