6 ms·

Who is the target market for this? Small/unproven apps that need to run some AI model, but won't/can't use hosted offerings by the literally dozens of race-to-z

Who is the target market for this? Small/unproven apps that need to run some AI model, but won't/can't use hosted offerings by the literally dozens of race-to-zero startups offering OSS models? We run plenty of our own models and hardware, so I get wanting to have control over the metal. I'm just trying to figure out who this is targeted at.

mrkurt 3y ago
We have some ideas but there's no clear answer yet. Probably people building hosting platforms. Maybe not obvious hosting platforms, but hosting platforms.
KTibow 3y ago
Fly is an edge network - in theory, if your GPUs are next to your servers and your servers are next to your users, your app will be very fast, as highlighted in the article. In practice this might not matter much since inference takes a long time anyway.
- joshxyz 3y ago
  this is crazy, this move alone cements fly as an edge player for the next 3 / 5 / 10 years.
- tptacek 3y ago
  We're really a couple things; the edge stuff was where we got started in 2020, but "fast booting VMs" is just as important to us now, and that's something that's useful whether or not you're doing edge stuff.
3kBPE 3y ago
[dead]
dathinab 3y ago
TL;DR: (skip to last paragraph) - having the GPU compute in the same data center or at least from the same cloud provider can be a huge plus - it's not that rare for various providers we have tried out to run out of available A100 GPUs, even with large providers we had issues like that multiple times (less an issue if you aren't locked to specific regions) - not all providers provide a usable scale down to zero "on demand" model, idk. how well it works with fly long term but that could be another point - race-to-zero startups have the tendency to not last, it's kind by design from a 100 of them just a very few survive - if you are already on fly and write a non-public tech demo which just gets evaluated a few times their GPU offering can act like a default don't think much about it solution (through you using e.g. Huggingface services would be often more likely) - A lot of companies can't run their own hardware for various reasons, at best they can rent a rack in another Datacenter but for small use use-cases this isn't always worth it. Similar there are use cases which do might A100s but only run them rarely (e.g. on weekly analytics data). Potentially less then 1h/w in which case race-to-zero pricing might not look interesting at all To sum up I think there are many small reasons why some companies, not just startups, might have interest in fly GPUs, especially if they are already on fly. But there is no single "that's why" argument, especially if you are already deploying to another cloud.
- qeternity 3y ago
  It's not like Fly has GPUs in every PoP...so there goes all the same datacenter stuff (unless you just want to be in the PoP with GPUs in which case...) But none of this answers my question. I'm trying to understand the intersection of things like "people who need GPU compute" and "people who need to scale down to zero". This can't be a very big market.
DreamGen 3y ago
I am not seeing any race-to-zero in the hosted offering space. Most charge multiples of what you would pay on GCP, and the public prices on GCP are already several times what you would pay as an enterprise customer.
- qeternity 3y ago
  I don't know what you think I'm talking about, or who is charging multiples of GCP? But I'm talking about hosted inference, where many startups are offering Mistral models cheaper than Mistral are.

mrkurt 3y ago

We have some ideas but there's no clear answer yet. Probably people building hosting platforms. Maybe not obvious hosting platforms, but hosting platforms.

KTibow 3y ago

Fly is an edge network - in theory, if your GPUs are next to your servers and your servers are next to your users, your app will be very fast, as highlighted in the article. In practice this might not matter much since inference takes a long time anyway.

joshxyz 3y ago
this is crazy, this move alone cements fly as an edge player for the next 3 / 5 / 10 years.
tptacek 3y ago
We're really a couple things; the edge stuff was where we got started in 2020, but "fast booting VMs" is just as important to us now, and that's something that's useful whether or not you're doing edge stuff.

3kBPE 3y ago

[dead]

dathinab 3y ago

TL;DR: (skip to last paragraph) - having the GPU compute in the same data center or at least from the same cloud provider can be a huge plus - it's not that rare for various providers we have tried out to run out of available A100 GPUs, even with large providers we had issues like that multiple times (less an issue if you aren't locked to specific regions) - not all providers provide a usable scale down to zero "on demand" model, idk. how well it works with fly long term but that could be another point - race-to-zero startups have the tendency to not last, it's kind by design from a 100 of them just a very few survive - if you are already on fly and write a non-public tech demo which just gets evaluated a few times their GPU offering can act like a default don't think much about it solution (through you using e.g. Huggingface services would be often more likely) - A lot of companies can't run their own hardware for various reasons, at best they can rent a rack in another Datacenter but for small use use-cases this isn't always worth it. Similar there are use cases which do might A100s but only run them rarely (e.g. on weekly analytics data). Potentially less then 1h/w in which case race-to-zero pricing might not look interesting at all To sum up I think there are many small reasons why some companies, not just startups, might have interest in fly GPUs, especially if they are already on fly. But there is no single "that's why" argument, especially if you are already deploying to another cloud.

qeternity 3y ago
It's not like Fly has GPUs in every PoP...so there goes all the same datacenter stuff (unless you just want to be in the PoP with GPUs in which case...) But none of this answers my question. I'm trying to understand the intersection of things like "people who need GPU compute" and "people who need to scale down to zero". This can't be a very big market.

DreamGen 3y ago

I am not seeing any race-to-zero in the hosted offering space. Most charge multiples of what you would pay on GCP, and the public prices on GCP are already several times what you would pay as an enterprise customer.

qeternity 3y ago
I don't know what you think I'm talking about, or who is charging multiples of GCP? But I'm talking about hosted inference, where many startups are offering Mistral models cheaper than Mistral are.