3 ms·

How many people need to host models like this? I'm having trouble seeing why I would need this muti-tenant model stuff if I'm building a consumer or b2b app In

by verdverm 10mo ago

How many people need to host models like this? I'm having trouble seeing why I would need this muti-tenant model stuff if I'm building a consumer or b2b app In other words, how many middlemen do you think you TAM is? You go on to say this is great for light workloads, because obviously at scale we run models very differently. So who is this for in the end?

pveldandi 10mo ago
Good question. This isn’t for single-model apps running steady traffic at high utilization. If you’re saturating GPUs 24/7, you’ll architect very differently. This is for teams that… • Serve many models with uneven traffic • Run per-customer fine-tunes • Offer model marketplaces • Do evaluation / experimentation at scale • Have spiky workloads • Don’t want idle GPU burn between requests A lot of SaaS AI products fall into that category. They aren’t OpenAI-scale. They’re running dozens of models with unpredictable demand. Lambda exists because not every workload is steady state. Same idea here.
- verdverm 10mo ago
  > A lot of SaaS AI products fall into that category. ... They’re running dozens of models with unpredictable demand. How do you know this? What are the numbers like? > Lambda exists because not every workload is steady state Vertex AI has all these models via API or hosting the same way. Same features already available with my current cloud provider. (traffic scaling, fine-tunes,all of the frontier and leading oss models)
  - pveldandi 10mo ago
    Vertex is great control plane. We’re not replacing them. What we focus on is the runtime layer underneath. You can run us behind Cloud Run or inside your existing GCP setup. The difference is at the GPU utilization level when you’re serving many models with uneven demand. If your workload is steady and high volume on a small set of models, the standard cloud stack works well. If you’re juggling dozens of models with spiky traffic, the economics start to look very different. As an example, we’re currently being tested inside GCP environments. Some teams are experimenting with running us behind their existing Google Cloud setup rather than replacing it. The idea isn’t to swap out Cloud Run or Vertex, but to improve the runtime efficiency underneath when serving multiple models with uneven demand.
    - verdverm 10mo ago
      Vertex AI is far more than a "control plane" I don't see anything you do that they don't already do for me. I suggest you do a deep dive on their offering as there seem to be gaps in your understanding of what features they have > economics start to look very different You need to put numbers to this, comparing against API calls at per-token pricing is a required comparison imo, because that is the more popular alternative to model hot-swapping for spikey or heterogeneous workloads

pveldandi 10mo ago

Good question. This isn’t for single-model apps running steady traffic at high utilization. If you’re saturating GPUs 24/7, you’ll architect very differently. This is for teams that… • Serve many models with uneven traffic • Run per-customer fine-tunes • Offer model marketplaces • Do evaluation / experimentation at scale • Have spiky workloads • Don’t want idle GPU burn between requests A lot of SaaS AI products fall into that category. They aren’t OpenAI-scale. They’re running dozens of models with unpredictable demand. Lambda exists because not every workload is steady state. Same idea here.

verdverm 10mo ago
> A lot of SaaS AI products fall into that category. ... They’re running dozens of models with unpredictable demand. How do you know this? What are the numbers like? > Lambda exists because not every workload is steady state Vertex AI has all these models via API or hosting the same way. Same features already available with my current cloud provider. (traffic scaling, fine-tunes,all of the frontier and leading oss models)
- pveldandi 10mo ago
  Vertex is great control plane. We’re not replacing them. What we focus on is the runtime layer underneath. You can run us behind Cloud Run or inside your existing GCP setup. The difference is at the GPU utilization level when you’re serving many models with uneven demand. If your workload is steady and high volume on a small set of models, the standard cloud stack works well. If you’re juggling dozens of models with spiky traffic, the economics start to look very different. As an example, we’re currently being tested inside GCP environments. Some teams are experimenting with running us behind their existing Google Cloud setup rather than replacing it. The idea isn’t to swap out Cloud Run or Vertex, but to improve the runtime efficiency underneath when serving multiple models with uneven demand.
  - verdverm 10mo ago
    Vertex AI is far more than a "control plane" I don't see anything you do that they don't already do for me. I suggest you do a deep dive on their offering as there seem to be gaps in your understanding of what features they have > economics start to look very different You need to put numbers to this, comparing against API calls at per-token pricing is a required comparison imo, because that is the more popular alternative to model hot-swapping for spikey or heterogeneous workloads