pveldandi

4 ms·

1.
by pveldandi 10mo ago
Happy to engage with you if Hulu have any additional questions. Thanks for all the feedback.that was helpful
2.
by pveldandi 10mo ago
Fair point. I’ll repost as a regular submission instead of Show HN. The goal was to demonstrate the runtime behavior behind multi-model serving rather than a polished end-user app. Appreciate the clarification.
3.
by pveldandi 10mo ago
It’s an open-core model. The control plane is already open source and can be deployed fairly easily. We’re not trying to replace in-house systems or hyperscalers. This can run on Kubernetes and integrate into existing infrastructure. The ru
4.
by pveldandi 10mo ago
It’s an open-core model. The control plane is already open source and can be deployed fairly easily. We’re not trying to replace in-house systems or hyperscalers. This can run on Kubernetes and integrate into existing infrastructure. The ru
5.
by pveldandi 10mo ago
The demo is live. It’s meant to show how snapshot restore works inside a multi-tenant runtime, not just a prompt playground. You can interact with the deployed models and observe how state is restored and managed across them. The focus is o
6.
by pveldandi 10mo ago
Happy to dig deeper and show how exactly it works under the hood. For context, here’s the main site where the architecture and deployment options are explained: https://inferx.net/
7.
by pveldandi 10mo ago
Also, For what it’s worth, this can be deployed both on-prem and in the cloud. Different teams have different constraints, so we’re trying to stay flexible on that.
8.
by pveldandi 10mo ago
Basic html? The core of what we built is at the runtime layer. We’re capturing CUDA graphs and restoring model state directly at the GPU execution level rather than just snapshotting containers. That’s what enables fast restores and higher
9.
by pveldandi 10mo ago
Vertex is great control plane. We’re not replacing them. What we focus on is the runtime layer underneath. You can run us behind Cloud Run or inside your existing GCP setup. The difference is at the GPU utilization level when you’re serving
10.
by pveldandi 10mo ago
Good question. This isn’t for single-model apps running steady traffic at high utilization. If you’re saturating GPUs 24/7, you’ll architect very differently. This is for teams that… • Serve many models with uneven traffic • Run per-cu
11.
by pveldandi 10mo ago
You can try it here: https://inferx.net:8443/demo/
12.
by pveldandi 10mo ago
Ollama is great for local workflows. What we’re focused on is multi-tenant, high-throughput serving where dozens of models share the same GPUs and scale to zero without keeping them resident. You’re right that HN expects something runnable.
13.
Show HN: How We Run 60 Hugging Face Models on 2 GPUs
4 points by pveldandi 10mo ago|20 comments
14.
by pveldandi 10mo ago
Lambda made compute cheap by treating code as ephemeral and execution state as disposable. LLM inference breaks that model. The execution state (weights shards, CUDA context, KV cache) is expensive and GPU-resident, so “scale to zero” usual
15.
Benchmark: A100 vs. H100 NVMe Random Read throughput during multi-GPU loading
1 points by pveldandi 11mo ago|0 comments
16.
by pveldandi 1y ago
Really interesting work. we’ve been building a container-native snapshotting system too, but focused on cold start reduction and multi-model orchestration for LLM inference. Different use case (sub-2s loading for large models), but very si
17.
Show HN: 50+ LLMs on 2 GPUs with 2-Second Swapping? We built AI-Native Runtime(github.com)
5 points by pveldandi 1y ago|0 comments
18.
Show HN: InferX - AI Lambda-Like Inference Function as a Service
2 points by pveldandi 1y ago|0 comments
19.
We're running 50 LLMs on 2 GPUs – no cold starts, no overprovisioning
4 points by pveldandi 2y ago|1 comments
20.
Show HN: InferX – an AI-native OS for running 50 LLMs per GPU with hot swapping
3 points by pveldandi 2y ago|2 comments