4 ms·
Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework
- Carrok 2y agoAs someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".
- vinni2 2y agoCan you share some of your wisdom on setting up a scalable inference infrastructure?
- Carrok 2y agoUse Ray Serve. https://docs.ray.io/en/latest/serve/index.html https://docs.ray.io/en/latest/serve/index.html
- ipsum2 2y agoAs someone who has run LLMs in production, using Ray is probably the worst idea. It's not optimized for language models, and is extremely slow. There's no KV-caching, model parallelism, and other basic table stakes features that are offered by Dynamo or other open source inference frameworks. Useful only if you have <1 QPS. Use SGLang, vLLM, or text-generation-inference instead.
- Carrok 2y agoThis is probably true, but unlike every Nvidia product we tried, it did, you know, reply to inference requests with actual output. That said, you can serve vLLM with Ray Serve. https://docs.ray.io/en/latest/serve/tutorials/vllm-example.html https://docs.ray.io/en/latest/serve/tutorials/vllm-example.h...
- erulabs 2y agoIt really depends on the task. If you have 1 massive job, Ray sucks and doesn't provide table stakes. If you have 50M tiny jobs, Ray and kuberay is great and serves as the backbone of several billion dollar products. Good for the goose, good for the gander...
- richardliaw 2y ago> If you have 1 massive job, Ray sucks and doesn't provide table stakes. Can you say more?
- islewis 2y agois this in reference to Triton?
- Carrok 2y agoAnd NIM, yes.
- aabhay 2y agoTriton is not that bad at all, considering the wide scope of systems it has to support (tensorrt, onnx, multiple generations of pytorch, cuda, python). It was much nicer than the old Torchserve project which was JVM based.
- eoskx 2y agoJust curious what your issues with Triton were. We've done OK with it using it to serve LLM models w/ a classifier head via HF Transformers pipeline & Flash Attention 2, as well as serving text generation models with the vLLM back-end.
- bytesandbits 2y agotriton is not that bad, TensorRT will give you nightmares
- dlewis1788 2y ago100% - probably why vLLM is now the default back-end in Dynamo.
- raffraffraff 2y agoI've done very little with Nvidia software, but what I have done puts me off ever doing it again. I quit a job partially because it involved trying to get their shit to work. (There were other factors, but that was definitely on the 'GTFO' side)
- bloomingkales 2y agoBuilt in Rust for performance and in Python for extensibility Omg, a team that knows how to selectively use tech as needed. Looking at the Rust web developers in corner.
- seangrogg 2y agoUnsure if the implication is that Rust is poorly suited for web development or what.
- gr_norm 2y agoIt is, in my opinion (as an avid Rust user!). The type errors from most of the major web frameworks/ORMs (diesel, sqlx) are just awful, more often than not. Usually some inscrutable thing involving Send/Sync. Or some hilariously complicated type/trait hackery on the part of the library, attempting to save me from the former, that I'm never going to figure out. Great language in many other settings, but not this one. At least not right now, but given my experience with async Rust in general, I'm not sure it ever will be.
- echelon 2y ago> Usually some inscrutable thing involving Send/Sync. What kind of queries are you writing? I never see SQLx emit anything like that. I always get back SQL errors. Column mismatches are what I hit most in development, and they're pretty explicit. The hairiest thing I see with SQLx is when I try to write custom type conversions for my own types to SQL fields. I sometimes have to delve into macros. But those errors are pretty self-explanatory too.
- deleted 2y ago[deleted]
- echelon 2y agoRust is emerging as one of the best web programming languages out there. Actix and Axum feel like Python's Flask. Rust has decent Redis and connection pool libraries, but the SQL space needs more work. Diesel SQL is too ORM-y (I've never liked ORMs). While SQLx allows you to write "typechecked" SQL, it still has really annoying edge cases (WHERE IN clauses can't be typechecked, type bindings can get hairy, etc.) I'm not very happy about the state of Rust's Elasticsearch libraries, either. Rust probably needs a Rails/Django-like framework too for those that prefer a framework-oriented development lifecycle. Rust also needs some observability frameworks. There are a few, but the choices are sparse. I'd give Rust a 7.5/10 for web programming, and as far as the promise of the language goes, I'd give it an 11/10. Developing in Actix and Axum feels amazing. It's honestly better than Go and Python. The other pieces (database, API clients, etc.) will presumably get better in time. And because of the way HTTP request flow logic is typically structured, 99.9% of the time you'll never hit Rust's borrow checker or have to worry about lifetimes. It's as if you've been given one of the best typed languages, best package managers, and nearly no tradeoffs. The server compiles down to a single static binary. It's multithreaded, and it's blindingly fast. I'm picking Rust for every new web service I write these days.
- deleted 2y ago[deleted]
- lmeyerov 2y agoSo this replaces triton for LLMs or?
- aabhay 2y agoThis is very narrowly focused on LLMs, whereas triton is still useful for running all kinds of ML models. In practice, Triton is a very poor choice for LLMs specifically because it has none of the required non negotiable features like KV caching built in.
- changtimwu 2y agosame question here. Just asked Grok for a comparsion https://grok.com/share/bGVnYWN5_fa210574-f27b-45ae-9d95-19edb5657014 https://grok.com/share/bGVnYWN5_fa210574-f27b-45ae-9d95-19ed...
- deleted 2y ago[deleted]
- saagarjha 2y agoNvidia not name products after existing things in the ML space challenge: IMPOSSIBLE More seriously, though: > OpenAI Compatible Frontend – High performance OpenAI compatible http api server written in Rust. Is this normal in this space? I know everyone has settled on copying the S3 API for block storage but I’m unsure if we’ve done the same for LLM serving.
- DeveloperErrata 2y agoIncreasingly so. Many other popular inference tools in this space also expose an OpenAI compatible API: VLLM, Llama.cpp, and LiteLLM all do.
- nitrogen99 2y agoI have deployed and developed on their Triton inferencing server and it was amazing. All very good C++ and well architected. This one has Rust, Go, Python and C++. Seriously? First, not many Rust devs in the AI community. How do you think you'll get community involvement. Ok, may be you don’t need it. Second, good luck maintaining such a polyglot system. I prefer at most 2-3 languages - main language (C++/Java), Python for extensibility and Shell, etc for deployment.