6 ms·
Nvidia Nemotron 3.5 Lightning and NeMo Switchyard
- thehamkercat 2mo ago> NeMo Switchyard, an open source library for smart routing > When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job How do routers like this handle prompt caching when you send the second request? Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.
- embedding-shape 2mo agoThe repo is probably a better entrypoint to it, bit more concise description than the press releases: https://github.com/NVIDIA-NeMo/Switchyard https://github.com/NVIDIA-NeMo/Switchyard (Notably: "Experimental software. Not for production use."). Unclear if they actually want you to deploy it or not, press release says yes, README says no, do with that what you will. Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+cache&type=code https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.
- thehamkercat 2mo agoI personally think it's snake-oil marketing with all these smart-model-routing products/projects prompt-cache won't work with these
- try-working 2mo agoTo keep it simple, forget about routers and imagine you're in Cursor using GPT for a while, reaching a cache of says 200k. You decide to switch to DeepSeek in the same session via the model picker, and continue as usual. What happens is that the cache for DeepSeek is created with the 200k + the incremental message. After this, cache can be kept warm for both models; two instances of the cache exists, one for GPT and one for DS. You switch back to GPT. The whole session is sent to the model with the 200k original from GPT and the incremental messages you sent to DS. The 200k is read from cache and the incrementals are new, and then added to the cache. Let's say every second message you switch between GPT and DS; cache was 200k and each incremental message is 1k. If you kept going with only GPT, cache hit rate would be 200k/(200k+1k) = 99.5%. When you switch between two models with warm cache, hit rate instead becomes 200k/(200k+2k) = 99%. Model routers work the same way. Keep the cache warm, replicate it in two places. For this reason, when you set up your model pool for routing, you want to keep the model pool small and differentiated. First principles of model routing: https://try.works/first-principles-of-model-routing https://try.works/first-principles-of-model-routing role-model router and protocol: https://github.com/try-works/role-model https://github.com/try-works/role-model note: edited to keep the answer to the below message clearer
- thehamkercat 2mo agoCan you explain how does it work? like how is the previous K/V cache used when you switch to another model? Source?
- hedgehog 2mo agoSee sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model).
- try-working 2mo agoedit: updated the answer above to be more qualitative instead
- hedgehog 2mo agoTo elaborate, because I don't think some of the people reading this understand the reason, typically a lot or most of the cost in "agentic" API usage is cached read + generation. Cached read costs scale with turn count, which multi-model switching doesn't increase, and of course generation gets cheaper if you do some of it with a cheaper model. When you switch models the "catching up" batch of messages is just a single prefill and then that goes into cache. You don't even need to have the same chat history across models so long as the view from each model's perspective looks like a series of appends. The main problem with model routing in my experience is that to work well the router needs to be pretty strong, maybe even moreso than any of the actual models in service. There are probably clever solutions to this but I haven't seen any that look better than just using sub-agents.
- richwater 2mo ago> which multi-model switching doesn't increase Given model A with cache C(a) and model B with C(b) Isn't this not true because the moment you switch models from A to B, you need to provide C(b) the latest conversation diff since C(b) last updated, say many turns ago?
- eli 2mo agoI've seen ones that are configurable to pick a trade off point between lower cost (cache stickiness) and routing performance (best model for that turn). But yeah I'm skeptical all this overhead is worth it.
- quinncom 2mo agoCaching should be possible as long as all the models use the same shared cache. The models don't even need to be running on the same server if the shared cache is distributed. I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.
- IanCal 2mo agoHow do caches work across models? I would have thought that was very model specific - if not I’ve really misunderstood what’s getting cached.
- armanckeser 2mo agoI am not sure the author of the comment you are replying to understands that LLM systems have prompt caches
- amluto 2mo agoHuh? Prompt caching isn’t about caching the literal text of the prompt. It’s about caching the result of running prefill on the prompt (or, equivalently, the result of generating the prompt one token at a time by autoregressive inference, or some combination of the above in the case of speculative decoding). This is often called the “KV” cache, and it is very model-specific.
- rufasterisco 2mo agohttps://github.com/NVIDIA-NeMo/Switchyard#routing-strategies https://github.com/NVIDIA-NeMo/Switchyard#routing-strategies Looks like your great question doesn’t have an answer, but looking at the routing strategies things get even more confused, since the proposed ones tend to rely on extra llm calls to determine which model to pick. The nice thing is that it makes sense for specific setups, less conversation oriented. As an example, you need to classify batches of data, and have many fine tuned models. Or you need to do speed to text and need to pick which whisper to use. You can write your own strategy, in that case an harness with subagents would be able to leverage this, picking the right model and then keeping its session sticky, but overall the lack of concern for caching points towards use cases where you do not gain much from it.
- WalterGR 2mo ago24 comments so far about Nemotron on this earlier submission: https://news.ycombinator.com/item?id=49257947 https://news.ycombinator.com/item?id=49257947
- XCSme 2mo agoThe new Meta 30B models seems A LOT better: https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvidia-nemotron-3-5-lightning-high/ https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...
- Tactical45 2mo agoAt what cost difference?
- XCSme 2mo agoI don't think it matters, if it's for local/on-device usage. The cost is similar vram footprint I guess (?)
- sleepyeldrazi 2mo agoloading the model would be similar vram footprint, correct, however the size of KV is based on 'active' params, not total params. So while at 1k ctx both will be in the same ballpark vram footprint-wise, at 100k the story will be very different. 27B at q4 kv for 256k takes ~8gb, while 35B at q4 kv around ~3.5gb, so at full precision kv those would be ~32gb and ~14gb (all ballparks, if you want exact numbers, its not hard to test). As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.
- XCSme 2mo agoI am asking mostly for running on a 3090. I think the tps difference between them (both fitting in vram) won't be more than 2x in practice. I would happily take 20tps over 40tps, if the model gets 3x more correct answers.
- sleepyeldrazi 2mo agoSpeed is (for the most part) active-parameter based, so a 30B-A3B model is roughly 10x the speed of a dense 30B (realistically closer to 8x) in the case when both fit. That's the proposition of MoE and why everyone is trying to make massive models with very few active params, so that they are still fast while having access to a lot of knowledge (at the cost of reasoning, as reasoning ability 'for the most part' comes from active params). You can test this by running this nemo or 35B on the 3090. I have and its very fun (but sadly a worse model than 27B, so I usually keep 27B on my 3090)
- average_bloke 2mo agoI would like to propose something: - problem: massive deluge of information because of AI - solution: human beings should adopt a minimalist style of communicating in writing. - e.g. this entire website page can be ten bullet points.
- fooker 2mo agok
- marsven_422 2mo ago[dead]
- encrux 2mo agoIn my opinion: the only way forward is zero-knowledge-proof authenticated social media. We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate. If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.
- kubelsmieci 2mo ago> We can’t have legitimate debate I'm not sure people really want that
- npunt 2mo ago[dead]
- ttoinou 2mo agoIs the network based on trust and peer to peer confirmation of private keys from who you know in real life that you validated isn’t a robot ? Or do you have something else in mind ?
- SMAAART 2mo ago[flagged]
- CurbStomper 2mo ago[dead]
- jmward01 2mo agoOne major consequence of the ramapocalypse, I think, is an even higher focus on small efficient models. I personally believe that the multi-trillion parameter models are fundamentally missing things and the push to smaller, more efficient will drive evolutionary structural changes that will lead to future gains
- schainks 2mo agoI am literally betting my company on this being true.
- jmward01 2mo agoWhat company? I am 100% focused on this as a concept in my own internal research.
- schainks 1mo agoworkingmemory.ai!
- oblio 2mo agoIt's a bad bet, historically. I'm having an extremely hard time thinking of companies that have prospered due to software optimization. Most of them were swept away by hardware advances, instead.
- jmward01 2mo agoThe 1980's US car industry comes to mind. Nearly wiped out because they refused to make efficient vehicles. SpaceX is arguably showing how a rethink towards efficient can take over an entire industry. I am sure there are strong examples in software as well but they aren't coming to mind. I think when successful, optimization really just means 'finally built right' and people forget the ridiculously inefficient ways before.
- sgc 2mo ago
- docheinestages 2mo agoThey conveniently decided not to include the Qwen range of models in the Artificial Analysis graph, except the out-of-league Max variant. At least be brave and honest.
- jadbox 2mo agoNemotron 3.5 Lightning runs on how little GPU vram? Can q4 run on 16gb?
- sleepyeldrazi 2mo agoNot by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?" Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp). Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).
- mark_l_watson 2mo agoI love the wave of new small model releases. Pleasantly surprising that an NVIDIA model runs so well on Apple Silicon using MLX! I was using nemotron-3.5-lightning:30b-mlx with OpenCode on my old (cheap) Mac this morning and no bad experiences except for running slowly.
- halfdeadcat 2mo agoGood luck getting it to run with NVFP4 on a DGX Spark, the very architecture Nvidia created that format for.
- Scaled 2mo agoCan you elaborate on the problem? The article claims support for dgx spark
- rsolva 2mo agoI have seen many a model card claim Spark compatibility, but without any proof or instructions. There are vibe-patched images of vLLM that kinda works with NVFP4 models, but I have not seen any support from NVIDIA. Does not seem to be a priority for some reason.
- Scaled 1mo agoThanks, I've been mulling over buying some of these so I appreciate the heads up.
- halfdeadcat 1mo agoAfter thrashing on it for a day, I could not get it going on DGX Spark with FP4 quantization. I find this irksome, since Nvidia created the NVFP4 specifically for Blackwell. The Nvidia cookbooks for this model are all for H100. I tried vLLM, Ollama and various patches. As of right now on DGX: you can feasibly do FP4 on dense models. But FP4 + MoE is a largely a dead-end.
- kentonv 2mo agoCoincidentally I've been playing with small (~30B) self-hostable models for coding tasks today -- specifically plugging them into Cloudflare OS (which I work on) and asking each to build a collaborative whiteboard. I'm finding that the Mixture-of-Experts (MoE) models (Qwen 3.6-35B, and Nemotron 3.5 Lightning) are, well, terrible at this. They just couldn't get the job done at all. Went way off the rails. They are really fast though! Whereas ~30B dense models (not MoE) are pretty decent. I tried Muse Glimmer, Gemma 4-31B, Qwen 3.6-27B, and Laguna XS[0]. They were all able to build a working collaborative whiteboard app, without any guidance (other than feeding back error logs to the model). I also asked each to then draw a monkey by calling the API of the whiteboard it has just built. Laguna drew random scribbles but the rest all managed to produce something monkey-like. (Frontier models in comparison will write the app in one shot with no errors at all.) Note that both Qwen 3.6 and Gemma 4 each have both MoE and dense variants. I find this very confusing, because e.g. ollama's model index typically only distinguishes variants by their size, but MoE vs. dense makes a huge difference in how they actually perform. IMO they should use a suffix, like Qwen 3.6-moe vs. Qwen 3.6-dense, or maybe Qwen 3.6-fast vs. Qwen 3.6-smart... [0] EDIT: Turns out Laguna XS is MoE, I misunderstood. It performed similarly to the dense models. But maybe this explains why it couldn't write code and think about monkey shapes at the same time!
- ChadNauseam 2mo agoI've been pretty impressed with Laguna. I downloaded their coding agent and have used it for a task here and there (the larger variant). Obviously it's nothing like a frontier LLM, but it surprised me with how good it was. And I think the model personality and way it talks is pretty pleasant
- trouve_search 2mo agoLaguna XS is MoE, however.
- kentonv 2mo agoOh! I think I missed that and assumed it wasn't because it performed similarly to the dense models. Interesting!
- macwhisperer 2mo agobig week for open models... seems like companies are noticing the 26-35b sweet spot... though I think a 12b-a1b-MoE model would be helpful for the 16gb folks
- deleted 2mo ago[deleted]
- HackerThemAll 2mo agoAn open source model from Nvidia is a free drug to later buy their chips.
- reindeer2 2mo ago[flagged]