4 ms·

I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For e

I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?

pizza234 2mo ago
> but are they functionally useful or helpful to anyone currently? Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do. MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro. The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
- ddevnyc 2mo ago
  Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?
  - dannyw 2mo ago
    MoE has nothing to do with multimodality. MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing. Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization. As for reasoning in non-text modalities, you might find this paper interesting :) https://huggingface.co/papers/2502.05171 https://huggingface.co/papers/2502.05171
    - dummydummy1234 2mo ago
      Wait I thought the router ends up specializing the experts? Like there is no explicit goal aside from each 'expert' getting roughly equal weight? And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference? I do not know what I am talking about, this is my limited understanding...
      - juliendorra 2mo ago
        You are right that at training the main goal is balancing between the sections to avoid certain paths becoming the only path. In the end the inference will be routed token by token to a mixture of say 3 or 4 sections of the model. The combination can change at each turn. It’s really a statistical optimization. As for many things in neural networks, the original intuition coming from anthropomorphism once implemented becomes something very non human!
      - ahepp 2mo ago
        My understanding of it is also pretty surface level, but I was not under the impression that it develops "expertise" in a particular subject matter, at least not in a way that's easy to harness. From what I've read, it develops expertise at the token level. Because the natural continuation of this is to say like, "Ok I want to load the bird detection expert and the navigation expert but leave the medieval European history expert behind", and my understanding is that this is not really how it works. At least at the moment.
    - ahepp 2mo ago
      > You can think of it like sharing. was this meant to read "sharding"?
    - xg15 2mo ago
      > There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization. Huh, always thought one would sort of require the other. For a good MoE model, wouldn't I want to minimize the "churn" between experts, i.e. the amount of time one expert model has to be swapped in for another expert model? That would be naturally the way if experts correspond to semantic categories. E.g. suppose I have a model that can answer questions in 100s of languages. Then while any of those languages might be requested by some caller, it's highly unlikely a caller will request all languages in the same session - realistically, there might be one or two languages in a session and those will then span the entire session. There will also be languages that are requested very often and others that are extremely rare. (Let's say my model also supports Klingon and Sindarin. Those are important for marketing reasons and because I genuinely like to make the occasional nerd happy - but practically, I get maybe a handful of requests for those every few months. So it would make sense to centralize the knowledge for those languages in some specific part of the model, so I can keep that part out of VRAM - and probably RAM as well - during the 99% of time where it's not needed) So wouldn't it make sense to make the expert models language specific here? Then you could take advantage of the fact that a language rarely changes inside a session and keep that expert in VRAM for the entire session. You could also avoid dragging parameters along with you for languages that are practically never used.
- bensyverson 2mo ago
  Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.
  - reactordev 2mo ago
    yes, I average 80-120 tok/s on my RTX 3080 with gemma 4 and faster with Qwen 3.5. The main use-case here is just code-monkey agents. I'm not looking for architectural guidance, but an agent to take a spec and complete it.
    - bensyverson 2mo ago
      And is this using conventional model loading (all in VRAM), or are you streaming it in some way?
  - cyanydeez 2mo ago
    in theory, QWEN3.6-35B-A3B should run as fast as a 3B model; and in practice, it can be just as dumb.
nickpsecurity 2mo ago
If I could justify wear and tear and electricity, I was willing to do something like this for batch processing. The batches would be a bunch of prompts whose outputs I'd look at the next day. Maybe common operations, like QA or refactoring, on whatever software I wrote. If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one try. I also would be using AI's as a supplement to, not replacement for, my own brain. So, issues with the outputs wouldn't be a problem because I'm just keeping what's helpful. If I still need to re-generate it all, it might still save money over time by avoiding cloud costs. Also, hardware that's already paid for is a sunk cost that doesn't inflate over time. Glitches in loading or destroying VM's might blow up into a big bill.
- parineum 2mo ago
  at 5 minutes per token, you could look at the results next week
  - bensyverson 2mo ago
    one week later: "It says 'You're absolutely right! Let me look at the seams so I'm checking, not guessing—' and I guess that's when my SSD melted."
    - hex4def6 2mo ago
      Turns out the SSD was load-bearing.
  - nickpsecurity 2mo ago
    That's way slower than I thought it would be. I struggle to imagine a use case. If you had a deadly condition, and no diagnosis worked, and a specific model had the answer... past that I wouldn't use it.
deleted 2mo ago
[deleted]
zapataband1 2mo ago
[flagged]
dsl 2mo ago
> but are they functionally useful or helpful to anyone currently? I've been building a SaaS that deals with data that can't be distributed to third parties. Some of the useful AI stuff I can add is not time sensitive and can run overnight. Things like this allow me to use higher quality models without selling my house for GPUs.

pizza234 2mo ago

> but are they functionally useful or helpful to anyone currently? Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do. MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro. The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).

ddevnyc 2mo ago
Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?
- dannyw 2mo ago
  MoE has nothing to do with multimodality. MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing. Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization. As for reasoning in non-text modalities, you might find this paper interesting :) https://huggingface.co/papers/2502.05171 https://huggingface.co/papers/2502.05171
  - dummydummy1234 2mo ago
    Wait I thought the router ends up specializing the experts? Like there is no explicit goal aside from each 'expert' getting roughly equal weight? And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference? I do not know what I am talking about, this is my limited understanding...
    - juliendorra 2mo ago
      You are right that at training the main goal is balancing between the sections to avoid certain paths becoming the only path. In the end the inference will be routed token by token to a mixture of say 3 or 4 sections of the model. The combination can change at each turn. It’s really a statistical optimization. As for many things in neural networks, the original intuition coming from anthropomorphism once implemented becomes something very non human!
    - ahepp 2mo ago
      My understanding of it is also pretty surface level, but I was not under the impression that it develops "expertise" in a particular subject matter, at least not in a way that's easy to harness. From what I've read, it develops expertise at the token level. Because the natural continuation of this is to say like, "Ok I want to load the bird detection expert and the navigation expert but leave the medieval European history expert behind", and my understanding is that this is not really how it works. At least at the moment.
  - ahepp 2mo ago
    > You can think of it like sharing. was this meant to read "sharding"?
  - xg15 2mo ago
    > There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization. Huh, always thought one would sort of require the other. For a good MoE model, wouldn't I want to minimize the "churn" between experts, i.e. the amount of time one expert model has to be swapped in for another expert model? That would be naturally the way if experts correspond to semantic categories. E.g. suppose I have a model that can answer questions in 100s of languages. Then while any of those languages might be requested by some caller, it's highly unlikely a caller will request all languages in the same session - realistically, there might be one or two languages in a session and those will then span the entire session. There will also be languages that are requested very often and others that are extremely rare. (Let's say my model also supports Klingon and Sindarin. Those are important for marketing reasons and because I genuinely like to make the occasional nerd happy - but practically, I get maybe a handful of requests for those every few months. So it would make sense to centralize the knowledge for those languages in some specific part of the model, so I can keep that part out of VRAM - and probably RAM as well - during the 99% of time where it's not needed) So wouldn't it make sense to make the expert models language specific here? Then you could take advantage of the fact that a language rarely changes inside a session and keep that expert in VRAM for the entire session. You could also avoid dragging parameters along with you for languages that are practically never used.
bensyverson 2mo ago
Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.
- reactordev 2mo ago
  yes, I average 80-120 tok/s on my RTX 3080 with gemma 4 and faster with Qwen 3.5. The main use-case here is just code-monkey agents. I'm not looking for architectural guidance, but an agent to take a spec and complete it.
  - bensyverson 2mo ago
    And is this using conventional model loading (all in VRAM), or are you streaming it in some way?
- cyanydeez 2mo ago
  in theory, QWEN3.6-35B-A3B should run as fast as a 3B model; and in practice, it can be just as dumb.

nickpsecurity 2mo ago

If I could justify wear and tear and electricity, I was willing to do something like this for batch processing. The batches would be a bunch of prompts whose outputs I'd look at the next day. Maybe common operations, like QA or refactoring, on whatever software I wrote. If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one try. I also would be using AI's as a supplement to, not replacement for, my own brain. So, issues with the outputs wouldn't be a problem because I'm just keeping what's helpful. If I still need to re-generate it all, it might still save money over time by avoiding cloud costs. Also, hardware that's already paid for is a sunk cost that doesn't inflate over time. Glitches in loading or destroying VM's might blow up into a big bill.

parineum 2mo ago
at 5 minutes per token, you could look at the results next week
- bensyverson 2mo ago
  one week later: "It says 'You're absolutely right! Let me look at the seams so I'm checking, not guessing—' and I guess that's when my SSD melted."
  - hex4def6 2mo ago
    Turns out the SSD was load-bearing.
- nickpsecurity 2mo ago
  That's way slower than I thought it would be. I struggle to imagine a use case. If you had a deadly condition, and no diagnosis worked, and a specific model had the answer... past that I wouldn't use it.

deleted 2mo ago

[deleted]

zapataband1 2mo ago

[flagged]

dsl 2mo ago

> but are they functionally useful or helpful to anyone currently? I've been building a SaaS that deals with data that can't be distributed to third parties. Some of the useful AI stuff I can add is not time sensitive and can run overnight. Things like this allow me to use higher quality models without selling my house for GPUs.