3 ms·

> This also helps distributes traffic as a side effect. I'm not sure it's "distributing" traffic so much as amplifying it.

• ninetyninenine 3y ago
  Two users. One user is talking about physics, the other about art. Two different models are utilized. Load is divided across 2 models. Load balancing is a feature for free and division is across subjects. Of course this is assuming each model owns it's own set of gpus.
  • apetresc 3y ago
    Ah okay, I misunderstood. I thought you meant inferencing on all the different models and letting them judge to what extent they're the best fit. What you're suggesting is just simply intent classification and using a specific model per intent. That's what everyone did _before_ LLMs.
    • ninetyninenine 3y ago
      Given the existence of specialized LLMs people can still do this.
  • felurx 3y ago
    You could use the same GPUs to run two instances of one model.