Maybe VLLM is better at inferencing MoE (also you can set the number of experts to use). In theory half of the model fits to RAM, so it should be GPU limited i

Maybe VLLM is better at inferencing MoE (also you can set the number of experts to use). In theory half of the model fits to RAM, so it should be GPU limited if memory management is smart.