3 ms·

I want something I can put in my own PC. GPUs are utterly insane in pricing, since for the good stuff you need at least 16GB but probably a lot more.

• Dylan16807 1y ago
  9060 XT 16GB, $360 5060 Ti 16GB, $450 If you want more than 16GB, that's when it gets bad. And you should be able to get two and load half your model into each. It should be about the same speed as if a single card had 32GB.
  • oblio 1y ago
    > And you should be able to get two and load half your model into each. It should be about the same speed as if a single card had 32GB. This seems super duper expensive and not really supported by the more reasonably priced Nvidia cards, though. SLI is deprecated, NVLink isn't available everywhere, etc.
    • Dylan16807 1y ago
      No, no, nothing like that. Every layer of an LLM runs separately and sequentially, and there isn't much data transfer between layers. If you wanted to, you could put each layer on a separate GPU with no real penalty. A single request will only run on one GPU at a time, so it won't go faster than a single GPU with a big RAM upgrade, but it won't go slower either.
      • oblio 1y ago
        Interesting, thank you for the feedback, it's definitely worth looking into!