6 ms·

I've only ever been a consumer of ChatGPT/Bard. Never set up any LLM stuff locally, but the idea is appealing to me. I have a ThinkStation P620 w/ThreadRipper

by akulbe 3y ago

I've only ever been a consumer of ChatGPT/Bard. Never set up any LLM stuff locally, but the idea is appealing to me. I have a ThinkStation P620 w/ThreadRipper Pro 3945WX (12c24t) with a GTX 1070 (and a second 1070 I could put in there) and there's 512GB of RAM on the box. Does this need to be bare metal, or can it run in VM? I'm currently running RHEL 9.2 w/KVM (as a VM host) with light usage so far.

davidy123 3y ago
You can run it with CPU or GPU in the VM, in the latter case you'd need to pass the GPU(s through, which depends what kind of VM you're using.
xrd 3y ago
Do you really have 512 GB of RAM on that box?
- akulbe 3y ago
  Yes. I really do. Do you need to see pictures to believe it?