3 ms·

I would like to extend the question: is anyone building a homelab for the specific purpose of training a LLM on their personal info? The choice of hardware (for

I would like to extend the question: is anyone building a homelab for the specific purpose of training a LLM on their personal info? The choice of hardware (for speed, cost, and noise concerns) seems important.

hushpiper 3y ago
I admittedly have not done this myself, but the hardware choices are less complex than you might think. The only truly important part is the GPU, and the number of GPUs on the consumer market that can handle a local LLM is presently quite small. It's pretty much dual-wielding 3090's or 4090's or bust. If you're running but not training a (relatively) small one, you can do well with just one. But if you want to run inference _and_ training on one large enough to be a consistent daily assistant (read: 70b+), cloud hardware for the training is just far more practical and economical. You're generally not gonna wanna drop the $$ for an A100 or two just to do a small training run every couple of weeks on a dataset made up of the amount of RLHF'd conversations one person might have with an LLM in that time.
spaceywilly 3y ago
I think a good use case for this would be auto generation of code documentation. There are many reasons to not want to upload your source code to a cloud AI service, but having an AI that was trained on your local code base so you could ask it “what does this foobar function do anyways?” would be killer.
rodrigodlu 3y ago
For me, nothing fancy, I just added extra ram to a gaming notebook to get enough speed on answers, since it already as a good nvidia card, keeping the api open for access from another laptop I have, via api inside my local network. I have an extra computer for services like filesharing, samba, nfs, git, firewall, etc, for instance caching the models I'm downloading with a squid proxy, so I can test several UIs downloading the same model over again. Not every UI is offering an easy way to set a single folder to store all gguf files, or it's lacking documentation. I'm already having a lot of fun. There's people already doing much more than this. I'm more worried about integrating and gluing in a way that will become transparent after the new year, local models or not. Also how to glue this with obsidian/logseq/neovim/etc in a way that I can use with fewest keystrokes possible, instead just uploading a gigantic context or sensible source code files.

hushpiper 3y ago

I admittedly have not done this myself, but the hardware choices are less complex than you might think. The only truly important part is the GPU, and the number of GPUs on the consumer market that can handle a local LLM is presently quite small. It's pretty much dual-wielding 3090's or 4090's or bust. If you're running but not training a (relatively) small one, you can do well with just one. But if you want to run inference _and_ training on one large enough to be a consistent daily assistant (read: 70b+), cloud hardware for the training is just far more practical and economical. You're generally not gonna wanna drop the $$ for an A100 or two just to do a small training run every couple of weeks on a dataset made up of the amount of RLHF'd conversations one person might have with an LLM in that time.

spaceywilly 3y ago

I think a good use case for this would be auto generation of code documentation. There are many reasons to not want to upload your source code to a cloud AI service, but having an AI that was trained on your local code base so you could ask it “what does this foobar function do anyways?” would be killer.

rodrigodlu 3y ago

For me, nothing fancy, I just added extra ram to a gaming notebook to get enough speed on answers, since it already as a good nvidia card, keeping the api open for access from another laptop I have, via api inside my local network. I have an extra computer for services like filesharing, samba, nfs, git, firewall, etc, for instance caching the models I'm downloading with a squid proxy, so I can test several UIs downloading the same model over again. Not every UI is offering an easy way to set a single folder to store all gguf files, or it's lacking documentation. I'm already having a lot of fun. There's people already doing much more than this. I'm more worried about integrating and gluing in a way that will become transparent after the new year, local models or not. Also how to glue this with obsidian/logseq/neovim/etc in a way that I can use with fewest keystrokes possible, instead just uploading a gigantic context or sensible source code files.