4 ms·
Note that while "open and free" is possible if a company bites the bullet, there is currently no way it would be self-hostable on consumer GPU
by lairv 4y ago
Note that while "open and free" is possible if a company bites the bullet, there is currently no way it would be self-hostable on consumer GPU
- williamcotton 4y agoWouldn't it be great if there was some way to do distributed training of models?
- lairv 4y agoActually there is the Petals [0] project which does that (I think it's more for inference and fine-tuning, not full training) and might be the best current approach for "self-hosted LLM", though it defeats the point of privacy/anonymity since everyone in the distributed swarm have access to your data [0] https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals
- heurist 4y agoIs there any concept of a "frontend" model that interfaces with a "backend" model? I.e. frontend takes the raw input, encodes it, backend processes the encoding, returns to frontend, which decodes for end use?
- jonathankoren 4y agoAll models have some sort of translation layer between the real world and world of statistics. If you mean a common input format that obfuscates the raw input, anything that deals in a latent space could work, but they’d be pretty generic. It would be interesting if you could make a system that does statistics and (even modeling if you get it to work) with homomorphic encrypted data. That way, even the host of the model couldn’t understand what was going on
- nawgz 4y agoI'm not sure I understand your question. How would this concept do anything to help with privacy preservation or the fact that ChatGPT runs on 800GB of VRAM?
- grogenaut 4y agoWhy is something runnable on 4x A10 (see parent sibling) but not 1x4090 in 4-8x the time
- ilyt 4y agoCoz it uses all of the memory so the moment you'd have to spill it to RAM or worse, disk it would be not x4 but x100+ slower
- grogenaut 4y agoSure but that would get you to one inferences around every second. I wait around 1-2 minutes with some online tools for 4 inferences. So seems like it might work fine for a local no data leak version?
- lairv 4y agoThe problem here is memory, so while it should be possible to run inference on a 4090, you would have to constantly move data from VRAM to RAM, though I don't have benchmarks to prove it I think it would be slower than 10x
- gmadsen 4y agodoes that mean it could at least run? even if a magnitude or two slower?
- heisenzombie 4y agoVRAM, you need enough. That’s sort of a binary thing: either you have enough or you don’t. 1x4090 does not.
- grogenaut 4y agoBut at 4x another GPU, they also don't have the vram, right thus 4...?