3 ms·
That's a tricky number. Does it run on an 80GB GPU, does it auto-shave some parameters to fit in 79.99GB like any articifially "intelligent" piece of code would
by dheera 3y ago
That's a tricky number. Does it run on an 80GB GPU, does it auto-shave some parameters to fit in 79.99GB like any articifially "intelligent" piece of code would do, or does it give up like an unintelligent piece of code?
- declaredapple 3y agoWhat? Are you asking if the framework automatically quantizes/prunes the model on the fly? Or are you suggesting the LLM itself should realize it's too big to run, and prune/quantize itself? Your references to "intelligent" almost leads me to the conclusion that you think the LLM should prune itself. Not only is this a chicken and egg problem, but LLMs are statistical models, they aren't inherently self bootstraping.
- dheera 3y agoI realize that, but I do think it's doable to bootstrap it on a cluster and teach itself to self-prune, and surprised nobody is actively working on this. I hate software that complains (about dependencies, resources) when you try to run it and I think that should be one of the first use cases for LLMs to get L5 autonomous software installation and execution.
- Red_Leaves_Flyy 3y agoMake your dreams a reality!
- lobocinza 3y agoWorst is software that doesn't complain but fails silently.
- 2099miles 3y agoThe LLM itself should realize it’s too big and only put the important parts on the gpu. If you’re asking questions about literature there’s no need to have all the params on the gpu, just tell it to put only the ones for literature on there.
- Jedd 3y agoAre you aware how Macs present memory? Their 'unified' memory approach means you could run an 80GB model on a 128GB machine. There's no concept of 'dedicated GPU memory' as per conventional amd64 arch machines.