5 ms·
Care to share some links? My lack of GPU is the main blocker for me from playing with local-only options. I have an old laptop with 16GB RAM and no GPU. Can I
by john2x 3y ago
Care to share some links? My lack of GPU is the main blocker for me from playing with local-only options.
I have an old laptop with 16GB RAM and no GPU. Can I run these models?
- PostOnce 3y agohttps://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp https://huggingface.co/TheBloke https://huggingface.co/TheBloke There's a LocalLLaMA subreddit, irc channels, and a whole big community around the web working on it on GitHub nd elsewhere. edit: I forgot to directly answer you: yes you can run these models. 16GB of plenty. Different quantizations give you different amounts of smarts and speed. There are tables that tell you how much RAM is needed per which quantization you choose, as well as how fast it can produce results (ms per token). e.g. https://github.com/ggerganov/llama.cpp#quantization https://github.com/ggerganov/llama.cpp#quantization where RAM required a little more than the file size, but there are tables that list it explicitly which I don't have immediately at hand.
- tensor 3y agoA reminder that llama isn't legal for the vast majority of use cases. Unless you signed their contract and then you can use it only for research purposes.
- rvcdbn 3y agoWe don’t actually know that it’s not legal. The copyrightability of model weights is an open legal question right now afaik.
- tensor 3y agoIt doesn't have to be copyrightable to be intellectual property.
- actionfromafar 3y agoPatents? Trademark? What do you mean?
- Rexxar 3y agoMaybe this: https://en.wikipedia.org/wiki/Database_right https://en.wikipedia.org/wiki/Database_right but it doesn't exist in every countries.
- twbarr 3y agoNo, but what is it? Not your lawyer, not legal advice, but it's not a trade secret, they've given it to researchers. It's not a trademark because it's not an origin identifier. The structure might be patentable, but the weights won't be. It's certainly not a mask work. It might have been a contract violation for the guy who redistributed it, but I'm not a party to that contract.
- Art9681 3y agoI'm going to play devil's advocate and state that a lot of what you mentioned will be relevant to a tiny part of the world that has the means to enforce this. The law will be forced to change as a response to AI. Many debates will be had. Many crap laws will be made by people grasping at straws but it's too late. Putting red tape around this technology puts that nation at a technological disadvantage. I would go as far as labeling a national security threat. I'm calling it now. Based on what I see today. Europe will position itself as a leader in AI legislation, and its economy will give way to the nations that want to enter the race and grab a chunk of the new economy. It's a Catch 22. You either gimp your own technological progress, or start a war with a nation that does not. Pretty sure Russia and China don't really care about the ethics behind it. There are plenty of nations capable enough in the same boat. Now what? OK, so in some hypothetical future China has an uncensored model with free reign over the internet. The US and Europe has banned this. What's stopping anyone from running the Chinese model? There isn't enough money in the world to enforce software laws. How long have they tried to take down The Pirate Bay? Pretty much every permutation of every software that's ever been banned can be found and ran with impunity if you have the technical knowledge to do so. No law exists that can prevent that. If it did, OpenAI wouldn't exist.
- PostOnce 3y agoOpenLLaMA is though. https://github.com/openlm-research/open_llama https://github.com/openlm-research/open_llama All of these are surmountable problems. We can beat OpenAI. We can drain their moat.
- donw 3y agoFor the above, are the RAM figures system RAM or GPU?
- PostOnce 3y agoCPU RAM
- kordlessagain 3y ago> We can drain their moat. I've got an AI powered sump pump if you need it.
- ignoramous 3y agoThey most certainly don't need / deserve the snark, to be sure, on hacker news of all places.
- tensor 3y agoAbsolutely, 100% agree. I just wouldn't touch the original LLaMA weights. There are many amazing open source models being built that should be used instead.
- niemandhier 3y agoIt’s not clear if their license terms would hold, for the moment just act and worry later. Update: That is only true for the legal system I am currently residing in. No idea about e.g. the US.
- lhl 3y agoThis is the most well-maintained list of commercially usable open LLMs: https://github.com/eugeneyan/open-llms https://github.com/eugeneyan/open-llms MPT, OpenLLaMA, and Falcon are probably the most generally useful. For code, Replit Code (specifically replit-code-instruct-glaive) and StarCoder (WizardCoder-15B) are the current top open models and both can be used commercially.
- jstummbillig 3y agoJust a heads up: If you are more interested in being effective than being an evangelist, beware. While you can run all kinds of GPTs locally, GPT-4 still smokes everything right now – and even it is not actually good enough to not be a lynchpin for a lot of cases yet.
- slaymaker1907 3y agoI guess ignoring copyright and treating the whole internet as your training data does have its advantages.
- dcow 3y agoYes? That’s the point. Who cares about an outdated concept that has no digital analog? All the artists have moved on already #midjourney.
- bombolo 3y agoWhen mirosoft will open up all of their source code, I will agree with you.
- raxxorraxor 3y agoNo, I doubt artists have moved on. And if they want no artificial gatekeeper, than it is #stablediffusion instead of #midjourney. I would argue that it creates better images too.
- logicchains 3y ago>GPT-4 still smokes everything right now Not if you want it to write adult (graphically pornographic or violent) content.
- tudorw 3y agohttps://gpt4all.io/index.html https://gpt4all.io/index.html
- yard2010 3y agoKeep in mind it doesn't relate to GPT4, the 4 in the name is for, not four. But I should try it. TBH openAI shady practices and MS behind them is just an anti trust waiting to happen and I don't want a part in this dystopia
- moffkalast 3y ago16GB of RAM can fit a 5 bit 13B model at best, they're second dumbest class of LLama model. If Open Orca turns out any good than that might be enough for the time being, but you'll need more RAM to use anything serious. Here's a handy model comparison chart (this is a coding benchmark, so coding-only models tend to rank higher): https://i.imgur.com/AqSjjj2.jpeg https://i.imgur.com/AqSjjj2.jpeg
- PostOnce 3y agoYour benchmark lacks the current #2 https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder It beats Claude and Bard. You could probably get a 4bit 15B model going in 16GB of RAM and be approaching GPT4 in capability. ...on an old laptop, lol Let's eat OpenAI's lunch! They deserve it for trying to steal this tech by "privatizing" a charity, hiding scientific data that was supposed to be shared with us by said charity whose purpose was to help us all, and dishonestly trying to persuade the government not to let us compete with them.
- moffkalast 3y agoYeah I mean I wouldn't really include coding models in this list since they're not general purpose models and have an obvious fine tuning edge compared to the rest. But WizardCoder is definitely something to look at as a Copilot replacement. I'd post a more well rounded benchmark but the problem is that all non-coding benchmarks are currently more or less complete garbage, especially the Vicuna benchmark that rates everything as 99.7% GPT 3.5 lol.
- PostOnce 3y agoThe benchmark you linked was to "programming performance", not generic LLM "intelligence". The situation for the little guy is wildly better than most people imagine.
- moffkalast 3y agoYep, that's what I'm saying, programming performance is seemingly very indicative of model inteligence (assuming it's tuned well enough to be able to run the benchmark at all). Coding is an exercise in problem solving and abstract thinking after all. There are exceptions of course, as there are a few models (e.g. Vicuna, Baize) that don't do well at coding at all but otherwise perform well for chat, and the coding models I mentioned that game the benchmark by sacrificing performance in all other areas. If you exclude those, it's very a accurate overall reasoning level comparison, at least it fits most to what I've seen their performance was for various tasks when testing out individual models. The only other valid benchmark that isn't coding are the SAT and LSAT tests that OpenAI runs on all of their models, but afaik there isn't an open version that would be widely used.