37 ms·
Excellent packaging OP! I just wanted to say 2 things relating to LLaMa: 1) 7B is unusable for anything really, in case you are hopeful; 2) 68B otoh is awesom
by ojosilva 4y ago
Excellent packaging OP! I just wanted to say 2 things relating to LLaMa:
1) 7B is unusable for anything really, in case you are hopeful;
2) 68B otoh is awesome ("at least DaVinci level").
I don't know if this is something FB/Meta planned strategically but this LLaMa-mania (LLaMania?) over the weekend is their November/2022 chatGPT moment. If they (Mark) take it seriously, it could become a strong hand in AI and a hint of how the industry could be shaped in the near future, with cloud models competing with local installs.
Think about it: who ever trains a popular, albeit closed model, can give it whatever bias it wishes with nearly no oversight. A dystopian and scary thought.
- oceanplexian 4y ago> Think about it: who ever trains a popular, albeit closed model, can give it whatever bias it wishes with nearly no oversight. A dystopian and scary thought. You have perfectly described what OpenAI did. They released a moralizing “biased” model behind a gated API with no oversight. The only dystopia is one in which corporations get to decide what is, or isn’t considered biased.
- EGreg 4y agoIs this 68B of RAM? How do you get access to that on a Macbook?
- junipertea 4y agoThat’s 68 billions of parameters. It probably does not fit on ram. Though If you encode each parameter using one byte, you would need 68GB RAM which you could get on workstations at this point.
- taf2 4y agoSeems to use about 40~ GB RAM here...
- terafo 4y agoIt fits, whisper.cpp uses 4 bit quantization, 13B model takes a little bit more than 8gb and around 9gb ram while inferencing.
- gymbeaux 4y agoEveryone with “only” 64GB of RAM is pouting today, including me
- Taek 4y agoYou can run llama using 4 bits per parameter, 64 GB of RAM is more than enough
- geysersam 4y ago4 bits is ridiculously little. I'm very curious what makes these models so robust to quantization.
- MacsHeadroom 4y agoRead The Case for 4 Bit Precision. https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720 Spoiler: it's the parameter count. As parameter count goes up, but depth matters less. It just so happens that at around 10B+ parameters you can quantize down to 4bit with essentially no downsides. Models are that big now. So there's no need to waste RAM by having unnecessary precision for each parameter.
- Taek 4y agoFor completeness, there's also another paper that demonstrated you get more power/accuracy per-bit at 4 bits than at any other level of precision (including 2 bits and 3 bits)
- MacsHeadroom 4y agoThat's the paper I referenced. But newer research is already challenging it. 'Int-4 llama is not enough [0] - Int-3 and beyond' suggests 3-bit is best for models larger than ~10B parameters when combining binning and GPTQ. [0] https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-int-3-and https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i...
- deleted 4y ago[deleted]
- boredemployee 4y agosorry for the extremely dumb question but is it possible to run the 68B model in a 8gb ram computer?
- infinityio 4y agoin general, assume 2GB per billion parameters - with quantisation you can get this down to <1GB (~500MB for 3 bit?), but even with that you'll only be able to run quantised llama-13B in the best case Having said that: if you are feeling incredibly patient you can technically run the 68B parameter model by swapping to disk, although it will not be a pleasant experience (think minutes or hours per token instead of tokens per second) Additionally worth noting pure CPU inference is much slower than GPU/TPU inference, so the output will be much slower than a ChatGPT-like service even if it does fit in your computer's RAM
- boredemployee 4y agothanks for explaining! How much GPU memory would work nice with 68B?
- ukd1 4y agothey said 2g per 1 billion....and it's called 68B...I presume that's 68 billion... 68*2...so at least 136g?
- vishal0123 4y ago68/2, not 682
- boredemployee 4y agoSo, if I understand correctly, that's what you need to run the best model? With GPU: VRAM + RAM >= 68/2 Without GPU: RAM >= 68/2
- 4y ago
- jacooper 4y agoSomething important, is that LLama was leaked, it was never directly published by Meta. So its basically piracy, and even if you got it officially, the license is very restrictive.
- deleted 4y ago[deleted]
- EMIRELADERO 4y agoI dispute that the model can be copyrightable in the first place.
- Breza 4y agoYou can restrict someone with a license even if you can't copyright the underlying technology.
- coldtea 4y agoAs long as the courts don't dispute it, then our disputes don't matter. They'd be no better than some "sovereign citizen" disputing their arrest...
- MacsHeadroom 4y agoThe idea that models can't be copyrighted isn't far fetched. The basic idea is that models are created by an automated process not by a person. The courts have already upheld that AI generated output is not copyrightable for this exact reason. So if you do not buy that it applies to models then you would have to explain the difference between the process which outputs bits into a model's layers (aka training) and the process which takes bits into the input layer and then dumps out the subsequent bits of the output layer (inference /generation). Then explain why that distinction is different in regards to the applicability of copyright.
- wongarsu 4y agoAll the arguments for why photographs are copyrighted would seem to apply. The photographer isn't painting the image, but his artistic input is still vital to creating the image. Same with training these models: the training is just an algorithm on some data, but choosing the right hyperparameters and training data is an artistic expression of the author, making copyright apply
- DennisP 4y ago> who ever trains a popular, albeit closed model, can give it whatever bias it wishes with nearly no oversight. That's true even if you can download the whole model. It's not like we can figure out what it's doing from looking at the weights. Training the model locally might avoid intentional bias, but that's what takes a huge GPU farm.