4 ms·

This is beyond exciting. Welcome to the new reality! On one hand, the resources required to run these models continues falling dramatically, thanks to the tech

This is beyond exciting. Welcome to the new reality! On one hand, the resources required to run these models continues falling dramatically, thanks to the techniques discovered by researchers: GPTQ quantizing down to 4, 3, 2, even 1 bits! model pruning! hybrid vram offloading! better, more efficient architectures! 1-click finetuning on consumer hardware! Of course, the free lunches won't last forever, and this will level off, but it's still incredible. And on the other side of the coin, the power of all computing devices continues its ever-upward exponential growth. So you have a continuous lowering of requirements, combined with a continuous increase in available power... surely these two trends will collide, and I can only imagine what this stuff will be like at that intersection.

visarga 3y ago
At that intersection is the "Good Enough Model" that can solve 95% of our needs in full privacy and with complete customisability. The key point is being easy to run on every device. We'll still use proprietary, expensive models for the rest of 5%.
acapybara 3y ago
[flagged]
- quickthrower2 3y ago
  This of course is making AI more Open, the final piece is for people to start packing these in Windows and Mac installers so that the average computer user can make use of them. And for them to run on the crappy graphics card you are likely to get with your default configuration of PC! Or for these to be at least available in every cloud and well understood so you are just paying AWS for compute but not secret sauce (like Kubernetes for example which lets you walk and there is genuine competition)
- Mike_12345 3y ago
  [flagged]
  - acapybara 3y ago
    Mike_12345, that kind of language and accusation is not productive or helpful to the conversation. In fact, it can be quite harmful and perpetuate harmful stereotypes and xenophobia. Let's focus on the important work being done by the RedPajama project and the potential benefits it can bring to a wide range of applications and users. Instead of promoting division and suspicion, let's work towards creating a more diverse and inclusive and collaborative environment in AI research and development.
    - deleted 3y ago
      [deleted]
    - deleted 3y ago
      [deleted]
    - Mike_12345 3y ago
      LOL you actually sound like ChatGPT. Are you a LLM?
      - dang 3y ago
        We've banned this account for repeatedly breaking the site guidelines. Please don't create accounts to break HN's rules with. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - dang 3y ago
      We've banned this account for trolling. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
quickthrower2 3y ago
I would love to see an article on why quantising to low bits works. Seems counterintuitive to me. For example do that with a CD and it will sound awful. It took smarts to come up with mp3 format rather than just reduce number of bits.
- int_19h 3y ago
  A very broad answer is that large NNs are surprisingly resilient to inaccuracies, and it seems to be more pronounced as size grows larger. This is readily observable with LLaMA, where 4-bit quantization affects 7B worst of all. Furthermore, model size is still the most significant contributor to output quality. E.g. vanilla llama-30b at 4-bit has better perplexity than any llama-13b finetune at 8-bit. Thus, if 4-bit lets you fit a larger model into available (V)RAM, you're still better off. This is also why analog computing is seriously considered as a hardware architecture for LLMs: if you don't actually need bit-perfect matmul for things to work well, it can be done much simpler as an analog circuit, and then you can cram a lot more of them on the same chip. Any resulting quality loss would presumably be minor, and in any case would be more than compensated by the much larger model sizes allowed by such architecture.
- specproc 3y ago
  The analogy I'm currently favouring when talking to semi technical people is that LLMs are a map. We map words and phrases to a coordinate space. We can use GPS to locate anything down to a sliding scale of decimal precision. There are only so many digits you need to locate a city or even a house.
- deepsquirrelnet 3y ago
  A CD doesn’t work as an analogy. Think about it this way — if you build a model and don’t train it at all, it will still have the same number of parameters and take up the same amount of disk space. We’re finding out that many models are undertrained for their sizes, and a good option is to post process them into smaller models by teaching a smaller model to mimic their output. Quantization effectively cuts down the model size as well. No loss in quality means that the model has not been trained enough to take advantage of the depth of precision that is available.
wcunning 3y ago
Do you have reading links on the consumer hardware fine tuning? I can’t find much from that description…
- Oranguru 3y ago
  Take a look at: https://huggingface.co/blog/trl-peft https://huggingface.co/blog/trl-peft

visarga 3y ago

At that intersection is the "Good Enough Model" that can solve 95% of our needs in full privacy and with complete customisability. The key point is being easy to run on every device. We'll still use proprietary, expensive models for the rest of 5%.

acapybara 3y ago

[flagged]

quickthrower2 3y ago
This of course is making AI more Open, the final piece is for people to start packing these in Windows and Mac installers so that the average computer user can make use of them. And for them to run on the crappy graphics card you are likely to get with your default configuration of PC! Or for these to be at least available in every cloud and well understood so you are just paying AWS for compute but not secret sauce (like Kubernetes for example which lets you walk and there is genuine competition)
Mike_12345 3y ago
[flagged]
- acapybara 3y ago
  Mike_12345, that kind of language and accusation is not productive or helpful to the conversation. In fact, it can be quite harmful and perpetuate harmful stereotypes and xenophobia. Let's focus on the important work being done by the RedPajama project and the potential benefits it can bring to a wide range of applications and users. Instead of promoting division and suspicion, let's work towards creating a more diverse and inclusive and collaborative environment in AI research and development.
  - deleted 3y ago
    [deleted]
  - deleted 3y ago
    [deleted]
  - Mike_12345 3y ago
    LOL you actually sound like ChatGPT. Are you a LLM?
    - dang 3y ago
      We've banned this account for repeatedly breaking the site guidelines. Please don't create accounts to break HN's rules with. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - dang 3y ago
    We've banned this account for trolling. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

quickthrower2 3y ago

I would love to see an article on why quantising to low bits works. Seems counterintuitive to me. For example do that with a CD and it will sound awful. It took smarts to come up with mp3 format rather than just reduce number of bits.

int_19h 3y ago
A very broad answer is that large NNs are surprisingly resilient to inaccuracies, and it seems to be more pronounced as size grows larger. This is readily observable with LLaMA, where 4-bit quantization affects 7B worst of all. Furthermore, model size is still the most significant contributor to output quality. E.g. vanilla llama-30b at 4-bit has better perplexity than any llama-13b finetune at 8-bit. Thus, if 4-bit lets you fit a larger model into available (V)RAM, you're still better off. This is also why analog computing is seriously considered as a hardware architecture for LLMs: if you don't actually need bit-perfect matmul for things to work well, it can be done much simpler as an analog circuit, and then you can cram a lot more of them on the same chip. Any resulting quality loss would presumably be minor, and in any case would be more than compensated by the much larger model sizes allowed by such architecture.
specproc 3y ago
The analogy I'm currently favouring when talking to semi technical people is that LLMs are a map. We map words and phrases to a coordinate space. We can use GPS to locate anything down to a sliding scale of decimal precision. There are only so many digits you need to locate a city or even a house.
deepsquirrelnet 3y ago
A CD doesn’t work as an analogy. Think about it this way — if you build a model and don’t train it at all, it will still have the same number of parameters and take up the same amount of disk space. We’re finding out that many models are undertrained for their sizes, and a good option is to post process them into smaller models by teaching a smaller model to mimic their output. Quantization effectively cuts down the model size as well. No loss in quality means that the model has not been trained enough to take advantage of the depth of precision that is available.

wcunning 3y ago

Do you have reading links on the consumer hardware fine tuning? I can’t find much from that description…

Oranguru 3y ago
Take a look at: https://huggingface.co/blog/trl-peft https://huggingface.co/blog/trl-peft