3 ms·

I have a hard time imagining anything that comes even close to ChatGPT being able to run on consumer hardware in the next couple of years

superkuh 4y ago
That's what I thought 2 weeks ago. I figured it'd be ~5 years before I could do anything at home. But already people have the leaked facebook llama weights running on CPU w/under 32 GB of system ram doing a token a second or so.
- riku_iki 4y ago
  that llama is likely much smaller than chatgtp
  - minxomat 4y ago
    The whole point of the LLaMa paper is that large models are undertrained and oversized.
    - riku_iki 4y ago
      not sure where did you get it, but they trained 65b llama too, which outperformed llama 7b on their benchmark.
      - ShamelessC 4y ago
        Here is the paper and synopsis: https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971 > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.
        riku_iki 4y ago
        what's your point? If you read actual paper you will see that 65B outperforms that 7B which was discussed above, and size matters.
        ShamelessC 4y ago
        I don’t think anyone said size doesn’t matter. The point is that by training for longer, models with fewer parameters will perform similarly to GPT-3 models with more parameters.
        riku_iki 4y ago
        probably not if gap is so large: 7B vs 175B. Also, all those benchmark are trash because they can't track data leaks in training data. For example they trained llama on github, where GSM8k eval data is located, of course model will perform well on GSM8K, because it memorized answers.
        ShamelessC 4y ago
        You don’t have to guess, the information is in the provided synopsis. I’m sure there are issues similar to your description. Nevertheless, you seem to be a staunch defender of GPT-3, which to me indicates some kind of bias? Like, who cares if LLaMA is better - in fact, isn’t that a good indicator of progress?
        riku_iki 4y ago
        > You don’t have to guess, the information is in the provided synopsis. yes, I checked benchmarks in paper, and there are many where gpt won over 7b llama. Also, it is not clean experiment, because models were trained on different datasets. > I’m sure there are issues similar to your description. Nevertheless, you seem to be a staunch defender of GPT-3, which to me indicates some kind of bias? Like, who cares if LLaMA is better - in fact, isn’t that a good indicator of progress? personality rants have been ignored.
mattnewton 4y ago
I can certainly imagine it after seeing https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Still a couple years out but moving way faster than I would have expected.
- abraxas 4y ago
  ChatGPT has 175B weights if I'm not mistaken. Llama 7B would not be in any way comparable.
  - superkuh 4y ago
    For people who think the number of parameters determines LLM coherence, well, that's a good rule of thumb. But there's an optimal training set data size to parameters count and gpt3 was trained on too little data. The LLM coming out now trained on more data with fewer parameters and achieve something close. Sure, the 7 billion parameter can't do long outputs. But the 13 billion one is not too bad. They're not a full replacement by any means but for many use cases a local service that is stupider is far preferable to a paid cloud service.
  - v64 4y ago
    One finding in the LLaMA paper [1] is that our current large models are undertrained. LLaMA with 13B params outperforms GPT-3 175B (not ChatGPT), but an "instruct" version of LLaMA was finetuned over the 65B model and did quite well. [1] https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf
KeplerBoy 4y ago
It's crazy, but it seems to be happening already. Granted, that's probably still a far-cry from Chat-GPT, but it seems inevitable a few years down the line. https://news.ycombinator.com/item?id=35100086 https://news.ycombinator.com/item?id=35100086
yieldcrv 4y ago
I could perceive something like in 1 or less. M3 Macbook with eGPU functionality restored in conjunction with more efficient programming would mean having enough memory available to all the processors. This would definitely count as consumer hardware. Custom built GPU-like devices with tons of RAM could become vogue. Kind of like the Nvidia A100 but even more purpose built for running LLMs or whatever models come next.
zamnos 4y ago
Moore's law isn't quite beaten yet, so the (hypothetical future) RTX 5090 and 6090 is gonna be insane. Combined with software optimization and refinement of the techniques, along with training != inference, means I think we'll see something better, runnable locally, in a couple of years. The leaps and bounds Stable Diffusion has gone is insane. Facebook's LLaMA is also seeing a similar growth from just having the model available.
- ihatepython 4y ago
  I hear that disco music is also experiencing tremendous growth, and is poised to make a comeback any day now.

superkuh 4y ago

That's what I thought 2 weeks ago. I figured it'd be ~5 years before I could do anything at home. But already people have the leaked facebook llama weights running on CPU w/under 32 GB of system ram doing a token a second or so.

riku_iki 4y ago
that llama is likely much smaller than chatgtp
- minxomat 4y ago
  The whole point of the LLaMa paper is that large models are undertrained and oversized.
  - riku_iki 4y ago
    not sure where did you get it, but they trained 65b llama too, which outperformed llama 7b on their benchmark.
    - ShamelessC 4y ago
      Here is the paper and synopsis: https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971 > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.
      - riku_iki 4y ago
        what's your point? If you read actual paper you will see that 65B outperforms that 7B which was discussed above, and size matters.
        ShamelessC 4y ago
        I don’t think anyone said size doesn’t matter. The point is that by training for longer, models with fewer parameters will perform similarly to GPT-3 models with more parameters.
        riku_iki 4y ago
        probably not if gap is so large: 7B vs 175B. Also, all those benchmark are trash because they can't track data leaks in training data. For example they trained llama on github, where GSM8k eval data is located, of course model will perform well on GSM8K, because it memorized answers.
        ShamelessC 4y ago
        You don’t have to guess, the information is in the provided synopsis. I’m sure there are issues similar to your description. Nevertheless, you seem to be a staunch defender of GPT-3, which to me indicates some kind of bias? Like, who cares if LLaMA is better - in fact, isn’t that a good indicator of progress?
        riku_iki 4y ago
        > You don’t have to guess, the information is in the provided synopsis. yes, I checked benchmarks in paper, and there are many where gpt won over 7b llama. Also, it is not clean experiment, because models were trained on different datasets. > I’m sure there are issues similar to your description. Nevertheless, you seem to be a staunch defender of GPT-3, which to me indicates some kind of bias? Like, who cares if LLaMA is better - in fact, isn’t that a good indicator of progress? personality rants have been ignored.

mattnewton 4y ago

I can certainly imagine it after seeing https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Still a couple years out but moving way faster than I would have expected.

abraxas 4y ago
ChatGPT has 175B weights if I'm not mistaken. Llama 7B would not be in any way comparable.
- superkuh 4y ago
  For people who think the number of parameters determines LLM coherence, well, that's a good rule of thumb. But there's an optimal training set data size to parameters count and gpt3 was trained on too little data. The LLM coming out now trained on more data with fewer parameters and achieve something close. Sure, the 7 billion parameter can't do long outputs. But the 13 billion one is not too bad. They're not a full replacement by any means but for many use cases a local service that is stupider is far preferable to a paid cloud service.
- v64 4y ago
  One finding in the LLaMA paper [1] is that our current large models are undertrained. LLaMA with 13B params outperforms GPT-3 175B (not ChatGPT), but an "instruct" version of LLaMA was finetuned over the 65B model and did quite well. [1] https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf

KeplerBoy 4y ago

It's crazy, but it seems to be happening already. Granted, that's probably still a far-cry from Chat-GPT, but it seems inevitable a few years down the line. https://news.ycombinator.com/item?id=35100086 https://news.ycombinator.com/item?id=35100086

yieldcrv 4y ago

I could perceive something like in 1 or less. M3 Macbook with eGPU functionality restored in conjunction with more efficient programming would mean having enough memory available to all the processors. This would definitely count as consumer hardware. Custom built GPU-like devices with tons of RAM could become vogue. Kind of like the Nvidia A100 but even more purpose built for running LLMs or whatever models come next.

zamnos 4y ago

Moore's law isn't quite beaten yet, so the (hypothetical future) RTX 5090 and 6090 is gonna be insane. Combined with software optimization and refinement of the techniques, along with training != inference, means I think we'll see something better, runnable locally, in a couple of years. The leaps and bounds Stable Diffusion has gone is insane. Facebook's LLaMA is also seeing a similar growth from just having the model available.

ihatepython 4y ago
I hear that disco music is also experiencing tremendous growth, and is poised to make a comeback any day now.