4 ms·

> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run i

> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for inference using Bloom: https://github.com/bigscience-workshop/petals/wiki/Launch-your-own-swarm https://github.com/bigscience-workshop/petals/wiki/Launch-yo... (You'll need around 200GB of GPU memory across the machines in the swarm)

adam_arthur 4y ago
Hmmm, 2-3 high end GPUs implies it's likely not very far off from mainstream. Maybe runnable on the average device within 10-20 years... perhaps even sooner if the model/software can be optimized?
- lolspace 4y ago
  20 years?
  - adam_arthur 4y ago
    In 20 years I'll still be alive and enjoying myself. Does 20 years seem long to you?
    - krsdcbl 4y ago
      it seems a bit long for achieving just running your own AI models locally, given it seems to be largely a question of vram and that you already _could_ do it today with a handful of graphic cards. 20y ago we had the GeForce Ti4400, current graphics cards now come with 100x the vram and 50-60x the bandwidth.
    - 5e92cb50239222b 4y ago
      That's one hell of an assumption. Many of my Russian friends were absolutely certain they'll be alive and well for at least the next 20 years not that long ago.
      - adam_arthur 4y ago
        If I'm dead then being able to run an AI locally doesn't matter anyway
      - simne 4y ago
        Sure, they will be alive if not come to Ukraine.
- dahdum 4y ago
  > 2-3 high end GPUs implies it's likely not very far off from mainstream Looks like FLOP/s per $ are doubling every ~3 years for high end cards, and 10x in ~10 years. So probably not that far off for desktop users. https://www.lesswrong.com/posts/c6KFvQcZggQKZzxr9/trends-in-gpu-price-performance https://www.lesswrong.com/posts/c6KFvQcZggQKZzxr9/trends-in-...
  - visarga 4y ago
    FLOPs remain unused when the bottleneck is bandwidth.
- bemmu 4y ago
  I’d be surprised if there weren’t any algo breakthroughs before that to make these several times faster (10x?). (such as are all of the weights really needed all of the time, or could you load different ones based on recent context?) If in 10 years on top of that you’d have that 10x faster hardware as well, you might be running GPT-3s as just a subcomponent of games/apps.
  - ETH_start 4y ago
    ChatGPT prompt: What's the difference between a component and subcomponent? A component is a self-contained unit of functionality, often in the form of a class or object, that represents a piece of an overall system or application. A subcomponent, on the other hand, is a smaller, simpler component that is used as part of a larger component. A subcomponent is typically used to break down a larger component into smaller, more manageable pieces, and is often used to create reusable building blocks for larger components. In other words, a subcomponent is a component that is used by another component.
lfkdev 4y ago
Two or three top GPUs? Thats basically nothing for a professinal project or even an investeded hobby
- throwifasd 4y ago
  [flagged]
  - ronsor 4y ago
    A few 4090Ti GPUs might be good enough, if they do indeed end up with 48GB of VRAM.
    - deleted 4y ago
      [deleted]
    - firsttimebigboy 4y ago
      According to wikipedia ChatGPT is built on GPT-3 which has 175B parameters requiring 800GB of memory, so a few 4090s will fall short.
      - samr71 4y ago
        Let's say the 4090ti is priced around $2500. 16 of them would set you back just $40 grand. That's quite a bit for the average consumer, but people spend that kinda dough on cars/boats/planes etc. It's well within the realm of dedicated hobbyists. For companies it's nothing. Someone will come along selling pre-builts for this purpose, if that's not already available for their current research/enterprise cards.
  - wokwokwok 4y ago
    You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L59 https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5... /shrug If you don't want to run it locally, don't. If the hardware you have doesn't support it, don't. ...but to say it's not possible is, again, bluntly, wrong.
corobo 4y ago
How badly is bad? What sort of output are we talking? I am asking as I once had a Markov-chain IRC bot* and while it often struggled to string together a sentence, it was quite hilarious sometimes. Absolutely pointless other than the occasional laugh. Can it form sentences or are those small models completely unusable for anything? I'm not thinking OpenAI level uses - sort of compare a Postgres cluster to a SQLite file (not literally, conceptually I guess). Can it be used for single tasks in any way? Could it figure out how to map search terms to URLs for a knowledge base type thing? Forgive me if these are silly questions. The extent of my knowledge in this field is asking ChatGPT questions and going "that's so cool" when it answers. * Your phone's predictive text except it finishes the sentence itself based on a word someone in chat used so that it felt on topic. In my case it also learned how to form sentences from other people talking in chat, in hindsight it's amazing I never had a Tay issue. https://en.m.wikipedia.org/wiki/Tay_(bot) https://en.m.wikipedia.org/wiki/Tay_(bot)
- wokwokwok 4y ago
  > or are those small models completely unusable for anything? Sadly, they really offer almost no value. For the effort, you’re better off with an NLP framework like spacy. You can play with the small neo gpt models on hugging face, eg. https://huggingface.co/EleutherAI/gpt-neo-125M https://huggingface.co/EleutherAI/gpt-neo-125M …but, the tldr is they’re cute to play with, but practically, the content they can generate is short, inconsistent and full of errors.
  - blagie 4y ago
    .... which is actually not of "almost no value." The value of smaller models is different. For example, I have anonymized data, with fields removed. The smaller models do fine for filling those fields in with plausible values. The smaller models do okay for zero-shot clustering of data in many cases (e.g. liberal versus conservative text), and if not, with minimal training. For generating statistics or probabilistic information about large numbers of text, they're great. GPT-3, they're not, but I use them in my day-to-day work quite a bit more than I thought I would. I bought a GPU for one purpose, and I find I spin it up a lot these days. I /really/ want to be able to use a large-scale language model locally, though. For the types of things I'd like it for, such as helping me draft emails, I don't trust OpenAI with my data.
anigbrowl 4y ago
How much of this is the language vs the vast amount of passably accurate domain knowledge? ChatGPT etc. seem magic because they can answer questions about virtually anything with a high degree of plausibility. It often gets specific facts wrong, but the general contours are correct. Many of us know a lot of trivia/specialist knowledge, but I don't think anyone is as broadly informed as ChatGPT appears to be. It's not clear where the language ends and the encyclopedic knowledge starts, but the latter must be taking up a very large amount of the space in the model.
- dragonwriter 4y ago
  > How much of this is the language vs the vast amount of passably accurate domain knowledge? LLMs don’t have domain knowledge, its all language.
  - anigbrowl 4y ago
    That's what I meant by 'It's not clear where the language ends and the encyclopedic knowledge starts,' since the model (and perhaps our brains) make little distinction. But the model seems to be storing an absolutely vast amount of information, beyond the the capability of any individual person to accumulate and recall. This is clearly not a prerequisite for language, even if the information is represented linguistically. Put another way, at age 20 I had read maybe 10-20% of what I've read since, but I was capable of reading comprehension and conversation even though my levels of knowledge and insight were much lower. By 'comprehension' I mean in the sense of being able to read a piece of text and answer questions about it or rewrite it, without necessarily having any priors about the topic; the kind of task we expect to be able to assign to a high school graduate. I'm wondering what the size of an 'ignorant' language model is, as a precursor to more curated/directed training. While the state of the art is very impressive, it's a bit like taking a feast for a thousand people and rendering it into a giant cube of spam. This strategy seems guaranteed to produce a succession of increasingly capable idiots savant but limits other avenues of exploration.
    - simne 4y ago

adam_arthur 4y ago

Hmmm, 2-3 high end GPUs implies it's likely not very far off from mainstream. Maybe runnable on the average device within 10-20 years... perhaps even sooner if the model/software can be optimized?

lolspace 4y ago
20 years?
- adam_arthur 4y ago
  In 20 years I'll still be alive and enjoying myself. Does 20 years seem long to you?
  - krsdcbl 4y ago
    it seems a bit long for achieving just running your own AI models locally, given it seems to be largely a question of vram and that you already _could_ do it today with a handful of graphic cards. 20y ago we had the GeForce Ti4400, current graphics cards now come with 100x the vram and 50-60x the bandwidth.
  - 5e92cb50239222b 4y ago
    That's one hell of an assumption. Many of my Russian friends were absolutely certain they'll be alive and well for at least the next 20 years not that long ago.
    - adam_arthur 4y ago
      If I'm dead then being able to run an AI locally doesn't matter anyway
    - simne 4y ago
      Sure, they will be alive if not come to Ukraine.
dahdum 4y ago
> 2-3 high end GPUs implies it's likely not very far off from mainstream Looks like FLOP/s per $ are doubling every ~3 years for high end cards, and 10x in ~10 years. So probably not that far off for desktop users. https://www.lesswrong.com/posts/c6KFvQcZggQKZzxr9/trends-in-gpu-price-performance https://www.lesswrong.com/posts/c6KFvQcZggQKZzxr9/trends-in-...
- visarga 4y ago
  FLOPs remain unused when the bottleneck is bandwidth.
bemmu 4y ago
I’d be surprised if there weren’t any algo breakthroughs before that to make these several times faster (10x?). (such as are all of the weights really needed all of the time, or could you load different ones based on recent context?) If in 10 years on top of that you’d have that 10x faster hardware as well, you might be running GPT-3s as just a subcomponent of games/apps.
- ETH_start 4y ago
  ChatGPT prompt: What's the difference between a component and subcomponent? A component is a self-contained unit of functionality, often in the form of a class or object, that represents a piece of an overall system or application. A subcomponent, on the other hand, is a smaller, simpler component that is used as part of a larger component. A subcomponent is typically used to break down a larger component into smaller, more manageable pieces, and is often used to create reusable building blocks for larger components. In other words, a subcomponent is a component that is used by another component.

lfkdev 4y ago

Two or three top GPUs? Thats basically nothing for a professinal project or even an investeded hobby

throwifasd 4y ago
[flagged]
- ronsor 4y ago
  A few 4090Ti GPUs might be good enough, if they do indeed end up with 48GB of VRAM.
  - deleted 4y ago
    [deleted]
  - firsttimebigboy 4y ago
    According to wikipedia ChatGPT is built on GPT-3 which has 175B parameters requiring 800GB of memory, so a few 4090s will fall short.
    - samr71 4y ago
      Let's say the 4090ti is priced around $2500. 16 of them would set you back just $40 grand. That's quite a bit for the average consumer, but people spend that kinda dough on cars/boats/planes etc. It's well within the realm of dedicated hobbyists. For companies it's nothing. Someone will come along selling pre-builts for this purpose, if that's not already available for their current research/enterprise cards.
- wokwokwok 4y ago
  You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L59 https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5... /shrug If you don't want to run it locally, don't. If the hardware you have doesn't support it, don't. ...but to say it's not possible is, again, bluntly, wrong.

corobo 4y ago

How badly is bad? What sort of output are we talking? I am asking as I once had a Markov-chain IRC bot* and while it often struggled to string together a sentence, it was quite hilarious sometimes. Absolutely pointless other than the occasional laugh. Can it form sentences or are those small models completely unusable for anything? I'm not thinking OpenAI level uses - sort of compare a Postgres cluster to a SQLite file (not literally, conceptually I guess). Can it be used for single tasks in any way? Could it figure out how to map search terms to URLs for a knowledge base type thing? Forgive me if these are silly questions. The extent of my knowledge in this field is asking ChatGPT questions and going "that's so cool" when it answers. * Your phone's predictive text except it finishes the sentence itself based on a word someone in chat used so that it felt on topic. In my case it also learned how to form sentences from other people talking in chat, in hindsight it's amazing I never had a Tay issue. https://en.m.wikipedia.org/wiki/Tay_(bot) https://en.m.wikipedia.org/wiki/Tay_(bot)

wokwokwok 4y ago
> or are those small models completely unusable for anything? Sadly, they really offer almost no value. For the effort, you’re better off with an NLP framework like spacy. You can play with the small neo gpt models on hugging face, eg. https://huggingface.co/EleutherAI/gpt-neo-125M https://huggingface.co/EleutherAI/gpt-neo-125M …but, the tldr is they’re cute to play with, but practically, the content they can generate is short, inconsistent and full of errors.
- blagie 4y ago
  .... which is actually not of "almost no value." The value of smaller models is different. For example, I have anonymized data, with fields removed. The smaller models do fine for filling those fields in with plausible values. The smaller models do okay for zero-shot clustering of data in many cases (e.g. liberal versus conservative text), and if not, with minimal training. For generating statistics or probabilistic information about large numbers of text, they're great. GPT-3, they're not, but I use them in my day-to-day work quite a bit more than I thought I would. I bought a GPU for one purpose, and I find I spin it up a lot these days. I /really/ want to be able to use a large-scale language model locally, though. For the types of things I'd like it for, such as helping me draft emails, I don't trust OpenAI with my data.

anigbrowl 4y ago

How much of this is the language vs the vast amount of passably accurate domain knowledge? ChatGPT etc. seem magic because they can answer questions about virtually anything with a high degree of plausibility. It often gets specific facts wrong, but the general contours are correct. Many of us know a lot of trivia/specialist knowledge, but I don't think anyone is as broadly informed as ChatGPT appears to be. It's not clear where the language ends and the encyclopedic knowledge starts, but the latter must be taking up a very large amount of the space in the model.

dragonwriter 4y ago
> How much of this is the language vs the vast amount of passably accurate domain knowledge? LLMs don’t have domain knowledge, its all language.
- anigbrowl 4y ago
  That's what I meant by 'It's not clear where the language ends and the encyclopedic knowledge starts,' since the model (and perhaps our brains) make little distinction. But the model seems to be storing an absolutely vast amount of information, beyond the the capability of any individual person to accumulate and recall. This is clearly not a prerequisite for language, even if the information is represented linguistically. Put another way, at age 20 I had read maybe 10-20% of what I've read since, but I was capable of reading comprehension and conversation even though my levels of knowledge and insight were much lower. By 'comprehension' I mean in the sense of being able to read a piece of text and answer questions about it or rewrite it, without necessarily having any priors about the topic; the kind of task we expect to be able to assign to a high school graduate. I'm wondering what the size of an 'ignorant' language model is, as a precursor to more curated/directed training. While the state of the art is very impressive, it's a bit like taking a feast for a thousand people and rendering it into a giant cube of spam. This strategy seems guaranteed to produce a succession of increasingly capable idiots savant but limits other avenues of exploration.
  - simne 4y ago