4 ms·

The article concludes: "Eventually, though, someone will release a program that’s nearly as capable as ChatGPT, and entirely open-source. An enterprising amate

by jbotz 4y ago

The article concludes: "Eventually, though, someone will release a program that’s nearly as capable as ChatGPT, and entirely open-source. An enterprising amateur will find a way to make it run for free on your laptop." Well not in the near future... Large language models like ChatGPT are called "large" for a reason, and they're too big for your laptop for the foreseeable future. I think you'd need a computer with several high-end GPUs that each have 128+GB of RAM to be able to run one of those. Maybe a laptop from 2030 will do. Otherwise a very good article.

sebzim4500 4y ago
FlexGen can already run GPT-3 size models on commodity hardware, albeit with high latency and fairly slow throughput (order of 1 token/s).
api 4y ago
You can run giant models on a high-end laptop today... they'll just be slooooowww since you'll be doing things like swapping data in/out and leveraging the CPU. If you don't mind waiting an hour for a prompt response it can work. It's the same as it's always been. Any general purpose computer is Turing complete. Spending more gets you faster results.
wyldfire 4y ago
> to be able to run one of those It's very important to distinguish the different use cases between training and inference. The amount of memory required to execute the ChatGPT model, once trained, is likely much, much less than 128GB.
- dxuh 4y ago
  Afaik you actually need much more. About 400GB just to load the trained model according to this tweet thread: https://twitter.com/tomgoldsteincs/status/1600196981955100694?lang=de https://twitter.com/tomgoldsteincs/status/160019698195510069... I am not quite sure how reliable the source is, but it makes sense that you at least need to store the 175 billion parameters that define the model in VRAM. I know that for a short-ish while GPUDirect storage is a thing, so that could help for sure, but it would definitely impact execution time as well.
  - moyix 4y ago
    That thread makes a bunch of assumptions that seem a bit dubious to me. We've known since Chinchilla that you don't need 175B parameters to get GPT-3 quality – a 70B model can outperform GPT3 [1]. And his numbers assume the model is loaded into GPU memory in FP16 (175B*2 = 350GB), but people have shown you can quantize down to 8-bit (and in some cases 4 bit) with almost no performance loss. So in 8-bit precision with a 70B model you need ~70GB of VRAM, which you can get with two A6000s on a desktop (each 48GB). And finally there are lots of other ways to get this down. Aside from quantization, people have also shown that you can do pruning – getting rid of many of the weights – again without much perf loss. You can also offload the weights to CPU RAM or an NVME and stream them in as needed [2]; it's slower but if you arrange things right the performance is not too bad. There are also ways to speed up inference using techniques like early exit [3], where you can skip running the whole model for some tokens that are easy to predict. Overall it feels like within a year or two a combination of better quantization/pruning, improved understanding of how to train smaller LLMs, and hardware improvements will put inference for ChatGPT-style models within reach of the average user. [1] https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b-greatly-outperforms-gpt-3-175b-and-gopher-280b-408b9b4510 https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b... [2] https://github.com/FMInference/FlexGen https://github.com/FMInference/FlexGen [3] https://ai.googleblog.com/2022/12/accelerating-text-generation-with.html https://ai.googleblog.com/2022/12/accelerating-text-generati...
CuriouslyC 4y ago
Few points: 1. Chinchilla has demonstrated models are currently unnecessarily large, and would benefit more from data scaling. 2. Models can be brought down in size massively by a combination of distillation and quantization. A GPT-3 equivalent with 50B parameters quantized to fp4 is 200gig, and it could probably be distilled to half of that or less while still being functional for the vast majority of prompts. That means ~100gig memory will be a target for devices in the near future. Once large language models are the main thing people buy GPUs (and even new computers) for, architectures will be redesigned to improve gpu -> memory bandwidth and latency. I wouldn't be surprised to see GPU integrated motherboards as a future premium tier offering, we're already running into heat and space issues with add-on cards and it should be possible to build a low latency bus to a unified system memory.
- sebzim4500 4y ago
  >1. Chinchilla has demonstrated models are currently unnecessarily large, and would benefit more from data scaling. Not convinced. It showed this for the original self supervised task, but it might be true that the spare parameters end up being useful for the later finetuning/RLHF stages.
- moyix 4y ago
  Many of these models are already being trained in FP16, and FP8 seems likely now that the H100s support it.
deleted 4y ago
[deleted]
netruk44 4y ago
I'll preface by saying that Mac is definitely not the platform for deep learning currently. However! The M2 MacBook Pros can optionally be equipped with 96 GB of RAM, all of which can be accessed by the GPU. Assuming that somebody, somewhere, is working on improving things for Mac, we may very likely already have the hardware to run at least a distilled version of ChatGPT locally on laptops. (And if not the MBP, then the M1 Mac Studio would be a good runner-up with 128 GB of memory, though that's obviously not a laptop)
l33tman 4y ago
You can run this today on a consumer GPU at slow speed, using swapping and 4-bit weights (which works surprisingly well and is the new hot topic now)

by jbotz 4y ago

sebzim4500 4y ago

FlexGen can already run GPT-3 size models on commodity hardware, albeit with high latency and fairly slow throughput (order of 1 token/s).

api 4y ago

You can run giant models on a high-end laptop today... they'll just be slooooowww since you'll be doing things like swapping data in/out and leveraging the CPU. If you don't mind waiting an hour for a prompt response it can work. It's the same as it's always been. Any general purpose computer is Turing complete. Spending more gets you faster results.

wyldfire 4y ago

> to be able to run one of those It's very important to distinguish the different use cases between training and inference. The amount of memory required to execute the ChatGPT model, once trained, is likely much, much less than 128GB.

dxuh 4y ago
Afaik you actually need much more. About 400GB just to load the trained model according to this tweet thread: https://twitter.com/tomgoldsteincs/status/1600196981955100694?lang=de https://twitter.com/tomgoldsteincs/status/160019698195510069... I am not quite sure how reliable the source is, but it makes sense that you at least need to store the 175 billion parameters that define the model in VRAM. I know that for a short-ish while GPUDirect storage is a thing, so that could help for sure, but it would definitely impact execution time as well.
- moyix 4y ago
  That thread makes a bunch of assumptions that seem a bit dubious to me. We've known since Chinchilla that you don't need 175B parameters to get GPT-3 quality – a 70B model can outperform GPT3 [1]. And his numbers assume the model is loaded into GPU memory in FP16 (175B*2 = 350GB), but people have shown you can quantize down to 8-bit (and in some cases 4 bit) with almost no performance loss. So in 8-bit precision with a 70B model you need ~70GB of VRAM, which you can get with two A6000s on a desktop (each 48GB). And finally there are lots of other ways to get this down. Aside from quantization, people have also shown that you can do pruning – getting rid of many of the weights – again without much perf loss. You can also offload the weights to CPU RAM or an NVME and stream them in as needed [2]; it's slower but if you arrange things right the performance is not too bad. There are also ways to speed up inference using techniques like early exit [3], where you can skip running the whole model for some tokens that are easy to predict. Overall it feels like within a year or two a combination of better quantization/pruning, improved understanding of how to train smaller LLMs, and hardware improvements will put inference for ChatGPT-style models within reach of the average user. [1] https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b-greatly-outperforms-gpt-3-175b-and-gopher-280b-408b9b4510 https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b... [2] https://github.com/FMInference/FlexGen https://github.com/FMInference/FlexGen [3] https://ai.googleblog.com/2022/12/accelerating-text-generation-with.html https://ai.googleblog.com/2022/12/accelerating-text-generati...

CuriouslyC 4y ago

Few points: 1. Chinchilla has demonstrated models are currently unnecessarily large, and would benefit more from data scaling. 2. Models can be brought down in size massively by a combination of distillation and quantization. A GPT-3 equivalent with 50B parameters quantized to fp4 is 200gig, and it could probably be distilled to half of that or less while still being functional for the vast majority of prompts. That means ~100gig memory will be a target for devices in the near future. Once large language models are the main thing people buy GPUs (and even new computers) for, architectures will be redesigned to improve gpu -> memory bandwidth and latency. I wouldn't be surprised to see GPU integrated motherboards as a future premium tier offering, we're already running into heat and space issues with add-on cards and it should be possible to build a low latency bus to a unified system memory.

sebzim4500 4y ago
>1. Chinchilla has demonstrated models are currently unnecessarily large, and would benefit more from data scaling. Not convinced. It showed this for the original self supervised task, but it might be true that the spare parameters end up being useful for the later finetuning/RLHF stages.
moyix 4y ago
Many of these models are already being trained in FP16, and FP8 seems likely now that the H100s support it.

deleted 4y ago

[deleted]

netruk44 4y ago

I'll preface by saying that Mac is definitely not the platform for deep learning currently. However! The M2 MacBook Pros can optionally be equipped with 96 GB of RAM, all of which can be accessed by the GPU. Assuming that somebody, somewhere, is working on improving things for Mac, we may very likely already have the hardware to run at least a distilled version of ChatGPT locally on laptops. (And if not the MBP, then the M1 Mac Studio would be a good runner-up with 128 GB of memory, though that's obviously not a laptop)

l33tman 4y ago

You can run this today on a consumer GPU at slow speed, using swapping and 4-bit weights (which works surprisingly well and is the new hot topic now)