4 ms·

Is a GPT3-like model currently available for discussion type use and works on a laptop? How might I go about getting started with locally running a model of thi

by cced 3y ago

Is a GPT3-like model currently available for discussion type use and works on a laptop? How might I go about getting started with locally running a model of this quality?

api 3y ago
The large Alpaca models can run on a M1 Pro MacBook or similar performance Linux PC with llama.cpp. You will need 32-64G of RAM and fast SSD for the large models. They show performance that reminds me of GPT-3 (not quite 3.5). There are some newer models out there I have not tried yet like the open llama, GPT4all, etc. so I’m not sure how good they are. I get the sense they are still GPT-3 level but are achieving that with less RAM. There’s a race on both for raw capability and optimization via pruning and quantization. The latter is important to make these things runnable locally without gigantic hardware. Lots of people have stuff with GPUs, fast CPUs, and 32-64G RAM. Few have huge workstations with hundreds of gigs of RAM. Unless progress stagnates I can see something approaching GPT-4 that can run on under $5k worth of hardware in a year or so. Open model progress seems to be lagging only 1-2 years behind big cloud hosted models.
panzi 3y ago
I heard people claiming this is at GPT 3 level: https://open-assistant.io/ https://open-assistant.io/ Don't know if true. It's open source.
infinityio 3y ago
The quality does not yet measure up exactly to ChatGPT (even 3.5), but yes it is possible Probably the fastest way to get started is to look into [0] - this only requires a beta chromium browser with WebGPU. For a more integrated setup, I am under the impression [1] is the main tool used. If you want to take a look at the quality possible before getting started, [2] is an online service by Hugging Face that hosts one of the best of the current generation of open models (OpenAssistant w/ 30B LLaMa) [0]: https://mlc.ai/web-llm/ https://mlc.ai/web-llm/ [1]: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui [2]: https://huggingface.co/chat https://huggingface.co/chat
- api 3y ago
  I downloaded a version of that openassistant model for llama.cpp and it’s at least on par with GPT-3 or a little beyond. It’s to the level of being generally useful.
diarized 3y ago
Google “We have no moat, and neither does OpenAI” https://news.ycombinator.com/item?id=35813322 https://news.ycombinator.com/item?id=35813322

by cced 3y ago

Is a GPT3-like model currently available for discussion type use and works on a laptop? How might I go about getting started with locally running a model of this quality?

api 3y ago

The large Alpaca models can run on a M1 Pro MacBook or similar performance Linux PC with llama.cpp. You will need 32-64G of RAM and fast SSD for the large models. They show performance that reminds me of GPT-3 (not quite 3.5). There are some newer models out there I have not tried yet like the open llama, GPT4all, etc. so I’m not sure how good they are. I get the sense they are still GPT-3 level but are achieving that with less RAM. There’s a race on both for raw capability and optimization via pruning and quantization. The latter is important to make these things runnable locally without gigantic hardware. Lots of people have stuff with GPUs, fast CPUs, and 32-64G RAM. Few have huge workstations with hundreds of gigs of RAM. Unless progress stagnates I can see something approaching GPT-4 that can run on under $5k worth of hardware in a year or so. Open model progress seems to be lagging only 1-2 years behind big cloud hosted models.

panzi 3y ago

I heard people claiming this is at GPT 3 level: https://open-assistant.io/ https://open-assistant.io/ Don't know if true. It's open source.

infinityio 3y ago

The quality does not yet measure up exactly to ChatGPT (even 3.5), but yes it is possible Probably the fastest way to get started is to look into [0] - this only requires a beta chromium browser with WebGPU. For a more integrated setup, I am under the impression [1] is the main tool used. If you want to take a look at the quality possible before getting started, [2] is an online service by Hugging Face that hosts one of the best of the current generation of open models (OpenAssistant w/ 30B LLaMa) [0]: https://mlc.ai/web-llm/ https://mlc.ai/web-llm/ [1]: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui [2]: https://huggingface.co/chat https://huggingface.co/chat

api 3y ago
I downloaded a version of that openassistant model for llama.cpp and it’s at least on par with GPT-3 or a little beyond. It’s to the level of being generally useful.

diarized 3y ago

Google “We have no moat, and neither does OpenAI” https://news.ycombinator.com/item?id=35813322 https://news.ycombinator.com/item?id=35813322