3 ms·

I'd love to do this offline, so I could feed it all my mail. Am I right that it's still going to be a while before we can do that? Or perhaps with a less good m

I'd love to do this offline, so I could feed it all my mail. Am I right that it's still going to be a while before we can do that? Or perhaps with a less good model than GPT-3?

rahimnathwani 4y ago
The things I've seen all use hosted language models. For example https://github.com/jerryjliu/gpt_index https://github.com/jerryjliu/gpt_index depends on LangChain, which wraps APIs from hosted LLMs: https://langchain.readthedocs.io/en/latest/reference/modules/llms.html https://langchain.readthedocs.io/en/latest/reference/modules... AFAIK there's no GPT-3-like LLM that's easy to run at home, because the number of parameters is so so large. Your gaming PC's GPU won't have enough RAM to hold the model. For example, gpt-neox-20b needs about 40GB of RAM: https://huggingface.co/EleutherAI/gpt-neox-20b/discussions/1#62ad9268d2543e176d3dd612 https://huggingface.co/EleutherAI/gpt-neox-20b/discussions/1...
- flemhans 4y ago
  I wouldn't mind throwing hardware after the problem, but I haven't yet found a "full" guide on how to set it up, it always ends up being something to run in Google Colab or similar.
  - ttt3ts 4y ago
    Hmm. I have full size bloom running on a server in my basement. It can be ran naively with about 400GB of ram. Using used hardware you can get that for about $1200. Still, with CPU inference, you're looking at about 5 mins per response. With optimization, I have it down to 140GB of ram. Trying to get it under 120GB without loosing too much accuracy so it can be ran on standard desktop consumer hardware (who's limits are usually 128GB). Given the lack of resources I have found I figured the general intrest was low? Maybe I will open source it and do a write up.
    - flemhans 4y ago
      I think that would be an incredibly interesting write-up. There are many applications where a 5-min response time would be more than adequate. It could slowly churn through the inbox, while I'm not looking. Or parse customer emails and suggest replies for a rep to potentially use.

I'd love to do this offline, so I could feed it all my mail. Am I right that it's still going to be a while before we can do that? Or perhaps with a less good model than GPT-3?

rahimnathwani 4y ago

The things I've seen all use hosted language models. For example https://github.com/jerryjliu/gpt_index https://github.com/jerryjliu/gpt_index depends on LangChain, which wraps APIs from hosted LLMs: https://langchain.readthedocs.io/en/latest/reference/modules/llms.html https://langchain.readthedocs.io/en/latest/reference/modules... AFAIK there's no GPT-3-like LLM that's easy to run at home, because the number of parameters is so so large. Your gaming PC's GPU won't have enough RAM to hold the model. For example, gpt-neox-20b needs about 40GB of RAM: https://huggingface.co/EleutherAI/gpt-neox-20b/discussions/1#62ad9268d2543e176d3dd612 https://huggingface.co/EleutherAI/gpt-neox-20b/discussions/1...

flemhans 4y ago
I wouldn't mind throwing hardware after the problem, but I haven't yet found a "full" guide on how to set it up, it always ends up being something to run in Google Colab or similar.
- ttt3ts 4y ago
  Hmm. I have full size bloom running on a server in my basement. It can be ran naively with about 400GB of ram. Using used hardware you can get that for about $1200. Still, with CPU inference, you're looking at about 5 mins per response. With optimization, I have it down to 140GB of ram. Trying to get it under 120GB without loosing too much accuracy so it can be ran on standard desktop consumer hardware (who's limits are usually 128GB). Given the lack of resources I have found I figured the general intrest was low? Maybe I will open source it and do a write up.
  - flemhans 4y ago
    I think that would be an incredibly interesting write-up. There are many applications where a 5-min response time would be more than adequate. It could slowly churn through the inbox, while I'm not looking. Or parse customer emails and suggest replies for a rep to potentially use.