3 ms·
Other than feeding sensitive documents? I’m not sure. OpenAI has certainly gone out of its way to eliminate the need for local LLMs- they offer fine-tuning of G
by gymbeaux 3y ago
Other than feeding sensitive documents? I’m not sure. OpenAI has certainly gone out of its way to eliminate the need for local LLMs- they offer fine-tuning of GPT 3.5 and 4, so it’s harder to argue you might get better results from a local model for a particular task with particular data. Though I don’t have personal experience with the ChatGPT fine-tuning.
It’s fun running models from HuggingFace on my computer. Finally, something that utilizes my computer’s 64GB of RAM and 24GB of VRAM. It’s neat seeing the immense performance difference between CPU (Ryzen 7 5700X) and GPU (RTX 3090) offloading.
I think as with most “Cloud vs on-prem” arguments, it comes down to cost vs convenience. Building an application on Azure or AWS is as easy as it gets, but if money is scarce, you can’t beat on-prem for raw resources.
I’m writing a program right now that will query ChatGPT 4 with… A LOT of tokens. We project it will cost between $5k and $15k and probably run for around 2 days. *OR* I could feed that same data through a local model running on the RTX 3090 and it’ll cost like $20 in electricity, and take maybe 6 days.
- brokenmachine 3y ago>I’m writing a program right now that will query ChatGPT 4 with… A LOT of tokens. What are you writing a program to do? That's more what I feel OP's question is about, and also what I'm interested in.
- gymbeaux 3y agoDocument comprehension/extracting certain fields from the document. There’s some overlap with what can be achieved with regex, but some of the “questions” for ChatGPT require an “intelligent” understanding of the documents. Think “what is this document about?” It’s good at that, but less-good at reliably extracting properties… which is where regex comes in handy I suppose.
- brokenmachine 3y agoThanks. I'm keen to hear how well it works for you.
- gymbeaux 3y agoSo far it’s mixed. Prompt engineering is important and makes all the difference, but for what we pay to hit the OpenAI API, I’m not particularly pleased.
- 2rsf 3y ago> Other than feeding sensitive documents? Everything in our developer's portal is sensitive to a degree, so a local LLM is required to allow talking to it
- datascienced 3y agoA local LLM wont be rate limited, so if you are an AI company you may prefer to secure (as in purchase outright!) your own servers for your AI workload rather than be at the mercy of a third party who could rate limit you. I guess that is not local per-se. For local it has to be privacy and having uncensored models (and maybe the combination of those things).
- gymbeaux 3y agoTrue, I forgot about the uncensored aspect. No “sorry, I can’t advise on that….” answers.