4 ms·

I dropped my ChatGPT plus subscription as of the end of July, I'm just using a local client with API access. It'll be much cheaper than $20/month for something

I dropped my ChatGPT plus subscription as of the end of July, I'm just using a local client with API access. It'll be much cheaper than $20/month for something that's still largely a toy for me. Unless OpenAI puts a minimum monthly fee on the API now that GPT4 will be available to everyone. I also got into local LLMs and for my purposes they're pretty great. It does help to have an M2 MAX with 96GB of RAM, but even the small models have improved tremendously.

drapado 3y ago
Which client are you using? Looking for recommendations
- rootusrootus 3y ago
  I use MacGPT. Obviously only a recommendation if you're using a Mac, but I am happy with it so far.
eurekin 3y ago
Please explain more. I toyed around with few initial Llama derived ones and they didn't seem very useful. I was interested most using them as some for of a coding assitant. Which models work ok for you and for what usages?
- CuriouslyC 3y ago
  Go to https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ and get a 30b uncensored quantized variant with instruction tuning. They work pretty well, pretty close to gpt-3.5 in most domains.
jon-wood 3y ago
I'm doing much the same, I've got a tiny Python script that I use to interact with GPT-4 to deal with work related programming questions. For my use case its even better than ChatGPT because I can prime it with a default prompt asking it to be succinct, answer only with code if the answer is some code (rather than providing commentary alongside), and let it know anything I'm asking is in the context of Python running on Linux. With that prompt answers that would previously have been several hundred words of SEO-optimised mush become 5 lines of code I can read to get the idea, and if I need further explanation I can ask for it.
cube2222 3y ago
For anybody else wanting to go down this path, I've created a little open-source desktop chat app you can use locally with the OpenAI API that supports prompt customization, nice code rendering, and tool use (like your local terminal)[0]. [0]: https://github.com/cube2222/cuttlefish https://github.com/cube2222/cuttlefish
tmaly 3y ago
I am plan on getting a RTX 4090 but I wish it had a bit more vram.
- eurekin 3y ago
  TBH, this is what most "open LLM" community is researching right now. Most LORAs or custom models are 7B and sometimes 13B quantized. They fit into 24 GB vram nicely

drapado 3y ago

Which client are you using? Looking for recommendations

rootusrootus 3y ago
I use MacGPT. Obviously only a recommendation if you're using a Mac, but I am happy with it so far.

eurekin 3y ago

Please explain more. I toyed around with few initial Llama derived ones and they didn't seem very useful. I was interested most using them as some for of a coding assitant. Which models work ok for you and for what usages?

CuriouslyC 3y ago
Go to https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ and get a 30b uncensored quantized variant with instruction tuning. They work pretty well, pretty close to gpt-3.5 in most domains.

jon-wood 3y ago

I'm doing much the same, I've got a tiny Python script that I use to interact with GPT-4 to deal with work related programming questions. For my use case its even better than ChatGPT because I can prime it with a default prompt asking it to be succinct, answer only with code if the answer is some code (rather than providing commentary alongside), and let it know anything I'm asking is in the context of Python running on Linux. With that prompt answers that would previously have been several hundred words of SEO-optimised mush become 5 lines of code I can read to get the idea, and if I need further explanation I can ask for it.

cube2222 3y ago

For anybody else wanting to go down this path, I've created a little open-source desktop chat app you can use locally with the OpenAI API that supports prompt customization, nice code rendering, and tool use (like your local terminal)[0]. [0]: https://github.com/cube2222/cuttlefish https://github.com/cube2222/cuttlefish

tmaly 3y ago

I am plan on getting a RTX 4090 but I wish it had a bit more vram.

eurekin 3y ago
TBH, this is what most "open LLM" community is researching right now. Most LORAs or custom models are 7B and sometimes 13B quantized. They fit into 24 GB vram nicely