7 ms·

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as muc

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

brucethemoose2 3y ago
If MeZO gets implemented, we are basically there: https://github.com/princeton-nlp/MeZO https://github.com/princeton-nlp/MeZO
- moffkalast 3y ago
  Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.
  - nl 3y ago
    According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.
    - moffkalast 3y ago
      Wow if that's true then it's genuinely a complete gamechanger for LLMs as a whole. You probably mean more like 0.3s per token, not per example, but that's still more like 1 or two minutes per training case, not like a day for 4 cases like it is now.
    - valval 3y ago
      I think more importantly, what would the fine tuning routine look like? It's a non-trivial task to dump all of your personal data into any LLM architecture.
    - f_devd 3y ago
      MeZO assumes a smooth parameter space, so you probably won't be able to do it with INT4/8 quantization, probably needs fp8 or smoother.
    - isoprophlex 3y ago
      If you go through the drudgery of integrating with all the existing channels (mail, Teams, discord, slack, traditional social media, texts, ...), such rapid finetuning speeds could enable an always up to date personality construct, modeled on you. Which is my personal holy grail towards making myself unnecessary; it'd be amazing to be doing some light gardening while the bot handles my coworkers ;)
      - deleted 3y ago
        [deleted]
      - vgb2k18 3y ago
        > while the bot handles my coworkers Or it handles their bots ;)
    - gliptic 3y ago
      I cannot find any such numbers in the paper. What the paper says is that MeZO converges much slower than SGD, and each step needs two forward passes. "As a limitation, MeZO takes many steps in order to achieve strong performance."
    - sp332 3y ago
      It's the same memory footprint as inference. It's not that fast, and the paper mentions some optimizations that could still be done.
      - nl 3y ago
        Yes you are right. I completely misread that!
rvz 3y ago
> ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. > I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that! I think you are setting yourself up for disappointment in the future.
- ulchar 3y ago
  > The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. How exactly could they meaningfully do that? Genuine question. The issue with the OpenAI business model is that the collaboration within academia and open source circles is creating innovations that are on track to out-pace the closed source approach. Does OpenAI have the pockets to buy the open source collaborators and researchers? I'm truly cynical about many aspects of the tech industry but this is one of those fights that open source could win for the betterment of everybody.
  - maxilevi 3y ago
    I agree with the spirit but saying that open source is on track to outpace OpenAI in innovation is just not true. Open source models are being compared to GPT3.5, none yet even get close to GPT4 quality and they finished that last year.
    - jart 3y ago
      We're basically surviving off the scraps companies like Facebook have been tossing off the table, like LLaMA. The fact that we're even allowed and able to use these things ourselves, at all, is a tremendous victory.
      - maxilevi 3y ago
        I agree
behnamoh 3y ago
I wonder if ClosedAI and other companies use the findings of the open source community in their products. For example, do they use QLORA to reduce the costs of training and inference? Do they quantize their models to serve non-subscribing consumers?
- danielbln 3y ago
  Not disagreeing with your points, but saying "ClosedAI" is about as clever as writing M$ for Microsoft back in the day, which is to say not very.
  - loa_in_ 3y ago
    I'd say saying M$ makes it harder for M$ to find out I'm talking about them in them in the indexed web because it's more ambiguous, that's all I need to know.
    - coolspot 3y ago
      If we are talking about indexing, writing M$ is easier to find in an index because it is a such unique token. MS can mean many things (e.g. Miss), M$ is less ambiguous.
  - rafark 3y ago
    I think it’s ironic that M$ made ClosedAI.
    - replygirl 3y ago
      Pedantic but that's not irony
      - rafark 3y ago
        Why do you think so? According to the dictionary, ironic could be something paradoxical or weird.
SparkyMcUnicorn 3y ago
Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.
- SkyPuncher 3y ago
  I think people want both. They want fine tuning for their style of communication and interaction. They want better rank and retrieval for rote information. In other words, it’s like having a spouse/partner. There are certain ways that we communicate that we simply know where the other person is at or what they actually mean.
  - SparkyMcUnicorn 3y ago
    Unless you want machine-readable responses, or some other very specific need, the benefits of a fine-tuned model aren't really going to be that much better than a prompt that asks for the style you want along with an example or two. It also raises the barrier to entry quite a bit, since the majority of computers that can run the model aren't capable of training on it. Even if you're using OpenAI's models, gpt-3.5-turbo is going to be much better (cheaper, bigger context window, higher quality) than any of their models that can be fine-tuned. But if you're able to fine-tune a local model, then a combination of fine-tuning and embedding is probably going to give you better results than embedding alone.
- dr_dshiv 3y ago
  How does this work?
  - deet 3y ago
    The parent is saying that "fine tuning", which has a specific meaning related to actually retraining the model itself (or layers at its surface) on a specialized set of data, is not what the GP is actually looking for. An alternative method is to index content in a database and then insert contextual hints into the LLM's prompt that give it extra information and detail with which to respond with an answer on-the-fly. That database can use semantic similarity (ie via a vector database), keyword search, or other ranking methods to decide what context to inject into the prompt. PrivateGPT is doing this method, reading files, extracting their content, splitting the documents into small-enough-to-fit-into-prompt bits, and then indexing into a database. Then, at query time, it inserts context into the LLM prompt The repo uses LangChain as boilerplate but it's pretty easily to do manually or with other frameworks. (PS if anyone wants this type of local LLM + document Q/A and agents, it's something I'm working on as supported product integrated into macOS, and using ggml; see profile)
ignoramous 3y ago
Can LLaMA be used for commerical purposes though (might limit external contributors)? I believe, FOSS alternatives like DataBricks Dolly / Together RedPajama / Eluether GPT NeoX (et al) is where the most progress is likely to be at.
- samwillis 3y ago
  Although llama.cpp started with the LLaMA model, it now supports many others.
- okhuman 3y ago
  This is a very good question that will be interesting how this develops. thanks for posting the alternatives list.
- detrites 3y ago
  May also be worth mentioning - UAE's Falcon, which apparently performs well (leads?). Falcon recently had its royalty-based commercial license modified to be fully open for free private and commercial use, via Apache 2.0: https://falconllm.tii.ae/ https://falconllm.tii.ae/
  - mistercow 3y ago
    Hugging Face has a demo of the 40B Falcon instruct model: https://huggingface.co/blog/falcon#demo https://huggingface.co/blog/falcon#demo It’s pretty good as models of that size go, although it doesn’t take a lot of playing around with it to find that there’s still a good distance between it and ChatGPT 3.5. (I do recommend editing the instructions before playing with it though; telling a model this size that it “always tells the truth” just seems to make it overconfident and stubborn)
- chaxor 3y ago
  Why is commercial necessary to run local models?
  - ignoramous 3y ago
shostack 3y ago
I've been trying to figure out what I might need to do in order to turn my Obsidian vault into a dataset to fine tune against. I'd invest a lot more into it now if I thought it would be a key to an AI learning about my the way it does in the movie Her.
- 58x14 3y ago
  I've been working on this for awhile now and I'd love to chat. I'll send you an email.
  - legendofbrando 3y ago
    I'm interested in this as well and have been exploring similarly. Would be super interesting to chat if you're up for it as well. Sending you an email to say hello.
- mydjtl 3y ago
  The holy grail. https://github.com/brianpetro/obsidian-smart-connections https://github.com/brianpetro/obsidian-smart-connections https://wfhbrian.com/introducing-smart-chat-transform-your-obsidian-notes-into-interactive-ai-powered-conversations/ https://wfhbrian.com/introducing-smart-chat-transform-your-o...

brucethemoose2 3y ago

If MeZO gets implemented, we are basically there: https://github.com/princeton-nlp/MeZO https://github.com/princeton-nlp/MeZO

moffkalast 3y ago
Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.
- nl 3y ago
  According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.
  - moffkalast 3y ago
    Wow if that's true then it's genuinely a complete gamechanger for LLMs as a whole. You probably mean more like 0.3s per token, not per example, but that's still more like 1 or two minutes per training case, not like a day for 4 cases like it is now.
  - valval 3y ago
    I think more importantly, what would the fine tuning routine look like? It's a non-trivial task to dump all of your personal data into any LLM architecture.
  - f_devd 3y ago
    MeZO assumes a smooth parameter space, so you probably won't be able to do it with INT4/8 quantization, probably needs fp8 or smoother.
  - isoprophlex 3y ago
    If you go through the drudgery of integrating with all the existing channels (mail, Teams, discord, slack, traditional social media, texts, ...), such rapid finetuning speeds could enable an always up to date personality construct, modeled on you. Which is my personal holy grail towards making myself unnecessary; it'd be amazing to be doing some light gardening while the bot handles my coworkers ;)
    - deleted 3y ago
      [deleted]
    - vgb2k18 3y ago
      > while the bot handles my coworkers Or it handles their bots ;)
  - gliptic 3y ago
    I cannot find any such numbers in the paper. What the paper says is that MeZO converges much slower than SGD, and each step needs two forward passes. "As a limitation, MeZO takes many steps in order to achieve strong performance."
  - sp332 3y ago
    It's the same memory footprint as inference. It's not that fast, and the paper mentions some optimizations that could still be done.
    - nl 3y ago
      Yes you are right. I completely misread that!

rvz 3y ago

> ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. > I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that! I think you are setting yourself up for disappointment in the future.

ulchar 3y ago
> The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. How exactly could they meaningfully do that? Genuine question. The issue with the OpenAI business model is that the collaboration within academia and open source circles is creating innovations that are on track to out-pace the closed source approach. Does OpenAI have the pockets to buy the open source collaborators and researchers? I'm truly cynical about many aspects of the tech industry but this is one of those fights that open source could win for the betterment of everybody.
- maxilevi 3y ago
  I agree with the spirit but saying that open source is on track to outpace OpenAI in innovation is just not true. Open source models are being compared to GPT3.5, none yet even get close to GPT4 quality and they finished that last year.
  - jart 3y ago
    We're basically surviving off the scraps companies like Facebook have been tossing off the table, like LLaMA. The fact that we're even allowed and able to use these things ourselves, at all, is a tremendous victory.
    - maxilevi 3y ago
      I agree

behnamoh 3y ago

I wonder if ClosedAI and other companies use the findings of the open source community in their products. For example, do they use QLORA to reduce the costs of training and inference? Do they quantize their models to serve non-subscribing consumers?

danielbln 3y ago
Not disagreeing with your points, but saying "ClosedAI" is about as clever as writing M$ for Microsoft back in the day, which is to say not very.
- loa_in_ 3y ago
  I'd say saying M$ makes it harder for M$ to find out I'm talking about them in them in the indexed web because it's more ambiguous, that's all I need to know.
  - coolspot 3y ago
    If we are talking about indexing, writing M$ is easier to find in an index because it is a such unique token. MS can mean many things (e.g. Miss), M$ is less ambiguous.
- rafark 3y ago
  I think it’s ironic that M$ made ClosedAI.
  - replygirl 3y ago
    Pedantic but that's not irony
    - rafark 3y ago
      Why do you think so? According to the dictionary, ironic could be something paradoxical or weird.

SparkyMcUnicorn 3y ago

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

SkyPuncher 3y ago
I think people want both. They want fine tuning for their style of communication and interaction. They want better rank and retrieval for rote information. In other words, it’s like having a spouse/partner. There are certain ways that we communicate that we simply know where the other person is at or what they actually mean.
- SparkyMcUnicorn 3y ago
  Unless you want machine-readable responses, or some other very specific need, the benefits of a fine-tuned model aren't really going to be that much better than a prompt that asks for the style you want along with an example or two. It also raises the barrier to entry quite a bit, since the majority of computers that can run the model aren't capable of training on it. Even if you're using OpenAI's models, gpt-3.5-turbo is going to be much better (cheaper, bigger context window, higher quality) than any of their models that can be fine-tuned. But if you're able to fine-tune a local model, then a combination of fine-tuning and embedding is probably going to give you better results than embedding alone.
dr_dshiv 3y ago
How does this work?
- deet 3y ago
  The parent is saying that "fine tuning", which has a specific meaning related to actually retraining the model itself (or layers at its surface) on a specialized set of data, is not what the GP is actually looking for. An alternative method is to index content in a database and then insert contextual hints into the LLM's prompt that give it extra information and detail with which to respond with an answer on-the-fly. That database can use semantic similarity (ie via a vector database), keyword search, or other ranking methods to decide what context to inject into the prompt. PrivateGPT is doing this method, reading files, extracting their content, splitting the documents into small-enough-to-fit-into-prompt bits, and then indexing into a database. Then, at query time, it inserts context into the LLM prompt The repo uses LangChain as boilerplate but it's pretty easily to do manually or with other frameworks. (PS if anyone wants this type of local LLM + document Q/A and agents, it's something I'm working on as supported product integrated into macOS, and using ggml; see profile)

ignoramous 3y ago

Can LLaMA be used for commerical purposes though (might limit external contributors)? I believe, FOSS alternatives like DataBricks Dolly / Together RedPajama / Eluether GPT NeoX (et al) is where the most progress is likely to be at.

samwillis 3y ago
Although llama.cpp started with the LLaMA model, it now supports many others.
okhuman 3y ago
This is a very good question that will be interesting how this develops. thanks for posting the alternatives list.
detrites 3y ago
May also be worth mentioning - UAE's Falcon, which apparently performs well (leads?). Falcon recently had its royalty-based commercial license modified to be fully open for free private and commercial use, via Apache 2.0: https://falconllm.tii.ae/ https://falconllm.tii.ae/
- mistercow 3y ago
  Hugging Face has a demo of the 40B Falcon instruct model: https://huggingface.co/blog/falcon#demo https://huggingface.co/blog/falcon#demo It’s pretty good as models of that size go, although it doesn’t take a lot of playing around with it to find that there’s still a good distance between it and ChatGPT 3.5. (I do recommend editing the instructions before playing with it though; telling a model this size that it “always tells the truth” just seems to make it overconfident and stubborn)
chaxor 3y ago
Why is commercial necessary to run local models?
- ignoramous 3y ago

shostack 3y ago

I've been trying to figure out what I might need to do in order to turn my Obsidian vault into a dataset to fine tune against. I'd invest a lot more into it now if I thought it would be a key to an AI learning about my the way it does in the movie Her.

58x14 3y ago
I've been working on this for awhile now and I'd love to chat. I'll send you an email.
- legendofbrando 3y ago
  I'm interested in this as well and have been exploring similarly. Would be super interesting to chat if you're up for it as well. Sending you an email to say hello.
mydjtl 3y ago
The holy grail. https://github.com/brianpetro/obsidian-smart-connections https://github.com/brianpetro/obsidian-smart-connections https://wfhbrian.com/introducing-smart-chat-transform-your-obsidian-notes-into-interactive-ai-powered-conversations/ https://wfhbrian.com/introducing-smart-chat-transform-your-o...