3 ms·
> I considered loading another wiki of something different, but it's less convincing as a proof of concept because it needs to allow addition, change and exclus
by brucethemoose2 3y ago
> I considered loading another wiki of something different, but it's less convincing as a proof of concept because it needs to allow addition, change and exclusion of small bits of data without the bot going offline for retraining 3 times a day.
I'm not sure how privateGPT works, but any sane implementation is going to store the data in a vector database, not constantly train it into the llm itself. Updating it is easy, fast, and can probably be done without going offline.
...But yeah, the key is somehow replicating your wiki closely enough for a demo. With access to the wiki, maybe you could copy the structure (but not any data) and populate it with dummy data from an LLM? If they are good at anything, its producing something convincing and totally made up.
EDIT: Also, if the backend is llama.cpp, recent advances will make it very easy to run, but questionable for commercial use depending on the model it uses. LLaMA itself is basically illegal to use, even if Facebook doesn't seem to care.
- ouraf 3y agoMy idea is that the end user is an employee, not a customer of the company. Since they have a general understanding and the questions won't be as broad, i hope one of the open models with a less strict license is enough for the task. Stability AI released a model with creative commons CC BY license (allows commercial use with attribuition, although Creative Commons is a weird license for a gigantic database) and there are some "LLaMA" clones (open llama, redpanda, vicuna, etc.) that i need to look deeper into to understand how it was done and how it didn't inherit the original's license
- brucethemoose2 3y agoAll of the LLaMA variants like Vicuna are finetunes, hence are tainted by base LLaMA. Stability AI's model is not good, and generally the LLaMA clones are not ready yet. There is a really cool model that actually outperforms LLaMA: Falcon 7B and Falcon 40B. But it is extremely new, and not even supported by llama.cpp (the backend many projects use) yet, though it is on the June roadmap. You would have to run the 4 bit variant on an Nvidia GPU, and you might have to finetune it for certain llm frameworks.
- throwawayadvsec 3y agowhy would it be supported by llama.cpp when it have nothing to do with it?
- brucethemoose2 3y agoshrug it is in the roadmap: https://github.com/ggerganov/llama.cpp/discussions/1729 https://github.com/ggerganov/llama.cpp/discussions/1729 The ggml model format is growing to support more than just llama, I think.