3 ms·

Local models have taken a mind boggling leap over the past months so i'm sure we'll be able to add layers soon by ourselves even on a laptop? Seriously this is

Local models have taken a mind boggling leap over the past months so i'm sure we'll be able to add layers soon by ourselves even on a laptop? Seriously this is not far from Chat GPT 3.5 in only 6.7GB's and runs on a Macbook Air: https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF But yeah current context windows are limiting.

TrevorJ 3y ago
I think with RAG it's pretty reasonable. Put your corpus in pinecone or some other vector store and relevant sections get injected along with your prompt which lessens the burden on context window.
rodrigodlu 3y ago
I was testing this one days ago. It seems fine to use as a base for extra finetuning, but failed hard questions that chatgpt nailed. One example was trying to use as a assistant to beat long games, without immediate rewards. I was trying to log and simultaneously get feedback playing Stardew Valley. gpt-3.5-turbo-1106 basically went along with me and my daughter in a coop session giving nice suggestions, sometimes with huge gaps, but easy enough to ask more about after giving more context. Mistral 7b and 13B was basically mixing up stardew valley with WoW and Genshin Impact, even giving a lot of context about the day I was, what the npcs answered, or things that I know on how to solve a certain quest. It straight made up non existing towns (stardew valley only has one) etc, etc. I was running the model on a separate gaming notebook, with nvidia, while playing the game on the one I'm using now.
- MyFirstSass 3y ago
  True, and makes sense that the logic is closing in but the breadth of the data is too narrow in 7GB's to ask questions about niche topics. Mistral hasn't released their own official 13B/30B's yet, but i'm really looking forward to what they can do. What is crazy is that Ultrafastbert, Speculative, Jacobi, or lookahead decoding could potentially speed up by up to 80x depending on size which could make GPT-4 like models feasible on entry level macs / Phones if similar wizardry is done memory wise. ..Yes im very optimistic after the insane progress over the last months with models like Mistral, Deepseek etc.
nerpderp82 3y ago
With RAG and fine tuning (which is cheap), you can fine tune a model on a daily basis so that one isn't trying to stuff everything in the context window.

TrevorJ 3y ago

I think with RAG it's pretty reasonable. Put your corpus in pinecone or some other vector store and relevant sections get injected along with your prompt which lessens the burden on context window.

rodrigodlu 3y ago

I was testing this one days ago. It seems fine to use as a base for extra finetuning, but failed hard questions that chatgpt nailed. One example was trying to use as a assistant to beat long games, without immediate rewards. I was trying to log and simultaneously get feedback playing Stardew Valley. gpt-3.5-turbo-1106 basically went along with me and my daughter in a coop session giving nice suggestions, sometimes with huge gaps, but easy enough to ask more about after giving more context. Mistral 7b and 13B was basically mixing up stardew valley with WoW and Genshin Impact, even giving a lot of context about the day I was, what the npcs answered, or things that I know on how to solve a certain quest. It straight made up non existing towns (stardew valley only has one) etc, etc. I was running the model on a separate gaming notebook, with nvidia, while playing the game on the one I'm using now.

MyFirstSass 3y ago
True, and makes sense that the logic is closing in but the breadth of the data is too narrow in 7GB's to ask questions about niche topics. Mistral hasn't released their own official 13B/30B's yet, but i'm really looking forward to what they can do. What is crazy is that Ultrafastbert, Speculative, Jacobi, or lookahead decoding could potentially speed up by up to 80x depending on size which could make GPT-4 like models feasible on entry level macs / Phones if similar wizardry is done memory wise. ..Yes im very optimistic after the insane progress over the last months with models like Mistral, Deepseek etc.

nerpderp82 3y ago

With RAG and fine tuning (which is cheap), you can fine tune a model on a daily basis so that one isn't trying to stuff everything in the context window.