4 ms·

I'm very excited about work being done in this area. In fact, I'm working on a product that does exactly this (runs in the background, local LLM, has access to

by dvt 3y ago

I'm very excited about work being done in this area. In fact, I'm working on a product that does exactly this (runs in the background, local LLM, has access to screen-space entities, can take certain actions). It feels pretty magical to use (here it's running on my 3090Ti; much slower but still serviceable on my M1 MBP): https://www.youtube.com/watch?v=JH1noETdQEA https://www.youtube.com/watch?v=JH1noETdQEA Currently using Mistral-7B-Instruct-v0.2, but working on a fine-tuning dataset which should make it work better with local applications interfaces (console, browser, email client, Slack, Discord, Word, Excel, etc.).

djohnston 3y ago
Your video is private!
- dvt 3y ago
  Whoops, meant to just make it unlisted, fixed.
lewismenelaws 3y ago
Cool! How hard would it be for it to take actions do you think?
- dvt 3y ago
  I've been working on this for the past month, and in its current state it runs some actions, but it's very context-dependent. For example, if I'm on a GitHub page and I say "clone this git repo," it works pretty flawlessly (git clones the repo in a scratch directory you set up in the settings), but if I'm reading a blog which references a git repo, it sometimes gets confused (may try to "clone" the blog URL for example), so I'm working through a few solutions, while trying to avoid multi-shot prompting. A lot of this involves pretty par for the course data cleaning. For example, you'd turn the user query into an embedding which you then compare to a few "contexts" (git workflows, research, creative work) to see which one matches it best, and then prune the raw screen data per that context, removing (or de-emphasizing) non-context-relevant information. So, in the above case, if I'm trying to clone a git repo, I don't care about non-git URLs (we can mask them, remove them, or whatever). Then, we feed the sanitized context to the LLM. Et Voilà !
richardkmichael 3y ago
This reminds me of https://www.openinterpreter.com/ https://www.openinterpreter.com/.
smravec 3y ago
Can you explain how do you plan to finetune it(how will you make that dataset , what data you will use...)? Always wondered how it is done
- dvt 3y ago
  Still a lot of work to be done before that point, but I'll write at least 1000 prompts (some synthetic, some hand-crafted) based on the "arrow notation" I posted here[1]. The fine-tuning itself is actually quite easy[2] as long as the data is properly formatted. After, I'll have to quantize the model (which I've never done before, but doesn't seem too hard). [1] https://www.reddit.com/r/LocalLLaMA/comments/1be5b4x/finetuning_mistralinstruct_for_function_calling/ https://www.reddit.com/r/LocalLLaMA/comments/1be5b4x/finetun... [2] https://exnrt.com/blog/ai/mistral-7b-fine-tuning/ https://exnrt.com/blog/ai/mistral-7b-fine-tuning/

by dvt 3y ago

djohnston 3y ago

Your video is private!

dvt 3y ago
Whoops, meant to just make it unlisted, fixed.

lewismenelaws 3y ago

Cool! How hard would it be for it to take actions do you think?

dvt 3y ago
I've been working on this for the past month, and in its current state it runs some actions, but it's very context-dependent. For example, if I'm on a GitHub page and I say "clone this git repo," it works pretty flawlessly (git clones the repo in a scratch directory you set up in the settings), but if I'm reading a blog which references a git repo, it sometimes gets confused (may try to "clone" the blog URL for example), so I'm working through a few solutions, while trying to avoid multi-shot prompting. A lot of this involves pretty par for the course data cleaning. For example, you'd turn the user query into an embedding which you then compare to a few "contexts" (git workflows, research, creative work) to see which one matches it best, and then prune the raw screen data per that context, removing (or de-emphasizing) non-context-relevant information. So, in the above case, if I'm trying to clone a git repo, I don't care about non-git URLs (we can mask them, remove them, or whatever). Then, we feed the sanitized context to the LLM. Et Voilà !

richardkmichael 3y ago

This reminds me of https://www.openinterpreter.com/ https://www.openinterpreter.com/.

smravec 3y ago

Can you explain how do you plan to finetune it(how will you make that dataset , what data you will use...)? Always wondered how it is done

dvt 3y ago
Still a lot of work to be done before that point, but I'll write at least 1000 prompts (some synthetic, some hand-crafted) based on the "arrow notation" I posted here[1]. The fine-tuning itself is actually quite easy[2] as long as the data is properly formatted. After, I'll have to quantize the model (which I've never done before, but doesn't seem too hard). [1] https://www.reddit.com/r/LocalLLaMA/comments/1be5b4x/finetuning_mistralinstruct_for_function_calling/ https://www.reddit.com/r/LocalLLaMA/comments/1be5b4x/finetun... [2] https://exnrt.com/blog/ai/mistral-7b-fine-tuning/ https://exnrt.com/blog/ai/mistral-7b-fine-tuning/