3 ms·

We need something like this on Linux, maybe powered by Vicuna. I’m not sure if the current batch of LLaMA variants is coherent enough to work as a digital assis

by valine 3y ago

We need something like this on Linux, maybe powered by Vicuna. I’m not sure if the current batch of LLaMA variants is coherent enough to work as a digital assistant, but my gut feeling is that a little fine tuning on tool use might be all thats needed.

Ajedi32 3y ago
The hard part I think will be integrating the LLM closely enough with the other programs on your machine that it's actually useful in that context, and not just a glorified chat window or a text interface to do things you can already do more easily with a KB+Mouse.
- valine 3y ago
  In a perfect world you give the LLM a python interpreter and it does the rest. Realistically, with the LLMs we have today, the right approach is probably to curate a set of APIs it's allowed to interact with. Some basic file system access and FFmpeg support would be extremely useful on it's own.
brucethemoose2 3y ago
Linux is fundamentally not monolithic like Windows, but maybe some DEs could expose hooks for LLMs to use. There is also the performance issue. Right now the task energy/memory usage of llama implementations is very high, and it takes some time to load into RAM and/or VRAM. It seems Microsoft is getting around this with cloud inference, and eats the hosting cost (for now). > little fine tuning on tool use might be all thats needed. Maybe I am interpreting this wrong, but LORA finetuning is extremely resource intense right now. There are practical alternatives though, like embedding databases people are just now setting up.
- trufas 3y ago
  Plugging a llm into dbus may take you surprisingly far.
- valine 3y ago
  Fine tuning is moderately intense. Going from base LLaMA to Vicuna 13B costs about $300.
  - brucethemoose2 3y ago
    The compute requirements are reasonable, but the memory requirements are extremely high, even with some upcoming breakthroughs like 4 bit bitsandbytes.
- lostmsu 3y ago
  How does accessibility work on Linux?

by valine 3y ago

Ajedi32 3y ago

The hard part I think will be integrating the LLM closely enough with the other programs on your machine that it's actually useful in that context, and not just a glorified chat window or a text interface to do things you can already do more easily with a KB+Mouse.

valine 3y ago
In a perfect world you give the LLM a python interpreter and it does the rest. Realistically, with the LLMs we have today, the right approach is probably to curate a set of APIs it's allowed to interact with. Some basic file system access and FFmpeg support would be extremely useful on it's own.

brucethemoose2 3y ago

Linux is fundamentally not monolithic like Windows, but maybe some DEs could expose hooks for LLMs to use. There is also the performance issue. Right now the task energy/memory usage of llama implementations is very high, and it takes some time to load into RAM and/or VRAM. It seems Microsoft is getting around this with cloud inference, and eats the hosting cost (for now). > little fine tuning on tool use might be all thats needed. Maybe I am interpreting this wrong, but LORA finetuning is extremely resource intense right now. There are practical alternatives though, like embedding databases people are just now setting up.

trufas 3y ago
Plugging a llm into dbus may take you surprisingly far.
valine 3y ago
Fine tuning is moderately intense. Going from base LLaMA to Vicuna 13B costs about $300.
- brucethemoose2 3y ago
  The compute requirements are reasonable, but the memory requirements are extremely high, even with some upcoming breakthroughs like 4 bit bitsandbytes.
lostmsu 3y ago
How does accessibility work on Linux?