7 ms·
Ask HN: What AI assistants are already bundled for Linux?
Are there any AI assistants which come bundled with one of the popular distros (Debian-like, Arch, …)?
Or anything which is free (at least as in beer) and readily bundled in distro-specific installation packages?
- throwaway8481 3y agoCould they just package ollama and build a Python GTK interface around the CLI interface?
- gtirloni 3y agoNone. Microsoft has Copilot in preview mode in Windows and it's not very integrated apart from a chat window. I doubt GNOME/KDE will be able to dedicate enough resources to adding an assistant that is well integrated with the desktop environment any time soon. A search in Fedora yields a single GSoC project[0] limited in scope to NetworkManager and it's not clear if anyone actually is working on that. If the use case you're interested in is actually having the LLM doing things for you in SaaS applications, that wouldn't need deep integration but, considering Google is yet to deliver a Google Drive client for Linux, I wouldn't hold my breath waiting for a native Linux AI-assisted assistant. Your best option right now is to interface with the assistants through their web interface and hope they have plugins/extensions to interact with things you want. Other than that, some people have built prototypes running LLMs locally that talk to things like Home Assistant. But again, no deep desktop integration. 0 - https://docs.fedoraproject.org/en-US/mentored-projects/gsoc/2024/ideas/ https://docs.fedoraproject.org/en-US/mentored-projects/gsoc/...
- lukan 3y ago"I wouldn't hold my breath waiting for a native Linux AI-assisted assistant" A simple chat window and a automated script to install a existing small modell should be doable, but sounds not very exciting to me. But mid term, having a locally run LLM and integrated into the OS that scans my files and can summarize folders for me, would be nice. I have big folders with mixed stuff, AI would be nice to sort that. I do believe some people are working on something like this, but the bulk of it is not OS specific. And not OSS.
- MuffinFlavored 3y ago> But mid term, having a locally run LLM and integrated into the OS that scans my files and can summarize folders for me But don't most LLMs have about a max 32k token context?
- lukan 3y agoThe idea is to have a personal AI, that is trained with my content, my files. My emails, my pictures, projects, notes, etc. How that can be implemented in the best way, is not my expertize, but I believe is subject of heavy research.
- AlienRobot 3y ago>scans my files and can summarize folders for me Can you tell me what exactly you want it to do? You have a bunch of files in folders and you want the AI to tell you what exactly?
- HKH2 3y agoNot OP, but when searching for files, spelling something wrong, or using the wrong synonym is a big problem. We're just used to computers being inflexible.
- AlienRobot 3y agoI mean, I see, but do you need an entire AI for this? This sounds like a specific problem that would be best solved by a specific algorithm.
- MuffinFlavored 3y ago> I wouldn't hold my breath waiting for a native Linux AI-assisted assistant. On Mac when I press Command + Space, it brings up Spotlight search That can't easily be added to be the equivalent of some kind of LLM prompt on GNOME/KDE/XFCE? I don't quite know what you'd ask it/do with it that would be of much value? Seems like a quicker way/a wrapper around either asking an LLM questions via CLI or basically Electron wrapping HTML (like this https://github.com/lencx/ChatGPT https://github.com/lencx/ChatGPT)?
- guerrilla 3y ago> That can't easily be added to be the equivalent of some kind of LLM prompt on GNOME/KDE/XFCE? Both GNOME and KDE have that already. Shouldn't be too hard to implement what you're thinking if the APIs/services are available.
- MuffinFlavored 3y agoOr wrap to a shell script if the LLM has a local "CLI" interface, right?
- wing-_-nuts 3y agoGiven the fact that one can control damned near everything over command line in linux, and command line is a much more stable interface than a gui, I'd guess that there's a great deal more potential for assistants in linux than windows. The other day I wanted to figure out how to turn my dock red if I dropped the vpn in gnome. I found the file that controlled my wireguard gnome shell extension and with the help of gpt3.5 and some very rudimentary js knowledge (I'm a backend dev, don't hate me), I was able to add a js function to toggle the color on vpn up / down events. This didn't even take me an hour to do and I'd never even thought to try it before GPT. Sure, things are janky now, but the future potential of LLMs with linux and OSS is huge.
- ragebol 3y agoRecently came across https://www.warp.dev/ https://www.warp.dev/ on HN, which includes a AI part for your terminal. That's a paid feature running nonlocal, but it's a start I suppose.
- AlienRobot 3y agoUnrelated, but is there something like Bonzi Buddy for linux? Not the spyware part, just the friendly looking clippy-esque character that can tell you about your new e-mails, weather, or whatever? I kind of wish I had something like that.
- bqmjjx0kac 3y agoClosest I can think of is xeyes.
- adr1an 3y agopip install llm # among other, to run local or not. Yet, KDE or Gnome are yet to integrate or develop a nice API for/ to any of these.
- dartos 3y agoOpen interpreter?
- 29athrowaway 3y agoThere is one you can install via Flatpak. That will work in most distros. https://flathub.org/apps/io.github.Bavarder.Bavarder https://flathub.org/apps/io.github.Bavarder.Bavarder
- louismerlin 3y agoNot as far as I know, but llamafiles are almost easier to get running than native packages. https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile
- brucethemoose2 3y agoOne thing about LLMs is that they are 6GB+ (and much larger for "smart" ones) just sitting in the background. They suck power and produce heat like nothing else, and they are finicky, especially at smaller sizes. Running one as a background desktop assistant is whole different animal than calling a Microsoft API.
- eurekin 3y agoI imagine such assistant would be on top of support charts for "bogs down RAM" and "sucks up battery" for most users
- diggan 3y ago> One thing about LLMs is that they are 6GB+ (and much larger for "smart" ones) just sitting in the background. They suck power and produce heat like nothing else, and Huh? That's not at all true. It's only using processing power (CPU) while it actually generates text, otherwise it sits and wait. Although yes it occupies memory (RAM or VRAM) if you don't unload it, but you can configure it to startup when you need it, and shut down when you don't.
- zajio1am 3y agoIf one uses llama.cpp in CPU mode, then models are mmaped, so they do not really occupy memory when not used (they are just in reclaimable page cache).
- diggan 3y agoDo anyone actually use the CPU for anything besides testing? Last time I tried it, it was horribly slow compared to GPU that there wasn't really any point to use for me, besides getting access to more memory.
- ingenieroariel 3y agoI have. On a Mac Studio with NixOS based Asahi Linux and 128Gb of RAM, mixtral 8x7b uses 49GB of RAM. At the same time I load airflow tasks that deal with world wide datasets (using ~60GB on 16 parallel streams with the performance cores) format is parquet and also mmaped. Computer still has 8 efficiency cores and the whole GPU for visualizing the maps using lonboard / browsing / etc. The computer uses 8-10W when idle, ~100W when running jobs or actively using the LLM and around ~200W when really using the GPU. This makes it very efficient energy wise in my book compared to the beast of keeping a modern CPU and nvidia GPU on when idle. My electricity bill is unaffected.
- gessha 3y agoShameless plug: I’m writing one for Ubuntu Touch and I’m planning on adding a desktop version as well. My frontend side is very weak so it’s going to be very barebones but contributions are welcome once it’s stable: https://github.com/gessha/llmtest https://github.com/gessha/llmtest
- bossyTeacher 3y agoWhy? They take lots of space and lots of computing power. Linux has always been about lightweight and a bundle only containing essential things. You can always install one if you need it but as it stands right now LLMs are not useful enough to warrant their bundling in a distro. Just my 2 cents
- yjftsjthsd-h 3y ago> Linux has always been about lightweight and a bundle only containing essential things. I really don't think that's true. There have always been distros that are based on being tiny, of course, but I think most of the normal distros are concerned with hitting a happy medium of size and features. Otherwise I can't imagine why anything would be shipping GNOME or KDE over LXDE, or why libreoffice would be installed by default. So the question is more where LLMs are on cost/benefit... which granted, may not be there yet, but I could easily see it turning into a checkbox at install time - "this machine has 16+GB of RAM; add SomeLLM?"
- bossyTeacher 3y agoyou can't really compare the utility of an LLM against libreoffice or similar. There is no comparison. Libreoffice is something that you would definitely use unlike an LLM.
- p10_user 3y agoI use LLMs to double check/write python pandas code so I don't need to use Libreoffice/spreadsheet software.
- bossyTeacher 3y agoBut distros are not made for you only. They are made based on what most of their targeted userbase needs. And for most an LLM is not needed
- TheCoreh 3y agoNot bundled in a distro-specific way, but ollama fairly straightforward to install and use: https://github.com/ollama/ollama/blob/main/docs/linux.md https://github.com/ollama/ollama/blob/main/docs/linux.md
- javvee 3y agoLinux options (possibly not bundled) from another thread https://news.ycombinator.com/item?id=39533494 https://news.ycombinator.com/item?id=39533494
- yjftsjthsd-h 3y agoNixOS has packages for llama-cpp and ollama: https://search.nixos.org/packages?channel=23.11&from=0&size=50&sort=relevance&type=packages&query=llama https://search.nixos.org/packages?channel=23.11&from=0&size=... Edit: And Arch packages ollama officially - https://archlinux.org/packages/?sort=&q=llama&maintainer=&flagged= https://archlinux.org/packages/?sort=&q=llama&maintainer=&fl... - and a few things in the AUR - https://aur.archlinux.org/packages?O=0&K=llama https://aur.archlinux.org/packages?O=0&K=llama
- ghthor 3y agoNixOS just got tabbyml[1] which is built on llama-cpp. Working on systemsd services the weekend and updating latest tabbyml release which supports rocm in addition to cuda [1] https://github.com/TabbyML/tabby https://github.com/TabbyML/tabby [2] https://github.com/NixOS/nixpkgs/pull/291744 https://github.com/NixOS/nixpkgs/pull/291744
- ghthor 3y agoSystemd service PR posted to nixpkgs https://github.com/NixOS/nixpkgs/pull/292873 https://github.com/NixOS/nixpkgs/pull/292873
- mdev23 3y agoCheck out https://gitlab.com/mdev1974008/doda https://gitlab.com/mdev1974008/doda - Pure terminal UI. Its just python.
- onemoresoop 3y agoI'm wondering if there's any AI assistant that could run on a headless RPI? And is easy to install?
- dartharva 3y agoIt's a fair assumption ChromeOS will be getting Gemini soon.
- crimsoneer 3y agolm studio has a good linux version and gives you everything you'd ever need.
- throawayonthe 3y ago[dead]
- xk_id 3y agoWhen Linux comes bundled with AI I will lose all my faith in humanity and never use a computer again.
- quickthrower2 3y agoJan.ai has a deb package. https://jan.ai/ https://jan.ai/
- deleted 3y ago[deleted]
- bzmrgonz 3y agoWouldn't it be ironic if chatgpt assistant on linux gives linux end-user desktop dominance? lol... Microsoft would be like, "you fools, you've doomed us all"
- lucabs 3y ago[dead]
- harishpillay 3y agoPerhaps this: https://github.com/yusufcanb/tlm https://github.com/yusufcanb/tlm? it is not distro bundled (yet), but I have it running on my Fedora Linux 39 running on a NUC with 16GB of RAM. Performance is good enough for me.