19 ms·

I want everything local – Building my offline AI workspace

1143 points by mkagenius 1y ago

shaky 1y ago
This is something that I think about quite a bit and am grateful for this write-up. The amount of friction to get privacy today is astounding.
- sneak 1y ago
  This writeup has nothing of the sort and is not helpful toward that goal.
  - frank_nitti 1y ago
    I'd assume they are referring to being able to run your own workloads in a home-built system, rather then surrendering that ownership to the tech giants alone
    - Imustaskforhelp 1y ago
      Also you get a sort of complete privacy that the data never leaves your home too whereas at best you would have to trust the AI cloud providers that they are not training or storing that data. Its just more freedom and privacy in that matter.
      - doctorpangloss 1y ago
        The entire stack involved sends so much telemetry.
        frank_nitti 1y ago
        This, in particular, is a big motivator and rewarding factor in getting local setup and working. Turning off the internet and seeing everything run end to end is a joy
        doctorpangloss 1y ago
        NVIDIA drivers send detailed telemetry. Windows and macOS send detailed telemetry. You have to install the pip packages and the models, which all come from websites, which collect detailed telemetry. You don’t think Microsoft gathers detailed telemetry on all your interactions with GitHub? The local setup doesn’t really help with that.
        frank_nitti 1y ago
        We might be talking about two different things. Yes, under normal circumstances the setup steps involve software that defaults to using telemetry -- though I'd be surprised if it's not possible anymore to achieve those in an air-gapped env using e.g. offline installers, zipped repos and wheel files, etc. My comment was referring to runtime workloads having no telemetry (because I unplugged the internet)
      - wkat4242 1y ago
        > whereas at best you would have to trust the AI cloud providers that they are not training or storing that data. Yeah, about that. They even illegally torrented entire databases, hide their crawlers. Crawl entire newspaper archives without permission. They didn't respect the rights of big media companies. But they're going to respect the little guy's of course because it says to in the T&Cs. Uh-huh. Also, openai already admitted that they do store "deleted" content and temporary chats.
        Imustaskforhelp 1y ago
        I agree but I was just (repeating?) some argument that I heard that if the companies would actually not follow on their premise that they are actually safe if they said so (think amazon bedrock tos policy which says such) Then it will cause an insane backlash and nobody would use the product. So it is in their interest to not train/record. But yes I also agree with you. They are already torrenting :/ So pretty sure if they can do illegal stuff scott free, they might do this too idk, And yeah this was why I was actually saying that local matters more tbh. You just get rid of such headache.
        wkat4242 1y ago
        > Then it will cause an insane backlash and nobody would use the product. So it is in their interest to not train/record. I don't think there would be that much backlash. People are getting hooked on it and many don't actually care about privacy. We know about Google, meta and people still use them. Not a big dent in openai usage either since their revelations. But I understand your point!
- Aurornis 1y ago
  > The amount of friction to get privacy today is astounding I don't understand this. It's easy to get a local LLM running with a couple commands in the terminal. There are multiple local LLM runners to choose from. This blog post introduces some additional tools for sandboxed code execution and browser automation, but you don't need those to get started with local LLMs. There are multiple local options. This one is easy to start with: https://ollama.com/ https://ollama.com/
  - apitman 1y ago
    > It's easy to get a local LLM running Easy for what percentage of people?
noelwelsh 1y ago
It's the hardware more than the software that is the limiting factor at the moment, no? Hardware to run a good LLM locally starts around $2000 (e.g. Strix Halo / AI Max 395) I think a few Strix Halo iterations will make it considerably easier.
- colecut 1y ago
  This is rapidly improving https://simonwillison.net/2025/Jul/29/space-invaders/ https://simonwillison.net/2025/Jul/29/space-invaders/
  - Imustaskforhelp 1y ago
    I hope it improves at such a steady rate! Please lets just hope that there is still room for improvement to packing even more improvements in such LLMS which can help the home labbing community in general.
- ramesh31 1y ago
  >Hardware to run a good LLM locally starts around $2000 (e.g. Strix Halo / AI Max 395) I think a few Strix Halo iterations will make it considerably easier. And "good" is still questionable. The thing that makes this stuff useful is when it works instantly like magic. Once you find yourself fiddling around with subpar results at slower speeds, essentially all of the value is gone. Local models have come a long way but there is still nothing even close to Claude levels when it comes to coding. I just tried taking the latest Qwen and GLM models for a spin through OpenRouter with Cline recently and they feel roughly on par with Claude 3.0. Benchmarks are one thing, but reality is a completely different story.
ahmedbaracat 1y ago
Thanks for sharing. Note that the GitHub at the end of the article is not working…
- mkagenius 1y ago
  Thanks for the heads up. It's fixed now - Coderunner-UI: https://github.com/instavm/coderunner-ui https://github.com/instavm/coderunner-ui Coderunner: https://github.com/instavm/coderunner https://github.com/instavm/coderunner
navbaker 1y ago
Open Web UI is a great alternative for a chat interface. You can point to an OpenAI API like vLLM or use the native Ollama integration and it has cool features like being able to say something like “generate code for an HTML and JavaScript pong game” and have it display the running code inline with the chat for testing
dmezzetti 1y ago
I built TxtAI with this philosophy in mind: https://github.com/neuml/txtai https://github.com/neuml/txtai
pyman 1y ago
Mr Stallman? Richard, is that you?
tcdent 1y ago
I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardware unjustifiable. Coupled with the dramatically inferior performance of the weights you would be running in a local environment, it's just not worth it. I expect this will change in the future, and am excited to invest in a local inference stack when the weights become available. Until then, you're idling a relatively expensive, rapidly depreciating asset.
- braooo 1y ago
  Running LLMs at home is a repeat of the mess we make with "run a K8s cluster at home" thinking You're not OpenAI or Google. Just use pytorch, opencv, etc to build the small models you need. You don't need Docker even! You can share over a simple code based HTTP router app and pre-shared certs with friends. You're recreating the patterns required to manage a massive data center in 2-3 computers in your closet. That's insane.
  - frank_nitti 1y ago
    For me, this is essential. On priciple, I won't pay money to be a software engineer. I never paid for cloud infrastructure out of pocket, but still became the go-to person and achieved lead architecture roles for cloud systems, because learning the FOSS/local tooling "the hard way" put me in a better position to understand what exactly my corporate employers can leverage with the big cash they pay the CSPs. The same is shaping up in this space. Learning the nuts and bolts of wiring systems together locally with whatever Gen AI workloads it can support, and tinkering with parts of the process, is the only thing that can actually keep me interested and able to excel on this front relative to my peers who just fork out their own money to the fat cats that own billions worth of compute. I'll continue to support efforts to keep us on the track of engineers still understanding and able to 'own' their technology from the ground up, if only at local tinkering scale
sneak 1y ago
Halfway through he gives up and uses remote models. The basic premise here is false. Also, the term “remote code execution” in the beginning is misused. Ironically, remote code execution refers to execution of code locally - by a remote attacker. Claude Code does in fact have that, but I’m not sure if that’s what they’re referring to.
- thepoet 1y ago
  The blog says more about keeping the user data private. The remote models in the context are operating blind. I am not sure why you are nitpicking, almost nobody reading the blog would take remote code execution in that context.
  - vunderba 1y ago
    The MCP aspect (for code/tool execution) is completely orthogonal to the issue of data privacy. If you put a remote LLM in the chain than it is 100% going to inadvertently send user data up to them at some point. e.g. if I attach a PDF to my context that contains private data, it WILL be sent to the LLM. I have no idea what "operating blind" means in this context. Connecting to a remote LLM means your outgoing requests are tied to a specific authenticated API key.
mark_l_watson 1y ago
That is fairly cool. I was talking about this on X yesterday: another angle however, I use a local web scraper and search engine via meilisearch the main tech web sites I am interested in. For my personal research I use three web search APIs, but there is some latency. Having a big chuck of the web that I am interested in available locally with close to zero latency is nice when running local models, my own MCP services that might need web search, etc.
luke14free 1y ago
you might want to check out what we built -> https://inference.sh https://inference.sh supports most major open source/weight models from wan 2.2 video, qwen image, flux, most llms, hunyan 3d etc.. works in a containerized way locally by allowing you to bring your own gpu as an engine (fully free) or allows you to rent remote gpu/pool from a common cloud in case you want to run more complex models. for each model we tried to add quantized/ggufs versions to even wan2.2/qwen image/gemma become possible to execute with as little as 8gb vram gpus. mcp support coming soon in our chat interface so it can access other apps from the ecosystem.
- pwn0 1y ago
  The website is very confusing. Where can I download the application? Is there a GitHub repository?
rshemet 1y ago
if you ever end up trying to take this in the mobile direction, consider running on-device AI with Cactus – https://cactuscompute.com/ https://cactuscompute.com/ Blazing-fast, cross-platform, and supports nearly all recent OS models.
- b0ner_t0ner 1y ago
  Is this your site? It's missing a <title> tag.
xt00 1y ago
Yea in an ideal world there would be a legal construct around AI agents in the cloud doing something on your behalf that could not be blocked by various stakeholders deciding they don't like the thing you are doing even if totally legal. Things that would be considered fair use, or maybe annoying to certain companies should not be easy for companies to just wholesale block by leveraging business relationships. Barring that, then yea, a local AI setup is the way to go.
sabareesh 1y ago
Here is my rig, running GLM 4.5 Air. Very impressed by this model https://sabareesh.com/posts/llm-rig/ https://sabareesh.com/posts/llm-rig/ https://huggingface.co/zai-org/GLM-4.5 https://huggingface.co/zai-org/GLM-4.5
mkummer 1y ago
Super cool and well thought out! I'm working on something similar focused on being able to easily jump between the two (cloud and fully local) using a Bring Your Own [API] Key model – all data/config/settings/prompts are fully stored locally and provider API calls are routed directly (never pass through our servers). Currently using mlc-llm for models & inference fully local in the browser (Qwen3-1.7b has been working great) [1] https://hypersonic.chat/ https://hypersonic.chat/
Imustaskforhelp 1y ago
I think I still prefer local but I feel like that's because that most AI inference is kinda slow or comparable to local. But I recently tried out cerebras or (I have heard about groq too) and honestly when you try things at 1000 tk/s or similar, your mental model really shifts and becomes quite impatient. Cerebras does say that they don't log your data or anything in general and you would have to trust me to say that I am not sponsored by them (Wish I was tho) Its just that they are kinda nice. But I still hope that we can someday actually have some meaningful improvements in speed too. Diffusion models seem to be really fast in architecture.
- vgb2k18 1y ago
  > Cerebras does say that they don't log your data or anything in general Unil a judge says they must log everything, indefinitely
retrocog 1y ago
Its all about context and purpose, isn't it? For certain lightweight uses cases, especially those concerning sensitive user data, a local implementation may make a lot of sense.
- accrual 1y ago
  My thoughts exactly. The recent GPT-OSS 20B parameter model was a nice upgrade, it really feels like having a local mini ChatGPT.
kaindume 1y ago
Self hosted and offline AI systems would be great for privacy but the hardware and electricity cost are much too high for most users. I am hoping for a P2P decentralized solution that runs on distributed hardware not controlled by a single corporation.
- user3939382 1y ago
  I’d settle for homomorphic encryption but that’s a long way off if ever
woadwarrior01 1y ago
> LLMs: Ollama for local models (also private models for now) Incidentally, I decided to try to Ollama macOS app yesterday, and the first thing it tries to do upon launch is try to connect to some google domain. Not very private. https://imgur.com/a/7wVHnBA https://imgur.com/a/7wVHnBA
- Aurornis 1y ago
  Automatic update checks https://github.com/ollama/ollama/blob/main/docs/faq.md https://github.com/ollama/ollama/blob/main/docs/faq.md
- abtinf 1y ago
  Yep, and I’ve noticed the same thing with in vscode with both the cline plugin and the copilot plugin. I configure them both to use local ollama, block their outbound connections via little snitch, and they just flat out don’t work without the ability to phone home or posthog. Super disappointing that Cline tries to do so much outbound comms, even after turning off telemetry in the settings.
- eric-burel 1y ago
  But can be audited which I'd buy everyday. It's probably not to hard to find network calls in a codebase if this task must be automated on update.
  - woadwarrior01 1y ago
    This is the macOS GUI, which IIUC is closed source.
adsharma 1y ago
https://github.com/adsharma/ask-me-anything https://github.com/adsharma/ask-me-anything Supports MLX on Apple silicon. Electron app. There is a CI to build downloadable binaries. Looking to make a v0.1 release.
andylizf 1y ago
This is fantastic work. The focus on a local, sandboxed execution layer is a huge piece of the puzzle for a private AI workspace. The `coderunner` tool looks incredibly useful. A complementary challenge is the knowledge layer: making the AI aware of your personal data (emails, notes, files) via RAG. As soon as you try this on a large scale, storage becomes a massive bottleneck. A vector database for years of emails can easily exceed 50GB. (Full disclosure: I'm part of the team at Berkeley that tackled this). We built LEANN, a vector index that cuts storage by ~97% by not storing the embeddings at all. It makes indexing your entire digital life locally actually feasible. Combining a local execution engine like this with a hyper-efficient knowledge index like LEANN feels like the real path to a true "local Jarvis." Code: https://github.com/yichuan-w/LEANN https://github.com/yichuan-w/LEANN Paper: https://arxiv.org/abs/2405.08051 https://arxiv.org/abs/2405.08051
- sebmellen 1y ago
  I know next to nothing about embeddings. Are there projects that implement this same “pruned graph” approach for cloud embeddings?
  - NJL3000 1y ago
    It’s in the works… been meaning to do a show HN moment to see if it flies or I Fall on my face..
- doctoboggan 1y ago
  > A vector database for years of emails can easily exceed 50GB. In 2025 I would consider this a relatively meager requirement.
  - andylizf 1y ago
    Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that point, it's no longer a "meager" requirement. It becomes a heavy "tax" on your primary drive, which is often non-upgradable on modern laptops. The goal for practical local AI shouldn't just be that it's possible, but that it's also lightweight and sustainable. That's the problem we focused on: making a comprehensive local knowledge base feasible without forcing users to dedicate half their SSD to a single index.
com2kid 1y ago
> Even with help from the "world's best" LLMs, things didn't go quite as smoothly as we had expected. They hallucinated steps, missed platform-specific quirks, and often left us worse off. This shows how little native app training data is even available. People rarely write blog posts about designing native apps, long winded medium tutorials don't exist, heck even the number of open source projects for native desktop apps is a small percentage compared to mobile and web apps. Historically Microsoft paid some of the best technical writers in the world to write amazing books on how to code for Windows (see: Charles Petzold), but now days that entire industry is almost dead. These types of holes in training data are going to be a larger and larger problem. Although this is just representative of software engineering in general - few people want to write native desktop apps because it is a career dead end. Back in the 90s knowing how to write Windows desktop apps was great, it was pretty much a promised middle class lifestyle with a pretty large barrier to entry (C/C++ programming was hard, the Windows APIs were not easy to learn, even though MS dumped tons of money into training programs), but things have changed a lot. Outside of the OS vendors themselves (Microsoft, Apple) and a few legacy app teams (Adobe, Autodesk, etc), very few jobs exist for writing desktop apps.
- thorncorona 1y ago
  I mean outside of HPC why would you when the browser is the world’s most ubiquitous VM?
  - esseph 1y ago
    Because the browser is gross and you can reclaim lot of performance and security when you don't need to use it.
    - esseph 1y ago
      I mean, why aren't the apps on your phone all just webapps, right? (Also, eww)
      - jakelazaroff 1y ago
gen2brain 1y ago
People are talking about AI everywhere, but where can we find documentation, examples, and proof of how it works? It all ends with chat. Which chat is better and cheaper? This local story is just using some publicly available model, but downloaded? When is this going to stop?
bling1 1y ago
On a similar vibe, we developed app.czero.cc to run an LLM inside your chrome browser on your machine hardware without installation (you do have to download the models). Hard to run big models, but it doesnt get more local than that without having to install anything.
btbuildem 1y ago
I didn't see any mention of the hardware OP is planning to run this on -- any hints?
vunderba 1y ago
Infra notwithstanding - I'd be interested in hearing how much success they actually had using a locally hosted MCP-capable LLM (and which ones in particular) because the E2E tests in the article seem to be against remote models like Claude.
ruler88 1y ago
At least you won't be needing a heater for the winter
mikeyanderson1 1y ago
We have this in closed alpha right now getting ready to roll out to our most active builders in the coming weeks at ThinkAgents.ai
techlatest_net 1y ago
[dead]
LastTrain 1y ago
I get it but I can’t get over the irony that you are using a tool that only works precisely because people don’t do this.
eric-burel 1y ago
An llm on your computer is a fun hobby, an llm in your SME for 10 people is a business idea. There are not enough resources on this topic at all and the need is growing extremely fast. Local LLMs are needed for many use cases and business where cloud is not possible.
deleted 1y ago
[deleted]
nenadg 1y ago
did this by running models in chroot
josephwegner 1y ago
See
jumploops 1y ago
I'm a little confused about your product branding vs. blog post? From the product homepage, I imagine you're running VMs in the cloud (a la Firecracker). From the blog post though, it looks like you're running Apple-specific VMs for local execution? As someone who's built the former, I'd love the latter for use with the new gpt-oss releases :)
- mkagenius 1y ago
  You are right, the product is almost the same as you described, intended for customers running LLM-generated code in their workflow. This was something close to our hearts, so we thought of building it for our local use and releasing it for like-minded individuals.
b0ner_t0ner 1y ago
To OP, your link for https://github.com/assistant-ui/assistant-ui https://github.com/assistant-ui/assistant-ui does not work.
eyespasm 1y ago
To be honest, I just want to make porn. My own porn, the way I want it. That’s what I’m waiting for. Why the heck do I need to scroll through pages of boring, vanilla, pedestrian porn on Pornhub or RedGIfs or XNXX when I can create exactly what I want? That’ll be a huge killer app when I can do it locally and in the privacy of my own home.
123sereusername 1y ago
[dead]
jarym 1y ago
Playing with local LLMs is indeed fun. I use Kasm workspaces[0] to run a desktop session with ollama running on the host. Gives me the isolation and lets me experiment with all manner of crazy things (I tried to make a computer-use AI but it wasn't very good) [0] https://kasmweb.com/ https://kasmweb.com/
nikolayasdf123 1y ago
local is so hot right now
nikolayasdf123 1y ago
local/edge is the most under-valued space at the moment. incredible computing power that dwarfs datacenters, zero latency, zero cost, private, distributed and resilient
- oblio 1y ago
  I guess you imagine a world like Skype supernodes (Skype gave that up more than a decade ago) or Tor nodes (Tor is used by a tiny fraction of internet users). Not saying it can't be done, but the effort is humongous.
  - nikolayasdf123 1y ago
    no, I mean I saw multiple companies at this point with their entier K8S cluster... is smaller than single new macbook pro :/ now, if you have 100,000 users with latest iPhone, say you use 10GB RAM in each, using A16 chip with 1.9 TFLOPS, each with 5G connection this is 1 Peta-Byte RAM + 0.25 Peta-FLOPs GPU + 4 TB / second bandwidth at zero cost (no-upfront, no-maintenance, users pay for, upgrade, and maintain their phones working, pay for internet, charging with electricity, cooling? - thanks!) ... it goes even wilder if you use macbooks ... and if you consider say mid-size town in China with population of 15 million, you go Exa-scale and consider that for now iPhones are just sitting idle. for now.
    - oblio 1y ago
      Things don't work like that. First of all iPhones have more like 6-8GB of RAM, 1-2 of which are already taken up by the system and system apps. Add some resident apps and maybe 1-2GB are already taken. Then of course during peak times, which are predictable but not guaranteed, 5-10% is maybe available. So out of your 10GB estimated per device, you actually average maybe 3GB. Similar story for the CPU and GPU. Then, availability: dead battery, no cell reception, airplane mode, etc, etc. And on top of that, in the context of battery charge and long term wear and tear, you're assuming people will just let you run Bitcoin mining nodes on them. You need a really solid incentive for people to loan you end user computing power for legitimate reasons.
willtemperley 1y ago
How would this compare to using Apple Foundation Models which execute on device? https://developer.apple.com/documentation/FoundationModels https://developer.apple.com/documentation/FoundationModels
dcreater 1y ago
Then using ollama is not the right choice. https://news.ycombinator.com/item?id=44814607 https://news.ycombinator.com/item?id=44814607
jychang 1y ago
Any way to install this via just a container? Similar to a `docker compose up -d` that a lot of projects offer. Just download the docker-compose.yml file into a folder, run the command, and you're running. If you want to delete everything, just `docker compose down` and delete the folder, and the container and everything is gone. Anything similar to that? I don't want to run a random install.sh on my machine that does god knows what.
- patmorgan23 1y ago
  I believe a flatpack or appimage is what you're looking for.
- mkagenius 1y ago
  There are similar commands for coderunner (not the UI frontend): container image pull instavm/coderunner container run --name coderunner --detach instavm/coderunner (for more comprehensive commands, see from line 51 https://github.com/instavm/coderunner/blob/main/install.sh#L51 https://github.com/instavm/coderunner/blob/main/install.sh#L...) Frontend (coderunner-ui) is not inside a docker as of now.
- cheschire 1y ago
  But you would pump your secrets into a docker AI?
  - oblio 1y ago
    Does Docker do that or are you speculating? Also - podman?
    - cheschire 1y ago
      I wasn't implying docker itself was the issue. The previous commenter said that they didn't want to run a shell script that does "god knows what". The implication being that they would not trust the writer of the shell script. They wanted a docker container that would setup this offline AI workspace for them, presumably so they could interact with the AI and feed "secrets" or otherwise private data into it. Obviously there are other use cases for an offline AI, but folks tend to let their guard down when they think something is offline-only, and they may not be as careful with .env values, or personal information, as they would with a SaaS frontier model. So I was pointing out that the contents of the docker container would be also doing "god knows what" with their data. Sure they would get the offline user experience but then what happens? More shell scripts? Background data calls? etc. And of course it depends on how they configure their docker container, but if they aren't willing to review an install shell script, they probably aren't looking to do any level of effort for configuring Docker. Hopefully that clarifies it.
PeterStuer 1y ago
The link to assistent ui in the article 404's. It should be https://github.com/assistant-ui/assistant-ui https://github.com/assistant-ui/assistant-ui
- mkagenius 1y ago
  My bad, I typed `-ai` instead of `-ui`. Its fixed now.
EagnaIonat 1y ago
You can get good models that run fine on M1 32GB laptops just using Ollama App. Or if you want numerous features on top of your local LLMS then Open WebUI would be my choice. https://docs.openwebui.com https://docs.openwebui.com
anupshinde 1y ago
What is the Apple hardware being used here? I see Apple Silicon but not the configuration.. what did I miss
- thebruce87m 1y ago
  Was looking for that too. Need to know whether I already own the hardware or can’t afford it.
jeffWrld 1y ago
[dead]
_the_inflator 1y ago
The socialist EU allows only AI that serves the governance purpose. The EU has rightfully acknowledged that freedom of AI is essentially freedom of speech. Hacking officially stopped being non-political in EU. https://artificialintelligenceact.eu/ https://artificialintelligenceact.eu/ Enjoy understanding this here: https://artificialintelligenceact.eu/article/3/ https://artificialintelligenceact.eu/article/3/ Measures of Innovations rank at... Article 57! https://artificialintelligenceact.eu/ai-act-explorer/ https://artificialintelligenceact.eu/ai-act-explorer/ I bet that soon, anyone involved with sophisticated AI systems will be system-checked and require a license. God bless you all out there and have phun!
- oblio 1y ago
  The EU isn't socialist, what are you going on about? And AI - if true AI - can be "end of times" type tech, you think it won't be regulated? This is not hackers playing with breadboards in the 60s, it's Project Manhattan in the 40s.
yichuan 1y ago
That's my vision, hope it can help. I think that if we combine all our personal data and organize it effectively, we can be 10 times more efficient. Long-term AI memory, all you speak and see will secretly be loaded to your own personal AI, and that can solve many difficulties, I think. https://x.com/YichuanM/status/1953886817906045211 https://x.com/YichuanM/status/1953886817906045211
shekhargulati 1y ago
I tried to port it to Docker and wrote a blog here https://shekhargulati.com/2025/08/09/making-coderunner-ui-work-with-docker-using-claude-code/ https://shekhargulati.com/2025/08/09/making-coderunner-ui-wo.... I used Claude Code to do the port. We used Datalayer Jupyter MCP Server instead of coderunner which uses Apple containers.
- mockingloris 1y ago
  At least, you are honest about augmenting the porting process. It's amazing what one can accomplish when they realize that with proper time, planning and a good grounding on building code/systems, that a lot more is possible. The takeaway for me is that because these tools are fast doesn't mean the task also needs to move as fast. At least till AGI, a sound human reasoning before hitting enter goes a long way. Thanks for sharing
k__ 1y ago
Half-OT: Anything useful that runs reasonably fast on a regular Intel CPU/GPU?
- oblio 1y ago
  I did a bunch of research and basically no. Unless you can work with sending a request in the evening and getting the result in the morning. And you'd need a lot of regular RAM because otherwise you start swapping at which point I think response times end up in days. This tech is in the Wild West days, for it to be usable by the average person on consumer hardware, I think we'll need to be in 2030+.
- ethan_smith 1y ago
  For Intel CPUs, Phi-2 (2.7B) and TinyLlama (1.1B) run reasonably well using llama.cpp with 4-bit quantization. GGUF models with INT4 quantization typically need ~2GB RAM per billion parameters, so even older machines can handle smaller models.
  - akawry 1y ago
    Take a look at ik_llama.cpp: https://github.com/ikawrakow/ik_llama.cpp https://github.com/ikawrakow/ik_llama.cpp CPU performance is much better than mainline llama, as well as having more quantization types available
Woodi 1y ago
So there are models to download but: a) on what data that things was trained ? b) any reproducible builds projects ? ;)
hollowonepl 1y ago
Yep, that is something I do also actively experiment with in home projects. Local NAS (Synology) with 28TB of RAIDed storage, local containers and VMs on it and local gitea and other devops and productivity tools. All that talks to my mac which runs editing, compiling, etc and lmstudio with local agent. Not best always with AI, I lack enough RAM but close to imagine how I will work in the future, end-to-end
zakki 1y ago
Curious with the hardware used in this article.
t_mann 1y ago
Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great for the purpose, but you don't really get them with enough RAM or at prices that don't really qualify as consumer-grade anymore. I've seen people create home clusters (eg using Exo [0]), but I wouldn't really call it practical (single digit token/sec for large models, and the price isn't exactly accessible either). Framework (the modular laptop company) has announced a desktop that can be configured up to 128GB unified RAM, but it's still going to come in at around 2-2.5k depending on your config. [0] https://github.com/exo-explore/exo https://github.com/exo-explore/exo
- zozbot234 1y ago
  What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project. Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.
  - graemep 1y ago
    You still need a lot of RAM though right? so its not going to be that cheap? What sort of specs do you need?
  - flanger 1y ago
    The founders of Exo ghosted the dev community and went closed-source. Nobody has heard from them. I wish people would stop recommending Exo (a tribute to their marketing) and check out GPUStack instead. Overall another rug pull by the devs as soon as they got traction.
    - zozbot234 1y ago
      Why can't that dev community just fork the project under a new name and maintain it properly? Picking up a third-party project is absolutely par for the course in FLOSS development.
hoppp 1y ago
Local is important for compliance with GDPR and closed source software I hate sending my code to openAI or my client's code. I find local llms to be usable for short snippets but still too slow for a lot of things. I just spent hours debugging code mistral ai gave me and had multiple errors, rtfm is still most of the times better than relying on an llm
brbcompiling 1y ago
Local AI is awesome, but without beefy hardware it’s like trying to run a marathon in flip-flops.
- mathiaspoint 1y ago
  If you have good flipflops you can walk miles without having to take a break. The other year Walmart had some really good George brand flipflops I used to wear everywhere.
evrennetwork 1y ago
[dead]
ljosifov 1y ago
In the same boat. I love running things localhost. It's been great fun, and I learned tons I didn't know before. I know remote models API-s are a must for any serious work where tons is to be done, produced fr. Still it warms my heart every time llama-server runs on, and serves my aging mbp. Recent MoEs run great on macs with loads of v/ram, and the power efficiency is scarcely believable.
synergy20 1y ago
a PC with rtx3090 is able to run many models locally with decent speed. or rtx4090 though it's more expensive(and power hungry)
solarkraft 1y ago
I’m all for this. This is the first effort I’ve seen attempting to solve the full stack - most local solutions I’ve seen look so DIY that I don’t have much hope I’ll be able to properly configure and operate them dependably. I think there’s room for an integrated solution with all the features we’re used to from commercial solutions: Web search (most important to me), voice mode (very handy), image recognition (useful in some cases), the killer feature being RAG on personal files.
cheesedoodle 1y ago
I’m trying to do something similar but hyper fine tune a model of choice for my specific local data source. For example, use existing code models to answer dquestions with code examples based on my private source files and documentation. I tried doing it with using Huggingface and Unsloth but keep getting OOM errors. Have anyone done this that runs locally against your own data?
- codazoda 1y ago
  I was just writing up a plan this morning. I use local models a fair amount, especially on trips. My plan is to build a $150 AI bot, host it in my bedroom, give it access to all my writing, and let the world access it.
unboxingelf 1y ago
In addition to self hosting, anonymous access to hosted inference is another interesting path. There’s a “A Decentralised LLM Routing Marketplace” being built out on nostr that leverages ecash. https://www.routstr.com/ https://www.routstr.com/
testuseraugust 1y ago
It would be nice to have something more modest like a local offline foreign language translator. Basically I'd like to be able to have an emacs "M-x translate-french-to-english" function. This should be easier than a full chat app but doesn't exist as far as I know.
kalasoo 1y ago
Agree I agree on this in every aspect AI or any technology serve users locally will eventually empower users in a great manner because users can fully understand what they want. Like a paper and pencil, which was not "cheap" in early history but eventually local. "AI" or any technology will function the same way eventually. why? 1. free to run and create (free == cheap, free == uncensored) 2. ambient everywhere
kalasoo 1y ago
Agree I agree on this in every aspect AI or any technology serve users locally will eventually empower users in a great manner because users can fully understand what they want. Like a paper and pencil, which was not "cheap" in early history but eventually "local". AI or any technology will function the same way eventually. why? 1. free to run and create (free == cheap, free == uncensored) 2. ambient everywhere
kenny239 1y ago
We need more projects like this maybe I'll help write some part in the near future.
felarof 1y ago
You should definitely checkout BrowserOS! -- https://github.com/browseros-ai/BrowserOS https://github.com/browseros-ai/BrowserOS
jama211 1y ago
Avoiding cloud dependency seems like a huge amount of effort to make life harder for yourself, just to end up with a situation where you have to rely on other parts of the cloud to do any other part of your work or business anyway. I mean, why stop there? Unplug yourself from the grid so you don’t have to depend on water or electricity just in case the companies that provide it stop working. Yeah, you could do that, but honestly the only world where you’re able to live off grid safely without being attacked and looted is a world where the rest of society hasn’t broken down and still has a grid to connect to. Similarly, the only world where you could succeed as a software developer is one where the cloud generally still functions so you may as well use cloud services that are convenient and right there. You’re not the military with military secrets, you don’t have anything to gain from being independent from the cloud. Interesting as a thought experiment, though.

1143 points by mkagenius 1y ago

This is something that I think about quite a bit and am grateful for this write-up. The amount of friction to get privacy today is astounding.

sneak 1y ago
This writeup has nothing of the sort and is not helpful toward that goal.
- frank_nitti 1y ago
  I'd assume they are referring to being able to run your own workloads in a home-built system, rather then surrendering that ownership to the tech giants alone
  - Imustaskforhelp 1y ago
    Also you get a sort of complete privacy that the data never leaves your home too whereas at best you would have to trust the AI cloud providers that they are not training or storing that data. Its just more freedom and privacy in that matter.
    - doctorpangloss 1y ago
      The entire stack involved sends so much telemetry.
      - frank_nitti 1y ago
        This, in particular, is a big motivator and rewarding factor in getting local setup and working. Turning off the internet and seeing everything run end to end is a joy
        doctorpangloss 1y ago
        NVIDIA drivers send detailed telemetry. Windows and macOS send detailed telemetry. You have to install the pip packages and the models, which all come from websites, which collect detailed telemetry. You don’t think Microsoft gathers detailed telemetry on all your interactions with GitHub? The local setup doesn’t really help with that.
        frank_nitti 1y ago
        We might be talking about two different things. Yes, under normal circumstances the setup steps involve software that defaults to using telemetry -- though I'd be surprised if it's not possible anymore to achieve those in an air-gapped env using e.g. offline installers, zipped repos and wheel files, etc. My comment was referring to runtime workloads having no telemetry (because I unplugged the internet)
    - wkat4242 1y ago
      > whereas at best you would have to trust the AI cloud providers that they are not training or storing that data. Yeah, about that. They even illegally torrented entire databases, hide their crawlers. Crawl entire newspaper archives without permission. They didn't respect the rights of big media companies. But they're going to respect the little guy's of course because it says to in the T&Cs. Uh-huh. Also, openai already admitted that they do store "deleted" content and temporary chats.
      - Imustaskforhelp 1y ago
        I agree but I was just (repeating?) some argument that I heard that if the companies would actually not follow on their premise that they are actually safe if they said so (think amazon bedrock tos policy which says such) Then it will cause an insane backlash and nobody would use the product. So it is in their interest to not train/record. But yes I also agree with you. They are already torrenting :/ So pretty sure if they can do illegal stuff scott free, they might do this too idk, And yeah this was why I was actually saying that local matters more tbh. You just get rid of such headache.
        wkat4242 1y ago
        > Then it will cause an insane backlash and nobody would use the product. So it is in their interest to not train/record. I don't think there would be that much backlash. People are getting hooked on it and many don't actually care about privacy. We know about Google, meta and people still use them. Not a big dent in openai usage either since their revelations. But I understand your point!
Aurornis 1y ago
> The amount of friction to get privacy today is astounding I don't understand this. It's easy to get a local LLM running with a couple commands in the terminal. There are multiple local LLM runners to choose from. This blog post introduces some additional tools for sandboxed code execution and browser automation, but you don't need those to get started with local LLMs. There are multiple local options. This one is easy to start with: https://ollama.com/ https://ollama.com/
- apitman 1y ago
  > It's easy to get a local LLM running Easy for what percentage of people?

noelwelsh 1y ago

It's the hardware more than the software that is the limiting factor at the moment, no? Hardware to run a good LLM locally starts around $2000 (e.g. Strix Halo / AI Max 395) I think a few Strix Halo iterations will make it considerably easier.

colecut 1y ago
This is rapidly improving https://simonwillison.net/2025/Jul/29/space-invaders/ https://simonwillison.net/2025/Jul/29/space-invaders/
- Imustaskforhelp 1y ago
  I hope it improves at such a steady rate! Please lets just hope that there is still room for improvement to packing even more improvements in such LLMS which can help the home labbing community in general.
ramesh31 1y ago
>Hardware to run a good LLM locally starts around $2000 (e.g. Strix Halo / AI Max 395) I think a few Strix Halo iterations will make it considerably easier. And "good" is still questionable. The thing that makes this stuff useful is when it works instantly like magic. Once you find yourself fiddling around with subpar results at slower speeds, essentially all of the value is gone. Local models have come a long way but there is still nothing even close to Claude levels when it comes to coding. I just tried taking the latest Qwen and GLM models for a spin through OpenRouter with Cline recently and they feel roughly on par with Claude 3.0. Benchmarks are one thing, but reality is a completely different story.

ahmedbaracat 1y ago

Thanks for sharing. Note that the GitHub at the end of the article is not working…

mkagenius 1y ago
Thanks for the heads up. It's fixed now - Coderunner-UI: https://github.com/instavm/coderunner-ui https://github.com/instavm/coderunner-ui Coderunner: https://github.com/instavm/coderunner https://github.com/instavm/coderunner

navbaker 1y ago

Open Web UI is a great alternative for a chat interface. You can point to an OpenAI API like vLLM or use the native Ollama integration and it has cool features like being able to say something like “generate code for an HTML and JavaScript pong game” and have it display the running code inline with the chat for testing

dmezzetti 1y ago

I built TxtAI with this philosophy in mind: https://github.com/neuml/txtai https://github.com/neuml/txtai

pyman 1y ago

Mr Stallman? Richard, is that you?

tcdent 1y ago

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardware unjustifiable. Coupled with the dramatically inferior performance of the weights you would be running in a local environment, it's just not worth it. I expect this will change in the future, and am excited to invest in a local inference stack when the weights become available. Until then, you're idling a relatively expensive, rapidly depreciating asset.

braooo 1y ago
Running LLMs at home is a repeat of the mess we make with "run a K8s cluster at home" thinking You're not OpenAI or Google. Just use pytorch, opencv, etc to build the small models you need. You don't need Docker even! You can share over a simple code based HTTP router app and pre-shared certs with friends. You're recreating the patterns required to manage a massive data center in 2-3 computers in your closet. That's insane.
- frank_nitti 1y ago
  For me, this is essential. On priciple, I won't pay money to be a software engineer. I never paid for cloud infrastructure out of pocket, but still became the go-to person and achieved lead architecture roles for cloud systems, because learning the FOSS/local tooling "the hard way" put me in a better position to understand what exactly my corporate employers can leverage with the big cash they pay the CSPs. The same is shaping up in this space. Learning the nuts and bolts of wiring systems together locally with whatever Gen AI workloads it can support, and tinkering with parts of the process, is the only thing that can actually keep me interested and able to excel on this front relative to my peers who just fork out their own money to the fat cats that own billions worth of compute. I'll continue to support efforts to keep us on the track of engineers still understanding and able to 'own' their technology from the ground up, if only at local tinkering scale

sneak 1y ago

Halfway through he gives up and uses remote models. The basic premise here is false. Also, the term “remote code execution” in the beginning is misused. Ironically, remote code execution refers to execution of code locally - by a remote attacker. Claude Code does in fact have that, but I’m not sure if that’s what they’re referring to.

thepoet 1y ago
The blog says more about keeping the user data private. The remote models in the context are operating blind. I am not sure why you are nitpicking, almost nobody reading the blog would take remote code execution in that context.
- vunderba 1y ago
  The MCP aspect (for code/tool execution) is completely orthogonal to the issue of data privacy. If you put a remote LLM in the chain than it is 100% going to inadvertently send user data up to them at some point. e.g. if I attach a PDF to my context that contains private data, it WILL be sent to the LLM. I have no idea what "operating blind" means in this context. Connecting to a remote LLM means your outgoing requests are tied to a specific authenticated API key.

mark_l_watson 1y ago

That is fairly cool. I was talking about this on X yesterday: another angle however, I use a local web scraper and search engine via meilisearch the main tech web sites I am interested in. For my personal research I use three web search APIs, but there is some latency. Having a big chuck of the web that I am interested in available locally with close to zero latency is nice when running local models, my own MCP services that might need web search, etc.

luke14free 1y ago

you might want to check out what we built -> https://inference.sh https://inference.sh supports most major open source/weight models from wan 2.2 video, qwen image, flux, most llms, hunyan 3d etc.. works in a containerized way locally by allowing you to bring your own gpu as an engine (fully free) or allows you to rent remote gpu/pool from a common cloud in case you want to run more complex models. for each model we tried to add quantized/ggufs versions to even wan2.2/qwen image/gemma become possible to execute with as little as 8gb vram gpus. mcp support coming soon in our chat interface so it can access other apps from the ecosystem.

pwn0 1y ago
The website is very confusing. Where can I download the application? Is there a GitHub repository?

rshemet 1y ago

if you ever end up trying to take this in the mobile direction, consider running on-device AI with Cactus – https://cactuscompute.com/ https://cactuscompute.com/ Blazing-fast, cross-platform, and supports nearly all recent OS models.

b0ner_t0ner 1y ago
Is this your site? It's missing a <title> tag.

xt00 1y ago

Yea in an ideal world there would be a legal construct around AI agents in the cloud doing something on your behalf that could not be blocked by various stakeholders deciding they don't like the thing you are doing even if totally legal. Things that would be considered fair use, or maybe annoying to certain companies should not be easy for companies to just wholesale block by leveraging business relationships. Barring that, then yea, a local AI setup is the way to go.

sabareesh 1y ago

Here is my rig, running GLM 4.5 Air. Very impressed by this model https://sabareesh.com/posts/llm-rig/ https://sabareesh.com/posts/llm-rig/ https://huggingface.co/zai-org/GLM-4.5 https://huggingface.co/zai-org/GLM-4.5

mkummer 1y ago

Super cool and well thought out! I'm working on something similar focused on being able to easily jump between the two (cloud and fully local) using a Bring Your Own [API] Key model – all data/config/settings/prompts are fully stored locally and provider API calls are routed directly (never pass through our servers). Currently using mlc-llm for models & inference fully local in the browser (Qwen3-1.7b has been working great) [1] https://hypersonic.chat/ https://hypersonic.chat/

Imustaskforhelp 1y ago

I think I still prefer local but I feel like that's because that most AI inference is kinda slow or comparable to local. But I recently tried out cerebras or (I have heard about groq too) and honestly when you try things at 1000 tk/s or similar, your mental model really shifts and becomes quite impatient. Cerebras does say that they don't log your data or anything in general and you would have to trust me to say that I am not sponsored by them (Wish I was tho) Its just that they are kinda nice. But I still hope that we can someday actually have some meaningful improvements in speed too. Diffusion models seem to be really fast in architecture.

vgb2k18 1y ago
> Cerebras does say that they don't log your data or anything in general Unil a judge says they must log everything, indefinitely

retrocog 1y ago

Its all about context and purpose, isn't it? For certain lightweight uses cases, especially those concerning sensitive user data, a local implementation may make a lot of sense.

accrual 1y ago
My thoughts exactly. The recent GPT-OSS 20B parameter model was a nice upgrade, it really feels like having a local mini ChatGPT.

kaindume 1y ago

Self hosted and offline AI systems would be great for privacy but the hardware and electricity cost are much too high for most users. I am hoping for a P2P decentralized solution that runs on distributed hardware not controlled by a single corporation.

user3939382 1y ago
I’d settle for homomorphic encryption but that’s a long way off if ever

woadwarrior01 1y ago

> LLMs: Ollama for local models (also private models for now) Incidentally, I decided to try to Ollama macOS app yesterday, and the first thing it tries to do upon launch is try to connect to some google domain. Not very private. https://imgur.com/a/7wVHnBA https://imgur.com/a/7wVHnBA

Aurornis 1y ago
Automatic update checks https://github.com/ollama/ollama/blob/main/docs/faq.md https://github.com/ollama/ollama/blob/main/docs/faq.md
abtinf 1y ago
Yep, and I’ve noticed the same thing with in vscode with both the cline plugin and the copilot plugin. I configure them both to use local ollama, block their outbound connections via little snitch, and they just flat out don’t work without the ability to phone home or posthog. Super disappointing that Cline tries to do so much outbound comms, even after turning off telemetry in the settings.
eric-burel 1y ago
But can be audited which I'd buy everyday. It's probably not to hard to find network calls in a codebase if this task must be automated on update.
- woadwarrior01 1y ago
  This is the macOS GUI, which IIUC is closed source.

adsharma 1y ago

https://github.com/adsharma/ask-me-anything https://github.com/adsharma/ask-me-anything Supports MLX on Apple silicon. Electron app. There is a CI to build downloadable binaries. Looking to make a v0.1 release.

andylizf 1y ago

This is fantastic work. The focus on a local, sandboxed execution layer is a huge piece of the puzzle for a private AI workspace. The `coderunner` tool looks incredibly useful. A complementary challenge is the knowledge layer: making the AI aware of your personal data (emails, notes, files) via RAG. As soon as you try this on a large scale, storage becomes a massive bottleneck. A vector database for years of emails can easily exceed 50GB. (Full disclosure: I'm part of the team at Berkeley that tackled this). We built LEANN, a vector index that cuts storage by ~97% by not storing the embeddings at all. It makes indexing your entire digital life locally actually feasible. Combining a local execution engine like this with a hyper-efficient knowledge index like LEANN feels like the real path to a true "local Jarvis." Code: https://github.com/yichuan-w/LEANN https://github.com/yichuan-w/LEANN Paper: https://arxiv.org/abs/2405.08051 https://arxiv.org/abs/2405.08051

sebmellen 1y ago
I know next to nothing about embeddings. Are there projects that implement this same “pruned graph” approach for cloud embeddings?
- NJL3000 1y ago
  It’s in the works… been meaning to do a show HN moment to see if it flies or I Fall on my face..
doctoboggan 1y ago
> A vector database for years of emails can easily exceed 50GB. In 2025 I would consider this a relatively meager requirement.
- andylizf 1y ago
  Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that point, it's no longer a "meager" requirement. It becomes a heavy "tax" on your primary drive, which is often non-upgradable on modern laptops. The goal for practical local AI shouldn't just be that it's possible, but that it's also lightweight and sustainable. That's the problem we focused on: making a comprehensive local knowledge base feasible without forcing users to dedicate half their SSD to a single index.

com2kid 1y ago

> Even with help from the "world's best" LLMs, things didn't go quite as smoothly as we had expected. They hallucinated steps, missed platform-specific quirks, and often left us worse off. This shows how little native app training data is even available. People rarely write blog posts about designing native apps, long winded medium tutorials don't exist, heck even the number of open source projects for native desktop apps is a small percentage compared to mobile and web apps. Historically Microsoft paid some of the best technical writers in the world to write amazing books on how to code for Windows (see: Charles Petzold), but now days that entire industry is almost dead. These types of holes in training data are going to be a larger and larger problem. Although this is just representative of software engineering in general - few people want to write native desktop apps because it is a career dead end. Back in the 90s knowing how to write Windows desktop apps was great, it was pretty much a promised middle class lifestyle with a pretty large barrier to entry (C/C++ programming was hard, the Windows APIs were not easy to learn, even though MS dumped tons of money into training programs), but things have changed a lot. Outside of the OS vendors themselves (Microsoft, Apple) and a few legacy app teams (Adobe, Autodesk, etc), very few jobs exist for writing desktop apps.

thorncorona 1y ago
I mean outside of HPC why would you when the browser is the world’s most ubiquitous VM?
- esseph 1y ago
  Because the browser is gross and you can reclaim lot of performance and security when you don't need to use it.
  - esseph 1y ago
    I mean, why aren't the apps on your phone all just webapps, right? (Also, eww)
    - jakelazaroff 1y ago

gen2brain 1y ago

People are talking about AI everywhere, but where can we find documentation, examples, and proof of how it works? It all ends with chat. Which chat is better and cheaper? This local story is just using some publicly available model, but downloaded? When is this going to stop?

bling1 1y ago

On a similar vibe, we developed app.czero.cc to run an LLM inside your chrome browser on your machine hardware without installation (you do have to download the models). Hard to run big models, but it doesnt get more local than that without having to install anything.

btbuildem 1y ago

I didn't see any mention of the hardware OP is planning to run this on -- any hints?

vunderba 1y ago

Infra notwithstanding - I'd be interested in hearing how much success they actually had using a locally hosted MCP-capable LLM (and which ones in particular) because the E2E tests in the article seem to be against remote models like Claude.

ruler88 1y ago

At least you won't be needing a heater for the winter

mikeyanderson1 1y ago

We have this in closed alpha right now getting ready to roll out to our most active builders in the coming weeks at ThinkAgents.ai

techlatest_net 1y ago

[dead]

LastTrain 1y ago

I get it but I can’t get over the irony that you are using a tool that only works precisely because people don’t do this.

eric-burel 1y ago

An llm on your computer is a fun hobby, an llm in your SME for 10 people is a business idea. There are not enough resources on this topic at all and the need is growing extremely fast. Local LLMs are needed for many use cases and business where cloud is not possible.

deleted 1y ago

[deleted]

nenadg 1y ago

did this by running models in chroot

josephwegner 1y ago

See

jumploops 1y ago

I'm a little confused about your product branding vs. blog post? From the product homepage, I imagine you're running VMs in the cloud (a la Firecracker). From the blog post though, it looks like you're running Apple-specific VMs for local execution? As someone who's built the former, I'd love the latter for use with the new gpt-oss releases :)

mkagenius 1y ago
You are right, the product is almost the same as you described, intended for customers running LLM-generated code in their workflow. This was something close to our hearts, so we thought of building it for our local use and releasing it for like-minded individuals.

b0ner_t0ner 1y ago

To OP, your link for https://github.com/assistant-ui/assistant-ui https://github.com/assistant-ui/assistant-ui does not work.

eyespasm 1y ago

To be honest, I just want to make porn. My own porn, the way I want it. That’s what I’m waiting for. Why the heck do I need to scroll through pages of boring, vanilla, pedestrian porn on Pornhub or RedGIfs or XNXX when I can create exactly what I want? That’ll be a huge killer app when I can do it locally and in the privacy of my own home.

123sereusername 1y ago

[dead]

jarym 1y ago

Playing with local LLMs is indeed fun. I use Kasm workspaces[0] to run a desktop session with ollama running on the host. Gives me the isolation and lets me experiment with all manner of crazy things (I tried to make a computer-use AI but it wasn't very good) [0] https://kasmweb.com/ https://kasmweb.com/

nikolayasdf123 1y ago

local is so hot right now

nikolayasdf123 1y ago

local/edge is the most under-valued space at the moment. incredible computing power that dwarfs datacenters, zero latency, zero cost, private, distributed and resilient

oblio 1y ago
I guess you imagine a world like Skype supernodes (Skype gave that up more than a decade ago) or Tor nodes (Tor is used by a tiny fraction of internet users). Not saying it can't be done, but the effort is humongous.
- nikolayasdf123 1y ago
  no, I mean I saw multiple companies at this point with their entier K8S cluster... is smaller than single new macbook pro :/ now, if you have 100,000 users with latest iPhone, say you use 10GB RAM in each, using A16 chip with 1.9 TFLOPS, each with 5G connection this is 1 Peta-Byte RAM + 0.25 Peta-FLOPs GPU + 4 TB / second bandwidth at zero cost (no-upfront, no-maintenance, users pay for, upgrade, and maintain their phones working, pay for internet, charging with electricity, cooling? - thanks!) ... it goes even wilder if you use macbooks ... and if you consider say mid-size town in China with population of 15 million, you go Exa-scale and consider that for now iPhones are just sitting idle. for now.
  - oblio 1y ago
    Things don't work like that. First of all iPhones have more like 6-8GB of RAM, 1-2 of which are already taken up by the system and system apps. Add some resident apps and maybe 1-2GB are already taken. Then of course during peak times, which are predictable but not guaranteed, 5-10% is maybe available. So out of your 10GB estimated per device, you actually average maybe 3GB. Similar story for the CPU and GPU. Then, availability: dead battery, no cell reception, airplane mode, etc, etc. And on top of that, in the context of battery charge and long term wear and tear, you're assuming people will just let you run Bitcoin mining nodes on them. You need a really solid incentive for people to loan you end user computing power for legitimate reasons.

willtemperley 1y ago

How would this compare to using Apple Foundation Models which execute on device? https://developer.apple.com/documentation/FoundationModels https://developer.apple.com/documentation/FoundationModels

dcreater 1y ago

Then using ollama is not the right choice. https://news.ycombinator.com/item?id=44814607 https://news.ycombinator.com/item?id=44814607

jychang 1y ago

Any way to install this via just a container? Similar to a `docker compose up -d` that a lot of projects offer. Just download the docker-compose.yml file into a folder, run the command, and you're running. If you want to delete everything, just `docker compose down` and delete the folder, and the container and everything is gone. Anything similar to that? I don't want to run a random install.sh on my machine that does god knows what.

patmorgan23 1y ago
I believe a flatpack or appimage is what you're looking for.
mkagenius 1y ago
There are similar commands for coderunner (not the UI frontend): container image pull instavm/coderunner container run --name coderunner --detach instavm/coderunner (for more comprehensive commands, see from line 51 https://github.com/instavm/coderunner/blob/main/install.sh#L51 https://github.com/instavm/coderunner/blob/main/install.sh#L...) Frontend (coderunner-ui) is not inside a docker as of now.
cheschire 1y ago
But you would pump your secrets into a docker AI?
- oblio 1y ago
  Does Docker do that or are you speculating? Also - podman?
  - cheschire 1y ago
    I wasn't implying docker itself was the issue. The previous commenter said that they didn't want to run a shell script that does "god knows what". The implication being that they would not trust the writer of the shell script. They wanted a docker container that would setup this offline AI workspace for them, presumably so they could interact with the AI and feed "secrets" or otherwise private data into it. Obviously there are other use cases for an offline AI, but folks tend to let their guard down when they think something is offline-only, and they may not be as careful with .env values, or personal information, as they would with a SaaS frontier model. So I was pointing out that the contents of the docker container would be also doing "god knows what" with their data. Sure they would get the offline user experience but then what happens? More shell scripts? Background data calls? etc. And of course it depends on how they configure their docker container, but if they aren't willing to review an install shell script, they probably aren't looking to do any level of effort for configuring Docker. Hopefully that clarifies it.

PeterStuer 1y ago

The link to assistent ui in the article 404's. It should be https://github.com/assistant-ui/assistant-ui https://github.com/assistant-ui/assistant-ui

mkagenius 1y ago
My bad, I typed `-ai` instead of `-ui`. Its fixed now.

EagnaIonat 1y ago

You can get good models that run fine on M1 32GB laptops just using Ollama App. Or if you want numerous features on top of your local LLMS then Open WebUI would be my choice. https://docs.openwebui.com https://docs.openwebui.com

anupshinde 1y ago

What is the Apple hardware being used here? I see Apple Silicon but not the configuration.. what did I miss

thebruce87m 1y ago
Was looking for that too. Need to know whether I already own the hardware or can’t afford it.

jeffWrld 1y ago

[dead]

_the_inflator 1y ago

The socialist EU allows only AI that serves the governance purpose. The EU has rightfully acknowledged that freedom of AI is essentially freedom of speech. Hacking officially stopped being non-political in EU. https://artificialintelligenceact.eu/ https://artificialintelligenceact.eu/ Enjoy understanding this here: https://artificialintelligenceact.eu/article/3/ https://artificialintelligenceact.eu/article/3/ Measures of Innovations rank at... Article 57! https://artificialintelligenceact.eu/ai-act-explorer/ https://artificialintelligenceact.eu/ai-act-explorer/ I bet that soon, anyone involved with sophisticated AI systems will be system-checked and require a license. God bless you all out there and have phun!

oblio 1y ago
The EU isn't socialist, what are you going on about? And AI - if true AI - can be "end of times" type tech, you think it won't be regulated? This is not hackers playing with breadboards in the 60s, it's Project Manhattan in the 40s.

yichuan 1y ago

That's my vision, hope it can help. I think that if we combine all our personal data and organize it effectively, we can be 10 times more efficient. Long-term AI memory, all you speak and see will secretly be loaded to your own personal AI, and that can solve many difficulties, I think. https://x.com/YichuanM/status/1953886817906045211 https://x.com/YichuanM/status/1953886817906045211

shekhargulati 1y ago

I tried to port it to Docker and wrote a blog here https://shekhargulati.com/2025/08/09/making-coderunner-ui-work-with-docker-using-claude-code/ https://shekhargulati.com/2025/08/09/making-coderunner-ui-wo.... I used Claude Code to do the port. We used Datalayer Jupyter MCP Server instead of coderunner which uses Apple containers.

mockingloris 1y ago
At least, you are honest about augmenting the porting process. It's amazing what one can accomplish when they realize that with proper time, planning and a good grounding on building code/systems, that a lot more is possible. The takeaway for me is that because these tools are fast doesn't mean the task also needs to move as fast. At least till AGI, a sound human reasoning before hitting enter goes a long way. Thanks for sharing

k__ 1y ago

Half-OT: Anything useful that runs reasonably fast on a regular Intel CPU/GPU?

oblio 1y ago
I did a bunch of research and basically no. Unless you can work with sending a request in the evening and getting the result in the morning. And you'd need a lot of regular RAM because otherwise you start swapping at which point I think response times end up in days. This tech is in the Wild West days, for it to be usable by the average person on consumer hardware, I think we'll need to be in 2030+.
ethan_smith 1y ago
For Intel CPUs, Phi-2 (2.7B) and TinyLlama (1.1B) run reasonably well using llama.cpp with 4-bit quantization. GGUF models with INT4 quantization typically need ~2GB RAM per billion parameters, so even older machines can handle smaller models.
- akawry 1y ago
  Take a look at ik_llama.cpp: https://github.com/ikawrakow/ik_llama.cpp https://github.com/ikawrakow/ik_llama.cpp CPU performance is much better than mainline llama, as well as having more quantization types available

Woodi 1y ago

So there are models to download but: a) on what data that things was trained ? b) any reproducible builds projects ? ;)

hollowonepl 1y ago

Yep, that is something I do also actively experiment with in home projects. Local NAS (Synology) with 28TB of RAIDed storage, local containers and VMs on it and local gitea and other devops and productivity tools. All that talks to my mac which runs editing, compiling, etc and lmstudio with local agent. Not best always with AI, I lack enough RAM but close to imagine how I will work in the future, end-to-end

zakki 1y ago

Curious with the hardware used in this article.

t_mann 1y ago

Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great for the purpose, but you don't really get them with enough RAM or at prices that don't really qualify as consumer-grade anymore. I've seen people create home clusters (eg using Exo [0]), but I wouldn't really call it practical (single digit token/sec for large models, and the price isn't exactly accessible either). Framework (the modular laptop company) has announced a desktop that can be configured up to 128GB unified RAM, but it's still going to come in at around 2-2.5k depending on your config. [0] https://github.com/exo-explore/exo https://github.com/exo-explore/exo

zozbot234 1y ago
What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project. Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.
- graemep 1y ago
  You still need a lot of RAM though right? so its not going to be that cheap? What sort of specs do you need?
- flanger 1y ago
  The founders of Exo ghosted the dev community and went closed-source. Nobody has heard from them. I wish people would stop recommending Exo (a tribute to their marketing) and check out GPUStack instead. Overall another rug pull by the devs as soon as they got traction.
  - zozbot234 1y ago
    Why can't that dev community just fork the project under a new name and maintain it properly? Picking up a third-party project is absolutely par for the course in FLOSS development.

hoppp 1y ago

Local is important for compliance with GDPR and closed source software I hate sending my code to openAI or my client's code. I find local llms to be usable for short snippets but still too slow for a lot of things. I just spent hours debugging code mistral ai gave me and had multiple errors, rtfm is still most of the times better than relying on an llm

brbcompiling 1y ago

Local AI is awesome, but without beefy hardware it’s like trying to run a marathon in flip-flops.

mathiaspoint 1y ago
If you have good flipflops you can walk miles without having to take a break. The other year Walmart had some really good George brand flipflops I used to wear everywhere.

evrennetwork 1y ago

[dead]

ljosifov 1y ago

In the same boat. I love running things localhost. It's been great fun, and I learned tons I didn't know before. I know remote models API-s are a must for any serious work where tons is to be done, produced fr. Still it warms my heart every time llama-server runs on, and serves my aging mbp. Recent MoEs run great on macs with loads of v/ram, and the power efficiency is scarcely believable.

synergy20 1y ago

a PC with rtx3090 is able to run many models locally with decent speed. or rtx4090 though it's more expensive(and power hungry)

solarkraft 1y ago

I’m all for this. This is the first effort I’ve seen attempting to solve the full stack - most local solutions I’ve seen look so DIY that I don’t have much hope I’ll be able to properly configure and operate them dependably. I think there’s room for an integrated solution with all the features we’re used to from commercial solutions: Web search (most important to me), voice mode (very handy), image recognition (useful in some cases), the killer feature being RAG on personal files.

cheesedoodle 1y ago

I’m trying to do something similar but hyper fine tune a model of choice for my specific local data source. For example, use existing code models to answer dquestions with code examples based on my private source files and documentation. I tried doing it with using Huggingface and Unsloth but keep getting OOM errors. Have anyone done this that runs locally against your own data?

codazoda 1y ago
I was just writing up a plan this morning. I use local models a fair amount, especially on trips. My plan is to build a $150 AI bot, host it in my bedroom, give it access to all my writing, and let the world access it.

unboxingelf 1y ago

In addition to self hosting, anonymous access to hosted inference is another interesting path. There’s a “A Decentralised LLM Routing Marketplace” being built out on nostr that leverages ecash. https://www.routstr.com/ https://www.routstr.com/

testuseraugust 1y ago

It would be nice to have something more modest like a local offline foreign language translator. Basically I'd like to be able to have an emacs "M-x translate-french-to-english" function. This should be easier than a full chat app but doesn't exist as far as I know.

kalasoo 1y ago

Agree I agree on this in every aspect AI or any technology serve users locally will eventually empower users in a great manner because users can fully understand what they want. Like a paper and pencil, which was not "cheap" in early history but eventually local. "AI" or any technology will function the same way eventually. why? 1. free to run and create (free == cheap, free == uncensored) 2. ambient everywhere

kalasoo 1y ago

Agree I agree on this in every aspect AI or any technology serve users locally will eventually empower users in a great manner because users can fully understand what they want. Like a paper and pencil, which was not "cheap" in early history but eventually "local". AI or any technology will function the same way eventually. why? 1. free to run and create (free == cheap, free == uncensored) 2. ambient everywhere

kenny239 1y ago

We need more projects like this maybe I'll help write some part in the near future.

felarof 1y ago

You should definitely checkout BrowserOS! -- https://github.com/browseros-ai/BrowserOS https://github.com/browseros-ai/BrowserOS

jama211 1y ago

Avoiding cloud dependency seems like a huge amount of effort to make life harder for yourself, just to end up with a situation where you have to rely on other parts of the cloud to do any other part of your work or business anyway. I mean, why stop there? Unplug yourself from the grid so you don’t have to depend on water or electricity just in case the companies that provide it stop working. Yeah, you could do that, but honestly the only world where you’re able to live off grid safely without being attacked and looted is a world where the rest of society hasn’t broken down and still has a grid to connect to. Similarly, the only world where you could succeed as a software developer is one where the cloud generally still functions so you may as well use cloud services that are convenient and right there. You’re not the military with military secrets, you don’t have anything to gain from being independent from the cloud. Interesting as a thought experiment, though.