4 ms·

I always put the agent harness in an ubuntu-based Docker, with a /workspace folder where it can work and occasionally some other stuff mounted as read-only. The

I always put the agent harness in an ubuntu-based Docker, with a /workspace folder where it can work and occasionally some other stuff mounted as read-only. The LLM server itself (llama-server) is running on a different more powerful computer on the local network, connected through Tailscale so I can also use it away from home. I honestly don't trust these things to not accidentally mess something up, otherwise. Now I think it's still technically possible to break out of that with some clever hacks? But the moment I see a model even vaguely considering that, I will never run it again. (I don't use Claude but currently Qwen3.8 27B)

Neywiny 7d ago
That's my approach too. I even added on a firewall container to the compose so it could fetch packages and that's it. It isn't impossible for it to exfiltrate data that way but I think I put a limit on the request size and limited to get requested so if it did it was relatively slow. But once it has all the tools it needs, that can be cut
- tosapple 7d ago
  you used to only 'need' debug.com.
javcasas 7d ago
How is Qwen3.8 27B behaving in comparison with, say, the free models available at OpenRouter or OpenCode? I'm interested in running models locally, and 27B is in the range of my budget.
- tripzilch 6d ago
  Oh it's terrible, it feels like a complete waste of time to me every time, making me wish I had just done the task myself, so at least it'd be present in my head and I could just work on in, instead of feeling winded and frustrated. Maybe I'm prompting it wrong. But to be honest I'm not super impressed either by the "frontier" models' ability to do a shitty job at coding, except much faster. Or maybe my setup (128GB amd strix halo box) isn't configured right and it could be faster, I dunno. I've already spent a few days on that, but it might take a few more. It's way more complicated than I expected. This week, I've had it look through and generate more complete documentation for Strudel (music live coding tool), cause a lot of functions/behavior in it are not in their official docs. It took about 2-3 days. I had expected it to be an overnight task. I did find the "opencode" harness to be a bit more performant than the "pi" harness. But maybe I've not configured "pi" right, I tried very hard, but when I installed "opencode" it just performed much better right out of the box. Especially running subagents just seemed to confuse the model in "pi". Either way, it made me realize that a large part of the "intelligence" and occasional "usefulness" of these tools are in the harness, not the weights. I didn't know there were free models on OpenRouter. I'm not really into renting tools that I'll become dependent on, so I never looked. But, I dunno. You're still hooking it in to your terminal, and they could in theory literally inject any command and take over your machine when you're not looking ... it still seems a bit like a crazy thing to do :) And it's not like I need LLMs to code or anything. To be completely honest I'm still waiting for when they get good, which everybody says is supposedly any day now. I've also tried Qwen3.6-35B/A3B a couple of times. I'd say it's about 4x faster, which is quite significant. Unfortunately it's also quite obviously more stupid and often fumbles its tool use. For me this adds up to taking about the same amount of time, multiplied by more frustration. I find it hard to give concrete tokens/sec numbers, because they seem to change a lot. When I give Qwen3.8-27B a test query in the browser chat interface (e.g. "explain fibonacci hash"), I currently can get it up to about 18 tok/s. The Qwen3.6-35B/A3B can get up to about 66 tok/s. But this is just what I use as an indication for when my settings are right, cause when I set it up in the coding harness, the numbers are wildly different (and generally slower).
  - jacobify 5d ago
    [dead]

Neywiny 7d ago

That's my approach too. I even added on a firewall container to the compose so it could fetch packages and that's it. It isn't impossible for it to exfiltrate data that way but I think I put a limit on the request size and limited to get requested so if it did it was relatively slow. But once it has all the tools it needs, that can be cut

tosapple 7d ago
you used to only 'need' debug.com.

javcasas 7d ago

How is Qwen3.8 27B behaving in comparison with, say, the free models available at OpenRouter or OpenCode? I'm interested in running models locally, and 27B is in the range of my budget.

tripzilch 6d ago
Oh it's terrible, it feels like a complete waste of time to me every time, making me wish I had just done the task myself, so at least it'd be present in my head and I could just work on in, instead of feeling winded and frustrated. Maybe I'm prompting it wrong. But to be honest I'm not super impressed either by the "frontier" models' ability to do a shitty job at coding, except much faster. Or maybe my setup (128GB amd strix halo box) isn't configured right and it could be faster, I dunno. I've already spent a few days on that, but it might take a few more. It's way more complicated than I expected. This week, I've had it look through and generate more complete documentation for Strudel (music live coding tool), cause a lot of functions/behavior in it are not in their official docs. It took about 2-3 days. I had expected it to be an overnight task. I did find the "opencode" harness to be a bit more performant than the "pi" harness. But maybe I've not configured "pi" right, I tried very hard, but when I installed "opencode" it just performed much better right out of the box. Especially running subagents just seemed to confuse the model in "pi". Either way, it made me realize that a large part of the "intelligence" and occasional "usefulness" of these tools are in the harness, not the weights. I didn't know there were free models on OpenRouter. I'm not really into renting tools that I'll become dependent on, so I never looked. But, I dunno. You're still hooking it in to your terminal, and they could in theory literally inject any command and take over your machine when you're not looking ... it still seems a bit like a crazy thing to do :) And it's not like I need LLMs to code or anything. To be completely honest I'm still waiting for when they get good, which everybody says is supposedly any day now. I've also tried Qwen3.6-35B/A3B a couple of times. I'd say it's about 4x faster, which is quite significant. Unfortunately it's also quite obviously more stupid and often fumbles its tool use. For me this adds up to taking about the same amount of time, multiplied by more frustration. I find it hard to give concrete tokens/sec numbers, because they seem to change a lot. When I give Qwen3.8-27B a test query in the browser chat interface (e.g. "explain fibonacci hash"), I currently can get it up to about 18 tok/s. The Qwen3.6-35B/A3B can get up to about 66 tok/s. But this is just what I use as an indication for when my settings are right, cause when I set it up in the coding harness, the numbers are wildly different (and generally slower).
- jacobify 5d ago
  [dead]