4 ms·
The problem with models like Qwen 3.6 35B (which really is an excellent model) is that my expectations of what a model can do have gone SO high now. Here's a p
by simonw 5mo ago
The problem with models like Qwen 3.6 35B (which really is an excellent model) is that my expectations of what a model can do have gone SO high now.
Here's a prompt I just ran against Claude Opus 4.7:
> Use python3 to experiment with whether the SQLite3 authorizer mechanism can be used to detect an INSERT OR REPLACE based just on running an explain query without examining the SQL string itself
Opus nailed it: https://claude.ai/share/c4212606-3fee-4b7c-bc97-505e0348ccac https://claude.ai/share/c4212606-3fee-4b7c-bc97-505e0348ccac
I tried the same thing against qwen/qwen3.5-35b-a3b running locally in lmstudio, with the Pi coding agent. At first it looked like it was going to do great! And then it fell apart over the course of several tool calls: https://gisthost.github.io/?8ae2f842df619fb7fd8f1ccd82fe41c7 https://gisthost.github.io/?8ae2f842df619fb7fd8f1ccd82fe41c7
I'm used to GPT-5.5 and Opus 4.7 handling that kind of prompt without any problems at all.
- deleted 5mo ago[deleted]
- whattheheckheck 5mo agoThats when your build a better Ralph loop around your llm for it to converge to an answer and not rely on 1 shots
- scribble0242 5mo agoThis worked for me with qwen3.6-36b-a3b even at a q4 quant. I ran pi in a docker container and it had to figure out how to install python as well. I used the same initial prompt you had without any additional. You talked about Qwen 3.6, but then said you tried Qwen 3.5 in lmstudio. Not sure if you meant Qwen 3.6. I ran with llama.cpp llama-server with the recommended settings from unsloth. I'm not an expert in SQLLite so I can't say if this is 100% correct, but it seemed directionally similar to the conclusion from claude. ### TL;DR - Authorizer + EXPLAIN: No — authorizer only sees SQLITE_INSERT, not VDBE opcodes - EXPLAIN opcode analysis alone: Yes — Delete opcode at position 10 is the unique signature of INSERT OR REPLACE / REPLACE I can't help but think the not-so-distant future will see language models expected on commodity personal computing devices.
- deleted 5mo ago[deleted]
- simonw 5mo agoOK that's a very good answer! Do you mind sharing the transcript?
- scribble0242 5mo agoSure I cleaned up the jsonl session file a little here: https://pastebin.com/PL9EPn9Y https://pastebin.com/PL9EPn9Y I tried it a second time, and it spent a lot of time trying to figure out some authorization issue, so definitely not a slam dunk. I might run it a few more times for science. But while this is a new model it's also quite lightweight, and as hardware adapts and improves it seems inevitable that for many use-cases a packaged language model running locally will do the trick.
- Balinares 5mo agoSo one of the prominent LLM advocates known for testing every model shared a prompt intended to exhibit Opus 4.7 capabilities, and Qwen 3.6 sorted it out okay? Interesting. Not saying they're equivalent, local models still decohere much quicker as the context grows in my experience. But... Interesting.
- lelandbatey 5mo agoSomething is definitely going wrong with your Qwen setup, in the link you posted it starts and ends with a compaction step due to a 4k token context limit. Qwen 35b supports I think up to 200k+ context limit (though I run only with 128k), that seems to be a major source of the problem.
- simonw 5mo agoGood call, I need to check if LM Studio is misconfigured.