3 ms·
Great post. A few choice quotes: > Often the model isn’t flaky at understanding the task. It’s flaky at expressing itself. You’re blaming the pilot for the lan
by chrisweekly 8mo ago
Great post. A few choice quotes:
> Often the model isn’t flaky at understanding the task. It’s flaky at expressing itself. You’re blaming the pilot for the landing gear.
> The model is the moat. The harness is the bridge. Burning bridges just means fewer people bother to cross. Treating harnesses as solved, or even inconsequential, is very short-sighted.
> The gap between “cool demo” and “reliable tool” isn’t model magic. It’s careful, rather boring, empirical engineering at the tool boundary.
- brendanmc6 8mo agoYou’re absolutely right! This isn’t your average engineering advice— it’s like painting the reader a vivid tapestry of the author’s mind.
- dimgl 8mo agoMy personal favorite: That’s not a threat. It’s free R&D.
- ramraj07 8mo agoGreat post indeed but let me ask you, put yourself in the LLM shoes. Now instead of reading through coherent lines of code that is exclusively about solving problems, you now have random characters before every line that mean something (because the presence of the edit tool implies it) but not about your actual problem. Do you reckon the LLM will be distracted a little bit? The benchmark deliberately sidestep the actual intelligence of the model on the task at hand, so while the author feels successful at their subtask its very possible they've failed at the war. This seems to be the beauty of AI engineering. The smarter you think you are about something the bigger the fall.