4 ms·
I think it depends - the actual thing to measure it to keep a developer in flow state. Many errors as well as latency break this. To be brief yes, accuracy come
by bhaktatejas922 1y ago
I think it depends - the actual thing to measure it to keep a developer in flow state. Many errors as well as latency break this. To be brief yes, accuracy comes first.
Quality is measured 2 main ways:
1) End-to-end: User query -> to task resolution. These are aider style benchmarks answering the question of actual task completion
2) Apply Quality: Syntax correctness, character diff, etc..
The error rate for large vs fast is around 2%. If you're doing code edits that are extremely complex or on obscure languages - large is the better option. There's also an auto option to route to the model we think is best for a task
- deepdarkforest 1y agoGlad to hear quality comes first! Then I assume you have some public benchmarks like the ones you mention that are reproducible? I could only find this graph https://docs.morphllm.com/guides/apply https://docs.morphllm.com/guides/apply but there is no mention of what it refers to, what data it used etc.
- candiddevmike 1y agoI don't believe anyone can be in some kind of "flow state" while waiting on LLM responses. I think it's funny that we complained for years about C and others being slow to compile and now folks are fine waiting seconds++ everytime they want to change something.
- bhaktatejas922 1y agohow so? Is your view that flow state at all isnt a thing, or just with using LLMs?
- candiddevmike 1y agoFlow state is 100% a thing, it's just impossible with LLMs (at least, for me). I can't be blocked waiting on things during a flow state or my mind starts wondering to other places.
- ada1981 1y agoI've had the opposite experience.
- bhaktatejas922 1y agosame
- bhaktatejas922 1y agoFast Apply definitely helps with keeping flow state and is a large part of Cursor's success Personally I work on multiple repos at a time to solve for this
- 0x457 1y agoI do it like simultaneous exhibition in chess: - Multiple repos or independent changes in monorepo - First round of changes idgaf about anything beyond public interface and unit tests - I review public interface and make changes if needed - I review unit tests it wrote to see that at least from the outside it looks alright. - here I either: - make more unit tests (features, edge cases and make it write code for it) - polish what it generate
- bhaktatejas922 1y agosounds like flow state to me
- 0x457 1y agooh it's fore sure is. But I use amazon q almost exclusively. One thing that gets me out of this state: when I have to do the math on "should I just do it myself" vs "keep refining prompt/context until this thing finally gets it right".
- bhaktatejas922 1y ago
- furyofantares 1y agoThis is gonna sound like some chad hype shit, but I've tried just working 2 different projects simultaneously and have had some incredible extended flow sessions. It felt like the old days of multitabling poker. I had tried doing it with different features in different worktrees in the same codebase but found flow much harder there. Lately I am also just spending a lot more time reworking code manually to keep the code in good shape. Still getting a ton of value out of the LLM doing a lot of work, but not exactly spending lots of time just waiting for it because I am dropping back down to manual mode frequently.
- throwaway2037 1y ago> we complained for years about C and others being slow to compile For C? I don't remember that, unless headers are poorly managed. C++? Definitely yes.
- bhaktatejas922 1y agoWhat do we not complain about if we're being honest?
- Aurornis 1y ago> the actual thing to measure it to keep a developer in flow state. Personally, I find flow state hard to achieve when I constantly have to switch modes to debugging LLM output or an edit error that I missed. When the majority of time is spent waiting for the main LLM to think, I will always wait a few extra seconds for a better edit than risk having to spend multiple cycles playing find-the-bug because something didn't get applied correctly somewhere.
- bhaktatejas922 1y agoLike most things its a tradeoff. Developer tolerance for errors is extremely low - but the error rate for Fast Apply is even lower