24 ms·

The ecosystem is 8 tools plus a claude code plugin, the unlock was composing those tools (I don't regularly use all 9). The 5x claim was from /insights (claude

The ecosystem is 8 tools plus a claude code plugin, the unlock was composing those tools (I don't regularly use all 9). The 5x claim was from /insights (claude code) Not for everyone, but it radically changed how I build. Senior engineer, 10+ years Now it's trivial to run multiple projects in parallel across claude sessions (this was not really manageable before using wheat) Genuinely don't remember the last time I opened a file locally

lmeyerov 6mo ago
It sounds like the answer is "No, there is no repeatable eval of the core AI coding productivity claim, definitely not on one of the many AI coding benchmarks in the community used for understanding & comparison, and there will not be"
- volatilityfund 6mo ago
  My data is from Anthropic Not sure how it works under the hood, probably a better question for them Perhaps you are misunderstanding the entire premise of this project, this is not an LLM
  - lmeyerov 6mo ago
    Maybe there's a fundamental miscommunication here of what evals are? Evals apply not just to LLMs but to skills, prompts, tools, and most things changing the behavior of compound AI systems, and especially like the productivity claims being put forth in this thread. The features in the post relate directly to heavily researched areas of agents that are regularly benchmarked and evaluated. They're not obscure, eg, another recent HN frontpage item benchmarked on research and planning.
    - aid-ninja 6mo ago
      your question makes sense, it's just not in current scope we are still benchmarking the compiler at scale and the LLM tools that were made were created as functional prototypes to showcase a single example of the compiler's use case since much of the unlock here is finding different applications for the compiler itself, we simply don't have the bandwidth to do much benchmarking on these projects on top of maintaining the repos themselves all the code is open source and there is nothing stopping anyone from running their own benchmarks if they were curious btw https://news.ycombinator.com/item?id=47733217 https://news.ycombinator.com/item?id=47733217
    - aid-ninja 6mo ago
      [dead]

lmeyerov 6mo ago

It sounds like the answer is "No, there is no repeatable eval of the core AI coding productivity claim, definitely not on one of the many AI coding benchmarks in the community used for understanding & comparison, and there will not be"

volatilityfund 6mo ago
My data is from Anthropic Not sure how it works under the hood, probably a better question for them Perhaps you are misunderstanding the entire premise of this project, this is not an LLM
- lmeyerov 6mo ago
  Maybe there's a fundamental miscommunication here of what evals are? Evals apply not just to LLMs but to skills, prompts, tools, and most things changing the behavior of compound AI systems, and especially like the productivity claims being put forth in this thread. The features in the post relate directly to heavily researched areas of agents that are regularly benchmarked and evaluated. They're not obscure, eg, another recent HN frontpage item benchmarked on research and planning.
  - aid-ninja 6mo ago
    your question makes sense, it's just not in current scope we are still benchmarking the compiler at scale and the LLM tools that were made were created as functional prototypes to showcase a single example of the compiler's use case since much of the unlock here is finding different applications for the compiler itself, we simply don't have the bandwidth to do much benchmarking on these projects on top of maintaining the repos themselves all the code is open source and there is nothing stopping anyone from running their own benchmarks if they were curious btw https://news.ycombinator.com/item?id=47733217 https://news.ycombinator.com/item?id=47733217
  - aid-ninja 6mo ago
    [dead]