5 ms·

Evals or GTFO

by lmeyerov 6mo ago

Evals or GTFO

deleted 6mo ago
[deleted]
deleted 6mo ago
[deleted]
deleted 6mo ago
[deleted]
volatilityfund 6mo ago
5x productivity boost in merged PRs (lots of open PR & merge rate goes down, but net positive) Starting to build custom tooling around new "friction" points in dev cycle (eng IC perspective)
- lmeyerov 6mo ago
  Evals let us agree on the baseline, measurement, etc, and compare if simple things others do perform just as well. For same reason, instead of 'works on my box' and 'my coding style', use one of the many community evals vs making up your own benchmark. That helps head off much of many of the unfalsifiable discussions & claims happening and moves everyone forward.
  - aid-ninja 6mo ago
    a rust version of that compiler (that the project runs on) ran at 480k claims/sec and it was able to deterministically resolve 83% of conflicts across 1 million concurrent agents (also 393,275x compression reduction @ 1m agents on input vs output, but different topics can make the compression vary) natively claude (and other LLM) will resolve conflicting claims at about 51% rate (based on internal research) the built in byzantine fault tolerance (again, in the compiler) is also pretty remarkable, it can correctly find the right answer even if 93% of the agents/data are malicious (with only 7% of agents/data telling us the correct information) basically the idea here is if you want to build autonomous at scale, you need to be able to resolve disagreement at scale and this project does a pretty nice job at doing that
    - lmeyerov 6mo ago
      My question was on claims like "5x productivity boost in merged PRs (lots of open PR & merge rate goes down, but net positive)", eg, does this change anything on swe-bench or any other standard coding eval?
      - volatilityfund 6mo ago
        The ecosystem is 8 tools plus a claude code plugin, the unlock was composing those tools (I don't regularly use all 9). The 5x claim was from /insights (claude code) Not for everyone, but it radically changed how I build. Senior engineer, 10+ years Now it's trivial to run multiple projects in parallel across claude sessions (this was not really manageable before using wheat) Genuinely don't remember the last time I opened a file locally
        lmeyerov 6mo ago
        It sounds like the answer is "No, there is no repeatable eval of the core AI coding productivity claim, definitely not on one of the many AI coding benchmarks in the community used for understanding & comparison, and there will not be"
        volatilityfund 6mo ago
        My data is from Anthropic Not sure how it works under the hood, probably a better question for them Perhaps you are misunderstanding the entire premise of this project, this is not an LLM
        lmeyerov 6mo ago
        Maybe there's a fundamental miscommunication here of what evals are? Evals apply not just to LLMs but to skills, prompts, tools, and most things changing the behavior of compound AI systems, and especially like the productivity claims being put forth in this thread. The features in the post relate directly to heavily researched areas of agents that are regularly benchmarked and evaluated. They're not obscure, eg, another recent HN frontpage item benchmarked on research and planning.

Evals or GTFO

deleted 6mo ago

[deleted]

deleted 6mo ago

[deleted]

deleted 6mo ago

[deleted]

volatilityfund 6mo ago

5x productivity boost in merged PRs (lots of open PR & merge rate goes down, but net positive) Starting to build custom tooling around new "friction" points in dev cycle (eng IC perspective)

lmeyerov 6mo ago
Evals let us agree on the baseline, measurement, etc, and compare if simple things others do perform just as well. For same reason, instead of 'works on my box' and 'my coding style', use one of the many community evals vs making up your own benchmark. That helps head off much of many of the unfalsifiable discussions & claims happening and moves everyone forward.
- aid-ninja 6mo ago
  a rust version of that compiler (that the project runs on) ran at 480k claims/sec and it was able to deterministically resolve 83% of conflicts across 1 million concurrent agents (also 393,275x compression reduction @ 1m agents on input vs output, but different topics can make the compression vary) natively claude (and other LLM) will resolve conflicting claims at about 51% rate (based on internal research) the built in byzantine fault tolerance (again, in the compiler) is also pretty remarkable, it can correctly find the right answer even if 93% of the agents/data are malicious (with only 7% of agents/data telling us the correct information) basically the idea here is if you want to build autonomous at scale, you need to be able to resolve disagreement at scale and this project does a pretty nice job at doing that
  - lmeyerov 6mo ago
    My question was on claims like "5x productivity boost in merged PRs (lots of open PR & merge rate goes down, but net positive)", eg, does this change anything on swe-bench or any other standard coding eval?
    - volatilityfund 6mo ago
      The ecosystem is 8 tools plus a claude code plugin, the unlock was composing those tools (I don't regularly use all 9). The 5x claim was from /insights (claude code) Not for everyone, but it radically changed how I build. Senior engineer, 10+ years Now it's trivial to run multiple projects in parallel across claude sessions (this was not really manageable before using wheat) Genuinely don't remember the last time I opened a file locally
      - lmeyerov 6mo ago
        It sounds like the answer is "No, there is no repeatable eval of the core AI coding productivity claim, definitely not on one of the many AI coding benchmarks in the community used for understanding & comparison, and there will not be"
        volatilityfund 6mo ago
        My data is from Anthropic Not sure how it works under the hood, probably a better question for them Perhaps you are misunderstanding the entire premise of this project, this is not an LLM
        lmeyerov 6mo ago
        Maybe there's a fundamental miscommunication here of what evals are? Evals apply not just to LLMs but to skills, prompts, tools, and most things changing the behavior of compound AI systems, and especially like the productivity claims being put forth in this thread. The features in the post relate directly to heavily researched areas of agents that are regularly benchmarked and evaluated. They're not obscure, eg, another recent HN frontpage item benchmarked on research and planning.