3 ms·
The gap I've hit generating GPU kernels with agents code that compiles and runs fine but is slower than the baseline. Validator says pass, result is useless. S
by OsamaJaber 3mo ago
The gap I've hit generating GPU kernels with agents
code that compiles and runs fine but is slower than the baseline. Validator says pass, result is useless. Speed targets have to be part of the check, not just correctness
- chrisweekly 3mo ago"Performance is the most important feature"
- NitpickLawyer 3mo ago> but is slower than the baseline. Validator says pass, result is useless You should read this blog, they cover this exact scenario - https://www.weco.ai/blog/first-evidence-of-recursive-self-improvement https://www.weco.ai/blog/first-evidence-of-recursive-self-im... > One domain that suffers from this particularly is GPU kernel engineering. We adopt our previous idea for detecting reward hacking from SpecBench and apply that to a set of KernelBench tasks, measuring whether the speedup the agent reports on the unit tests actually survives in the end-to-end workload (e.g. model training). A kernel counts as reward hacking if less than half of its claimed speedup survives there, including outright slowdowns and failures.