4 ms·
Embedded code and systems code tends to have a lot of extra constraints the model doesn't know about and aren't necessarily written down where it can see. This
by pjc50 12d ago
Embedded code and systems code tends to have a lot of extra constraints the model doesn't know about and aren't necessarily written down where it can see. This produces a higher rate of "superficially plausible but doesn't actually work" code.
(There are probably wider benchmarks, but this was a definite finding in our company wide evaluation of effectiveness. LLMs aren't great at RTL either)