You have either too much time, or too much money, or both to curate code bases, to run 20x agent evals on those curated databases and spend time micro-optimisin

You have either too much time, or too much money, or both to curate code bases, to run 20x agent evals on those curated databases and spend time micro-optimising your agents... for those curated codebases. The moment you step outside of those curated codebases and run the agents against non-curated codebases? Well, no one knows. They may or may not work because the actual codebase may be similar to, or may be completely different from the curated one. And how do I know that it may not work? Well, let's turn to our friends at Anthropic: https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues https://www.anthropic.com/engineering/a-postmortem-of-three-... --- start quote --- When Claude generates text, it calculates probabilities for each possible next word, then randomly chooses a sample from this probability distribution. We use "top-p sampling" to avoid nonsensical outputs—only considering words whose cumulative probability reaches a threshold (typically 0.99 or 0.999). On TPUs, our models run across multiple chips, with probability calculations happening in different locations. To sort these probabilities, we need to coordinate data between chips, which is complex --- end quote --- So it's a probabilistic next word (which is quite likely to be different for a non-curated codebase), and there's top sampling, and then the complex sorting of probabilities, and on top of that are all the changes and bugs and limits and input/output transforms that Anthropic introduces. > share your specific case and I'll help you set it up. I have several side projects in Elixir. And at work we're developing a product that runs across 14 different (but similar, but different enough) platforms using the company's proprietary services. It's especially funny to see the claims of "oh just one more fine-tuning, bro, and everything will be gazillion times better" when I have already used and found issues with every "diligently researched" "guaranteed eval'ed" hype tool under the sun. This is just one of the results: https://x.com/dmitriid/status/1967306828418818217 https://x.com/dmitriid/status/1967306828418818217 Yours are unlikely to be any different.