Yeah, I believe that lots of startups don’t have evals either, but as soon as you get paying customers you’re gonna need something to prevent accidentally regre

Yeah, I believe that lots of startups don’t have evals either, but as soon as you get paying customers you’re gonna need something to prevent accidentally regressing as you tune your scaffolding, swap in newer models, etc. This is a big chasm that I could well believe a lot of founders fail to cross. It’s really easy to build an impressive-looking tech demo, much harder to get and retain paying customers and continuously improve. But! Plenty of companies are actually doing this hard work. See for example this post: https://news.ycombinator.com/item?id=46025683 https://news.ycombinator.com/item?id=46025683