2 ms·
> Eg how do you build representative evals and measure forward progress? This assumes that those companies do evaluations. In my experience, seeing a huge amou
by iLoveOncall 11mo ago
> Eg how do you build representative evals and measure forward progress?
This assumes that those companies do evaluations. In my experience, seeing a huge amount of internal AI projects at my company (FAANG), there's not even 5% that have any sort of eval in place.
- theptip 11mo agoYeah, I believe that lots of startups don’t have evals either, but as soon as you get paying customers you’re gonna need something to prevent accidentally regressing as you tune your scaffolding, swap in newer models, etc. This is a big chasm that I could well believe a lot of founders fail to cross. It’s really easy to build an impressive-looking tech demo, much harder to get and retain paying customers and continuously improve. But! Plenty of companies are actually doing this hard work. See for example this post: https://news.ycombinator.com/item?id=46025683 https://news.ycombinator.com/item?id=46025683
- iLoveOncall 11mo agoLol ok so all of this is just a ploy to advertise your thing xD