3 ms·
> just prompt engineering This dismisses a lot of actual hard work. The scaffolding required to get SOTA performance is non-trivial! Eg how do you build repre
by theptip 11mo ago
> just prompt engineering
This dismisses a lot of actual hard work. The scaffolding required to get SOTA performance is non-trivial!
Eg how do you build representative evals and measure forward progress?
Also, tool calling, caching, etc is beyond what folks normally call “prompt engineering”.
If you think it’s trivial though - go build a startup and raise a seed round, the money is easy to come by if you can show results.
- submeta 11mo agoThis should be the top comment.
- jasonjmcghee 11mo agoprompt engineering + CRUD is likely much more fair. And many companies are "just CRUD".
- bayarearefugee 11mo agoThe money is easy to come by because wealthy investors, while they don't want to pay any more in taxes, are desperate to find possible returns in an economy that sucks outside of ballooning healthcare and the AI bubble... not because they need the money but because NUMBER MUST GO UP. And more so than even most VC markets, raising for an "AI" company is more about who you know than what results you can show. If anyone is actually showing significant results, where's the actual output of the AI-driven software boom (beyond just LLMs making coders more efficient by being a better google)? I don't see any real signs of it. All I see is people doing after market modifications on the shovels, I've yet to see any of the end users of these shovels coming down from the hills with sacks of real gold.
- theptip 11mo agoWhat’s your opinion on any of the plethora of unicorns in domain-specific AI, like Harvey? ($100m ARR from what I could find on a cursory search) https://www.forbes.com/sites/iainmartin/2025/10/29/legal-ai-startup-harvey-raises-150-million-at-8-billion-valuation/ https://www.forbes.com/sites/iainmartin/2025/10/29/legal-ai-...
- FridgeSeal 11mo agoI’m yet to be convinced it (Harvey) is anything other than a a prompt and some streamlined Rag. Law is slow and conservative, they were likely just the first to get a enterprise sales team.
- iLoveOncall 11mo ago> Eg how do you build representative evals and measure forward progress? This assumes that those companies do evaluations. In my experience, seeing a huge amount of internal AI projects at my company (FAANG), there's not even 5% that have any sort of eval in place.
- theptip 11mo agoYeah, I believe that lots of startups don’t have evals either, but as soon as you get paying customers you’re gonna need something to prevent accidentally regressing as you tune your scaffolding, swap in newer models, etc. This is a big chasm that I could well believe a lot of founders fail to cross. It’s really easy to build an impressive-looking tech demo, much harder to get and retain paying customers and continuously improve. But! Plenty of companies are actually doing this hard work. See for example this post: https://news.ycombinator.com/item?id=46025683 https://news.ycombinator.com/item?id=46025683
- iLoveOncall 11mo agoLol ok so all of this is just a ploy to advertise your thing xD
- biophysboy 11mo agoThis is like when people say that you should short the market if you think its going to crash. People have different risk premiums.
- ra 11mo agoI'm with you. I don't think anyone appreciates the effort that goes into a good measurable, repeatable eval / improvement process unless they've been through it in anger themselves.