TL;DR: There's a lot of really crappy, rushed, "AI" that accumulated the last 18 months. Yet AI is awesome. It's *really* stunning to live it. I cannot stress t

TL;DR: There's a lot of really crappy, rushed, "AI" that accumulated the last 18 months. Yet AI is awesome. It's *really* stunning to live it. I cannot stress this enough. You could probably do an GPT wrapper startup today, and as long as you don't rush, roundly beat every single incumbent within a year. Context: I'm a sole developer that quit my job at Google 7 months ago, decided to 18 months ago after getting first-hand look at how search x AI was being built. I just wrapped up 2 days of final benchmarking for release. 3-search-query RAG scores 97% on the USMLE, 6 points above "SOTA": Gemini 1.5 tuned for med that does 4 rounds of N answers each round, then 3 search queries to resolve differences. How am I possibly roundly beating that? Rushing. Lack of attention to basic details while just getting to the desired outcome, "we finetuned for med and beat GPT-4". And that's *Google*, infinitely resourced. And Perplexity? AI startup darling? 76%. That's the $40/month version. The free version is at 66%. I'm absolutely stunned, it didn't seem great, I didn't know it was actively much worse than just using ChatGPT. If that's how Google and Perplexity are going, I can't even imagine the shenanigans that are going down at companies like these. They don't have infinite resources, expertise, or have it as a core competency. There's probably more effort put into who gets to work on the AI thing than working on the AI thing. (I did legal & med benchmarking, for legal to compare TFA, perplexity free 58%, perplexity pro 67%, llama 8B w/no internet 65%, gpt4o x internet 90%)