4 ms·
I am in full agreement that LLMs themselves seem to be beginning to level out. Their capabilities do indeed appear to be following a sigmoid curve rather than a
by lukev 2y ago
I am in full agreement that LLMs themselves seem to be beginning to level out. Their capabilities do indeed appear to be following a sigmoid curve rather than an exponential one, which is entirely unsurprising.
That doesn't mean there's not a lot of juice left to squeeze out of what's available now. Not just from RAG and agent systems, but also integrating neuro-symbolic techniques.
We can do this already just with prompt manipulation and integration with symbolic compute systems: I gave a talk on this at Clojure Conj just the other week (https://youtu.be/OxzUjpihIH4 https://youtu.be/OxzUjpihIH4, apologies for the self promotion but I do think it's relevant.).
And that's just using existing LLMs. If we start researching and training them specifically for compatibility with neuro-symbolic data (e.g, directly tokenizing and embedding ontologies and knowledge graphs), it could unlock a tremendous amount of capability.
- joe_the_user 2y agoEven more, each earlier explosion of AI optimism involved tech that barely panned-out at all. For investors, something that's yielded things of significant utility, is yielding more and promises the potential of far more if X or Y hurdle is cleared, is a pretty appealing thing. I respect Marcus' analysis of the technology. But a lot of AI commentators have become habituated to shouting "AI winter" every time the tech doesn't live up to promises. Now that some substance is clearly present in AI, I can't imagine people stop trying to get a further payoff for the foreseeable future.
- cratermoon 2y ago> For investors, something that's yielded things of significant utility what exactly have investors gotten in return for their investment?
- PopePompus 2y agoA product which will significantly improve the productivity of programmers, if nothing else. That may not be a good return on investment, but I think it is undeniable that recent AI advances have nonzero value for coding.
- la64710 2y agoThen who cares ? If AI gets to do all the cool things and I am left to wash dishes and do my laundry. F** AI.
- jitl 2y agoRight now the programming AI that’s cheap enough for me to use is really good at fixing unbalanced parenthesis and indenting my code correctly. It more or less reduces my VIM motions by 80%. I am still doing something cool, and also still doing my laundry. Just the doing cool things part is a bit easier and less tedious. I think it makes me a fair bit more productive without robbing me of any agency.
- bathtub365 2y agoYou need AI to do basic code formatting? That’s been a feature of IDE’s and text editors for years.
- jitl 2y agoIve never met an IDE that automatically fixes unbalanced parens/braces or that auto-indents or completes bookkeeping comments like labeling SQL param indexes or something, this is stuff I’d use some VIM movement sequences before in Intellij or VS Code but now I just raise an eyebrow, it seems to intuit what I want, and then I press tab a few times ¯\_(ツ)_/¯ It’s kind of like how “learn to rank” used to eat the gains of all the specific optimizations Google used to do for search - before I used to use a bunch of plugins/workflows/explicit actions for a variety of text manipulation, now I just use tab; AI subsumed many of more specific or niche features.
- player1234 2y agoSounds like 100s of billions worth of a killer app. /s
- thefz 2y agoThose who think AI will make them better peogrammers make me think about the kind of day to day job they have. If a prompt is going to solve your problem, are you anything more than an entry level programmer? AI will not think for you and it's clear it is garbage against complexity.
- YetAnotherNick 2y agoI tracked ELO rating in Chatbot Arena for GPT-4/o series models over around 1.5 years(which are almost always highest rated), and at least on this metric it not only seems to be not stagnated, but also growth seems to be increasing[1] [1]: https://imgur.com/a/r5qgfQJ https://imgur.com/a/r5qgfQJ
- zamadatix 2y agoSomething seems quite off with the metric. Why would 4o recently increase on itself at a rate ~17x faster than 4o increased on 4 in that graph? E.g. ELO is a competitive metric, not an absolute metric, so someone could post the same graph with the claim the cause was "many new LLMs are being added to the system are not performing better than previous large models like they used to" (not saying it is or isn't, just saying the graph itself doesn't give context that LLMs are actually advancing at different rates or not).
- YetAnotherNick 2y agoChatbot arena also has H2H win rate for each pair of models for non tied results[1], so as to detect the global drift. e.g the gpt-4o released on 2024/09/03 wins 69% of the times with respect to gpt-4o released on 2024/05/13 in blind test. [1]: https://lmarena.ai/ https://lmarena.ai/