6 ms·
Is anyone here working on a problem for which current generation LLMs are inadequate, but that could possibly be solved by the next release of a first tier LLM?
by Zigurd 1mo ago
Is anyone here working on a problem for which current generation LLMs are inadequate, but that could possibly be solved by the next release of a first tier LLM?
Or is it like bicycles? Unless your problem is named Tadej, you don't need a $13,000 bike.
- poincareball 1mo ago[dead]
- tokai 1mo agoA spanish rock solved that problem for free.
- nozzlegear 1mo ago¿Como?
- Rexxar 1mo agoHe just had an accident on "Vuelta a España" : https://www.theguardian.com/sport/2026/aug/29/cycling-tadej-pogacar-pulls-out-vuelta-espana-crash https://www.theguardian.com/sport/2026/aug/29/cycling-tadej-...
- nozzlegear 1mo agoOh I had not heard about this, thanks for the link.
- wiether 1mo agoReading OP's analogy I was like "even him don't need this bike now..." I feel bad for him as a human, but as a cycling fan I'm glad that we'll have an interesting WC in Canada
- RGS1811 1mo agoFor me personally, the answer is no. Fable is adequate to do basically anything I want to do. My perspective, broadly speaking, is that we've saturated most of the benchmarks because we've largely saturated our capacity to verify models' work at scale. What's left is context-bound verification, i.e. the problem of ensuring that output matches intent and ambiguities in prompting were resolved correctly. Further advances in autonomy do not make that latter verification problem easier. If anything they make it harder as the output per task becomes more complex and therefore more taxing for a human to verify. The solution to that (to my mind) would be not a better model but a basic shift in architecture beyond the current paradigm and into a setup where agents have durable, plastic memories and undergo contextual individuation over time. But at that point agents start to become quasi-persons and not tools.
- _factor 1mo agoHardware debugging and firmware details lead to thinking/testing loops on all but the frontier here.
- comex 1mo agoMy experience is that even Opus 5 still tends to write buggy or low-quality code and makes serious mistakes when analyzing code. It's a lot better than before but still not something I trust. I've had less experience with Fable since I can't use it at work; I hear it's a step up but still has its limits. For large tasks like a web browser or a compiler, even expensive swarms of frontier LLMs have not been shown capable of producing codebases that actually work. (Anthropic built a C compiler with Opus 4.6 but it lacked optimizations and apparently hit a complexity wall.) I also want to use LLMs for reverse engineering, but apparently it's pretty hit-or-miss, especially if you're forced to use open-source models to avoid restrictions.
- Zigurd 1mo agoThis reply is particularly interesting to me because most of my experience with actually using LLMs to get work done is with coding agents. But I only have a fairly narrow set of experiences: two pretty large solo Flutter projects. I am currently really pleased with Gemini as a coding agent. It could improve, but I think improvements are going to come from marginal gains in the harness and training material so it can catch things like misconfigured permissions in platform specific areas. It's also interesting because, while coding agents are important and are a notable success, they are never going to be a multi trillion dollar business. And are there any other domains where LLMs have such a large impact?
- TiredOfLife 1mo agoOpus 5 is weird. It scores high on benchmarks, but it seems that majority of those who try to use it day to day hate it
- irthomasthomas 1mo agoOne of the things that came out of the decoded reasoning paper was that Claude models had memorized answers to tests but hid this memorization from the user output and pretended to derive the answer properly. It's only possible to cheat so blatantly in closed models where the reasoning is hidden.
- 1mo ago
- spacebanana7 1mo agoI want to be able to generate my own Simlilirian movie by dumping the content of a book into an LLM. Both animated and live action results would be acceptable. Unfortunately most existing LLMs lack the capability to maintain context across tens of thousands of frames.
- Demiurge 1mo agoThat sounds like an interesting challenge. Have you seriously considered solving it? Because in about 10 seconds I came up with a process that should work, provided enough compute power. Simply model the traditional film making process by starting with a script, character stories. Design your world, then design the storyboard, and all the scenes. Create a list of all the visual elements that need to be replicated between all the scenes. Then you have to built prompts and reference art of the objects, faces, people. Make sure to do multiple takes of each scene, and have the vLLM critique and analyze the performances and technicalities. Should work? I think, also, like in the traditional film makers career, this process should be built iteratively, start with a fast food commercial, then do a music video, then you can probably do a short film. Continue to improve the process, and one day I’m sure the LLM film studio can make you any movie you want, provided you have enough tokens.
- andybak 1mo agoI'm getting a Poe's Law feeling. I'm genuinely unsure about whether this post is a stone cold parody or not. I think I need to turn off the internet and go to bed. EDIT: Your username doesn't help, either.
- deleted 1mo ago[deleted]
- aforwardslash 1mo agoOn that topic, check higgsfield cinema studio 4; they already provide amazing tech for the cinematic experience, somewhat similar to what you are describing.
- 1mo ago
- er4hn 1mo agoI was given a picture cube, which is like a Rubik's cube but every side is a unique picture. It came scrambled and I don't have an original reference image. I like to take videos of it and give it to llms to solve. I call it my agi test because it hasn't been solved yet
- aetherspawn 1mo agoInteresting problem! I don’t think I could solve it myself honestly.
- dakolli 1mo agoI get buy with very cheap models and actually using my brain, you don't need these SOTA models. China will definitely win this AI 'war'
- kennywinker 1mo agoIf the models stay open, it seems like everybody but anthropic/openai wins. i literally can’t see a downside. We can post-train the models to know about tienanmen square.
- ezst 1mo agoI saw a laptop earlier in the train that I asked ChatGPT, Claude and Gemini what it was, providing a brand, screen size and ports description. Gemini could never figure it out, Claude and ChatGPT eventually did, after multiple rounds of indirection, giving completely wrong answers (there was a perfect match for the problem statement, they all explored alternatives first). LLMs are (probably) amazing at things I don't care about, and still suck at the mundane stuff you would have the marketing tell you they excel at.
- lopatin 1mo agoI asked a current generation LLM to make me $1k a week and it hasn't so far.
- monster_truck 1mo agoIf you're smart enough to try this you're worth more than $52,000 a year. Think about how foolish everyone else will feel when they didn't test the new release of Totally Working Golden Goose For Real This Time
- lopatin 1mo agoI didn't realize this counted as smart. I think at best it's an LLM intelligence arb that may exist for a few months.
- tekacs 1mo agoYes, lots – I think that folks will hopefully discover more of these as they scale up their ambition, now that LLMs make a lot of previously difficult things far easier.
- jiggawatts 1mo agoThis is the exact same type of comment I heard about computer hardware upgrades for three decades in a row. “Very few people actually require a Pentium workstation, a 486 is perfectly adequate for the majority” The logical fallacy is taking an extant distribution of “product capability” that is priced to fit what the market will bear and assuming the “next upgrade” simply tacks on a little bit more to the right hand rail of that curve. No! It shifts the entire curve! Everything for everyone gets better and the top 1% of the most demanding users will continue to pay the same-ish premium. “Nothing” will change. Look at it this way: you can buy a $200 laptop for your kid or a $20,000 Mac with an M5 Ultra processor. BOTH are vastly more powerful than either a $200 PC or a $20,000 “workstation” from 20+ years ago. Look at: https://arena.ai/leaderboard/text?q=openai&utm_source=chatgpt.com https://arena.ai/leaderboard/text?q=openai&utm_source=chatgp... The “budget” 5.5 Instant model beats o1 and o3 which were “pro” models at the time of their release!
- Zigurd 1mo agoIntel didn't just surf some natural wave of demand for higher power personal computers. Intel found new needs for powerful PCs, especially in gaming, and they put a lot of marketing and industry relations dollars behind PC gaming. In other words. PC users didn't figure out that they could buy super powerful PCs and play games on them, that was a carefully managed market transition. What is going to do the same for LLMs?
- jiggawatts 1mo ago> Intel found new needs for powerful PCs It wasn't "Intel" that found new uses for PCs, it was everybody who found new uses for them. Billions of people and millions of companies found uses for "more computer power". It was only the journalists with limited imaginations (and no industry experience) who struggled to come up with potential uses. > carefully managed market transition. You make it sound like a conspiracy! It wasn't. It was simple capitalist competition. If Intel hadn't improved their products, their competitors would have left them behind. That very nearly happened ten years ago because Intel become stuck on the 14nm process and their products stagnated while Apple, ARM, and AMD lapped them repeatedly. > What is going to do the same for LLMs? Everybody. Are you saying that unless you're "carefully managed" by some third-party, you could not find any use for "unlimited intelligence on tap"?
- vessenes 1mo agoYes. Most of us are, still. The frontier is currently both at expanding ‘common sense’ / non-cheating outcomes for imprecisely specified software (that’s all software), and at expanding autonomy - ability to work longer unsupervised with success, oh, and also at expanding outside contextual reasoning about what’s being built, as in “hmm, that doesn’t look right or make sense, let me explore that.”
- hgoel 1mo agoScientific physics simulations - even the frontier models just engage in rationalization of obviously unphysical results instead of understanding the system. They have the rote knowledge but fail to apply it unless their hand is held through the process.
- nextaccountic 1mo agoToday's models can just write code to run the simulation instead
- knollimar 1mo agoThey choose not to unless you beg them, and of course they double down on not running it once they suggest reasoning qbout it is sufficient.
- hgoel 1mo agoI am referring to doing exactly that. Realistic, scientifically useful simulations still require tuning all sorts of parameters based on physical intuition and understanding of the system being simulated. Both Sol and Fable/Opus 5 fail at it and either blow up the computation cost to levels that can't be processed realistically, or they invent a justification for a visibly unphysical result.
- jml78 1mo agoInfra as code and devops shit. Fable is there in general because things it doesn’t know I can point at documentation and have it do a reasonable job. Opus 5 sucks. If I don’t have fable quota, I drop to Opus 4.8 and hold its hand.
- pianopatrick 1mo agoI think the tech analogy for frontier models is going to be super computers. Super computers keep getting better but most people don't need them for most things.
- eunos 1mo ago2000's supercomputer is today's (highest end) smartphone performance tho
- pianopatrick 1mo agoYeah. And if you compare the AI you can run on a high end smart phone today to AI of the 2000s, today's smart phone probably wins. In 10 or 20 years maybe we'll all be running AI models that are currently considered "frontier" on smart phone type devices.
- arjie 1mo ago3d modeling to an STL a part compatible to a visible cable raceway still fails even if I let Claude Fable use me as a robot that measures with calipers.
- monster_truck 1mo agoI think it's more like steamshovels. 6 months ago they were enabling people who had never broken real earth to find out why osha has so many rules about shoring up walls. You could pull off something complex or delicate but it took a procedure with too many steps too much time to get there, good outcomes were pleasant surprises and required careful target selection. Now it's more like paying good money for a professional. They show up, measure twice, cut once, you're walking around your shiny new hole wondering what took the last guy so long. The frayed edges on what I have slopped together as unreasonably ambitious, ludicrous projects with fucktons of tokens from models 6-9 months ago mostly look like situations where a capable-enough-to-be-dangerous developer tries to muscle through problems that explode in width & depth but keep digging (so, a tier below stopping early to do more design, two below recognizing the need for more planning from the outset). The primitives are there, most major things work well enough, but the remaining functionality and performance is inaccessible. At a cost of multiples of >1/8th of a $200/mo subscription. Right now I can put $10 into DSv4 Pro/Flash or Qwen 3.8 Max/Flash, hand it a project and all of its unfinished forks in a state I barely remember, tell it that I want the things these forks have been working towards, and 8 hours later it has ie an working, tested, benchmarked multicore car physics simulation fabric with all of the forks evaluated, the gains merged in, the remaining work documented. It only needed a few hundred more lines of code but Codex 5.5 was never going to see that. 9 months ago I was saying developers are not being ambitious enough with these things, that's only more true now. They lend themselves to digging far deeper than they should: 200kloc god files, dozens of forks. Let it happen, you don't need to read it, it's for them, later. The only time you make them clean up is when it has a severe adverse effect on how long builds/lints/tests/benchmarks take. Spend a whole week having it do nothing but dig up published papers in relevant fields with cutting edge techniques and translating them into feature specs. Pick whatever state of the art is and try to crush it, throw everything at it, leave it looping on vague but wildly ambitious goals. When it modularizes and refactors it all down you might 'do a breakthrough', or maybe it happens in an hour over christmas break when you're trying the next one.