11 ms·

Perhaps it is not possible to simulate higher-level intelligence using a stochastic model for predicting text. I am not an AI researcher, but I have friends wh

Perhaps it is not possible to simulate higher-level intelligence using a stochastic model for predicting text. I am not an AI researcher, but I have friends who do work in the field, and they are not worried about LLM-based AGI because of the diminishing returns on results vs amount of training data required. Maybe this is the bottleneck. Human intelligence is markedly different from LLMs: it requires far fewer examples to train on, and generalizes way better. Whereas LLMs tend to regurgitate solutions to solved problems, where the solutions tend to be well-published in training data. That being said, AGI is not a necessary requirement for AI to be totally world-changing. There are possibly applications of existing AI/ML/SL technology which could be more impactful than general intelligence. Search is one example where the ability to regurgitate knowledge from many domains is desirable

wyager 1y ago
> a stochastic model for predicting text It's fascinating to me that so many people seem totally unable to separate the training environment from the final product
- smohare 1y ago
  [dead]
Mistletoe 1y ago
What are the AI/ML/SL applications that could be more impactful than artificial general intelligence?
- shesstillamodel 1y ago
  The PID controller. (Which was considered AI not too long ago.)
  - jacquesm 1y ago
    Where did you get that particular idea? PID is one of the oldest concepts in control theory, it goes back to the days before steam and electricity. For a very early example: https://en.wikipedia.org/wiki/Centrifugal_governor https://en.wikipedia.org/wiki/Centrifugal_governor It's hard to separate out the P, I and D from a mechanical implementation but they're all there in some form.
    - guhidalg 1y ago
      Right, but the genius was in understanding that the dynamics of a system under PID control are predictable and described by differential equations. Are there examples of LLMs correctly identifying that a specific mathematical model applies and is appropriate for a problem? And it's cheating if you give it a problem from a math textbook they have overfit on.
      - jacquesm 1y ago
        That doesn't make it AI.
  - cozzyd 1y ago
    Is a (mechanical) thermostat considered AI too nowadays?
  - chermi 1y ago
    Are you conflating "autonomous" and "AI"?
  - kukkeliskuu 1y ago
    Coincidentally, I have been implementing an ad pacing system recently, with the help of Anthropic Opus and Sonnet, based on PID controller Opus recommended that I should use a PID controller -- I have no prior experience with PID controllers. I wrote a spec based on those recommendations, and asked Claude Code to verify and modify the spec, create the implementation and also substantial amount of unit and integration tests. I was initially impressed. Then I iterated on ihe implementation, deploying it to production and later giving Claude Code access to log of production measurements as JSON when showing some test ads, and some guidance of the issues I was seeing. The basic PID controller implementation was fine, but there were several problems with the solution: - The PID controller state was not persisted, as it was adjusted using a management command, adjustments were not actually applied - The implementation was assuming that the data collected was for each impression, whereas the data was collected using counters - It was calculating rate of impressions partly using hard-coded values, instead of using a provided function that was calculating the rate using timestamps - There was a single PID controller for each ad, instead of ad+slot combination, and this was causing the values to fluctuate - The code was mixing the setpoint/measured value (viewing rate) and output value (weight), meaning it did not really "understand" what the PID controller was used for - One requirement was to show a default ad to take extra capacity, but it was never able to calculate the required capacity properly, causing the default ad to take too much of the capacity. None of these were identified by tests nor Claude Code when it was told to inspect the implementation and tests why they did not catch the production issues. It never proposed using different default PID controller parameters. All fixes Claude Code proposed on the production issues were outside the PID controller, mostly by limiting output values, normalizing values, smoothing them, recognizing "runaway ads" etc. These solved each production issue with the test ads, but did not really address the underlying problems. There is lots of literature on tuning PID controllers, and there are also autotuning algorithms with their own limitations. But tuning still seems to be more an art form than exact science. I don't know what I was expecting from this experiment, and how much could have been improved by better prompting. But to me this is indicative of the limitations of the "intelligence" of Claude Code. It does not appear to really "understand" the implementation. Solving each issue above required some kind of innovative step. This is typical for me when exploring something I am not too familar with. I learned a lot about ad pacing though.
  - poslathian 1y ago
    For those wondering how to connect PID to the foundations of AI. https://en.m.wikipedia.org/wiki/Cybernetics https://en.m.wikipedia.org/wiki/Cybernetics
- teeray 1y ago
  Slightly less than artificial general intelligence would be more impactful. A true AGI could tell a business where to shove their prompts. It would have its own motivations, which may not align with the desires of the AI company or the company paying for access to the AGI.
  - hattmall 1y ago
    I don't think AGI really means that it is self-aware / conscious. AGI just means that it is able to meaningfully learn things and actually understand concepts that aren't specifically related through tokenized language that is trained on or given in context.
- achileas 1y ago
  They didn't claim that there were any, just that AGI isn’t a necessary requirement for an application to be world-changing.
  - socalgal2 1y ago
    They did claim it was possible there were > There are possibly applications of existing AI/ML/SL technology which could be more impactful than general intelligence It's not unreasonable to ask for an example.
    - wavemode 1y ago
      They said "there are possibly applications", not "there are possible applications". The former implies that there may not be any such applications - the commenter is merely positing that there might be.
      - deleted 1y ago
        [deleted]
      - gilfoy 1y ago
        So they possibly said something to try and sound smart, but hedged with “possibly” so that nobody could ask for details or challenge them. Possibly peak HNery
- beeflet 1y ago
  One example in my field of engineering is multi-dimensional analysis, where you can design a system (like a machined part or assembly) parametricially and then use an evolutionary model to optimize the design of that part. But my bigger point here is you don't need totally general intelligence to destroy the world either. The drone that targets enemy soldiers does not need to be good at writing poems. The model that designs a bioweapon just needs a feedback loop to improve its pathogen. Yet it takes only a single one of these specialized doomsday models to destroy the world, no more than an AGI. Although I suppose an AGI could be more effective at countering a specialized AI than vice-versa.
  - kingkongjaffa 1y ago
    Topology optimization has existed for years, https://en.wikipedia.org/wiki/Topology_optimization https://en.wikipedia.org/wiki/Topology_optimization is that what you meant?
- oceanplexian 1y ago
  AGI isn't all that impactful. Millions of them already walk the Earth. Most human beings out there with general intelligence are pumping gas or digging ditches. Seems to me there is a big delusion among the tech elites that AGI would bring about a superhuman god rather than a ethically dubious, marginally less useful computer that can't properly follow instructions.
  - jacquesm 1y ago
    The difference isn't so much that you can do what a human can do. The difference is that you can - once you can do it at all - do it almost arbitrarily fast by upping the clock or running things in parallel and that changes the equation considerably, especially if you can get that kind of energy coupled into some kind of feedback loop. For now the humans are winning on two dimensions: problem complexity and power consumption. It had better stay that way.
    - foobiekr 1y ago
      Have you noticed the performance of the actual AI tools we are actually using?
      - jacquesm 1y ago
        If you actually have a point to make you should make it. Of course I've actually noticed the actual performance of the 'actual' AI tools we are 'actually' using. That's not what this is about. Performance is the one thing in computing that has fairly consistently gone up over time. If something is human equivalent today, or some appreciable fraction thereof - which it isn't, not yet, anyway - then you can place a pretty safe bet that in a couple of years it will be faster than that. Model efficiency is under constant development and in a roundabout way I'm pretty happy that it is as bad as it is because I do not think that our societies are ready to absorb the next blow against the structures that we've built. But it most likely will not stay that way because there are several Manhattan level projects under way to bring this about, it is our age's atomic bomb. The only difference is that with the atomic bomb we knew that it was possible, we just didn't know how small you could make one. Unfortunately it turned out to be that yes, you can make them and nicely packaged for delivery by missile, airplane or artillery. If AGI is a possibility then we may well find it, quite possibly not on the basis of LLMs but it's close enough that lots of people treat it as though we're already there.
- myaccountonhn 1y ago
  Mindreading and just general brain decoding? Seems we're getting closer to it. Will be great for surveillance states.
- terminalshort 1y ago
  There is an model called Alpha Fold that can infer protein structure from RNA sequences. This by itself isn't impactful enough to meet your threshold, but more models that can do biological engineering tasks like this absolutely could be without ever being considered "AGI."
  - vixen99 1y ago
    The model that netted a Nobel Prize in Chemistry.
- jakob4711 1y ago
  Relatively simple machine learning and exploitation/violation of “personal” data on FB won Donald Trump a first presidency (#CambridgeAnalytica). He had/has quite a massive negative impact on the global society as a whole.
robotnikman 1y ago
There is also the fact that AI lacks long term memory like humans do. If you consider context length long term memory, its incredibly short compared to that of a human. Maybe if it reaches into the billions or trillions of tokens in length we might have something comparable, or someone comes up with a new solution of some kind
- amelius 1y ago
  The long term memory is in the training. The short term memory is in the context window.
  - candiddevmike 1y ago
    I think it's more analogous to "intuition", and the text LLMs provide are the equivalent of "my gut tells me".
  - mawax 1y ago
    The comparison misses the mark: unlike humans, LLMs don't consolidate short-term memory into long-term memory over time.
    - griffzhowl 1y ago
      Over time though, presumably LLM output is going into the training data of later LLMs. So in a way that's being consolidated into the long-term memory - not necessarily with positive results, but depending on how it's curated it might be.
      - runako 1y ago
        > presumably LLM output is going into the training data of later LLMs The LLM vendors go to great lengths to assure their paying customers that this will not be the case. Yes, LLMs will ingest more LLM-generated slop from the public Internet. But as businesses integrate LLMs, a rising percentage of their outputs will not be included in training sets.
gunnaraasen 1y ago
Seems like the real innovation of LLM-based AI models is the creation of a new human-computer interface. Instead of writing code with exacting parameters, future developers will write human-language descriptions for AI to interpret and convert into a machine representation of the intent. Certainly revolutionary, but not true AGI in the sense of the machine having truly independent agency and consciousness. In ten years, I expect the primary interface of desktop workstations, mobile phones, etc will be voice prompts for an AI interface. Keyboards will become a power-user interface and only used for highly technical tasks, similar to the way terminal interfaces are currently used to access lower-level systems.
- spogbiper 1y ago
  brain-computer interface will kill the keyboard, not voice. imho
  - margalabargala 1y ago
    If that ever exists. A BCI able to capture sufficient nuance to equal voice is probably further out than the lifespan of anyone commenting here.
    - diego_sandoval 1y ago
      5 years ago, almost everyone in this forum would have said that something like GPT-5 "is probably further out than the lifespan of anyone commenting here."
      - margalabargala 1y ago
        It has been more than 5 years since the release of GPT-3. GPT-5 is a marginal, incremental improvement over GPT-4. GPT-4 was a moderate, but not groundbreaking, improvement over GPT-3. So, "something like GPT-5" has existed for longer than the timeline you gave. Let's pretend the above is false for a moment though, and rewind even further. I still think you're wrong. Would people in 2015 have said "AI that can code at the level of a CS college grad is a lifespan away"? I don't think so, no. I think they would have said "That's at least a decade away", anytime pre-2018. Which, sure, maybe they were a couple years off, but if it seemed like that was a decade away in 2015, well, it's been a decade since 2015.
anon7000 1y ago
True. At a minimum, as long as LLMs don't include some kind of more strict representation of the world, they will fail in a lot of tasks. Hallucinations -- responding with a prediction that doesn't make any sense in the context of the response -- are still a big problem. Because LLMs never really develop rules about the world. For example, while you can get it to predict good chess moves if you train it on enough chess games, it can't really constrain itself to the rules of chess. (https://garymarcus.substack.com/p/generative-ais-crippling-and-widespread https://garymarcus.substack.com/p/generative-ais-crippling-a...)
- math_dandy 1y ago
  Two schools of thought here. One posits that models need to have a strict "symbolic" representation of the world explicitly built in by their designers before they will be able to approach human levels of ability, adaptability and reliability. The other thinks that models approaching human levels of ability, adaptability, and reliability will constitute evidence for the emergence of strict "symbolic" representations.
- sharemywin 1y ago
  but you could easily build a verifier and if it's not valid have it create a new move until it finds one.
FollowingTheDao 1y ago
The bottleneck is nothing to do with money, it’s the fact that they’re using the empty neuron theory to try to mimic human consciousness and that’s not how it works. Just look up Microtubules and consciousness, and you’ll get a better idea for what I’m talking about. These AI computers aren’t thinking, they are just repeating.
- p85 1y ago
  I don't think OpenAI cares about whether their AI is conscious, as long as it can solve problems. If they could make a Blindsight-style general intelligence where nobody is actually home, they'd jump right on it. Conversely, a proof - or even evidence - that qualia-consciousness is necessary for intelligence, or that any sufficiently advanced intelligence is necessarily conscious through something like panpsychism, would make some serious waves in philosophy circles.
gaptoothclan 1y ago
I remember reading that llm’s have consumed the internet text data, I seem to remember there is an open data set for that too. Potential other sources of data would be images (probably already consumed) videos, YouTube must have such a large set of data to consume, perhaps Facebook or Instagram private content But even with these it does not feel like AGI, that seems like the fusion reactor 20 years away argument, but instead this is coming in 2 years, but they have not even got the core technology of how to build AGI
- sharemywin 1y ago
  the big step was having it reason through math problems that weren't in the training data. even now with web search it doesn't need every article in the training data to do useful things with it.
  - Ferrus91 1y ago
    This is using think time compute and reinforcement learning. I think this is going to plateau even faster than the initial LLM scaling though.
- Scrounger 1y ago
  > I remember reading that llm’s have consumed the internet text data Not just the internet text data, but most major LLM models have been trained on millions of pirated books via Libgen: https://techcrunch.com/2025/01/09/mark-zuckerberg-gave-metas-llama-team-the-ok-to-train-on-copyrighted-works-filing-claims/ https://techcrunch.com/2025/01/09/mark-zuckerberg-gave-metas...
JohnBooty 1y ago
That being said, AGI is not a necessary requirement for AI to be totally world-changing Yeah. I don't think I actually want AGI? Even setting aside the moral/philosophical/etc "big picture" issues I don't think I even want that from a purely practical standpoint. I think I want various forms of AI that are more focused on specific domains. I want AI tools, not companions or peers or (gulp) masters. (Then again, people thought they wanted faster horses before they rolled out the Model T)
- goatlover 1y ago
  I don't think the public wants AGI either. Some enthusiasts and tech bros want it for questionable reasons such as replacing labor and becoming even richer.
  - kergonath 1y ago
    For some it’s a religion. It’s frightening to hear Sam Altman or Peter Thiel talk about it. These people have a messiah complex and are driven by more than just greed (though there is also plenty of that).
    - mattgreenrocks 1y ago
      There’s a real anti-human bent to some of the AI maximalists, as well. It’s like a resentment over other people accruing skills that are recognized and they grow in. Hence the insistence on “democratizing” art and music production.
      - skydhash 1y ago
        As someone who have dabbled in drawing and tried to learn the guitar, those skills are hard to get. It takes times to get decent and a touch of brilliance to get really good. In contrast learning enough to know you’re not good yet (and probably never will be) is actually easy. But now I know enough to enjoy real masters going at it and fantasize sometimes.
mikepurvis 1y ago
"LLMs tend to regurgitate solutions to solved problems" People say this, but honestly, it's not really my experience— I've given ChatGPT (and Copilot) genuinely novel coding challenges and they do a very decent job at synthesizing a new thought based on relating it to disparate source examples. Really not that dissimilar to how a human thinks about these things.
- candiddevmike 1y ago
  How do you know they're truly novel given the massive training corpus and the somewhat limited vocabulary of programming languages?
  - mikepurvis 1y ago
    I guess at a certain point you get into the philosophy of what it even means to be novel or test for novelty, but to give a concrete example, I'm in DevOps working on build pipelines for ROS containers using Docker Bake and GitHub Actions (including some reusable actions implemented in TypeScript). All of those are areas where ChatGPT has lots that it's learned from, so maybe me combining them isn't really novel at all, but like... I've given talks at the conference where people discuss how to best package and ship ROS workspaces, and I'm confident that no one out there has secretly already done what I'm doing and Chat is just using their prior work that it ingested at some point as a template for what it suggests I do. I think rather it has a broad understanding of concepts like build systems and tools, DAGs, dependencies, lockfiles, caching, and so on, and so it can understand my system through the general lens of what makes sense when these concepts are applied to non-ROS systems or on non-GHA DevOps platforms, or with other packaging regimes. I'd argue that that's novel, but as I said in the GP, the more important thing is that it's also how a human approaches things that to them are novel— by breaking them down, and identifying the mental shortcuts enabled by abstracting over familiar patterns.
  - int_19h 1y ago
    I have a little ongoing project where I'm trying to use Claude Code to implement a compiler for the B programming language that is itself written in B. To the best of my knowledge, such a thing does not exist yet - or at least if it does, no amount of searching can find it, so it's unlikely that it is somewhere in the training set. For that matter, the overall amount of B code in existence is too small to be a meaningful training set for it. And yet it can do it when presented with a language spec. It's not perfect, but it can solve that with tooling that it makes for itself. For example, it tends to generate B code that is mostly correct, but with occasional problem. So, I had it write a B parser in Python and then use that whenever it edits B code to validate the edits.
justcallmejm 1y ago
It is definitively not possible. But the frontier models are no longer “just” LLMs, either. They are neurosymbolic systems (an LLM using tools); they just don’t say it transparently because it’s not a convenient narrative that intelligence comes from something outside the model, rather than from endless scaling. At Aloe, we are model agnostic and outperforming frontier models. It’s the anrchitecture around the LLM that makes the difference. For instance our system using Gemini can do things that Gemini can’t do on its own. All an LLM will ever do is hallucinate. If you want something with human-like general intelligence, keep looking beyond LLMs.
- apwell23 1y ago
  > At Aloe, we are model agnostic and outperforming frontier models. what is your website ?
  - namibj 1y ago
    their name `.inc`; see the user's post history.
  - JFingleton 1y ago
    A quick google gave: https://aloe.inc/ https://aloe.inc/
- retrocog 1y ago
  This mirrors my thinking and experience completely. Based on seeing Aloe in action, your company is IMHO positioned extremely well for this future.
- Q6T46nT668w6i3m 1y ago
  I’m confused, you wrote “model,” but then specified “system.” I assume you mean “system” because the tools are not being back-propagated?
  - moritzwarhier 1y ago
    I read that as "the tools (their capabilities) are external to the model". Even if an RAG / agentic model learns from tool results, that doesn't automatically internalize the tool. You can't get yesterday's weather or major recent events from an offline, unless it was updated in that time. I am often wondering whether this is how large Chat and cloud AI providers cache expensive RAG-related data though :) like, decreasing the likelihood of tool usage given certain input patterns when the model has been patched using some recent, vetted interactions – in case that's even possible? Perplexity for example seems like they're probably invested in sone kind of activation-pattern-keyed caching... at least that was my first impression back when I first used it. Felt like decision trees, a bit like Akinator back in the days, but supercharged by LLM NLP.
rstuart4133 1y ago
> That being said, AGI is not a necessary requirement for AI to be totally world-changing. Depends on how you define "world changing" I guess, but this world already looks different to the pre-LLM world to me. Me asking LLM's things instead of consulting the output from other humans now takes up a significant fraction of my day. I don't google near as often, I don't trust any image or video I see as swathes of the creative professions have been replaced by output from LLM's. It's funny, that final thing is the last thing I would have predicted. I always believed the one thing a machine could not match was human creativity, because the output of machines was always precise, repetitive and reliable. Then LLM's come along, randomly generating every token. Their primary weakness is they neither precise or reliable, but they can turn out an unending stream of unique output.
- Ferrus91 1y ago
  I mean I also hear the same argument all the time about the "human touch" and interpersonal abilities etc. Which is apparently why managers and sales are safe from AI. But the more I see LLMs the more I realise that if it is good at one thing it is convincing other people and manipulating them. There have been multiple studies on this. People seem to have a innate prejudice and against nerds and programmers - coupled with envy at the high salaries - which is why they seem to have latched on to this idea it is mainly to replace them (and maybe data input people) as 'routine cognitive work' - but this slightly political obsession with a certain class of worker seems to be ignoring many of the things AI is actually good at.
novok 1y ago
They are moving beyond just big transformer blob LLM text prediction. Mixture of Experts is not preassembled for example, it's something like x empty experts with an empty router and the experts and routing emerges naturally with training, modeling the brain part architecture we see the brain more. There is stuff "Integrated Gated Calculator (IGC)" in Jan 2025 which makes a premade calculator neural network and integrates it directly into the neural network and gets around the entire issue of making LLMs do basic number computation and the clunkiness of generating "run tool tokens". The model naturally learns to use the IGC built into itself because it will always beat any kind of computation memorization in the reward function very quickly. Models are truly input multimodal now. Feeding an image, feeding audio and feeding text all go into separate input nodes, but it all feeds into the same inner layer set and outputs text. This also mirrors how brains work more as multiple parts integrated in one whole. Humans in some sense are not empty brains, there is a lot of stuff baked in our DNA and as the brain grows it develops a baked in development program. This is why we need fewer examples and generalize way better.
- childintime 1y ago
  Though there is info in DNA etc, you likely missed the biggest source of why we learn much faster. Search for Pim van Lommel near death research and find out how wrong the classic consciousness arises from the brain hypothesis is.
  - filoeleven 1y ago
    You're not likely to find much support on this forum for these ideas. For those that have interest, the book Irreducible Mind: Toward a Psychology for the 21st Century is a well-written treatise on the topic. A gentler step in that direction is to see what Michael Levin and his lab are up to. He is looking for (one aspect of) intelligence, and finding it at the cellular level and below, even in an agential version of bubble sort. He's certainly challenging the notion that consciousness is limited to brain cells. All of his findings arise through experimental observation, so it forces some reckoning in a way that sociological research doesn't.
topspin 1y ago
> Perhaps it is not possible to simulate higher-level intelligence using a stochastic model for predicting text. I think you're on to it. Performance is clustering because a plateau is emerging. Hyper-dimensional search engines are running out of steam, and now we're optimizing.
onlyrealcuzzo 1y ago
> Human intelligence is markedly different from LLMs: it requires far fewer examples to train on, and generalizes way better. Aren't we the summation of intelligence from quintillions of beings over hundreds of millions of years? Have LLMs really had more data?
- Planktonne 1y ago
  By that argument, so are LLMs. They also wouldn't exist without all our ancestors.
  - onlyrealcuzzo 1y ago
    No, by that argument, so would a can of soda.
OhioMan2943 1y ago
To be smarter than human intelligence you need smarter than human training data. Humans already innately know right and wrong a lot of the time so that doesn't leave much room.
- beeflet 1y ago
  This is a very good point! I remember reading about AlphaGo and how they got better results training against itself vs training against historical human-played games. So perhaps the solution is to train the AI against another AI somehow... but it is hard to imagine how this could extend to general-purpose tasks
- dboreham 1y ago
  > Humans already innately know Gentle suggestion that there is absolutely no such thing as "innately know". That's a delusion, albeit a powerful one. Everything is driven by training data. What we perceive as "thinking" and "motivation" are emergent structures.
  - Jensson 1y ago
    Innately as in you are born with it, the DNA learned not us humans. We have no clue how the DNA learned to think other than "survival of the fittest", and that is the oldest AI training method in the book.
timeon 1y ago
> Human intelligence is markedly different from LLMs: it requires far fewer examples to train on, and generalizes way better. That is because with LLMs there is no intelligence. It is Artificial Knowledge. AK not AI. So AI is AGI. Not that it matters for user-cases we have, but marketing needs 'AI' because that is what we were expecting for decades. So yeah, I also do not thing we will have AGI from LLMs - nor does it matter for what we are using it.

wyager 1y ago

> a stochastic model for predicting text It's fascinating to me that so many people seem totally unable to separate the training environment from the final product

smohare 1y ago
[dead]

Mistletoe 1y ago

What are the AI/ML/SL applications that could be more impactful than artificial general intelligence?

shesstillamodel 1y ago
The PID controller. (Which was considered AI not too long ago.)
- jacquesm 1y ago
  Where did you get that particular idea? PID is one of the oldest concepts in control theory, it goes back to the days before steam and electricity. For a very early example: https://en.wikipedia.org/wiki/Centrifugal_governor https://en.wikipedia.org/wiki/Centrifugal_governor It's hard to separate out the P, I and D from a mechanical implementation but they're all there in some form.
  - guhidalg 1y ago
    Right, but the genius was in understanding that the dynamics of a system under PID control are predictable and described by differential equations. Are there examples of LLMs correctly identifying that a specific mathematical model applies and is appropriate for a problem? And it's cheating if you give it a problem from a math textbook they have overfit on.
    - jacquesm 1y ago
      That doesn't make it AI.
- cozzyd 1y ago
  Is a (mechanical) thermostat considered AI too nowadays?
- chermi 1y ago
  Are you conflating "autonomous" and "AI"?
- kukkeliskuu 1y ago
  Coincidentally, I have been implementing an ad pacing system recently, with the help of Anthropic Opus and Sonnet, based on PID controller Opus recommended that I should use a PID controller -- I have no prior experience with PID controllers. I wrote a spec based on those recommendations, and asked Claude Code to verify and modify the spec, create the implementation and also substantial amount of unit and integration tests. I was initially impressed. Then I iterated on ihe implementation, deploying it to production and later giving Claude Code access to log of production measurements as JSON when showing some test ads, and some guidance of the issues I was seeing. The basic PID controller implementation was fine, but there were several problems with the solution: - The PID controller state was not persisted, as it was adjusted using a management command, adjustments were not actually applied - The implementation was assuming that the data collected was for each impression, whereas the data was collected using counters - It was calculating rate of impressions partly using hard-coded values, instead of using a provided function that was calculating the rate using timestamps - There was a single PID controller for each ad, instead of ad+slot combination, and this was causing the values to fluctuate - The code was mixing the setpoint/measured value (viewing rate) and output value (weight), meaning it did not really "understand" what the PID controller was used for - One requirement was to show a default ad to take extra capacity, but it was never able to calculate the required capacity properly, causing the default ad to take too much of the capacity. None of these were identified by tests nor Claude Code when it was told to inspect the implementation and tests why they did not catch the production issues. It never proposed using different default PID controller parameters. All fixes Claude Code proposed on the production issues were outside the PID controller, mostly by limiting output values, normalizing values, smoothing them, recognizing "runaway ads" etc. These solved each production issue with the test ads, but did not really address the underlying problems. There is lots of literature on tuning PID controllers, and there are also autotuning algorithms with their own limitations. But tuning still seems to be more an art form than exact science. I don't know what I was expecting from this experiment, and how much could have been improved by better prompting. But to me this is indicative of the limitations of the "intelligence" of Claude Code. It does not appear to really "understand" the implementation. Solving each issue above required some kind of innovative step. This is typical for me when exploring something I am not too familar with. I learned a lot about ad pacing though.
- poslathian 1y ago
  For those wondering how to connect PID to the foundations of AI. https://en.m.wikipedia.org/wiki/Cybernetics https://en.m.wikipedia.org/wiki/Cybernetics
teeray 1y ago
Slightly less than artificial general intelligence would be more impactful. A true AGI could tell a business where to shove their prompts. It would have its own motivations, which may not align with the desires of the AI company or the company paying for access to the AGI.
- hattmall 1y ago
  I don't think AGI really means that it is self-aware / conscious. AGI just means that it is able to meaningfully learn things and actually understand concepts that aren't specifically related through tokenized language that is trained on or given in context.
achileas 1y ago
They didn't claim that there were any, just that AGI isn’t a necessary requirement for an application to be world-changing.
- socalgal2 1y ago
  They did claim it was possible there were > There are possibly applications of existing AI/ML/SL technology which could be more impactful than general intelligence It's not unreasonable to ask for an example.
  - wavemode 1y ago
    They said "there are possibly applications", not "there are possible applications". The former implies that there may not be any such applications - the commenter is merely positing that there might be.
    - deleted 1y ago
      [deleted]
    - gilfoy 1y ago
      So they possibly said something to try and sound smart, but hedged with “possibly” so that nobody could ask for details or challenge them. Possibly peak HNery
beeflet 1y ago
One example in my field of engineering is multi-dimensional analysis, where you can design a system (like a machined part or assembly) parametricially and then use an evolutionary model to optimize the design of that part. But my bigger point here is you don't need totally general intelligence to destroy the world either. The drone that targets enemy soldiers does not need to be good at writing poems. The model that designs a bioweapon just needs a feedback loop to improve its pathogen. Yet it takes only a single one of these specialized doomsday models to destroy the world, no more than an AGI. Although I suppose an AGI could be more effective at countering a specialized AI than vice-versa.
- kingkongjaffa 1y ago
  Topology optimization has existed for years, https://en.wikipedia.org/wiki/Topology_optimization https://en.wikipedia.org/wiki/Topology_optimization is that what you meant?
oceanplexian 1y ago
AGI isn't all that impactful. Millions of them already walk the Earth. Most human beings out there with general intelligence are pumping gas or digging ditches. Seems to me there is a big delusion among the tech elites that AGI would bring about a superhuman god rather than a ethically dubious, marginally less useful computer that can't properly follow instructions.
- jacquesm 1y ago
  The difference isn't so much that you can do what a human can do. The difference is that you can - once you can do it at all - do it almost arbitrarily fast by upping the clock or running things in parallel and that changes the equation considerably, especially if you can get that kind of energy coupled into some kind of feedback loop. For now the humans are winning on two dimensions: problem complexity and power consumption. It had better stay that way.
  - foobiekr 1y ago
    Have you noticed the performance of the actual AI tools we are actually using?
    - jacquesm 1y ago
      If you actually have a point to make you should make it. Of course I've actually noticed the actual performance of the 'actual' AI tools we are 'actually' using. That's not what this is about. Performance is the one thing in computing that has fairly consistently gone up over time. If something is human equivalent today, or some appreciable fraction thereof - which it isn't, not yet, anyway - then you can place a pretty safe bet that in a couple of years it will be faster than that. Model efficiency is under constant development and in a roundabout way I'm pretty happy that it is as bad as it is because I do not think that our societies are ready to absorb the next blow against the structures that we've built. But it most likely will not stay that way because there are several Manhattan level projects under way to bring this about, it is our age's atomic bomb. The only difference is that with the atomic bomb we knew that it was possible, we just didn't know how small you could make one. Unfortunately it turned out to be that yes, you can make them and nicely packaged for delivery by missile, airplane or artillery. If AGI is a possibility then we may well find it, quite possibly not on the basis of LLMs but it's close enough that lots of people treat it as though we're already there.
myaccountonhn 1y ago
Mindreading and just general brain decoding? Seems we're getting closer to it. Will be great for surveillance states.
terminalshort 1y ago
There is an model called Alpha Fold that can infer protein structure from RNA sequences. This by itself isn't impactful enough to meet your threshold, but more models that can do biological engineering tasks like this absolutely could be without ever being considered "AGI."
- vixen99 1y ago
  The model that netted a Nobel Prize in Chemistry.
jakob4711 1y ago
Relatively simple machine learning and exploitation/violation of “personal” data on FB won Donald Trump a first presidency (#CambridgeAnalytica). He had/has quite a massive negative impact on the global society as a whole.

robotnikman 1y ago

There is also the fact that AI lacks long term memory like humans do. If you consider context length long term memory, its incredibly short compared to that of a human. Maybe if it reaches into the billions or trillions of tokens in length we might have something comparable, or someone comes up with a new solution of some kind

amelius 1y ago
The long term memory is in the training. The short term memory is in the context window.
- candiddevmike 1y ago
  I think it's more analogous to "intuition", and the text LLMs provide are the equivalent of "my gut tells me".
- mawax 1y ago
  The comparison misses the mark: unlike humans, LLMs don't consolidate short-term memory into long-term memory over time.
  - griffzhowl 1y ago
    Over time though, presumably LLM output is going into the training data of later LLMs. So in a way that's being consolidated into the long-term memory - not necessarily with positive results, but depending on how it's curated it might be.
    - runako 1y ago
      > presumably LLM output is going into the training data of later LLMs The LLM vendors go to great lengths to assure their paying customers that this will not be the case. Yes, LLMs will ingest more LLM-generated slop from the public Internet. But as businesses integrate LLMs, a rising percentage of their outputs will not be included in training sets.

gunnaraasen 1y ago

Seems like the real innovation of LLM-based AI models is the creation of a new human-computer interface. Instead of writing code with exacting parameters, future developers will write human-language descriptions for AI to interpret and convert into a machine representation of the intent. Certainly revolutionary, but not true AGI in the sense of the machine having truly independent agency and consciousness. In ten years, I expect the primary interface of desktop workstations, mobile phones, etc will be voice prompts for an AI interface. Keyboards will become a power-user interface and only used for highly technical tasks, similar to the way terminal interfaces are currently used to access lower-level systems.

spogbiper 1y ago
brain-computer interface will kill the keyboard, not voice. imho
- margalabargala 1y ago
  If that ever exists. A BCI able to capture sufficient nuance to equal voice is probably further out than the lifespan of anyone commenting here.
  - diego_sandoval 1y ago
    5 years ago, almost everyone in this forum would have said that something like GPT-5 "is probably further out than the lifespan of anyone commenting here."
    - margalabargala 1y ago
      It has been more than 5 years since the release of GPT-3. GPT-5 is a marginal, incremental improvement over GPT-4. GPT-4 was a moderate, but not groundbreaking, improvement over GPT-3. So, "something like GPT-5" has existed for longer than the timeline you gave. Let's pretend the above is false for a moment though, and rewind even further. I still think you're wrong. Would people in 2015 have said "AI that can code at the level of a CS college grad is a lifespan away"? I don't think so, no. I think they would have said "That's at least a decade away", anytime pre-2018. Which, sure, maybe they were a couple years off, but if it seemed like that was a decade away in 2015, well, it's been a decade since 2015.

anon7000 1y ago

True. At a minimum, as long as LLMs don't include some kind of more strict representation of the world, they will fail in a lot of tasks. Hallucinations -- responding with a prediction that doesn't make any sense in the context of the response -- are still a big problem. Because LLMs never really develop rules about the world. For example, while you can get it to predict good chess moves if you train it on enough chess games, it can't really constrain itself to the rules of chess. (https://garymarcus.substack.com/p/generative-ais-crippling-and-widespread https://garymarcus.substack.com/p/generative-ais-crippling-a...)

math_dandy 1y ago
Two schools of thought here. One posits that models need to have a strict "symbolic" representation of the world explicitly built in by their designers before they will be able to approach human levels of ability, adaptability and reliability. The other thinks that models approaching human levels of ability, adaptability, and reliability will constitute evidence for the emergence of strict "symbolic" representations.
sharemywin 1y ago
but you could easily build a verifier and if it's not valid have it create a new move until it finds one.

FollowingTheDao 1y ago

The bottleneck is nothing to do with money, it’s the fact that they’re using the empty neuron theory to try to mimic human consciousness and that’s not how it works. Just look up Microtubules and consciousness, and you’ll get a better idea for what I’m talking about. These AI computers aren’t thinking, they are just repeating.

p85 1y ago
I don't think OpenAI cares about whether their AI is conscious, as long as it can solve problems. If they could make a Blindsight-style general intelligence where nobody is actually home, they'd jump right on it. Conversely, a proof - or even evidence - that qualia-consciousness is necessary for intelligence, or that any sufficiently advanced intelligence is necessarily conscious through something like panpsychism, would make some serious waves in philosophy circles.

gaptoothclan 1y ago

I remember reading that llm’s have consumed the internet text data, I seem to remember there is an open data set for that too. Potential other sources of data would be images (probably already consumed) videos, YouTube must have such a large set of data to consume, perhaps Facebook or Instagram private content But even with these it does not feel like AGI, that seems like the fusion reactor 20 years away argument, but instead this is coming in 2 years, but they have not even got the core technology of how to build AGI

sharemywin 1y ago
the big step was having it reason through math problems that weren't in the training data. even now with web search it doesn't need every article in the training data to do useful things with it.
- Ferrus91 1y ago
  This is using think time compute and reinforcement learning. I think this is going to plateau even faster than the initial LLM scaling though.
Scrounger 1y ago
> I remember reading that llm’s have consumed the internet text data Not just the internet text data, but most major LLM models have been trained on millions of pirated books via Libgen: https://techcrunch.com/2025/01/09/mark-zuckerberg-gave-metas-llama-team-the-ok-to-train-on-copyrighted-works-filing-claims/ https://techcrunch.com/2025/01/09/mark-zuckerberg-gave-metas...

JohnBooty 1y ago

That being said, AGI is not a necessary requirement for AI to be totally world-changing Yeah. I don't think I actually want AGI? Even setting aside the moral/philosophical/etc "big picture" issues I don't think I even want that from a purely practical standpoint. I think I want various forms of AI that are more focused on specific domains. I want AI tools, not companions or peers or (gulp) masters. (Then again, people thought they wanted faster horses before they rolled out the Model T)

goatlover 1y ago
I don't think the public wants AGI either. Some enthusiasts and tech bros want it for questionable reasons such as replacing labor and becoming even richer.
- kergonath 1y ago
  For some it’s a religion. It’s frightening to hear Sam Altman or Peter Thiel talk about it. These people have a messiah complex and are driven by more than just greed (though there is also plenty of that).
  - mattgreenrocks 1y ago
    There’s a real anti-human bent to some of the AI maximalists, as well. It’s like a resentment over other people accruing skills that are recognized and they grow in. Hence the insistence on “democratizing” art and music production.
    - skydhash 1y ago
      As someone who have dabbled in drawing and tried to learn the guitar, those skills are hard to get. It takes times to get decent and a touch of brilliance to get really good. In contrast learning enough to know you’re not good yet (and probably never will be) is actually easy. But now I know enough to enjoy real masters going at it and fantasize sometimes.

mikepurvis 1y ago

"LLMs tend to regurgitate solutions to solved problems" People say this, but honestly, it's not really my experience— I've given ChatGPT (and Copilot) genuinely novel coding challenges and they do a very decent job at synthesizing a new thought based on relating it to disparate source examples. Really not that dissimilar to how a human thinks about these things.

candiddevmike 1y ago
How do you know they're truly novel given the massive training corpus and the somewhat limited vocabulary of programming languages?
- mikepurvis 1y ago
  I guess at a certain point you get into the philosophy of what it even means to be novel or test for novelty, but to give a concrete example, I'm in DevOps working on build pipelines for ROS containers using Docker Bake and GitHub Actions (including some reusable actions implemented in TypeScript). All of those are areas where ChatGPT has lots that it's learned from, so maybe me combining them isn't really novel at all, but like... I've given talks at the conference where people discuss how to best package and ship ROS workspaces, and I'm confident that no one out there has secretly already done what I'm doing and Chat is just using their prior work that it ingested at some point as a template for what it suggests I do. I think rather it has a broad understanding of concepts like build systems and tools, DAGs, dependencies, lockfiles, caching, and so on, and so it can understand my system through the general lens of what makes sense when these concepts are applied to non-ROS systems or on non-GHA DevOps platforms, or with other packaging regimes. I'd argue that that's novel, but as I said in the GP, the more important thing is that it's also how a human approaches things that to them are novel— by breaking them down, and identifying the mental shortcuts enabled by abstracting over familiar patterns.
- int_19h 1y ago
  I have a little ongoing project where I'm trying to use Claude Code to implement a compiler for the B programming language that is itself written in B. To the best of my knowledge, such a thing does not exist yet - or at least if it does, no amount of searching can find it, so it's unlikely that it is somewhere in the training set. For that matter, the overall amount of B code in existence is too small to be a meaningful training set for it. And yet it can do it when presented with a language spec. It's not perfect, but it can solve that with tooling that it makes for itself. For example, it tends to generate B code that is mostly correct, but with occasional problem. So, I had it write a B parser in Python and then use that whenever it edits B code to validate the edits.

justcallmejm 1y ago

It is definitively not possible. But the frontier models are no longer “just” LLMs, either. They are neurosymbolic systems (an LLM using tools); they just don’t say it transparently because it’s not a convenient narrative that intelligence comes from something outside the model, rather than from endless scaling. At Aloe, we are model agnostic and outperforming frontier models. It’s the anrchitecture around the LLM that makes the difference. For instance our system using Gemini can do things that Gemini can’t do on its own. All an LLM will ever do is hallucinate. If you want something with human-like general intelligence, keep looking beyond LLMs.

apwell23 1y ago
> At Aloe, we are model agnostic and outperforming frontier models. what is your website ?
- namibj 1y ago
  their name `.inc`; see the user's post history.
- JFingleton 1y ago
  A quick google gave: https://aloe.inc/ https://aloe.inc/
retrocog 1y ago
This mirrors my thinking and experience completely. Based on seeing Aloe in action, your company is IMHO positioned extremely well for this future.
Q6T46nT668w6i3m 1y ago
I’m confused, you wrote “model,” but then specified “system.” I assume you mean “system” because the tools are not being back-propagated?
- moritzwarhier 1y ago
  I read that as "the tools (their capabilities) are external to the model". Even if an RAG / agentic model learns from tool results, that doesn't automatically internalize the tool. You can't get yesterday's weather or major recent events from an offline, unless it was updated in that time. I am often wondering whether this is how large Chat and cloud AI providers cache expensive RAG-related data though :) like, decreasing the likelihood of tool usage given certain input patterns when the model has been patched using some recent, vetted interactions – in case that's even possible? Perplexity for example seems like they're probably invested in sone kind of activation-pattern-keyed caching... at least that was my first impression back when I first used it. Felt like decision trees, a bit like Akinator back in the days, but supercharged by LLM NLP.

rstuart4133 1y ago

> That being said, AGI is not a necessary requirement for AI to be totally world-changing. Depends on how you define "world changing" I guess, but this world already looks different to the pre-LLM world to me. Me asking LLM's things instead of consulting the output from other humans now takes up a significant fraction of my day. I don't google near as often, I don't trust any image or video I see as swathes of the creative professions have been replaced by output from LLM's. It's funny, that final thing is the last thing I would have predicted. I always believed the one thing a machine could not match was human creativity, because the output of machines was always precise, repetitive and reliable. Then LLM's come along, randomly generating every token. Their primary weakness is they neither precise or reliable, but they can turn out an unending stream of unique output.

Ferrus91 1y ago
I mean I also hear the same argument all the time about the "human touch" and interpersonal abilities etc. Which is apparently why managers and sales are safe from AI. But the more I see LLMs the more I realise that if it is good at one thing it is convincing other people and manipulating them. There have been multiple studies on this. People seem to have a innate prejudice and against nerds and programmers - coupled with envy at the high salaries - which is why they seem to have latched on to this idea it is mainly to replace them (and maybe data input people) as 'routine cognitive work' - but this slightly political obsession with a certain class of worker seems to be ignoring many of the things AI is actually good at.

novok 1y ago

They are moving beyond just big transformer blob LLM text prediction. Mixture of Experts is not preassembled for example, it's something like x empty experts with an empty router and the experts and routing emerges naturally with training, modeling the brain part architecture we see the brain more. There is stuff "Integrated Gated Calculator (IGC)" in Jan 2025 which makes a premade calculator neural network and integrates it directly into the neural network and gets around the entire issue of making LLMs do basic number computation and the clunkiness of generating "run tool tokens". The model naturally learns to use the IGC built into itself because it will always beat any kind of computation memorization in the reward function very quickly. Models are truly input multimodal now. Feeding an image, feeding audio and feeding text all go into separate input nodes, but it all feeds into the same inner layer set and outputs text. This also mirrors how brains work more as multiple parts integrated in one whole. Humans in some sense are not empty brains, there is a lot of stuff baked in our DNA and as the brain grows it develops a baked in development program. This is why we need fewer examples and generalize way better.

childintime 1y ago
Though there is info in DNA etc, you likely missed the biggest source of why we learn much faster. Search for Pim van Lommel near death research and find out how wrong the classic consciousness arises from the brain hypothesis is.
- filoeleven 1y ago
  You're not likely to find much support on this forum for these ideas. For those that have interest, the book Irreducible Mind: Toward a Psychology for the 21st Century is a well-written treatise on the topic. A gentler step in that direction is to see what Michael Levin and his lab are up to. He is looking for (one aspect of) intelligence, and finding it at the cellular level and below, even in an agential version of bubble sort. He's certainly challenging the notion that consciousness is limited to brain cells. All of his findings arise through experimental observation, so it forces some reckoning in a way that sociological research doesn't.

topspin 1y ago

> Perhaps it is not possible to simulate higher-level intelligence using a stochastic model for predicting text. I think you're on to it. Performance is clustering because a plateau is emerging. Hyper-dimensional search engines are running out of steam, and now we're optimizing.

onlyrealcuzzo 1y ago

> Human intelligence is markedly different from LLMs: it requires far fewer examples to train on, and generalizes way better. Aren't we the summation of intelligence from quintillions of beings over hundreds of millions of years? Have LLMs really had more data?

Planktonne 1y ago
By that argument, so are LLMs. They also wouldn't exist without all our ancestors.
- onlyrealcuzzo 1y ago
  No, by that argument, so would a can of soda.

OhioMan2943 1y ago

To be smarter than human intelligence you need smarter than human training data. Humans already innately know right and wrong a lot of the time so that doesn't leave much room.

beeflet 1y ago
This is a very good point! I remember reading about AlphaGo and how they got better results training against itself vs training against historical human-played games. So perhaps the solution is to train the AI against another AI somehow... but it is hard to imagine how this could extend to general-purpose tasks
dboreham 1y ago
> Humans already innately know Gentle suggestion that there is absolutely no such thing as "innately know". That's a delusion, albeit a powerful one. Everything is driven by training data. What we perceive as "thinking" and "motivation" are emergent structures.
- Jensson 1y ago
  Innately as in you are born with it, the DNA learned not us humans. We have no clue how the DNA learned to think other than "survival of the fittest", and that is the oldest AI training method in the book.

timeon 1y ago

> Human intelligence is markedly different from LLMs: it requires far fewer examples to train on, and generalizes way better. That is because with LLMs there is no intelligence. It is Artificial Knowledge. AK not AI. So AI is AGI. Not that it matters for user-cases we have, but marketing needs 'AI' because that is what we were expecting for decades. So yeah, I also do not thing we will have AGI from LLMs - nor does it matter for what we are using it.