9 ms·

If you believe the bitter lesson, all the handwavy "engineering" is better done with more data. Someone likely would have written the same thing as this 8 years

by andy99 1y ago

If you believe the bitter lesson, all the handwavy "engineering" is better done with more data. Someone likely would have written the same thing as this 8 years ago about what it would take to get current LLM performance. So I don't buy the engineering angle, I also don't think LLMs will scale up to AGI as imagined by Asimov or any of the usual sci-fi tropes. There is something more fundamental missing, as in missing science, not missing engineering.

deleted 1y ago
[deleted]
Bukhmanizer 1y ago
This isn’t really what the bitter lesson says.
hnuser123456 1y ago
Even more fundamental than science, there is missing philosophy, both in us regarding these systems, and in the systems themselves. An AGI implemented by an LLM needs to, at the minimum, be able to self-learn by updating its weights, self-finetune, otherwise it quickly hits a wall between its baked-in weights and finite context window. What is the optimal "attention" mechanism for choosing what to self-finetune with, and with what strength, to improve general intelligence? Surely it should focus on reliable academics, but which academics are reliable? How can we reliably ensure it studies topics that are "pure knowledge", and who does it choose to be, if we assume there is some theoretical point where it can autonomously outpace all of the world's best human-based research teams?
- fragmede 1y ago
  A system that self-updates its weights is so obvious the only question is who will be the first to get there?
  - tmountain 1y ago
    I’m no expert, but it seems like self updating weights requires a grounded understanding of the underlying subject matter, and this seems like a problem current LLM systems.
  - soulofmischief 1y ago
    It's not always as useful as you think from the perspective of a business trying to sell an automated service to users who expect reliability. Now you have to worry about waking up in the middle of the night to rewind your model to a last known good state, leading to real data loss as far as users are concerned. Data and functionality become entwined and basically you have to keep these systems on tight rails so that you can reason about their efficacy and performance, because any surgery on functionality might affect learned data, or worse, even damage a memory. It's going to take a long time to solve these problems.
  - fuckaj 1y ago
    True. In the same way as making noises down a telephone line is the obvious way to build a million dollar business.
  - danenania 1y ago
    I’m not sure that self-updating weights is really analogous to “continuous learning” as humans do it. A memory data structure that the model can search efficiently might be a lot closer. Self-updating weights could be more like epigenetics.
    - Jensson 1y ago
      Human neurons are self updating though, we aren't running on our genes each cell is using our genes to determine how to connect to other cells and then the cell learns how to process some information there based on what it hears from its connected cells. So, genes would be a meta model that then updates weights in the real model so it can learn how to process new kinds of things, and for stuff like facts you can use an external memory just like humans does. Without updating the weights in the model you will never be able to learn to process new things like a new kind of math etc, since you learn that not by memorizing facts but by making new models for it.
    - imtringued 1y ago
      In spiking neural networks, the model weights are equivalent to dendrites/synapses, which can form anew and decay during your lifetime.
    - HarHarVeryFunny 1y ago
      There's a difference between memory and learning. Would you rather your illness was diagnosed by a doctor or by a plumber with access to a stack of medical books ? Learning is about assimilating lots of different sources of information, reconciling the differences, trying things out for yourself, learning from your mistakes, being curious about your knowledge gaps and contradictions, and ultimately learning to correctly predict outcomes/actions based on everything you have learnt. You will soon see the difference in action as Anthropic apparently agree with you that memory can replace learning, and are going to be relying on LLMs with longer compressed context (i.e. memory) in place of ability to learn. I guess this'll be Anthropic's promised 2027 "drop-in replacement remote worker" - not an actual plumber unfortunately (no AGI), but an LLM with a stack of your company's onboarding material. It'll have perfect (well, "compressed") recall of everything you've tried to teach it, or complained about, but will have learnt nothing from that.
  - deleted 1y ago
    [deleted]
  - emporas 1y ago
    But then it is a specialized intelligence, specialized to altering it's weights. Reinforcement Learning doesn't work as well when the goal is not easily defined. It does wonders for games, but anything else? Someone has to specify the goals, a human operator or another A.I. The second A.I. better be an A.G.I. itself, otherwise it's goals will not be significant enough for us to care.
  - imtringued 1y ago
    I wonder when there will be proofs in theoretical computer science that an algorithm is AGI-complete, the same way there are proofs of NP-completeness. Conjecture: A system that self updates its weights according to a series of objective functions, but does not suffer from catastrophic forgetting (performance only degrades due to capacity limits, rather than from switching tasks) is AGI-complete. Why? Because it could learn literally anything!
  - HarHarVeryFunny 1y ago
    Sure, it's obvious, but it's only one of the missing pieces required for brain-like AGI, and really upends the whole LLM-as-AI way of doing things. Runtime incremental learning is still going to be based on prediction failure, but now it's no longer failure to predict the training set, but rather requires closing the loop and having (multi-modal) runtime "sensory" feedback - what were the real-world results of the action the AGI just predicted (generated)? This is no longer an auto-regressive model where you can just generate (act) by feeding the model's own output back in as input, but instead you now need to continually gather external feedback to feed back into your new incremental learning algorithm. For a multi-modal model the feedback would have to include image/video/audio data as well as text, but even if initial implementations of incremental learning systems restricted themselves to text it still turns the whole LLM-based way of interacting with the model on it's head - the model generates text-based actions to throw out into the world, and you now need to gather the text-based future feedback to those actions. With chat the feedback is more immediate, but with something like software development far more nebulous - the model makes a code edit, and the feedback only comes later when compiling, running, debugging, etc, or maybe when trying to refactor or extend the architecture in the future. In corporate use the response to an AGI-generated e-mail or message might come in many delayed forms, with these then needing to be anticipated, captured, and fed back into the model. Once you've replaced the simple LLM prompt-response mode of interaction with one based on continual real-world feedback, and designed the new incremental (Bayesian?) learning algorithm to replace SGD, maybe the next question is what model is being updated, and where does this happen? It's not at all clear that the idea of a single shared (between all users) model will work when you have millions of model instances all simultaneously doing different things and receiving different feedback on different timescales... Maybe the incremental learning now needs to be applied to a user-specific model instance (perhaps with some attempt to later share & re-distribute whatever it has learnt), even if that is still cloud based. So... a lot of very fundamental changes need to be made, just to support self-learning and self-updates, and we haven't even discussed all the other equally obvious differences between LLMs and a full cognitive architecture that would be needed to support more human-like AGI.
- ants_everywhere 1y ago
  > there is missing philosophy I doubt it. Human intelligence evolved from organisms much less intelligent than LLMs and no philosophy was needed. Just trial and error and competition.
  - solid_fuel 1y ago
    We are trying to get there without a few hundred million years of trial and error. To do that we need to lower the search space, and to do that we do actually need more guiding philosophy and a better understanding of intelligence.
    - fzzzy 1y ago
      Lower the search space or increase the search speed
      - balamatom 1y ago
        Instead what they usually do is lower the fidelity and think they've done what you said. Which results in them getting eaten. Once eaten, they can't learn from mistakes no mo. Their problem. Because if we don't mix up "intelligence" the phenomenon of increasingly complex self-organization in living systems, with "intelligence" our experience of being able to mentally model complex phenomena in order to interact with them, then it becomes easy to see how the search speed you talk of is already growing exponentially. In fact, that's all it does. Culture goes faster than genetic selection. Printing goes faster than writing. Democracy is faster than theocracy. Radio is faster than post. A computer is faster than a brain. LLMs are faster than trained monkeys and complain less. All across the planet, systems bootstrap themselves into more advanced systems as soon as I look at 'em, and I presume even when I don't. OTOH, all the metaphysics stuff about "sentience" and "sapience" that people who can't tell one from the other love to talk past each other about - all that only comes into view if one were to what's happening with the search space if the search speed is increasing at a forever increasing rate. Such as, whether the search space is finite, whether it's mutable, in what order to search, is it ethical to operate from quantized representations of it, funky sketchy scary stuff the lot of it. One's underlying assumptions about this process determine much of one's outlook on life as well as complex socially organized activities. One usually receives those through acculturation and may be unaware of what they say exactly.
- Uehreka 1y ago
  Nah. The real philosophical headache is that we still haven’t solved the hard problem of consciousness, and we’re disappointed because we hoped in our hearts (if not out loud) that building AI would give us some shred of insight into the rich and mysterious experience of life we somehow incontrovertibly perceive but can’t explain. Instead we got a machine that can outwardly present as human, can do tasks we had thought only humans can do, but reveals little to us about the nature of consciousness. And all we can do is keep arguing about the goalposts as this thing irrevocably reshapes our society, because it seems bizarre that we could be bested by something so banal and mechanical.
  - galangalalgol 1y ago
    I think Metzinger nailed it, we aren't conscious at all. We confuse the map for the territory in thinking the model we build to predict our other models is us. We are a collection of models a few of which create the illusion of consciousness. Someone is going to connect a handful of already existing models in a way that gives an AI the same illusion sooner rather than later. That will be an interesting day.
    - EMIRELADERO 1y ago
      I don't see how your explanation leads to consciousness not being a thing. Consciousness is whatever process/mechanisms there are that as a whole produce our subjective experience and all its sensations, including but not limited to touch, vision, smell, taste, pain, etc.
      - frabcus 1y ago
        You've missed our consciousness of our inner experiences. They are more varied than just perception at the footlights of our consciousness (cf Hurlburt): Imagination, inner voice, emotion, unsymbolized conceptual thinking as well as (our reconstructed view of our) perception.
        EMIRELADERO 1y ago
- nikkwong 1y ago
  I found it strange that John Carmack and Ilya Sutskever both left prestigious positions within their companies to pursue AGI as if they had some proprietary insight that the rest of industry hadn't caught on to. To make as bold of a career move that publicly would mean you'd have to have some ultra serious conviction that everyone else was wrong or naive and you were right. That move seemed pompous to me at the time; but I'm an industry outsider so what do I know. And now, I still don't know; the months go by and as far as I'm aware they're still pursuing these goals but I wonder how much conviction they still have.
  - therobots927 1y ago
    The simple explanation is that they got high on their own supply. They deluded themselves into thinking an LLM was on the verge of consciousness.
    - nprateem 1y ago
      The simpler answer is they could convince VCs to give them boat loads of cash by sounding like they can.
  - jasonwatkinspdx 1y ago
    With Carmack it's consciously a dilliante project. He's been effectively retired for quite some time. It's clear at some point he no longer found game and graphics engine internals motivation, possibly because the industry took the path he was advocating against back in the day. For a while he was focused on Armadillo aerospace, and they got some cool stuff accomplished. That was also something of a knowing pet project, and when they couldn't pivot to anything that looked like commercial viability he just put it in hibernation. Carmack may be confident (ne arrogant) enough to think he does have something unique to offer with AGI, but I don't think he's under any illusions it's anything but another pet project.
    - defen 1y ago
      > possibly because the industry took the path he was advocating against back in the day What path did he advocate? And what path did the industry take instead?
- joe_the_user 1y ago
  Well, Original 80s AI was based on mathematical logic. And while that might not encompass all philosophy, it certainly was a product of philosophy broadly speaking - some analytical philosophers could endorse. But it definitely failed and failed because it could process uncertainty (imo). I think also if you closely, classical philosophy wasn't particularly amenable to uncertainty either. If anything, I would say that AI has inherited its failure from philosophy's failure and we should look to alternative approaches (from Cybernetics to Bergson to whatever) for a basis for it.
  - smohare 1y ago
    [dead]
deleted 1y ago
[deleted]
sdenton4 1y ago
Too bad about all those chumps designing better, faster architectures and kernels to make models run faster...
energy123 1y ago
What will it scale up to if not AGI? OpenAI has a synthetic data flywheel. What are the asymptotics of this flywheel assuming no qualitative additional breakthrough?
- supermatt 1y ago
  What will shouting louder achieve if not wisdom?
- richyg840 1y ago
  It's established science (see Chomsky) that a probablistic model will never achieve something approaching AGI
- HarHarVeryFunny 1y ago
  Did GPT-1 scale up to be a database ? No - it scaled up to be GPT-2 Did GPT-2 scale up to be an expert system ? No - it scaled up to be GPT-3 .. Did GPT-4 scale up to become AGI ? No - it scaled up to be GPT-5 Moreover, the differences between each new version are becoming increasingly less. We're reaching an asymptote because the more data you've trained on, natural or synthetic, the less is the impact of any incremental additions. If you scale up an LLM big enough, then essentially what you'll get is GPT-5.
justcallmejm 1y ago
The missing science to engineer intelligence is composable program synthesis. Aloe (https://aloe.inc https://aloe.inc) recently released a GAIA score demonstrating how CPS dramatically outperforms other generalist agents (OpenAI's deep research, Manus, and Genspark) on tasks similar to those a knowledge worker would perform. I'd argue it's because intelligence has been treated as a ML/NN engineering problem that we've had the hyper focus on improving LLMs rather than the approach articulated in the essay. Intelligence must be built from a first principles theory of what intelligence actually is.
- joe_the_user 1y ago
  CPS sounds interesting but your link goes to a teaser trailer and a waiting list. It's kind of hard to expect much from that.
aaron695 1y ago
[dead]
joe_the_user 1y ago
So the "Bitter Lesson" paper actually came up recently and I was surprised to discover that what it claimed was sensible and not at "all you need is data" or "data is inherently better" The first line and the conclusion is: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." [1] I don't necessary agree with it's examples or the direction it vaguely points at. But it's basic statement seems sound. And I would say that there's lot of opportunity for engineer, broadly speaking, in the process of creating "general methods that leverage computation" (IE, that scale). What the bitter lesson page was roughly/really about was earlier "AI" methods based on logic-programming and which including information on the problem domain in the code itself. And finally, the "engineering" the paper talks about actually is pro-Bitter lesson as far as I can tell. It's taking data routing and architectural as "engineering" and here I agree this won't work - but for the opposite reason - specifically 'cause I don't just data routing/process will be enough. [1]https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
tomrod 1y ago
Aye. Missing are self correction (world models/action and response observation), coherence over the long term, and self-scaling. The 3rd are what all the SV types are worried about, except maybe Yann LeCun who is worried about the first and second. Hinton thinks the 3rd is inevitable/already here and humanity is doomed. It's an odd arena.
whatever1 1y ago
The counter argument is that we were working with thermodynamics before knowing the theory. Famously the steam engine came before the first law of thermodynamics. Sometimes engineering is like that. Using something that you don’t understand exactly how it works.
ksec 1y ago
> There is something more fundamental missing I am thinking we need a foundation, something that is concrete and explicit and doesn't do hallucination. But has very limited knowledge outside of absolute Maths and basic physics.
samrus 1y ago
> If you believe the bitter lesson, all the handwavy "engineering" is better done with more data Id say better model architechture than more data. A human can learn to do things more complex than an LLM with less data. I think modelling the world as a static system to be representation learned in an unsupervised fashion is blocked on the static assumption. The world is dynamical, that should be reflected in the base model But yeah, definitely not an engineering problem. Thats like saying the reason a crow isnt as smart as a person is becauss they dont have the hands to type of keyboards. But its also not because they havent seen enough of the world like your saying. Its be ause their brain isnt complex enough
tim333 1y ago
The bitter lesson was "general methods that leverage computation" win rather than more data. Like rather than just LLMs you could maybe try applying something like AlphaEvolve to finding better algorithms/systems (https://news.ycombinator.com/item?id=43985489 https://news.ycombinator.com/item?id=43985489).
grumbel 1y ago
> all the handwavy "engineering" is better done with more data. How long until that gets more reliable than a simple database? How long until it can execute code faster than a CPU running a program? A lot of the stuff humans accomplish is through technology, not due to growing a bigger brain. Even something seemingly basic like a math equation benefits drastically from being written down with pen&paper instead of being juggled in the human brain itself (see Extended mind thesis). And when it comes to something like running a 3D engine, there is pretty much no hope of doing it with just your brain. Maybe we will get AIs smart enough that they can write their own tools, but for that to happen, we still need the infrastructure that allows them writing the tools in the first place. The way they can access Python is a start, but there is still a lack of persistence that lets them keep their accomplishments for future runs, be it in the form of a digital notepad or dynamic updating of weights.
dtagames 1y ago
Indeed. The Bitter Lesson has proved true so far. This sounds like going back to the 60s expert systems concept we're trying to get away from. The author also just describes RAG. That certainly isn't AGI, which probably isn't achievable at all.
audunw 1y ago
8 years is a pretty short perspective. The current growth phase was unlocked by more engineering. We could’ve had some of these kinds of capabilities decades ago, as illustrated by how cutting edge AI research is now trickling down all the way to microcontrollers with 64MHz CPU and kBs of RAM. Once we got the “Attention is all you need” paper I don’t remember anyone saying we couldn’t get better results by throwing more data and compute at it. But now we’ve pretty much thrown all the data and all (as much as we can reasonably manufacture) at it. So clearly we’re at the end of that phase.
cowboylowrez 1y ago
I agree with your comment and the article. LLMs should be part of the answer, but the core of the progress should probably dive back into neural networks in general. Language is how we communicate as well as with other senses, but right now we're stuck at LLMs that just seem to be blown out elizas trained with other actual humans work. I remember early on, training of simple neural networks was done with rules in their environment and they evolved behavior according to criteria set, like genetic algorithms. I think the current LLMs are getting a "filtered" view of the environment, and that filter behaves like the average IQ of netizens lol
- deleted 1y ago
  [deleted]
HarHarVeryFunny 1y ago
I think the gist of TFA is just that we need a new architecture/etc not scaling. I suppose one can argue about whether designing a new AGI-capable architecture and learning algorithm(s) is a matter of engineering (applying what we already know) or research, but I think saying we need new scientific discoveries is going to far. Neural nets seems to be the right technology, and we've now amassed a ton of knowledge and intuition about what neural nets can do and how to design with them. If there was any doubt, then LLMs, even if not brain-like, have proved the power of prediction as a learning technique - intelligence essentially is just successful prediction. It seems pretty obvious that the rough requirements for an neural-net architecture for AGI are going to be something like our own neocortex and thalamo-cortical loop - something that learns to predict based on sensory feedback and prediction failure, including looping and working memory. Built-in "traits" like curiosity (prediction failure => focus) and boredom will be needed so that this sort of autonomous AGI puts itself into leaning situations and is capable of true innovation. The major piece to be designed/discovered isn't so much the architecture as the incremental learning algorithm, and I think if someone like Google-DeepMind focused their money, talent and resources on this then they could fairly easily get something that worked and could then be refined. Demis Hassabis has recently given an estimate of human-level AGI in 5 years, but has indicated that a pre-trained(?) LLM may still be one component of it, so not clear exactly what they are trying to build in that time frame. Having a built-in LLM is likely to prove to be a mistake where the bitter lesson applies - better to build something capable of self-learning and just let it learn.
- x1f604 1y ago
  > Demis Hassabis has recently given an estimate of human-level AGI in 5 years He said 50% chance of AGI in 5 years.
  - HarHarVeryFunny 1y ago
    Yes, and I wonder how many "5 year" project estimates, even for well understood engineering endeavors, end up being accurate to within 50% (obviously overruns are more common then the opposite)? I took his "50% 5-year" estimate as essentially a project estimate for something semi-concrete they are working towards. That sort of timeline and confidence level doesn't seem to allow for a whole lot of unknowns and open-ended research problems to be solved, but OTOH who knows if he is giving his true opinion, or where those numbers came from.
cyanydeez 1y ago
It seems similar to the fermi paradox. The underlying assumption is that it exists in the first place. Or rather, one must first accept an axiom. In fermi, its that interstellar signals can be detected and further travel is possible. In AGI, its that intelligence is a isolateable process which we can bootstrap in minimal time. Both assumes human progress are templates of unlimited exponential growth.
interstice 1y ago
Sometimes I think the fundamental thing could be as ‘simple’ as something like introducing l an attention/event loo, flush to memory, emotion driven motivation. There are quite a few fairly obvious things that llms don’t have that might be best not to add just in case.

by andy99 1y ago

deleted 1y ago

[deleted]

Bukhmanizer 1y ago

This isn’t really what the bitter lesson says.

hnuser123456 1y ago

Even more fundamental than science, there is missing philosophy, both in us regarding these systems, and in the systems themselves. An AGI implemented by an LLM needs to, at the minimum, be able to self-learn by updating its weights, self-finetune, otherwise it quickly hits a wall between its baked-in weights and finite context window. What is the optimal "attention" mechanism for choosing what to self-finetune with, and with what strength, to improve general intelligence? Surely it should focus on reliable academics, but which academics are reliable? How can we reliably ensure it studies topics that are "pure knowledge", and who does it choose to be, if we assume there is some theoretical point where it can autonomously outpace all of the world's best human-based research teams?

fragmede 1y ago
A system that self-updates its weights is so obvious the only question is who will be the first to get there?
- tmountain 1y ago
  I’m no expert, but it seems like self updating weights requires a grounded understanding of the underlying subject matter, and this seems like a problem current LLM systems.
- soulofmischief 1y ago
  It's not always as useful as you think from the perspective of a business trying to sell an automated service to users who expect reliability. Now you have to worry about waking up in the middle of the night to rewind your model to a last known good state, leading to real data loss as far as users are concerned. Data and functionality become entwined and basically you have to keep these systems on tight rails so that you can reason about their efficacy and performance, because any surgery on functionality might affect learned data, or worse, even damage a memory. It's going to take a long time to solve these problems.
- fuckaj 1y ago
  True. In the same way as making noises down a telephone line is the obvious way to build a million dollar business.
- danenania 1y ago
  I’m not sure that self-updating weights is really analogous to “continuous learning” as humans do it. A memory data structure that the model can search efficiently might be a lot closer. Self-updating weights could be more like epigenetics.
  - Jensson 1y ago
    Human neurons are self updating though, we aren't running on our genes each cell is using our genes to determine how to connect to other cells and then the cell learns how to process some information there based on what it hears from its connected cells. So, genes would be a meta model that then updates weights in the real model so it can learn how to process new kinds of things, and for stuff like facts you can use an external memory just like humans does. Without updating the weights in the model you will never be able to learn to process new things like a new kind of math etc, since you learn that not by memorizing facts but by making new models for it.
  - imtringued 1y ago
    In spiking neural networks, the model weights are equivalent to dendrites/synapses, which can form anew and decay during your lifetime.
  - HarHarVeryFunny 1y ago
    There's a difference between memory and learning. Would you rather your illness was diagnosed by a doctor or by a plumber with access to a stack of medical books ? Learning is about assimilating lots of different sources of information, reconciling the differences, trying things out for yourself, learning from your mistakes, being curious about your knowledge gaps and contradictions, and ultimately learning to correctly predict outcomes/actions based on everything you have learnt. You will soon see the difference in action as Anthropic apparently agree with you that memory can replace learning, and are going to be relying on LLMs with longer compressed context (i.e. memory) in place of ability to learn. I guess this'll be Anthropic's promised 2027 "drop-in replacement remote worker" - not an actual plumber unfortunately (no AGI), but an LLM with a stack of your company's onboarding material. It'll have perfect (well, "compressed") recall of everything you've tried to teach it, or complained about, but will have learnt nothing from that.
- deleted 1y ago
  [deleted]
- emporas 1y ago
  But then it is a specialized intelligence, specialized to altering it's weights. Reinforcement Learning doesn't work as well when the goal is not easily defined. It does wonders for games, but anything else? Someone has to specify the goals, a human operator or another A.I. The second A.I. better be an A.G.I. itself, otherwise it's goals will not be significant enough for us to care.
- imtringued 1y ago
  I wonder when there will be proofs in theoretical computer science that an algorithm is AGI-complete, the same way there are proofs of NP-completeness. Conjecture: A system that self updates its weights according to a series of objective functions, but does not suffer from catastrophic forgetting (performance only degrades due to capacity limits, rather than from switching tasks) is AGI-complete. Why? Because it could learn literally anything!
- HarHarVeryFunny 1y ago
  Sure, it's obvious, but it's only one of the missing pieces required for brain-like AGI, and really upends the whole LLM-as-AI way of doing things. Runtime incremental learning is still going to be based on prediction failure, but now it's no longer failure to predict the training set, but rather requires closing the loop and having (multi-modal) runtime "sensory" feedback - what were the real-world results of the action the AGI just predicted (generated)? This is no longer an auto-regressive model where you can just generate (act) by feeding the model's own output back in as input, but instead you now need to continually gather external feedback to feed back into your new incremental learning algorithm. For a multi-modal model the feedback would have to include image/video/audio data as well as text, but even if initial implementations of incremental learning systems restricted themselves to text it still turns the whole LLM-based way of interacting with the model on it's head - the model generates text-based actions to throw out into the world, and you now need to gather the text-based future feedback to those actions. With chat the feedback is more immediate, but with something like software development far more nebulous - the model makes a code edit, and the feedback only comes later when compiling, running, debugging, etc, or maybe when trying to refactor or extend the architecture in the future. In corporate use the response to an AGI-generated e-mail or message might come in many delayed forms, with these then needing to be anticipated, captured, and fed back into the model. Once you've replaced the simple LLM prompt-response mode of interaction with one based on continual real-world feedback, and designed the new incremental (Bayesian?) learning algorithm to replace SGD, maybe the next question is what model is being updated, and where does this happen? It's not at all clear that the idea of a single shared (between all users) model will work when you have millions of model instances all simultaneously doing different things and receiving different feedback on different timescales... Maybe the incremental learning now needs to be applied to a user-specific model instance (perhaps with some attempt to later share & re-distribute whatever it has learnt), even if that is still cloud based. So... a lot of very fundamental changes need to be made, just to support self-learning and self-updates, and we haven't even discussed all the other equally obvious differences between LLMs and a full cognitive architecture that would be needed to support more human-like AGI.
ants_everywhere 1y ago
> there is missing philosophy I doubt it. Human intelligence evolved from organisms much less intelligent than LLMs and no philosophy was needed. Just trial and error and competition.
- solid_fuel 1y ago
  We are trying to get there without a few hundred million years of trial and error. To do that we need to lower the search space, and to do that we do actually need more guiding philosophy and a better understanding of intelligence.
  - fzzzy 1y ago
    Lower the search space or increase the search speed
    - balamatom 1y ago
      Instead what they usually do is lower the fidelity and think they've done what you said. Which results in them getting eaten. Once eaten, they can't learn from mistakes no mo. Their problem. Because if we don't mix up "intelligence" the phenomenon of increasingly complex self-organization in living systems, with "intelligence" our experience of being able to mentally model complex phenomena in order to interact with them, then it becomes easy to see how the search speed you talk of is already growing exponentially. In fact, that's all it does. Culture goes faster than genetic selection. Printing goes faster than writing. Democracy is faster than theocracy. Radio is faster than post. A computer is faster than a brain. LLMs are faster than trained monkeys and complain less. All across the planet, systems bootstrap themselves into more advanced systems as soon as I look at 'em, and I presume even when I don't. OTOH, all the metaphysics stuff about "sentience" and "sapience" that people who can't tell one from the other love to talk past each other about - all that only comes into view if one were to what's happening with the search space if the search speed is increasing at a forever increasing rate. Such as, whether the search space is finite, whether it's mutable, in what order to search, is it ethical to operate from quantized representations of it, funky sketchy scary stuff the lot of it. One's underlying assumptions about this process determine much of one's outlook on life as well as complex socially organized activities. One usually receives those through acculturation and may be unaware of what they say exactly.
Uehreka 1y ago
Nah. The real philosophical headache is that we still haven’t solved the hard problem of consciousness, and we’re disappointed because we hoped in our hearts (if not out loud) that building AI would give us some shred of insight into the rich and mysterious experience of life we somehow incontrovertibly perceive but can’t explain. Instead we got a machine that can outwardly present as human, can do tasks we had thought only humans can do, but reveals little to us about the nature of consciousness. And all we can do is keep arguing about the goalposts as this thing irrevocably reshapes our society, because it seems bizarre that we could be bested by something so banal and mechanical.
- galangalalgol 1y ago
  I think Metzinger nailed it, we aren't conscious at all. We confuse the map for the territory in thinking the model we build to predict our other models is us. We are a collection of models a few of which create the illusion of consciousness. Someone is going to connect a handful of already existing models in a way that gives an AI the same illusion sooner rather than later. That will be an interesting day.
  - EMIRELADERO 1y ago
    I don't see how your explanation leads to consciousness not being a thing. Consciousness is whatever process/mechanisms there are that as a whole produce our subjective experience and all its sensations, including but not limited to touch, vision, smell, taste, pain, etc.
    - frabcus 1y ago
      You've missed our consciousness of our inner experiences. They are more varied than just perception at the footlights of our consciousness (cf Hurlburt): Imagination, inner voice, emotion, unsymbolized conceptual thinking as well as (our reconstructed view of our) perception.
      - EMIRELADERO 1y ago
nikkwong 1y ago
I found it strange that John Carmack and Ilya Sutskever both left prestigious positions within their companies to pursue AGI as if they had some proprietary insight that the rest of industry hadn't caught on to. To make as bold of a career move that publicly would mean you'd have to have some ultra serious conviction that everyone else was wrong or naive and you were right. That move seemed pompous to me at the time; but I'm an industry outsider so what do I know. And now, I still don't know; the months go by and as far as I'm aware they're still pursuing these goals but I wonder how much conviction they still have.
- therobots927 1y ago
  The simple explanation is that they got high on their own supply. They deluded themselves into thinking an LLM was on the verge of consciousness.
  - nprateem 1y ago
    The simpler answer is they could convince VCs to give them boat loads of cash by sounding like they can.
- jasonwatkinspdx 1y ago
  With Carmack it's consciously a dilliante project. He's been effectively retired for quite some time. It's clear at some point he no longer found game and graphics engine internals motivation, possibly because the industry took the path he was advocating against back in the day. For a while he was focused on Armadillo aerospace, and they got some cool stuff accomplished. That was also something of a knowing pet project, and when they couldn't pivot to anything that looked like commercial viability he just put it in hibernation. Carmack may be confident (ne arrogant) enough to think he does have something unique to offer with AGI, but I don't think he's under any illusions it's anything but another pet project.
  - defen 1y ago
    > possibly because the industry took the path he was advocating against back in the day What path did he advocate? And what path did the industry take instead?
joe_the_user 1y ago
Well, Original 80s AI was based on mathematical logic. And while that might not encompass all philosophy, it certainly was a product of philosophy broadly speaking - some analytical philosophers could endorse. But it definitely failed and failed because it could process uncertainty (imo). I think also if you closely, classical philosophy wasn't particularly amenable to uncertainty either. If anything, I would say that AI has inherited its failure from philosophy's failure and we should look to alternative approaches (from Cybernetics to Bergson to whatever) for a basis for it.
- smohare 1y ago
  [dead]

deleted 1y ago

[deleted]

sdenton4 1y ago

Too bad about all those chumps designing better, faster architectures and kernels to make models run faster...

energy123 1y ago

What will it scale up to if not AGI? OpenAI has a synthetic data flywheel. What are the asymptotics of this flywheel assuming no qualitative additional breakthrough?

supermatt 1y ago
What will shouting louder achieve if not wisdom?
richyg840 1y ago
It's established science (see Chomsky) that a probablistic model will never achieve something approaching AGI
HarHarVeryFunny 1y ago
Did GPT-1 scale up to be a database ? No - it scaled up to be GPT-2 Did GPT-2 scale up to be an expert system ? No - it scaled up to be GPT-3 .. Did GPT-4 scale up to become AGI ? No - it scaled up to be GPT-5 Moreover, the differences between each new version are becoming increasingly less. We're reaching an asymptote because the more data you've trained on, natural or synthetic, the less is the impact of any incremental additions. If you scale up an LLM big enough, then essentially what you'll get is GPT-5.

justcallmejm 1y ago

The missing science to engineer intelligence is composable program synthesis. Aloe (https://aloe.inc https://aloe.inc) recently released a GAIA score demonstrating how CPS dramatically outperforms other generalist agents (OpenAI's deep research, Manus, and Genspark) on tasks similar to those a knowledge worker would perform. I'd argue it's because intelligence has been treated as a ML/NN engineering problem that we've had the hyper focus on improving LLMs rather than the approach articulated in the essay. Intelligence must be built from a first principles theory of what intelligence actually is.

joe_the_user 1y ago
CPS sounds interesting but your link goes to a teaser trailer and a waiting list. It's kind of hard to expect much from that.

aaron695 1y ago

[dead]

joe_the_user 1y ago

So the "Bitter Lesson" paper actually came up recently and I was surprised to discover that what it claimed was sensible and not at "all you need is data" or "data is inherently better" The first line and the conclusion is: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." [1] I don't necessary agree with it's examples or the direction it vaguely points at. But it's basic statement seems sound. And I would say that there's lot of opportunity for engineer, broadly speaking, in the process of creating "general methods that leverage computation" (IE, that scale). What the bitter lesson page was roughly/really about was earlier "AI" methods based on logic-programming and which including information on the problem domain in the code itself. And finally, the "engineering" the paper talks about actually is pro-Bitter lesson as far as I can tell. It's taking data routing and architectural as "engineering" and here I agree this won't work - but for the opposite reason - specifically 'cause I don't just data routing/process will be enough. [1]https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

tomrod 1y ago

Aye. Missing are self correction (world models/action and response observation), coherence over the long term, and self-scaling. The 3rd are what all the SV types are worried about, except maybe Yann LeCun who is worried about the first and second. Hinton thinks the 3rd is inevitable/already here and humanity is doomed. It's an odd arena.

whatever1 1y ago

The counter argument is that we were working with thermodynamics before knowing the theory. Famously the steam engine came before the first law of thermodynamics. Sometimes engineering is like that. Using something that you don’t understand exactly how it works.

ksec 1y ago

> There is something more fundamental missing I am thinking we need a foundation, something that is concrete and explicit and doesn't do hallucination. But has very limited knowledge outside of absolute Maths and basic physics.

samrus 1y ago

> If you believe the bitter lesson, all the handwavy "engineering" is better done with more data Id say better model architechture than more data. A human can learn to do things more complex than an LLM with less data. I think modelling the world as a static system to be representation learned in an unsupervised fashion is blocked on the static assumption. The world is dynamical, that should be reflected in the base model But yeah, definitely not an engineering problem. Thats like saying the reason a crow isnt as smart as a person is becauss they dont have the hands to type of keyboards. But its also not because they havent seen enough of the world like your saying. Its be ause their brain isnt complex enough

tim333 1y ago

The bitter lesson was "general methods that leverage computation" win rather than more data. Like rather than just LLMs you could maybe try applying something like AlphaEvolve to finding better algorithms/systems (https://news.ycombinator.com/item?id=43985489 https://news.ycombinator.com/item?id=43985489).

grumbel 1y ago

> all the handwavy "engineering" is better done with more data. How long until that gets more reliable than a simple database? How long until it can execute code faster than a CPU running a program? A lot of the stuff humans accomplish is through technology, not due to growing a bigger brain. Even something seemingly basic like a math equation benefits drastically from being written down with pen&paper instead of being juggled in the human brain itself (see Extended mind thesis). And when it comes to something like running a 3D engine, there is pretty much no hope of doing it with just your brain. Maybe we will get AIs smart enough that they can write their own tools, but for that to happen, we still need the infrastructure that allows them writing the tools in the first place. The way they can access Python is a start, but there is still a lack of persistence that lets them keep their accomplishments for future runs, be it in the form of a digital notepad or dynamic updating of weights.

dtagames 1y ago

Indeed. The Bitter Lesson has proved true so far. This sounds like going back to the 60s expert systems concept we're trying to get away from. The author also just describes RAG. That certainly isn't AGI, which probably isn't achievable at all.

audunw 1y ago

8 years is a pretty short perspective. The current growth phase was unlocked by more engineering. We could’ve had some of these kinds of capabilities decades ago, as illustrated by how cutting edge AI research is now trickling down all the way to microcontrollers with 64MHz CPU and kBs of RAM. Once we got the “Attention is all you need” paper I don’t remember anyone saying we couldn’t get better results by throwing more data and compute at it. But now we’ve pretty much thrown all the data and all (as much as we can reasonably manufacture) at it. So clearly we’re at the end of that phase.

cowboylowrez 1y ago

I agree with your comment and the article. LLMs should be part of the answer, but the core of the progress should probably dive back into neural networks in general. Language is how we communicate as well as with other senses, but right now we're stuck at LLMs that just seem to be blown out elizas trained with other actual humans work. I remember early on, training of simple neural networks was done with rules in their environment and they evolved behavior according to criteria set, like genetic algorithms. I think the current LLMs are getting a "filtered" view of the environment, and that filter behaves like the average IQ of netizens lol

deleted 1y ago
[deleted]

HarHarVeryFunny 1y ago

I think the gist of TFA is just that we need a new architecture/etc not scaling. I suppose one can argue about whether designing a new AGI-capable architecture and learning algorithm(s) is a matter of engineering (applying what we already know) or research, but I think saying we need new scientific discoveries is going to far. Neural nets seems to be the right technology, and we've now amassed a ton of knowledge and intuition about what neural nets can do and how to design with them. If there was any doubt, then LLMs, even if not brain-like, have proved the power of prediction as a learning technique - intelligence essentially is just successful prediction. It seems pretty obvious that the rough requirements for an neural-net architecture for AGI are going to be something like our own neocortex and thalamo-cortical loop - something that learns to predict based on sensory feedback and prediction failure, including looping and working memory. Built-in "traits" like curiosity (prediction failure => focus) and boredom will be needed so that this sort of autonomous AGI puts itself into leaning situations and is capable of true innovation. The major piece to be designed/discovered isn't so much the architecture as the incremental learning algorithm, and I think if someone like Google-DeepMind focused their money, talent and resources on this then they could fairly easily get something that worked and could then be refined. Demis Hassabis has recently given an estimate of human-level AGI in 5 years, but has indicated that a pre-trained(?) LLM may still be one component of it, so not clear exactly what they are trying to build in that time frame. Having a built-in LLM is likely to prove to be a mistake where the bitter lesson applies - better to build something capable of self-learning and just let it learn.

x1f604 1y ago
> Demis Hassabis has recently given an estimate of human-level AGI in 5 years He said 50% chance of AGI in 5 years.
- HarHarVeryFunny 1y ago
  Yes, and I wonder how many "5 year" project estimates, even for well understood engineering endeavors, end up being accurate to within 50% (obviously overruns are more common then the opposite)? I took his "50% 5-year" estimate as essentially a project estimate for something semi-concrete they are working towards. That sort of timeline and confidence level doesn't seem to allow for a whole lot of unknowns and open-ended research problems to be solved, but OTOH who knows if he is giving his true opinion, or where those numbers came from.

cyanydeez 1y ago

It seems similar to the fermi paradox. The underlying assumption is that it exists in the first place. Or rather, one must first accept an axiom. In fermi, its that interstellar signals can be detected and further travel is possible. In AGI, its that intelligence is a isolateable process which we can bootstrap in minimal time. Both assumes human progress are templates of unlimited exponential growth.

interstice 1y ago

Sometimes I think the fundamental thing could be as ‘simple’ as something like introducing l an attention/event loo, flush to memory, emotion driven motivation. There are quite a few fairly obvious things that llms don’t have that might be best not to add just in case.