11 ms·

How Organisms Come to Know the World: Fundamental Limits on AGI

59 points by yamrzou 4y ago

visarga 4y ago
There are two ways to learn: from imitation (language models are stochastic parrots) and from validation (problem solving, playing games). Many people only remember the first and forget the second. But the second one has the key to increasing intelligence. I don't know what the authors mean by "current algorithmic frame of AI research" because it is very diverse. But evolutionary methods work well with neural networks, they are not just next token predictors. > Evolution through Large Models https://arxiv.org/abs/2206.08896 https://arxiv.org/abs/2206.08896 It is true that simply using a dataset or a bunch of environments is not enough. > General Intelligence Requires Rethinking Exploration https://arxiv.org/abs/2211.07819 https://arxiv.org/abs/2211.07819 But when sufficient data can be generated from acting in the environment, then a model can surpass human level, like AlphaZero.
- usrbinbash 4y ago
  > But when sufficient data can be generated from acting in the environment, then a model can surpass human level, like AlphaZero. Yes, in a very very very very very limited environment like a board game. Let's up the ante just a tiny bit and say the AI has to be able to learn how to play Go. But it also has to learn how to get and setup the board, and the cups holding the Go-Stones. It has to place them on the table, or find the table where someone else placed them. It has to figure out how to place the stones. And it has to do so regardless of what else is on the table, what table it is, where the table is, what the light conditions are, or if its playing with stones, chips, painted pebbles or peanuts vs. roasted almonds (yes, I played Go that way once :D ) Now, how much more difficult does this make the task to an AI? Bear in mind that all these changes to the task are absolutely trivial to the only (G)eneral(I)ntelligence that we know (humans). We effortlessly combine so much prior knowledge about movement, our limbs, the way physical objects work, how to gather and integrate information into our model of the world, the ability to formulate and adapt goals _and their similarity to prior goals_ and the solutions to those, that all these tasks I just added to the problem are absolutely trivial, and the only hard exercise is where to place stones on the board. That's how far our current systems are from an AGI.
  - mannykannot 4y ago
    > Now, how much more difficult does this make the task to an AI? Given that I can do all these things, but I can't beat Lee Sedol, I suspect they don't add an insurmountable level of difficulty to the problem. The claim here is not that algorithmic AI is very [very...] difficult; it is that it is impossible.
    - usrbinbash 4y ago
      > I suspect they don't add an insurmountable level of difficulty to the problem. Not for you, because to humans, moving, finding things, gathering information, formulating new goals, etc. are common tasks. And we generalized them to a point where we can integrate them with other tasks, even such which are completely abstract to our daily lives like a game where little stones are put on a grid. But to a machine? Is it easier to make a machine play Go or make it find arbitrary objects while walking around? Is gathering information on its own difficult? Is goal formulation difficult? Is moving through arbitrary spaces difficult? And how difficult are all these tasks relative to one another? > The claim here is not that algorithmic AI is very [very...] difficult; it is that it is impossible. As I understood it, impossible in the current research paradigms. And given the difference in complexity between the task of playing a board game, and playing a board game as I described above, this seems very likely.
      - mannykannot 4y ago
        >> The claim here is not that algorithmic AI is very [very...] difficult; it is that it is impossible. > As I understood it, impossible in the current research paradigms. You can't get around the fact that they are making a claim that something is impossible. If they merely said it was difficult, or the 'something' was something of no interest, there would be nothing here worth discussing.
      - naasking 4y ago
        > As I understood it, impossible in the current research paradigms. No, they literally say that biological "affordances" cannot be captured by algorithms that run on Turing machines.
- ENGNR 4y ago
  I agree with the two ways in general What if, the second way is easy? So long as you can understand and interface with the world effectively, which the first provides. From there what if it was just act and observe and a fitness function
  - deleted 4y ago
    [deleted]
elil17 4y ago
> When implementing an algorithm as a computer program by means of some kind of formal language (including those based on recursive functional programming paradigms), we must introduce specific data and code structures, their properties and interactions, as well as the set of operations we are allowed to perform on them, in order to represent the objects and relations that are relevant for our computation. In other words, we must provide a precisely defined ontology on which the program can operate deductively, e. g., by drawing inferences or by ordering tasks for solving a given problem. In an algorithmic framework, novelty can only be represented combinatorially: it manifests as new combinations, mergers, and relations between objects in a (potentially vast, but predefined) space of possibilities. This means that an algorithm cannot discover or generate truly novel properties or relations that were not (at least implicitly) considered in its original ontology. Therefore, an algorithm operating in a deductive manner cannot jury-rig, since it cannot find new causal properties of an object that were not already inherent in its logical premises. This argument is not convincing to me. In fact, there seem to be real world counterexamples to their argument. If the AI agent is operating, on say, text, why is that space “potentially vast, but predefined”? A text stream of arbitrary length is arbitrarily vast. An AI agent could, conceivably, define new ontologies textually. As an example, this person used ChatGPT to invent a new language: https://maximumeffort.substack.com/p/i-taught-chatgpt-to-invent-a-language https://maximumeffort.substack.com/p/i-taught-chatgpt-to-inv... That strikes me as an example of generating “truly novel properties or relations that were not (at least implicitly) considered in its original ontology,” albeit in a rudimentary way.
- dr_dshiv 4y ago
  Gorp!
- Garlef 4y ago
  >> This means that an algorithm cannot discover or generate truly novel properties or relations that were not (at least implicitly) considered in its original ontology. > A text stream of arbitrary length... Maybe you could state this in some form of "turing completeness": As soon as a program has [vast enough output space] and allows for [enough recursive recombinations] then it can produce [everything]. In the wording of the article: If everything is implicitly considered, then there's nothing that can not be produced.
estevaoam 4y ago
I find it amusing that so many people is skeptic about general AI, trying to come up with arguments that it will never happen, while at the same time we do not understand how our own intelligence work. Go figure out.
- bamboozled 4y ago
  Well then it's funny to think we can build something we don't even have a good definition for? Sounds almost a bit insane?
  - hexomancer 4y ago
    We didn't know what electricity really was when we created electric lightbulbs.
    - timid_oshima 4y ago
      Hadn’t it been thoroughly studied and understood for over 100 years at that point? Ben Franklin was studying it, after all.
      - hexomancer 4y ago
        Electron was discovered 1897 but the first lightbulb was 1802. There are more examples as well, as another user has commented, we didn't know what fire is until relatively recently but we have been using it for thousands of years.
        timid_oshima 4y ago
        That sounds a bit early - whats your source? This one says at least mid 19th century for not practical electric lights https://www.energy.gov/articles/history-light-bulb https://www.energy.gov/articles/history-light-bulb But either way, is knowing the electron knowing electricity? There are so many properties of it that can be known and manipulated without that insight- and indeed they built up that understanding to reach practical engineering and use of electricity. That’s what I think is being gotten at wrt intelligence. “Knowing” something isn’t necessarily about being aware of its smaller parts.
graderjs 4y ago
I've always thought to develop AGI you really need an evolutionary iterative system that can explore an environment with sensory input and reward feedback for actions. In other words a synthetic analog of a real organism in the real world. Few interesting things come: 1) either try to bootstrap AGI in robots exploring the real world with senses, or try to bootstrap AGI in online robots who have some sort of online senses like scanning text and images, but then you need a set of online 'threats' to that robot, so that it can have reward feedback--it can't just exist 'for free' it has to 'fight' to live, otherwise no growth; and 2) you basically need emotions, because emotions are, essentially, our compressed, immediate internal responses to aggregate experiences in the world--they give us a way to interface with our reward feedback, so turns out you need emotions for AGI too, to have an effective interface with reward feedback in a complex environment; and 3) you need a complex environment with plenty of threats in order to drive evolution, and obtaining rewards has to be sufficiently ill-defined, difficult but possible that one has to figure out some unique solution for oneself; the world is just as important as the 'brain' for AGI to develop. If you (horribly) took a new born baby and kept it in a white featureless room (or in some sort of contrived abstract environment), and met its survival needs , it would grow up skewed and probably not very 'generally intelligent.' AGI doesn't have to be the same as consciousness...but you need something that 'lives in a world' and has limitations that it has to figure out how to overcome, and emotional responses in order to represent and respond to experiences, in order to develop it. When you realize this, it's kind of funny to think of the trope that AGI is always represented like this "robotic emotionless logical", but I think that's impossible. That's more like nth-gen ChatGPT, than real AGI. I think real AGI needs to be emotional--maybe not the same as human emotions, but it has to have them, and has to have a world to grow in.
- kordlessagain 4y ago
  I think you are right, but keep in mind the perceptions we have are built on an internal model of the "real world" and we operate mind on that, not the real world. For example, if you were to imagine an apple in mind, what is it that you see, if anything?[1] Are you seeing a real apple, that you saw before and saved as the reference apple, or are you seeing an apple that your mind synthesized and then gives to you for perception? And, where does this vision synthesis occur? Is it external, as Theosophy claims[2], or is it internal? The comment above about gathering entropy from the real world is interesting. So, is it that consciousness "emerges" from the mashup of real entropy gathered with sensors, then modeled internally with computation, or is it contained in the real world already, in the form of another model (like a soul model or some such) and therefore comes from outside us? Or is it a little of both? Emotional states could be just another model, which set the working conditions in a given moment. It could also be possible the real world, from which our sensors gather information, is a very complex model in-and-of-itself. [1] https://en.wikipedia.org/wiki/Aphantasia https://en.wikipedia.org/wiki/Aphantasia [2] https://en.wikipedia.org/wiki/Mental_body https://en.wikipedia.org/wiki/Mental_body
Jack000 4y ago
I think thinking in terms of affordances and goals already makes a lot of assumptions about how a potential AGI would work, but imo the kind of "radical emergence" talked about in the article is totally within the realm of possible applications of modern machine learning. If you think about how a language model works, it's not totally clear that it should work at all. Humans use sentences to communicate pre-existing ideas - ie. we already have the object, subject and action in our heads prior to forming the sentence. In order to generate realistic sentences, an LLM needs some ability to plan ahead, so that when generating the logits for the first token, it needs some idea of how the sentence will end. But if you think about it, nothing about next-token prediction should necessarily lead to this planning ability. It can easily get into a feedback loop of predict the same token over and over (and small LMs do in fact do this) We also didn't tell it explicitly that sentences should have objects, subjects and actions, these behaviors are purely emergent from the task of next token prediction. In a similar vein, I think artificial agents would not need explicit goals or affordances, just a sufficiently complex environment that a large model would not overfit on.
- davidmanheim 4y ago
  Yeah, and the idea that these programs can't have affordances is silly reliance on a definition that ignores what can happen. It clearly proves too much - humans have a limited action space - they can only move muscles. So they can't truly explore a larger action space, so they cannot be general intelligences. But more specifically, if something is within an LLM's action space, whether or not you call it an affordance doesn't change whether it gets explored and used. And perhaps you'd argue that their action space is limited because they are in a box. But hook an LLM up to the internet to allow it to query and retrieve data, and suddenly the action space is essentially infinite. So the limitation isn't the model, it's how the model was hobbled by being denied access to the world.
barneygale 4y ago
> We elaborate on the relation between affordances and algorithms—defined as computational processes that can run on universal Turing machines—ultimately arriving at the conclusion that identifying and leveraging affordances goes beyond algorithmic computation. Brains are physical systems, and physical systems can be simulated by algorithms. Right? Do the authors think this 'jury-rigging' is some non-physical process?
- dinkumthinkum 4y ago
  What do you mean by simulation? You can simulate an airplane without considering air resistance; it’s not necessarily an accurate simulation.
- posterboy 4y ago
  >... implying that algorithmic attempts at creating such dynamics in the field of artificial life (aLife) are doomed to fail. I guess they are so meta they wondered if a dedicated AI could output an AGI agent, algorithmically. As far as their colloquial definition which speaks of jury-rigging is truly intelligent, I suspect that the road to AGI would require so much human input and reboots (have you tried turning it on and off again?) that it is not predictable by any physical theory. So it's non-physical in a strong definition of physics. On the other hand, since they argue against classical computing, it would necessarily rely on innovative developments in various hard sciences.
mannykannot 4y ago
The authors concede that what they call algorithmic AI could, in principle, mimic organic intelligence arbitrarily closely. To maintain that this is not intelligence without begging the question, they need to have some relevant difference between the two which is not simply saying that Turing machines and living organisms are different categories. From the discussion (and especially section 7) it is clear that all their attempts to defend their position will fall back on one thing: Turing machines are deterministic. The simplest response to that is that as soon as a machine (including a digital computer) is interacting with the real world, it is tapping into a source of entropy and its behavior is no longer determined fully by its programming. Alternatively, if one holds that the universe is deterministic, then living organisms are, which also defeats the author's last stand.
- joshspankit 4y ago
  It’s funny to me to see this juxtaposed with the arguments from the other side: Society had long argued that animals were not intelligent, and that instincts were deterministic.
  - pnf 4y ago
    Instincts in animals (including humans) are deterministic insofar as they determine the behavior of the animal. A bird may choose one stick or another but it can't choose not to build a nest. A human can. This may be unique to humans. Heidegger refers to this as world-openness, meaning our world is not closed by our instincts. Unlike most animals, humans don't possess enough instincts to survive in nature. We are "insufficiently-determined". Hence, to survive, we augment our nature with our culture, which acts as a second nature within which we can act unthinkingly (as animals do). Without this second nature we shall surely die (hence the furious culture wars). Not incidentally, this is one of the anthropological arguments against state of nature political theories: there's no such thing and never has been in the history of the species. Animals may be more or less undetermined but a bee will make honey and an ant will gather food regardless of any individual instance of bee or ant intelligence. The problem with the above objection is that there is no account of "meaning" in what an AGI does. In other words, it's still a computer. To say that the entropy of the system will determine the entity's indeterminism is akin to arguments saying Schrodinger's cat is evidence for the possibility of human free will. It's a category error. Unless you want to argue that humans are also computers (which is the position of many strict materialists) then there is still a huge gap to account for in terms of meaning that can't be defined away satisfyingly. We all know what it's like to be an intelligence. At best, people say we can't know if an AGI also knows that because we can't inspect their "minds". This argument is justified by recourse to the principle of the multiple instantiability of intelligence. If we admit that non-carbon-based life forms could be intelligent (e.g., extraterrestrials) then we may admit that intelligence could be in silica as well. Without the ability to introspect other minds, we assume they are like ours and not simply "zombies". The above arguments (reducible to "Well, how do you KNOW anyone else has a mind?") are based in a kind of skeptical solipsism. The part about intelligence being demarcated by non-determinism and the way that argument fails is a side-step of the hard problem of consciousness.
novaRom 4y ago
Limited speed of signal transmission, limited amount of matter and energy per volume of space, topology of physical space, quantum uncertainty, tunneling, and other quantum effects.
Recursing 4y ago
See also: On the Impossibility of Supersized Machines https://arxiv.org/abs/1703.10987 https://arxiv.org/abs/1703.10987
- naasking 4y ago
  Great parody paper.
deleted 4y ago
[deleted]
wcoenen 4y ago
Their argument that Turing machines cannot be free from immediacy doesn't make sense to me. They say that "there are no actions that emanate from the historicity of internal organization". I have no idea what that means. Is this an argument about free will? If so, what would be the magic ingredient that saves human brains from immediacy? It can't be memory, because Turing machines have memory as well. From the article: > it is generally assumed that input-output processing is performed by some sort of algorithm that can be implemented on a universal Turing machine. The problem is that such algorithmic systems have no freedom from immediacy, since all their outputs are determined entirely—even though often in intricate and probabilistic ways—by the inputs of the system. There are no actions that emanate from the historicity of internal organization. There is, therefore, no agency at all in an AI “agent.”
- logicchains 4y ago
  >If so, what would be the magic ingredient that saves human brains from immediacy? It can't be memory, because Turing machines have memory as well. The universe is not deterministic. There is one brain state at time T, but at time T+1 there are many possible brain states, all with different possibilities. What determines which state at time T+1 the consciousness at time T perceives itself as progressing to? If we define this as a function, which associates states at time T with states at times T+1, then this function loosely fits the definition of "free will". We can't actually construct this function or say anything about it, but we can say it can't not exist (as consciousness perceives itself as progressing to a single future state), which by the law of the excluded middle implies it must exist. The definition doesn't however prevent it from essentially being completely random.
  - davidmanheim 4y ago
    "The universe is not deterministic." Maybe not. I think that's a big claim, and it's being made, not proved. But let's say it's right, ignoring the massive burden of proof. We can hook up a quantum random number generator to GPT-3, and suddenly it can be conscious? If that's the whole argument, it seems like a useless point anyways.
  - 4y ago
tomq 4y ago
The authors make a classic mistake: the brain is not comparable to a Turing Machine, it is a finite state machine. There’s no infinite tape in a brain.
- jesuscript 4y ago
  Explain this.
  - naasking 4y ago
    https://news.ycombinator.com/item?id=34299889 https://news.ycombinator.com/item?id=34299889
- dinkumthinkum 4y ago
  The brain is not a finite state machine. If it is then it is clearly equivalent to a Turing machine. It seems like your position maybe is just based on there not being “infinite tape” or something but human intelligence is not anything like any computational model we have conceived, or at least no one has presented any such evidence.
  - naasking 4y ago
    > The brain is not a finite state machine. Yes it is, all finite volumes contain finite information. This is a consequence of physics called the Bekenstein Bound. This means the brain can be fully captured by a finite state machine. It has a very large state space, but it's still finite.
    - sambapa 4y ago
      Brain can create new states, so by induction it is not a fsm
      - naasking 4y ago
        What exactly is an example of a new state?
iandanforth 4y ago
If you get bored by biological exceptionalism and hairsplitting you can skip this article. There's nothing new here.
alfor 4y ago
I think we mostly recognise pattern of the world. We train ourselves by minimizing surprise, improving our predictive capabilities. The same is true of AI at the moment. The affordance is another pattern (tool) we train ourselves all our life with many tools: hammer, knife, cisors, tweezers and more complex ones: car, backhoe, computers. There is a lot of overlap from one tool to another one, most of the pattern transfer easily from one to another. That’s why we don’t have to retrain ourselves if we change the size of the knife we use, it’s a tiny adjustment done in the first few seconds of use.
grantcas 4y ago
[dead]

59 points by yamrzou 4y ago

There are two ways to learn: from imitation (language models are stochastic parrots) and from validation (problem solving, playing games). Many people only remember the first and forget the second. But the second one has the key to increasing intelligence. I don't know what the authors mean by "current algorithmic frame of AI research" because it is very diverse. But evolutionary methods work well with neural networks, they are not just next token predictors. > Evolution through Large Models https://arxiv.org/abs/2206.08896 https://arxiv.org/abs/2206.08896 It is true that simply using a dataset or a bunch of environments is not enough. > General Intelligence Requires Rethinking Exploration https://arxiv.org/abs/2211.07819 https://arxiv.org/abs/2211.07819 But when sufficient data can be generated from acting in the environment, then a model can surpass human level, like AlphaZero.

usrbinbash 4y ago
> But when sufficient data can be generated from acting in the environment, then a model can surpass human level, like AlphaZero. Yes, in a very very very very very limited environment like a board game. Let's up the ante just a tiny bit and say the AI has to be able to learn how to play Go. But it also has to learn how to get and setup the board, and the cups holding the Go-Stones. It has to place them on the table, or find the table where someone else placed them. It has to figure out how to place the stones. And it has to do so regardless of what else is on the table, what table it is, where the table is, what the light conditions are, or if its playing with stones, chips, painted pebbles or peanuts vs. roasted almonds (yes, I played Go that way once :D ) Now, how much more difficult does this make the task to an AI? Bear in mind that all these changes to the task are absolutely trivial to the only (G)eneral(I)ntelligence that we know (humans). We effortlessly combine so much prior knowledge about movement, our limbs, the way physical objects work, how to gather and integrate information into our model of the world, the ability to formulate and adapt goals _and their similarity to prior goals_ and the solutions to those, that all these tasks I just added to the problem are absolutely trivial, and the only hard exercise is where to place stones on the board. That's how far our current systems are from an AGI.
- mannykannot 4y ago
  > Now, how much more difficult does this make the task to an AI? Given that I can do all these things, but I can't beat Lee Sedol, I suspect they don't add an insurmountable level of difficulty to the problem. The claim here is not that algorithmic AI is very [very...] difficult; it is that it is impossible.
  - usrbinbash 4y ago
    > I suspect they don't add an insurmountable level of difficulty to the problem. Not for you, because to humans, moving, finding things, gathering information, formulating new goals, etc. are common tasks. And we generalized them to a point where we can integrate them with other tasks, even such which are completely abstract to our daily lives like a game where little stones are put on a grid. But to a machine? Is it easier to make a machine play Go or make it find arbitrary objects while walking around? Is gathering information on its own difficult? Is goal formulation difficult? Is moving through arbitrary spaces difficult? And how difficult are all these tasks relative to one another? > The claim here is not that algorithmic AI is very [very...] difficult; it is that it is impossible. As I understood it, impossible in the current research paradigms. And given the difference in complexity between the task of playing a board game, and playing a board game as I described above, this seems very likely.
    - mannykannot 4y ago
      >> The claim here is not that algorithmic AI is very [very...] difficult; it is that it is impossible. > As I understood it, impossible in the current research paradigms. You can't get around the fact that they are making a claim that something is impossible. If they merely said it was difficult, or the 'something' was something of no interest, there would be nothing here worth discussing.
    - naasking 4y ago
      > As I understood it, impossible in the current research paradigms. No, they literally say that biological "affordances" cannot be captured by algorithms that run on Turing machines.
ENGNR 4y ago
I agree with the two ways in general What if, the second way is easy? So long as you can understand and interface with the world effectively, which the first provides. From there what if it was just act and observe and a fitness function
- deleted 4y ago
  [deleted]

elil17 4y ago

> When implementing an algorithm as a computer program by means of some kind of formal language (including those based on recursive functional programming paradigms), we must introduce specific data and code structures, their properties and interactions, as well as the set of operations we are allowed to perform on them, in order to represent the objects and relations that are relevant for our computation. In other words, we must provide a precisely defined ontology on which the program can operate deductively, e. g., by drawing inferences or by ordering tasks for solving a given problem. In an algorithmic framework, novelty can only be represented combinatorially: it manifests as new combinations, mergers, and relations between objects in a (potentially vast, but predefined) space of possibilities. This means that an algorithm cannot discover or generate truly novel properties or relations that were not (at least implicitly) considered in its original ontology. Therefore, an algorithm operating in a deductive manner cannot jury-rig, since it cannot find new causal properties of an object that were not already inherent in its logical premises. This argument is not convincing to me. In fact, there seem to be real world counterexamples to their argument. If the AI agent is operating, on say, text, why is that space “potentially vast, but predefined”? A text stream of arbitrary length is arbitrarily vast. An AI agent could, conceivably, define new ontologies textually. As an example, this person used ChatGPT to invent a new language: https://maximumeffort.substack.com/p/i-taught-chatgpt-to-invent-a-language https://maximumeffort.substack.com/p/i-taught-chatgpt-to-inv... That strikes me as an example of generating “truly novel properties or relations that were not (at least implicitly) considered in its original ontology,” albeit in a rudimentary way.

dr_dshiv 4y ago
Gorp!
Garlef 4y ago
>> This means that an algorithm cannot discover or generate truly novel properties or relations that were not (at least implicitly) considered in its original ontology. > A text stream of arbitrary length... Maybe you could state this in some form of "turing completeness": As soon as a program has [vast enough output space] and allows for [enough recursive recombinations] then it can produce [everything]. In the wording of the article: If everything is implicitly considered, then there's nothing that can not be produced.

estevaoam 4y ago

I find it amusing that so many people is skeptic about general AI, trying to come up with arguments that it will never happen, while at the same time we do not understand how our own intelligence work. Go figure out.

bamboozled 4y ago
Well then it's funny to think we can build something we don't even have a good definition for? Sounds almost a bit insane?
- hexomancer 4y ago
  We didn't know what electricity really was when we created electric lightbulbs.
  - timid_oshima 4y ago
    Hadn’t it been thoroughly studied and understood for over 100 years at that point? Ben Franklin was studying it, after all.
    - hexomancer 4y ago
      Electron was discovered 1897 but the first lightbulb was 1802. There are more examples as well, as another user has commented, we didn't know what fire is until relatively recently but we have been using it for thousands of years.
      - timid_oshima 4y ago
        That sounds a bit early - whats your source? This one says at least mid 19th century for not practical electric lights https://www.energy.gov/articles/history-light-bulb https://www.energy.gov/articles/history-light-bulb But either way, is knowing the electron knowing electricity? There are so many properties of it that can be known and manipulated without that insight- and indeed they built up that understanding to reach practical engineering and use of electricity. That’s what I think is being gotten at wrt intelligence. “Knowing” something isn’t necessarily about being aware of its smaller parts.

graderjs 4y ago

I've always thought to develop AGI you really need an evolutionary iterative system that can explore an environment with sensory input and reward feedback for actions. In other words a synthetic analog of a real organism in the real world. Few interesting things come: 1) either try to bootstrap AGI in robots exploring the real world with senses, or try to bootstrap AGI in online robots who have some sort of online senses like scanning text and images, but then you need a set of online 'threats' to that robot, so that it can have reward feedback--it can't just exist 'for free' it has to 'fight' to live, otherwise no growth; and 2) you basically need emotions, because emotions are, essentially, our compressed, immediate internal responses to aggregate experiences in the world--they give us a way to interface with our reward feedback, so turns out you need emotions for AGI too, to have an effective interface with reward feedback in a complex environment; and 3) you need a complex environment with plenty of threats in order to drive evolution, and obtaining rewards has to be sufficiently ill-defined, difficult but possible that one has to figure out some unique solution for oneself; the world is just as important as the 'brain' for AGI to develop. If you (horribly) took a new born baby and kept it in a white featureless room (or in some sort of contrived abstract environment), and met its survival needs , it would grow up skewed and probably not very 'generally intelligent.' AGI doesn't have to be the same as consciousness...but you need something that 'lives in a world' and has limitations that it has to figure out how to overcome, and emotional responses in order to represent and respond to experiences, in order to develop it. When you realize this, it's kind of funny to think of the trope that AGI is always represented like this "robotic emotionless logical", but I think that's impossible. That's more like nth-gen ChatGPT, than real AGI. I think real AGI needs to be emotional--maybe not the same as human emotions, but it has to have them, and has to have a world to grow in.

kordlessagain 4y ago
I think you are right, but keep in mind the perceptions we have are built on an internal model of the "real world" and we operate mind on that, not the real world. For example, if you were to imagine an apple in mind, what is it that you see, if anything?[1] Are you seeing a real apple, that you saw before and saved as the reference apple, or are you seeing an apple that your mind synthesized and then gives to you for perception? And, where does this vision synthesis occur? Is it external, as Theosophy claims[2], or is it internal? The comment above about gathering entropy from the real world is interesting. So, is it that consciousness "emerges" from the mashup of real entropy gathered with sensors, then modeled internally with computation, or is it contained in the real world already, in the form of another model (like a soul model or some such) and therefore comes from outside us? Or is it a little of both? Emotional states could be just another model, which set the working conditions in a given moment. It could also be possible the real world, from which our sensors gather information, is a very complex model in-and-of-itself. [1] https://en.wikipedia.org/wiki/Aphantasia https://en.wikipedia.org/wiki/Aphantasia [2] https://en.wikipedia.org/wiki/Mental_body https://en.wikipedia.org/wiki/Mental_body

Jack000 4y ago

I think thinking in terms of affordances and goals already makes a lot of assumptions about how a potential AGI would work, but imo the kind of "radical emergence" talked about in the article is totally within the realm of possible applications of modern machine learning. If you think about how a language model works, it's not totally clear that it should work at all. Humans use sentences to communicate pre-existing ideas - ie. we already have the object, subject and action in our heads prior to forming the sentence. In order to generate realistic sentences, an LLM needs some ability to plan ahead, so that when generating the logits for the first token, it needs some idea of how the sentence will end. But if you think about it, nothing about next-token prediction should necessarily lead to this planning ability. It can easily get into a feedback loop of predict the same token over and over (and small LMs do in fact do this) We also didn't tell it explicitly that sentences should have objects, subjects and actions, these behaviors are purely emergent from the task of next token prediction. In a similar vein, I think artificial agents would not need explicit goals or affordances, just a sufficiently complex environment that a large model would not overfit on.

davidmanheim 4y ago
Yeah, and the idea that these programs can't have affordances is silly reliance on a definition that ignores what can happen. It clearly proves too much - humans have a limited action space - they can only move muscles. So they can't truly explore a larger action space, so they cannot be general intelligences. But more specifically, if something is within an LLM's action space, whether or not you call it an affordance doesn't change whether it gets explored and used. And perhaps you'd argue that their action space is limited because they are in a box. But hook an LLM up to the internet to allow it to query and retrieve data, and suddenly the action space is essentially infinite. So the limitation isn't the model, it's how the model was hobbled by being denied access to the world.

barneygale 4y ago

> We elaborate on the relation between affordances and algorithms—defined as computational processes that can run on universal Turing machines—ultimately arriving at the conclusion that identifying and leveraging affordances goes beyond algorithmic computation. Brains are physical systems, and physical systems can be simulated by algorithms. Right? Do the authors think this 'jury-rigging' is some non-physical process?

dinkumthinkum 4y ago
What do you mean by simulation? You can simulate an airplane without considering air resistance; it’s not necessarily an accurate simulation.
posterboy 4y ago
>... implying that algorithmic attempts at creating such dynamics in the field of artificial life (aLife) are doomed to fail. I guess they are so meta they wondered if a dedicated AI could output an AGI agent, algorithmically. As far as their colloquial definition which speaks of jury-rigging is truly intelligent, I suspect that the road to AGI would require so much human input and reboots (have you tried turning it on and off again?) that it is not predictable by any physical theory. So it's non-physical in a strong definition of physics. On the other hand, since they argue against classical computing, it would necessarily rely on innovative developments in various hard sciences.

mannykannot 4y ago

The authors concede that what they call algorithmic AI could, in principle, mimic organic intelligence arbitrarily closely. To maintain that this is not intelligence without begging the question, they need to have some relevant difference between the two which is not simply saying that Turing machines and living organisms are different categories. From the discussion (and especially section 7) it is clear that all their attempts to defend their position will fall back on one thing: Turing machines are deterministic. The simplest response to that is that as soon as a machine (including a digital computer) is interacting with the real world, it is tapping into a source of entropy and its behavior is no longer determined fully by its programming. Alternatively, if one holds that the universe is deterministic, then living organisms are, which also defeats the author's last stand.

joshspankit 4y ago
It’s funny to me to see this juxtaposed with the arguments from the other side: Society had long argued that animals were not intelligent, and that instincts were deterministic.
- pnf 4y ago
  Instincts in animals (including humans) are deterministic insofar as they determine the behavior of the animal. A bird may choose one stick or another but it can't choose not to build a nest. A human can. This may be unique to humans. Heidegger refers to this as world-openness, meaning our world is not closed by our instincts. Unlike most animals, humans don't possess enough instincts to survive in nature. We are "insufficiently-determined". Hence, to survive, we augment our nature with our culture, which acts as a second nature within which we can act unthinkingly (as animals do). Without this second nature we shall surely die (hence the furious culture wars). Not incidentally, this is one of the anthropological arguments against state of nature political theories: there's no such thing and never has been in the history of the species. Animals may be more or less undetermined but a bee will make honey and an ant will gather food regardless of any individual instance of bee or ant intelligence. The problem with the above objection is that there is no account of "meaning" in what an AGI does. In other words, it's still a computer. To say that the entropy of the system will determine the entity's indeterminism is akin to arguments saying Schrodinger's cat is evidence for the possibility of human free will. It's a category error. Unless you want to argue that humans are also computers (which is the position of many strict materialists) then there is still a huge gap to account for in terms of meaning that can't be defined away satisfyingly. We all know what it's like to be an intelligence. At best, people say we can't know if an AGI also knows that because we can't inspect their "minds". This argument is justified by recourse to the principle of the multiple instantiability of intelligence. If we admit that non-carbon-based life forms could be intelligent (e.g., extraterrestrials) then we may admit that intelligence could be in silica as well. Without the ability to introspect other minds, we assume they are like ours and not simply "zombies". The above arguments (reducible to "Well, how do you KNOW anyone else has a mind?") are based in a kind of skeptical solipsism. The part about intelligence being demarcated by non-determinism and the way that argument fails is a side-step of the hard problem of consciousness.

novaRom 4y ago

Limited speed of signal transmission, limited amount of matter and energy per volume of space, topology of physical space, quantum uncertainty, tunneling, and other quantum effects.

Recursing 4y ago

See also: On the Impossibility of Supersized Machines https://arxiv.org/abs/1703.10987 https://arxiv.org/abs/1703.10987

naasking 4y ago
Great parody paper.

deleted 4y ago

[deleted]

wcoenen 4y ago

Their argument that Turing machines cannot be free from immediacy doesn't make sense to me. They say that "there are no actions that emanate from the historicity of internal organization". I have no idea what that means. Is this an argument about free will? If so, what would be the magic ingredient that saves human brains from immediacy? It can't be memory, because Turing machines have memory as well. From the article: > it is generally assumed that input-output processing is performed by some sort of algorithm that can be implemented on a universal Turing machine. The problem is that such algorithmic systems have no freedom from immediacy, since all their outputs are determined entirely—even though often in intricate and probabilistic ways—by the inputs of the system. There are no actions that emanate from the historicity of internal organization. There is, therefore, no agency at all in an AI “agent.”

logicchains 4y ago
>If so, what would be the magic ingredient that saves human brains from immediacy? It can't be memory, because Turing machines have memory as well. The universe is not deterministic. There is one brain state at time T, but at time T+1 there are many possible brain states, all with different possibilities. What determines which state at time T+1 the consciousness at time T perceives itself as progressing to? If we define this as a function, which associates states at time T with states at times T+1, then this function loosely fits the definition of "free will". We can't actually construct this function or say anything about it, but we can say it can't not exist (as consciousness perceives itself as progressing to a single future state), which by the law of the excluded middle implies it must exist. The definition doesn't however prevent it from essentially being completely random.
- davidmanheim 4y ago
  "The universe is not deterministic." Maybe not. I think that's a big claim, and it's being made, not proved. But let's say it's right, ignoring the massive burden of proof. We can hook up a quantum random number generator to GPT-3, and suddenly it can be conscious? If that's the whole argument, it seems like a useless point anyways.
- 4y ago

tomq 4y ago

The authors make a classic mistake: the brain is not comparable to a Turing Machine, it is a finite state machine. There’s no infinite tape in a brain.

jesuscript 4y ago
Explain this.
- naasking 4y ago
  https://news.ycombinator.com/item?id=34299889 https://news.ycombinator.com/item?id=34299889
dinkumthinkum 4y ago
The brain is not a finite state machine. If it is then it is clearly equivalent to a Turing machine. It seems like your position maybe is just based on there not being “infinite tape” or something but human intelligence is not anything like any computational model we have conceived, or at least no one has presented any such evidence.
- naasking 4y ago
  > The brain is not a finite state machine. Yes it is, all finite volumes contain finite information. This is a consequence of physics called the Bekenstein Bound. This means the brain can be fully captured by a finite state machine. It has a very large state space, but it's still finite.
  - sambapa 4y ago
    Brain can create new states, so by induction it is not a fsm
    - naasking 4y ago
      What exactly is an example of a new state?

iandanforth 4y ago

If you get bored by biological exceptionalism and hairsplitting you can skip this article. There's nothing new here.

alfor 4y ago

I think we mostly recognise pattern of the world. We train ourselves by minimizing surprise, improving our predictive capabilities. The same is true of AI at the moment. The affordance is another pattern (tool) we train ourselves all our life with many tools: hammer, knife, cisors, tweezers and more complex ones: car, backhoe, computers. There is a lot of overlap from one tool to another one, most of the pattern transfer easily from one to another. That’s why we don’t have to retrain ourselves if we change the size of the knife we use, it’s a tiny adjustment done in the first few seconds of use.

grantcas 4y ago

[dead]