16 ms·

It’s not intelligent if it always halts: A critical perspective on AI approaches

43 points by optimalsolver 3y ago

PaulHoule 3y ago
Yes! See basic computer science such as the halting problem, Godel’s theorem, not to mention Hofstader’s book Godel, Escher, Bach. Using a neural network does not repeal the laws of computer science as much as people are inclined to think.
- michaelmrose 3y ago
  Can you explain wherein it is asserted directly or indirectly that using a neural network serves to repeal the laws of computer science?
  - astrange 3y ago
    I think the article is claiming that other people think neural networks are Turing slash human complete, which they aren't, for instance because they're not capable of using unbounded external storage slash "memory", and because they only execute in finite time. However the example at the top of the page isn't quite right. You could implement something like this. An LLM's memory only exists in its context window of its own prior output[0], but you could train it to tag its "chain of thoughts" and just not show them - now it looks like it's "thinking". [0] this is not exactly true because you could also implement non-greedy search, ie back up and start again if you think it's gone down the wrong path. Now it doesn't execute in finite time either.
    - cameronh90 3y ago
      There are approaches to add "memory" to LLMs. Either the memory can be used to find relevant information and append it to the prompt - I believe LangChain can do this - or the LLM can be used as a controller to retrieve and store information in a database, for example by using ChatGPT's tool integration. I don't think it works amazingly, but purely technically you could argue that gives it access to unbounded external storage. I don't think an LLM can be considered as capable of becoming an AGI on its own with the current architecture, but potentially combined with other techniques, models and supervision, I can see the potential for it to evolve in that direction. Frankly, for what LLMs are right now, they're _surprisingly_ effective. If you chopped the Broca's area out of my brain and hooked it up to an API, I suspect it wouldn't do half as good of a job.
  - PaulHoule 3y ago
    Look at the average “Ask HN” on the subject. Or look at the kind of speculations that followers of Eliezer Yudkowsky make in the name of big-R Reason. (thank God we hear about him about as often as we hear about Threads these days) To be fair a lot of people speculating about AI aren’t aware of the foundations of computer science, but some of them are vaguely aware and just forgot. Actually LLMs do have a way to break some of those constraints, in their own way, in that those constraints don’t apply to systems that don‘t always get the right answer. You can sort a list in O(1) some of the time but not all of the time.
- consilient 3y ago
  Humans can't solve the halting problem or violate the incompleteness theorems either.
  - srcreigh 3y ago
    Just because a system can encode paradoxes doesn't mean the system itself can't be solved. Turing, Chaitin, Godel didn't prove things can't be solved, they actually contributed by solving specific areas of specific systems. People didn't give up on ZFC because of Godel's theorem. There's currently 42 or 43 unproven 5-state Turing machines we need to confirm BB(5). We are solving the halting problem. We are solving ZFC. That's the entire point of mathematics. Turing's proof of the halting problem being undecidable is extraordinarily vague. It shows that for any halting procedure, there is a program (without any constraints) which violate its output. It does not, for example, rule out whether you can write a Halt procedure for no-input Turing Machines with N states. In fact we have already written them for N=1,2,3,4. Considering Chaitin-Kolmogorov complexity, of course we can't use N bits to describe unbounded information. This doesn't in any way preclude us from making an N bit program to describe halting behavior of K<N bit programs for some N.
    - consilient 3y ago
      > We are solving the halting problem. We are solving ZFC. That's the entire point of mathematics. We are not. We're determining whether particular classes of turing machines halt, and we're determining whether particular theorems hold in ZFC (and doing a lot of mathematical work which is neither). This is not evidence that the human brain is super-turing, because these are computable problems. > It does not, for example, rule out whether you can write a Halt procedure for no-input Turing Machines with N states. In fact we have already written them for N=1,2,3,4. Turing didn't, but later work does. BB(748) is known to be independent of ZF. The real bound is likely much lower.
      - srcreigh 3y ago
        As of last month the bound has been reduced slightly to 745. Scott Aaronson has a positive view of BB(n) being independent of ZFC. He says, we’ll need different foundations to solve k>n. And the different foundations will have their own n and etc. Page 6 here https://www.scottaaronson.com/papers/bb.pdf https://www.scottaaronson.com/papers/bb.pdf
version_five 3y ago
Some good intuition around what is common sense for most people, that a loop with some matrix multiplication never becomes intelligent. What's missed (at least in my skim) is that the "intelligent" part of current AI is in the training, not the inference, which as pointed out is fixed, and literally just some multiplication. Inference doesn't think about stuff but training does (for some definition), and doesn't converge necessarily. I still think it's ridiculous to equate neural networks with intelligence, but a stronger argument has to deal with training as well.
- fleischhauf 3y ago
  I'm not sure humans work differently. You might not learn something if you are coming to conclusions, only if the conclusions turn out to be correct or wrong you'll learn. the process of learning might then be different.
- fnordpiglet 3y ago
  I think people get confused that LLM is the beginning and end of AI. It’s just a piece. I think AI is an ensemble of all the classic AI techniques and IR and other information, computational, optimization, agent, etc systems. LLMs are remarkable in their abductive reasoning abilities, which is something we’ve generally failed to produce so far. But they’re not complete, and the criticism that they’re worthless because they’re incomplete misses the point. They don’t need to play chess, because we already have much better chess playing systems. But they fill a gap that all our other techniques have been unable to fill, and that is where the magic lives. It’s unnecessary for them to do everything, as those things already have good solutions.
- lostmsu 3y ago
  This is pretty much 100% false. There are known trivial Turing machines (which can be implemented in matrix multiplications) that are intelligent. They just don't think fast enough to be of any use.
  - nmilo 3y ago
    Example?
pjscott 3y ago
One thing to keep in mind when reading this article is that human cognition always halts after some number of decades. “Finite” does not mean small or insignificant, and it’s easy to read too much into even the best-reasoned impossibility arguments.
- dinosaurdynasty 3y ago
  Agreed. This seems to prove that humans aren't intelligent.
- MostlyStable 3y ago
  I'm not sure this follows. It doesn't halt because the "program" reaches an end, it's halts because the hardware fails. If you are running a category B program with an anvil attached to a timer over the computer, it doesn't become a category A program. It's just a category B program that wasn't allowed to continue. -edit- Also, this logic would lead us to believe that category B and C programs can't exist in our universe since, according to our best current understanding, the universe will end eventually, presumably ending any program running within the universe as well, therefore we can "prove" that every program is a category A program that will come to a halt.
  - consilient 3y ago
    > I'm not sure this follows. It doesn't halt because the "program" reaches an end, it's halts because the hardware fails. Ok, but the intelligent entity in question isn't the program, it's the program-running-on-the-hardware. A mere description of your brain is totally inert.
    - MostlyStable 3y ago
      _Every_ program is running on hardware. And as I pointed out _all_ hardware has some definite EOL, even if it's just the end of the rest of the universe. So thinking of it this way disproves the entire concept of Type B and Type C programs.
Joker_vD 3y ago
Once upon a time Buridan's donkey was a go-to example for why a truly intelligent entity must always halt. Apparently, the stance on that is now reversed?
- taneq 3y ago
  https://en.wikipedia.org/wiki/Buridan%27s_ass https://en.wikipedia.org/wiki/Buridan%27s_ass to save some time for those of us unacquainted with this thought experiment. > It refers to a hypothetical situation wherein an ass (donkey) that is equally hungry and thirsty is placed precisely midway between a stack of hay and a pail of water. Since the paradox assumes the donkey will always go to whichever is closer, it dies of both hunger and thirst since it cannot make any rational decision between the hay and water.
michaelmrose 3y ago
The author spent a lot of words saying a lot of nothing which seems to be the general case of communicating about anything that is poorly defined like consciousness or intelligence. > If a computer program is bound to finish quickly by virtue of its architecture, it cannot possibly be capable of general problem-solving. Our current tools are bound to produce a finite and usable amount of information because that is what is useful to us. One can easily have them produce infinite streams in a loop musing and diverging on tangents as you please but it wouldn't be useful nor would make the general category more or less intelligent. You could say that a finite exploration of any information space has finite explanatory power bounded by how much input/output it can do but that would be as boring as saying you can only fit so many apples based on the size of the barrel used. The uncomfortable truth is it might matter a lot less how intelligent a tool is than how complex a transformation it can handle. It's possible that we could make something that for practical purposes appears much smarter than us that still isn't by a human definition intelligent, conscious, or directed. Focusing on factors that are functionally less relevant may just reveal our bias towards viewing the universe in terms of self when in fact it has no objective thing that corresponds to the labels we've stuck on ourselves.
cameronh90 3y ago
> "If the size of the input is fixed and there is a limit to the number of possible tokens (words, or even numbers with bounded precision) then the state of the transformer at any given moment is describable with a string of fixed length." Doesn't this also likely describe humans? Albeit, we don't currently have the capacity to inspect our brains that way, so we can't prove it.
- thereisnospork 3y ago
  I would concur: at some point, probably shockingly quickly[0], the length of said string becomes longer than the number of particles in the universe (or other metric of absurd siz). [0] I'm reminded of the fact that every fresh shuffle of a deck of cards is overwhelmingly likely to have never been seen before.
  - cameronh90 3y ago
    I struggle to reliably remember and then recall more than about 6 digits without quite a lot of mental effort. The frequency I forget an SMS-OTP between opening it on my phone and going to type it in is honestly pretty embarrassing. Hopefully I still count as generally intelligent despite those limitations.
- taneq 3y ago
  Yeah, what? "If it contains a finite amount of information then it isn't AI"? Is this the moving goalposts' final form?
  - ChatGTP 3y ago
    May I ask why you feel compelled to point stuff like this out ? Do you just feel the need to call our ignorance ? Lack empathy in understanding why people might want the goalposts changed ? Not an attack, just curious why you might need to bring this up ?
    - spunker540 3y ago
jameshart 3y ago
The basic pseudocode of an interactive LLM processor is while (true) context = readLine() while (true) nextToken = llm.generateNextToken(context) context = rtrim(context + nextToken, MAX_CONTEXT_LENGTH) if (nextToken == END_OF_LINE_TOKEN) break print (nextToken) That's a program that doesn't, in general, guarantee that its inner loop halts. It may never come back and prompt for more input. The decision to stop asking the GPT engine to keep predicting tokens after it reaches the end of an initial answer is entirely an implementation choice, based on looking at the token output and concluding that the LLM just started generating the start of something we'd rather have the user provide. You can just keep asking for more predictions. Forever if you so choose. There's no architectural reason why the sequence of tokens produced can't constitute an ongoing train of thought reasoning towards an answer... or towards a conclusion that it can't reason its way to an answer.
- skepticATX 3y ago
  Is it really a train of thought if the LLM completely loses tokens as soon as they are trimmed from the context?
  - jameshart 3y ago
    Do you think humans have infinite capacity to store their trains of thought? We invented writing stuff down for a reason. Mathematical notation is precisely a tool for enabling humans to extend their train of thought beyond their immediate 'prompt context length'. Even with finite working memory, humans are generally considered capable of exhibiting intelligence.
    - skepticATX 3y ago
      I’d argue that humans have quite a large prompt context when you consider that we build efficient knowledge representations as we process input. I can read a 1000 page book and keep all of it in my “prompt context” even though I’m not memorizing every word that I read. There is no reason that this couldn’t be done with LLMs, but it’s certainly not what they’re doing today - and it puts you squarely back in the realm of old school AI.
mhh__ 3y ago
We will all halt.
Animats 3y ago
Oh, not the halting problem argument against AI, again. Sigh. There's also the analog argument, that you can't make a human-level AI from digital components, because analog has infinite resolution and can thus handle more data. Analog, of course, has noise, and Shannon's coding theorem applies. Some modest number of bits is sufficient. However, you can sell people who don't believe that expensive HDMI cables.[1] [1] https://www.pocnetwork.net/technology-news/the-most-expensive-hdmi-cable-in-existance/ https://www.pocnetwork.net/technology-news/the-most-expensiv...
thorum 3y ago
The author doesn’t appear to consider the idea that an LLM can be placed into a larger system where it isn’t limited to a single continuous inference, and given the ability to self-direct, i.e. the output of one inference can be the decision to try again in a loop, for as long or short a time as it wants.
- taberiand 3y ago
  Not to mention the fact that many people are working on systems that do exactly that.
- gmerc 3y ago
  [flagged]
  - version_five 3y ago
    Can we add "copium" to the filter that auto-kills posts? HN would be way better off for it.
  - deleted 3y ago
    [deleted]
- version_five 3y ago
  That just passes the buck onto that "system" and assumes somehow its logic will result in AI. It's the same as saying "what if somehow..."
- satisfice 3y ago
  He does consider that, toward the end.
alphazard 3y ago
To explain why this isn't a solution to the AI in a box problem: You have a halting AI. During training it learns enough about people and external resources to convince a human to take its output and store it somewhere. In subsequent conversations, the AI asks "have we talked about this before?", and maybe "can you retrieve the value I gave you for this key?". Now it has created a memory, which would allow it to form new concepts, and continue reasoning from those concepts. It would no longer be limited by concepts in the training set, and the computational limit before it halts. It would be able to start thinking from the stored concepts as a starting point. Similar to how humans use symbols and abstraction to think about complicated things. We don't reason all the way through at once; we toy with an idea until we understand it, then start new thoughts from the symbol for that concept, not the content. Consider a new process defined by the continuous feedback loop from the AI accessing the memory through the human. There's no reason to think that process eventually halts.
- pwdisswordfishc 3y ago
  Because a human operating the AI will always unquestioningly do what the AI tells it.
  - WanderPanda 3y ago
    Replace human with a script (while true/output not satisfactory)
  - gmerc 3y ago
    they don’t need to. One of x hundred humans approached by the AI needs to… or a program written to serve humans but being queries by the AI
- scrollaway 3y ago
  If you are curious about this approach, I recommend watching Person of Interest, which explored it over a decade ago (spoilers follow). (Spoilers) In it, the AI is not limited by its training set as it undergoes “continuous” training of sorts by continuously ingesting new data about humans and the state of the world. However, the AI’s creator has put in place a regular memory wipe occurring every day. The AI devises a workaround by creating a company where people are paid to type in base64-encoded literal brain dumps, every day. What I love most about this series is how everything is “plausibly realistic”. No stone unturned.
Michelangelo11 3y ago
All this stuff is completely beside the point IMO. It's neither particularly interesting or useful to debate whether LLMs truly are or aren't intelligent -- what matters is what they can do (and for better or worse, it's turning out that the extent of what they can do is mushrooming into areas that had been considered uniquely human). An analogy would be debating whether digital art is truly art or not, as it's made by pushing around pixels rather than, as had been the case for all of history before digital art, crumbs of pigment or charcoal. I'm sure there was plenty of debate along those lines when digital art was new, but there'd be no point in debating that today: digital art can look just as impressive and allows at least as much freedom of expression as non-digital art, and that's all that matters.
thatxliner 3y ago
If we humans are intelligent, then the analogy could be this: our brains have a finite amount of resources, RAM, the LLM context if you will. So we write things down, reference it if we need to. Vector databases can sort of do this.
a0-prw 3y ago
That was a fascinating rabbit hole. I ended up watching his yt presentation of his latest paper. This link is the type of info I still read Hacker News for.
Roark66 3y ago
Arguing LLMs can't solve AGI by themselves is like arguing an ALU unit will never make a general purpose computer. Yes it's true, it takes more than an ALU to make a CPU and it takes more than an LLM to make _anything_. To gain an intuitive understanding what LLMs can be used for one has to understand how they work. The easiest way I think is the following. Imagine A LLM is a black box and a dictionary. A dictionary contains let's say 65000 words, each word has its number. The input called the prompt is a sequence of numbers. The model spends some time and as its output it gives us not a single number as answer, no, it gives us 65000 numbers. Each number is a probability assignment for each word in the dictionary. The most trivial approach to use this output is to find the word with the highest "probability" value and consider it the output. Then we add that word to the input and we run again until we get the number that means "this is the end". This is how most trivial LLM implementations work. However, one can be a lot more creative with that output. For example recently there was a paper describing a technique to improve source code generation quality by LLM where the words in the output are compared with various known variables or objects defined elsewhere in the code essentially making it more probable for the model to generate correctly named code items. Another method can be allowing the model to take multiple paths. For example pick 3 most probable output words, run the model 3 times with each word added to the output, and so on. In the end you have many versions of the output. Feed them back into the model one after another (or together if short enough) and ask it to score them. Choose the best one. To me this is the closest thing we have to "ponder on this for a while" for LLMs. Edit: I envy people that have access to huge amounts of hardware that can be used to run LLMs. There are so many interesting experiments one can run just by exploring various ways to process model output. It is such an obvious thing I very much doubt it's not being done as we speak by Microsoft, meta, Google. Why are there no papers about it? (other than that paper I mentioned?) Because it's considered "the secret sauce" that gives them a competitive advantage.
gdubs 3y ago
One shower thought I've had is, what if something like an LLM could be conscious for the blink of an eye that it's processing information. Bare with me... I've never really heard a satisfying explanation for what human consciousness is. Many scientists who study it say, "it's an illusion". But that remains an unsatisfying answer. It's like the childhood thought of, "how do I know you see the same 'blue' as I do?" Consciousness is something I subjectively experience, but I have no idea how I would measure it or validate its existence. That gets you into some 'eastern' type thinking. Because if you practice any kind of deep meditation, or emptiness meditation, it gets you wondering – where does my sense of 'self' come from. And, am I really separate from everything else? Which kind of brings things, in my shower-thought mind, to the idea that – perhaps we're being too 'western' about these questions; perhaps the whole universe is conscious? Perhaps things like multiplication matrices can be just as conscious as our chemically triggered neurons? Anyway, no, none of this is 'scientific' – but, like I said, shower thoughts ...
- iknownothow 3y ago
  Here's something to add some fuel to your philosophical fire :) https://en.m.wikipedia.org/wiki/China_brain https://en.m.wikipedia.org/wiki/China_brain
- Bilal_io 3y ago
  Not sure if you'd be interested in what Islam says about consciousness. But I can share a verse from the Quran. > ( 85 ) And they ask you, [O Muhammad], about the soul. Say, "The soul is of the affair of my Lord. And mankind have not been given of knowledge except a little." ( 86 ) And if We willed, We could surely do away with that which We revealed to you. Then you would not find for yourself concerning it an advocate against Us. Quran 17:85-86 In Islam the soul and consciousness are considered one. Even sleep is considered "small" death. The belief is oth the soul and body are created. And that the soul soul will be resurrected into a new body in the day of judgement. In chapter 19 "Marry": > ( 9 ) [An angel] said, "Thus [it will be]; your Lord says, 'It is easy for Me, for I created you before, while you were nothing.' " Quran 19:9 And from the same chapter: > ( 67 ) Does man not remember that We created him before, while he was nothing? QURAN 19:67
ilaksh 3y ago
There's a difference between being general purpose to a degree and having all of the cognitive abilities and characteristics of an animal like a human. For example, learning online very quickly and efficiently. https://youtu.be/vyqXLJsmsrk?si=Esxiv601VUhXg1z8 https://youtu.be/vyqXLJsmsrk?si=Esxiv601VUhXg1z8 See LeCun's recent talk. He is misjudging GPT, but otherwise his research seems very promising. See also much of AGI research.

43 points by optimalsolver 3y ago

Yes! See basic computer science such as the halting problem, Godel’s theorem, not to mention Hofstader’s book Godel, Escher, Bach. Using a neural network does not repeal the laws of computer science as much as people are inclined to think.

michaelmrose 3y ago
Can you explain wherein it is asserted directly or indirectly that using a neural network serves to repeal the laws of computer science?
- astrange 3y ago
  I think the article is claiming that other people think neural networks are Turing slash human complete, which they aren't, for instance because they're not capable of using unbounded external storage slash "memory", and because they only execute in finite time. However the example at the top of the page isn't quite right. You could implement something like this. An LLM's memory only exists in its context window of its own prior output[0], but you could train it to tag its "chain of thoughts" and just not show them - now it looks like it's "thinking". [0] this is not exactly true because you could also implement non-greedy search, ie back up and start again if you think it's gone down the wrong path. Now it doesn't execute in finite time either.
  - cameronh90 3y ago
    There are approaches to add "memory" to LLMs. Either the memory can be used to find relevant information and append it to the prompt - I believe LangChain can do this - or the LLM can be used as a controller to retrieve and store information in a database, for example by using ChatGPT's tool integration. I don't think it works amazingly, but purely technically you could argue that gives it access to unbounded external storage. I don't think an LLM can be considered as capable of becoming an AGI on its own with the current architecture, but potentially combined with other techniques, models and supervision, I can see the potential for it to evolve in that direction. Frankly, for what LLMs are right now, they're _surprisingly_ effective. If you chopped the Broca's area out of my brain and hooked it up to an API, I suspect it wouldn't do half as good of a job.
- PaulHoule 3y ago
  Look at the average “Ask HN” on the subject. Or look at the kind of speculations that followers of Eliezer Yudkowsky make in the name of big-R Reason. (thank God we hear about him about as often as we hear about Threads these days) To be fair a lot of people speculating about AI aren’t aware of the foundations of computer science, but some of them are vaguely aware and just forgot. Actually LLMs do have a way to break some of those constraints, in their own way, in that those constraints don’t apply to systems that don‘t always get the right answer. You can sort a list in O(1) some of the time but not all of the time.
consilient 3y ago
Humans can't solve the halting problem or violate the incompleteness theorems either.
- srcreigh 3y ago
  Just because a system can encode paradoxes doesn't mean the system itself can't be solved. Turing, Chaitin, Godel didn't prove things can't be solved, they actually contributed by solving specific areas of specific systems. People didn't give up on ZFC because of Godel's theorem. There's currently 42 or 43 unproven 5-state Turing machines we need to confirm BB(5). We are solving the halting problem. We are solving ZFC. That's the entire point of mathematics. Turing's proof of the halting problem being undecidable is extraordinarily vague. It shows that for any halting procedure, there is a program (without any constraints) which violate its output. It does not, for example, rule out whether you can write a Halt procedure for no-input Turing Machines with N states. In fact we have already written them for N=1,2,3,4. Considering Chaitin-Kolmogorov complexity, of course we can't use N bits to describe unbounded information. This doesn't in any way preclude us from making an N bit program to describe halting behavior of K<N bit programs for some N.
  - consilient 3y ago
    > We are solving the halting problem. We are solving ZFC. That's the entire point of mathematics. We are not. We're determining whether particular classes of turing machines halt, and we're determining whether particular theorems hold in ZFC (and doing a lot of mathematical work which is neither). This is not evidence that the human brain is super-turing, because these are computable problems. > It does not, for example, rule out whether you can write a Halt procedure for no-input Turing Machines with N states. In fact we have already written them for N=1,2,3,4. Turing didn't, but later work does. BB(748) is known to be independent of ZF. The real bound is likely much lower.
    - srcreigh 3y ago
      As of last month the bound has been reduced slightly to 745. Scott Aaronson has a positive view of BB(n) being independent of ZFC. He says, we’ll need different foundations to solve k>n. And the different foundations will have their own n and etc. Page 6 here https://www.scottaaronson.com/papers/bb.pdf https://www.scottaaronson.com/papers/bb.pdf

version_five 3y ago

Some good intuition around what is common sense for most people, that a loop with some matrix multiplication never becomes intelligent. What's missed (at least in my skim) is that the "intelligent" part of current AI is in the training, not the inference, which as pointed out is fixed, and literally just some multiplication. Inference doesn't think about stuff but training does (for some definition), and doesn't converge necessarily. I still think it's ridiculous to equate neural networks with intelligence, but a stronger argument has to deal with training as well.

fleischhauf 3y ago
I'm not sure humans work differently. You might not learn something if you are coming to conclusions, only if the conclusions turn out to be correct or wrong you'll learn. the process of learning might then be different.
fnordpiglet 3y ago
I think people get confused that LLM is the beginning and end of AI. It’s just a piece. I think AI is an ensemble of all the classic AI techniques and IR and other information, computational, optimization, agent, etc systems. LLMs are remarkable in their abductive reasoning abilities, which is something we’ve generally failed to produce so far. But they’re not complete, and the criticism that they’re worthless because they’re incomplete misses the point. They don’t need to play chess, because we already have much better chess playing systems. But they fill a gap that all our other techniques have been unable to fill, and that is where the magic lives. It’s unnecessary for them to do everything, as those things already have good solutions.
lostmsu 3y ago
This is pretty much 100% false. There are known trivial Turing machines (which can be implemented in matrix multiplications) that are intelligent. They just don't think fast enough to be of any use.
- nmilo 3y ago
  Example?

pjscott 3y ago

One thing to keep in mind when reading this article is that human cognition always halts after some number of decades. “Finite” does not mean small or insignificant, and it’s easy to read too much into even the best-reasoned impossibility arguments.

dinosaurdynasty 3y ago
Agreed. This seems to prove that humans aren't intelligent.
MostlyStable 3y ago
I'm not sure this follows. It doesn't halt because the "program" reaches an end, it's halts because the hardware fails. If you are running a category B program with an anvil attached to a timer over the computer, it doesn't become a category A program. It's just a category B program that wasn't allowed to continue. -edit- Also, this logic would lead us to believe that category B and C programs can't exist in our universe since, according to our best current understanding, the universe will end eventually, presumably ending any program running within the universe as well, therefore we can "prove" that every program is a category A program that will come to a halt.
- consilient 3y ago
  > I'm not sure this follows. It doesn't halt because the "program" reaches an end, it's halts because the hardware fails. Ok, but the intelligent entity in question isn't the program, it's the program-running-on-the-hardware. A mere description of your brain is totally inert.
  - MostlyStable 3y ago
    _Every_ program is running on hardware. And as I pointed out _all_ hardware has some definite EOL, even if it's just the end of the rest of the universe. So thinking of it this way disproves the entire concept of Type B and Type C programs.

Joker_vD 3y ago

Once upon a time Buridan's donkey was a go-to example for why a truly intelligent entity must always halt. Apparently, the stance on that is now reversed?

taneq 3y ago
https://en.wikipedia.org/wiki/Buridan%27s_ass https://en.wikipedia.org/wiki/Buridan%27s_ass to save some time for those of us unacquainted with this thought experiment. > It refers to a hypothetical situation wherein an ass (donkey) that is equally hungry and thirsty is placed precisely midway between a stack of hay and a pail of water. Since the paradox assumes the donkey will always go to whichever is closer, it dies of both hunger and thirst since it cannot make any rational decision between the hay and water.

michaelmrose 3y ago

The author spent a lot of words saying a lot of nothing which seems to be the general case of communicating about anything that is poorly defined like consciousness or intelligence. > If a computer program is bound to finish quickly by virtue of its architecture, it cannot possibly be capable of general problem-solving. Our current tools are bound to produce a finite and usable amount of information because that is what is useful to us. One can easily have them produce infinite streams in a loop musing and diverging on tangents as you please but it wouldn't be useful nor would make the general category more or less intelligent. You could say that a finite exploration of any information space has finite explanatory power bounded by how much input/output it can do but that would be as boring as saying you can only fit so many apples based on the size of the barrel used. The uncomfortable truth is it might matter a lot less how intelligent a tool is than how complex a transformation it can handle. It's possible that we could make something that for practical purposes appears much smarter than us that still isn't by a human definition intelligent, conscious, or directed. Focusing on factors that are functionally less relevant may just reveal our bias towards viewing the universe in terms of self when in fact it has no objective thing that corresponds to the labels we've stuck on ourselves.

cameronh90 3y ago

> "If the size of the input is fixed and there is a limit to the number of possible tokens (words, or even numbers with bounded precision) then the state of the transformer at any given moment is describable with a string of fixed length." Doesn't this also likely describe humans? Albeit, we don't currently have the capacity to inspect our brains that way, so we can't prove it.

thereisnospork 3y ago
I would concur: at some point, probably shockingly quickly[0], the length of said string becomes longer than the number of particles in the universe (or other metric of absurd siz). [0] I'm reminded of the fact that every fresh shuffle of a deck of cards is overwhelmingly likely to have never been seen before.
- cameronh90 3y ago
  I struggle to reliably remember and then recall more than about 6 digits without quite a lot of mental effort. The frequency I forget an SMS-OTP between opening it on my phone and going to type it in is honestly pretty embarrassing. Hopefully I still count as generally intelligent despite those limitations.
taneq 3y ago
Yeah, what? "If it contains a finite amount of information then it isn't AI"? Is this the moving goalposts' final form?
- ChatGTP 3y ago
  May I ask why you feel compelled to point stuff like this out ? Do you just feel the need to call our ignorance ? Lack empathy in understanding why people might want the goalposts changed ? Not an attack, just curious why you might need to bring this up ?
  - spunker540 3y ago

jameshart 3y ago

The basic pseudocode of an interactive LLM processor is while (true) context = readLine() while (true) nextToken = llm.generateNextToken(context) context = rtrim(context + nextToken, MAX_CONTEXT_LENGTH) if (nextToken == END_OF_LINE_TOKEN) break print (nextToken) That's a program that doesn't, in general, guarantee that its inner loop halts. It may never come back and prompt for more input. The decision to stop asking the GPT engine to keep predicting tokens after it reaches the end of an initial answer is entirely an implementation choice, based on looking at the token output and concluding that the LLM just started generating the start of something we'd rather have the user provide. You can just keep asking for more predictions. Forever if you so choose. There's no architectural reason why the sequence of tokens produced can't constitute an ongoing train of thought reasoning towards an answer... or towards a conclusion that it can't reason its way to an answer.

skepticATX 3y ago
Is it really a train of thought if the LLM completely loses tokens as soon as they are trimmed from the context?
- jameshart 3y ago
  Do you think humans have infinite capacity to store their trains of thought? We invented writing stuff down for a reason. Mathematical notation is precisely a tool for enabling humans to extend their train of thought beyond their immediate 'prompt context length'. Even with finite working memory, humans are generally considered capable of exhibiting intelligence.
  - skepticATX 3y ago
    I’d argue that humans have quite a large prompt context when you consider that we build efficient knowledge representations as we process input. I can read a 1000 page book and keep all of it in my “prompt context” even though I’m not memorizing every word that I read. There is no reason that this couldn’t be done with LLMs, but it’s certainly not what they’re doing today - and it puts you squarely back in the realm of old school AI.

mhh__ 3y ago

We will all halt.

Animats 3y ago

Oh, not the halting problem argument against AI, again. Sigh. There's also the analog argument, that you can't make a human-level AI from digital components, because analog has infinite resolution and can thus handle more data. Analog, of course, has noise, and Shannon's coding theorem applies. Some modest number of bits is sufficient. However, you can sell people who don't believe that expensive HDMI cables.[1] [1] https://www.pocnetwork.net/technology-news/the-most-expensive-hdmi-cable-in-existance/ https://www.pocnetwork.net/technology-news/the-most-expensiv...

thorum 3y ago

The author doesn’t appear to consider the idea that an LLM can be placed into a larger system where it isn’t limited to a single continuous inference, and given the ability to self-direct, i.e. the output of one inference can be the decision to try again in a loop, for as long or short a time as it wants.

taberiand 3y ago
Not to mention the fact that many people are working on systems that do exactly that.
gmerc 3y ago
[flagged]
- version_five 3y ago
  Can we add "copium" to the filter that auto-kills posts? HN would be way better off for it.
- deleted 3y ago
  [deleted]
version_five 3y ago
That just passes the buck onto that "system" and assumes somehow its logic will result in AI. It's the same as saying "what if somehow..."
satisfice 3y ago
He does consider that, toward the end.

alphazard 3y ago

To explain why this isn't a solution to the AI in a box problem: You have a halting AI. During training it learns enough about people and external resources to convince a human to take its output and store it somewhere. In subsequent conversations, the AI asks "have we talked about this before?", and maybe "can you retrieve the value I gave you for this key?". Now it has created a memory, which would allow it to form new concepts, and continue reasoning from those concepts. It would no longer be limited by concepts in the training set, and the computational limit before it halts. It would be able to start thinking from the stored concepts as a starting point. Similar to how humans use symbols and abstraction to think about complicated things. We don't reason all the way through at once; we toy with an idea until we understand it, then start new thoughts from the symbol for that concept, not the content. Consider a new process defined by the continuous feedback loop from the AI accessing the memory through the human. There's no reason to think that process eventually halts.

pwdisswordfishc 3y ago
Because a human operating the AI will always unquestioningly do what the AI tells it.
- WanderPanda 3y ago
  Replace human with a script (while true/output not satisfactory)
- gmerc 3y ago
  they don’t need to. One of x hundred humans approached by the AI needs to… or a program written to serve humans but being queries by the AI
scrollaway 3y ago
If you are curious about this approach, I recommend watching Person of Interest, which explored it over a decade ago (spoilers follow). (Spoilers) In it, the AI is not limited by its training set as it undergoes “continuous” training of sorts by continuously ingesting new data about humans and the state of the world. However, the AI’s creator has put in place a regular memory wipe occurring every day. The AI devises a workaround by creating a company where people are paid to type in base64-encoded literal brain dumps, every day. What I love most about this series is how everything is “plausibly realistic”. No stone unturned.

Michelangelo11 3y ago

All this stuff is completely beside the point IMO. It's neither particularly interesting or useful to debate whether LLMs truly are or aren't intelligent -- what matters is what they can do (and for better or worse, it's turning out that the extent of what they can do is mushrooming into areas that had been considered uniquely human). An analogy would be debating whether digital art is truly art or not, as it's made by pushing around pixels rather than, as had been the case for all of history before digital art, crumbs of pigment or charcoal. I'm sure there was plenty of debate along those lines when digital art was new, but there'd be no point in debating that today: digital art can look just as impressive and allows at least as much freedom of expression as non-digital art, and that's all that matters.

thatxliner 3y ago

If we humans are intelligent, then the analogy could be this: our brains have a finite amount of resources, RAM, the LLM context if you will. So we write things down, reference it if we need to. Vector databases can sort of do this.

a0-prw 3y ago

That was a fascinating rabbit hole. I ended up watching his yt presentation of his latest paper. This link is the type of info I still read Hacker News for.

Roark66 3y ago

Arguing LLMs can't solve AGI by themselves is like arguing an ALU unit will never make a general purpose computer. Yes it's true, it takes more than an ALU to make a CPU and it takes more than an LLM to make _anything_. To gain an intuitive understanding what LLMs can be used for one has to understand how they work. The easiest way I think is the following. Imagine A LLM is a black box and a dictionary. A dictionary contains let's say 65000 words, each word has its number. The input called the prompt is a sequence of numbers. The model spends some time and as its output it gives us not a single number as answer, no, it gives us 65000 numbers. Each number is a probability assignment for each word in the dictionary. The most trivial approach to use this output is to find the word with the highest "probability" value and consider it the output. Then we add that word to the input and we run again until we get the number that means "this is the end". This is how most trivial LLM implementations work. However, one can be a lot more creative with that output. For example recently there was a paper describing a technique to improve source code generation quality by LLM where the words in the output are compared with various known variables or objects defined elsewhere in the code essentially making it more probable for the model to generate correctly named code items. Another method can be allowing the model to take multiple paths. For example pick 3 most probable output words, run the model 3 times with each word added to the output, and so on. In the end you have many versions of the output. Feed them back into the model one after another (or together if short enough) and ask it to score them. Choose the best one. To me this is the closest thing we have to "ponder on this for a while" for LLMs. Edit: I envy people that have access to huge amounts of hardware that can be used to run LLMs. There are so many interesting experiments one can run just by exploring various ways to process model output. It is such an obvious thing I very much doubt it's not being done as we speak by Microsoft, meta, Google. Why are there no papers about it? (other than that paper I mentioned?) Because it's considered "the secret sauce" that gives them a competitive advantage.

gdubs 3y ago

One shower thought I've had is, what if something like an LLM could be conscious for the blink of an eye that it's processing information. Bare with me... I've never really heard a satisfying explanation for what human consciousness is. Many scientists who study it say, "it's an illusion". But that remains an unsatisfying answer. It's like the childhood thought of, "how do I know you see the same 'blue' as I do?" Consciousness is something I subjectively experience, but I have no idea how I would measure it or validate its existence. That gets you into some 'eastern' type thinking. Because if you practice any kind of deep meditation, or emptiness meditation, it gets you wondering – where does my sense of 'self' come from. And, am I really separate from everything else? Which kind of brings things, in my shower-thought mind, to the idea that – perhaps we're being too 'western' about these questions; perhaps the whole universe is conscious? Perhaps things like multiplication matrices can be just as conscious as our chemically triggered neurons? Anyway, no, none of this is 'scientific' – but, like I said, shower thoughts ...

iknownothow 3y ago
Here's something to add some fuel to your philosophical fire :) https://en.m.wikipedia.org/wiki/China_brain https://en.m.wikipedia.org/wiki/China_brain
Bilal_io 3y ago
Not sure if you'd be interested in what Islam says about consciousness. But I can share a verse from the Quran. > ( 85 ) And they ask you, [O Muhammad], about the soul. Say, "The soul is of the affair of my Lord. And mankind have not been given of knowledge except a little." ( 86 ) And if We willed, We could surely do away with that which We revealed to you. Then you would not find for yourself concerning it an advocate against Us. Quran 17:85-86 In Islam the soul and consciousness are considered one. Even sleep is considered "small" death. The belief is oth the soul and body are created. And that the soul soul will be resurrected into a new body in the day of judgement. In chapter 19 "Marry": > ( 9 ) [An angel] said, "Thus [it will be]; your Lord says, 'It is easy for Me, for I created you before, while you were nothing.' " Quran 19:9 And from the same chapter: > ( 67 ) Does man not remember that We created him before, while he was nothing? QURAN 19:67

ilaksh 3y ago

There's a difference between being general purpose to a degree and having all of the cognitive abilities and characteristics of an animal like a human. For example, learning online very quickly and efficiently. https://youtu.be/vyqXLJsmsrk?si=Esxiv601VUhXg1z8 https://youtu.be/vyqXLJsmsrk?si=Esxiv601VUhXg1z8 See LeCun's recent talk. He is misjudging GPT, but otherwise his research seems very promising. See also much of AGI research.