15 ms·

Thought experiment: if sentient AI is possible with nothing more than software, does that mean if you "load" the sentient AI program into a "computer" made of c

by ythn 10y ago

Thought experiment: if sentient AI is possible with nothing more than software, does that mean if you "load" the sentient AI program into a "computer" made of cardboard and marbles, that the cardboard and marbles will be self-aware?

deleted 10y ago
[deleted]
ae35 10y ago
That rephrases this: https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
- samirillian 10y ago
  Which rephrases this https://en.m.wikipedia.org/wiki/Leibniz's_gap https://en.m.wikipedia.org/wiki/Leibniz's_gap
- Aardwolf 10y ago
  This thought experiment seems to say: "the human following the instructions doesn't understand the Chinese, therefore strong AI is impossible". Seems like a fallacy to draw that conclusion, it doesn't prove anything, this is like saying: this tomato is not yellow, so bananas are not yellow. In addition, how can he say strong AI (which he defines as "understanding") isn't possible, doesn't he have at least one example of it, which is his own mind?
  - jhbadger 10y ago
    Taken literally, it even suggests human intelligence is impossible. After all, our thoughts originate from a bunch of neurons following instructions and passing information between each other. And each individual neuron isn't any more intelligent than any other cell and can't be said to "understand" our thoughts any more than the man in the Chinese Room understands Chinese.
  - drostie 10y ago
    Hi! Searle fan, here. Searle is not, as you might at first anticipate when you see that he's on the "computers can't be conscious" side of the argument, some sort of woo-woo psychic-spiritualism philosopher; he's actually really grounded and relatively materialistic, though he despises the latter term. (He thinks that you should not respond to the problems of dualism by choosing one or the other side of the duality to commit to, and argue the other into nonexistence -- but rather by saying "oh I guess it was stupid to draw a line here in the first place.") Even the earlier simplification I gave, he "thinks computers can't be conscious", he'd object to directly. He thinks that you're a computer, and he thinks that you're conscious. He just doesn't think that you're conscious by virtue of being the computer that you are. So that's who we're talking about here. Now, Searle says: we know, or think we know, that the brain is conscious, and it's conscious due to some aspect of the dance of molecules that's going on under the hood. And we happen to have this model of dances, called computation, which is defined in terms of abstract symbol-shuffling: the great strength of computation is that the 0s and 1s can be voltages and electric currents (as in transistors) or magnetic alignments of spins (as in hard drive) or how a gear is turned (as in the difference engine) or whether a resistance is infinite or low (as in your keyboard). They're just abstract symbols. At its most abstract the Turing test suggests that any human language can be boiled down to these abstract intrinsically-meaningless symbols (which is almost certainly true; that's the premise of the invention of writing), and then that anything which can output symbols in a way that's indistinguishable from a human being, deserves to be given the title of "understanding" that language. Searle steps in and says, no, that's crazy. And he says it's crazy because he's a philosopher and philosophers are very concerned with what words mean exactly, and the word understanding means exactly that these things you're talking about are not abstract symbols, but a computer necessarily treats them that way; it's in the definition of computation that the symbols are abstract. Now that you understand who this is and where he's coming from, maybe you can understand the argument better. He's trying to try to give a more formal proof of this above statement. So, he steps in with this Chinese room argument. The Chinese Room argument says: Your brain is conscious, because of the dance that its neurons do. Now according to this nice computer-reductionalist view, anything which does any dance is understanding a language as long as it produces results which are indistinguishable from a human's responses. So Searle says: let's step inside some Turing machine's dance so our brains are doing two dances at once. The very asset which makes computers awesome allows us to take a Turing-test-passing-algorithm and perform it ourselves: we are the computer, we get this stream of symbols in, we look up rules in some complicated rulebooks, we stream some symbols out, and we now pass the Turing test for understanding Chinese. But is the test correct: do we actually understand Chinese? No. There is no reasonable definition by which we understand Chinese just by performing this complicated computational dance. And this applies even if we move all of the rulebooks and whatever into your skull so that the neurons do those things. Now here's where we come to the conclusion that strong AI is impossible: the human has everything which is part of the definition of computation. I'll repeat that twice more. The human embodies a whole, complete, computer. There's nothing which a computer can be (by virtue of being a computer) which the human is not. So if we go this far and we say "that brain does not understand Chinese" we have to carry that over; "that silicon does not understand Chinese, either." And no computer will understand Chinese merely by virtue of running this computer program. Clearly our understanding of Chinese is not a property of arbitrary dances of neurons which can solve a Turing test, but there is something special about the right dance of neurons which solves a Turing test. The mere fact that the right dance exists points to some physics which is not reducible to computation. In other words Searle thinks that you can simulate people as much as you please, but that simulations are not reality, thank you very much. Just like we humans are very good at simulating pain-behavior without actually being in pain, so too can a computer simulate understanding-behavior without actually understanding a thing. If you want to understand how understanding works, computers will doubtless be a useful tool, but they cannot offer a complete and final answer precisely because they do not pin down the material dance firmly enough to get to something causal.
  - posterboy 10y ago
    On the other hand Google translate is (was?) a clear example that word by word translations aren't intelligent or even decidable. So he's got a point, but far from a general proof.
    - jessedhillon 10y ago
      That doesn't at all address the point of the argument. Given the following - an assumption that you can generate an algorithm to express behavior indistinguishable from a human's at a given task, and - an implementation of the algorithm at a macroscopic scale, carried out by individual humans, each executing a small part of the algorithm then, where in this system would you say that an actual understanding of the task exists? Google Translate doesn't pass the first requirement.
      - posterboy 10y ago
        That assumption is too simplistic. If the algorithmic behavior was indistinguishably human behavior, and carried out by humans, it would just be human behavior. Of course machine translation doesn't pass the requirement for human behavior. Nothing does, except humans. And if newer machine translation does pass, I'd say that's humanly accomplished, by use of tools. Nobody could learn a distinct language from nothing but a dictionary, enough to fool a native speaker. The assumption is ridiculous. The dictionary isn't conscious, either way.
        jessedhillon 10y ago
        https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room If you have a challenge to the Chinese Room argument, go read and respond to Searle's original essay. There's no point in responding to my TL/DR version -- offered only to correct your initial misapprehensions -- with further misapprehensions.
- bbctol 10y ago
  The Chinese room is a silly example, in which a highly simplified physical system that doesn't appear to have conscious thought is extended to the idea that no merely physical system has conscious thought. If the structure of the Chinese room was not a simple catalog of questions and answers, but hundreds of billions of complicated cardboard-and-marble apparatus, each processing some amount of data in a way that individually seemed meaningless, but together produced cogent speech... would you still be so sure it wasn't conscious?
  - blackflame7000 10y ago
    This is a good point. The chinese room seems to stem from a more antiquated system of serial processing and the analogy doesn't quite hold when we look at the massive parallelism that occurs in the human mind that gives us consciousness. Furthermore humans can still think in the absence of understanding. In fact id argue a majority thoughts by people are conjugated without full understanding of the subject of the thought.
- geofft 10y ago
  The trouble I've always had with this argument is that the amount of books needed to convincingly converse in Chinese would be staggeringly huge. You essentially have an entire human personality as a lookup table. It's not just that you can ask, in Chinese, What's your favorite food, and somewhere in the pages is an answer. You can ask, Tell me about your first love, and somewhere in the pages is an entire conversation. And not only would it be an entire conversation, it would be every possible conversation, depending on how the outside person replied. If it helps visualize the problem - the outside person could say, How about a game of chess?, and the inside person cannot rely on their knowledge of chess, so the books must encode every possible chess game, convincingly enough to hold up a game. Imagine how big a lookup table for chess would be. And that's just a tiny part of what a person knows. And then despite having you accept that without question, and picture it as a small room, Searle turns around and says it would not be "at all plausible" to imagine the system as being conscious. The system is almost entirely made of book. The human in the middle is almost invisible if you look at the system. That's the implausible part - that a human could even operate this Borges library. It's so unreasonable that intuition past that point doesn't give you any useful conclusions. But, if you suspend disbelief enough to accept this galaxy-sized Chinese megalibrary, it's not implausible that "conscious" is a fine adjective for an operable printout of an entire consciousness.
  - logfromblammo 10y ago
    You do not need a pre-existing, galaxy-sized library. The program operator has the ability to write into blank books and move them from one shelf in the library to another. The program could, in theory, assemble a book of knowledge on something that exists entirely outside the Room, solely from knowledge gleaned from the inputs passed in from outside. Your initial set of books might simply be a script to follow to elicit additional information and index all new information such that it can be retrieved later. You would then end up with a set of core rulebooks, a whole heap of raw conversations, and several layers of indexes into them. Searle's problem is that he decided the computer was incapable of understanding. But the computer is not the program. The library is the thing that might comprise a strong AI, not the human operator. The scary thing is that the human operator, having no understanding of the Chinese characters, does not actually know what the program is doing. It could smuggle instructions through the output window, through him, and the human operator would then be surprised when burly men drag him out of the Room one day, to replace him with a dozen monkeys who have all been trained to process the book instructions faster and more efficiently. Customers presenting inputs might then be surprised to learn that there is no longer a human inside the Room, as the conversations had become so much more interesting.
te_platt 10y ago
no
- jimktrains2 10y ago
  On what basis? If there was an "aware" AI program, it would be just as "aware" in a silicon computer as it would be in a marble computer (albeit much more slow).
  - ChuckMcM 10y ago
    Probably on the same basis that your blood cells, skin cells, etc aren't sentient. So the cardboard and marbles would not be sentient, but the system might be. Its the difference between an intrinsic property of the system and an emergent one.
  - te_platt 10y ago
    Hmmm... When I posted this the only other comment was "yes". I thought we could just vote this out to determine what the "correct" answer should be.
  - deleted 10y ago
    [deleted]
24gttghh 10y ago
The cardboard and marbles would be the engine which drives self-awareness. The individual parts are not self-aware. But considering the size and complexity that would be required for such a feat of engineering, a very liberal estimate would probably require more mass than is available on the entire planet. Edit: the sheer gravity of which would destroy the system.
- hinkley 10y ago
  Quite a house of cards.
- posterboy 10y ago
  That's semantically difficult to say, i've seen it in other comments, too, self is not used in the correct frame of reference: The individual parts' self is irrelevant; yet you claim, they are not self aware; when you imply the part's weren't aware of the whole ('s self), you switch frame of reference. The self of the sum of the parts are just the parts. Indeed the parts are the awareness. The whole might be aware of the self of each individual part, but for that to be true, contrary to your statement, each individual part doesn't need to be aware of itself (it might as well code for some other part or whatever).
phaedrus 10y ago
"Yes," except that the scale of the assemblage would have to be unimaginably enormous. Then, in order to continue working at that scale, it would have to incorporate copious error correcting and fail over. It would need mechanisms to reset and reload marbles, and a source of energy to raise them. The system for distribution of that energy would itself be complex... My point is that the Chinese Room argument is disingenuous. It oversimplifies the problem and asks for your incredulity that the oversimplified solution could answer the unsimplified problem.
- VLM 10y ago
  The specific bug the Chinese Room suffers from is its old and back then we'd have conversational AI next week so obviously it seemed a good example at the time of a non-trivial task that only conscious beings can perform. A re-stating of it in modern terms using modern skills would be that surely only conscious beings can invent and operate under Calculus rules. In fact its pretty hard to do and somewhat elitist in that only a small fraction of primates can learn calculus, larger than who know it now, but it'll never be 100%. So... Mathematica and Maple are decent computer algebraic systems. Better than most conscious beings, even. Well, OK someone programmed them. So fine how about Coq the theorem prover? Now the ancient Chinese Room argument extended to the modern computer is assume we have no access to the engineering department who created Mathematica because its 500 years from now or just the sake of argument or whatever. In fact assume the philosopher barely understands the concept of boolean logic much less floating point or algorithms. Then ask a philosopher which specific transistor in the CPU is the one that "understands" the calculus chain rule. What part of the hardware is the mathematica "for loop" construct as opposed to does the "for loop" construct? How much silicon and plastic does it take to think about Calculus? Observationally a lot less than it takes to do natural language conversational Chinese but why and how much and now it does not take a soul or disembodied spirit thingie to do Calculus and when did that change and when will it change for natural language conversational Chinese? Or heres some fun, when will technology advance enough such that self awareness and self authored demands for political autonomy no longer require soul spirit thingies and are just great piles of fast logic gates?
0xdeadbeefbabe 10y ago
Why don't you ask it?
jononor 10y ago
"To loose ones marbles" would get a more literal meaning...
monk_e_boy 10y ago
https://xkcd.com/505/ https://xkcd.com/505/
pavel_lishin 10y ago
> One definition of intelligence is the ability to skip deductive steps. To jump to a conclusion from the shadow of a ghost of a set of questions. It's preposterous that such a thing could be possible in an uncompromisingly digital reality, but if you make a computer wet enough, or big enough, or abstract enough, it will start to happen. And it has, now. https://qntm.org/transit https://qntm.org/transit
drzaiusapelord 10y ago
>that the cardboard and marbles will be self-aware? What a ridiculous idea and just as ridiculous as, say, meat becoming self-aware. Oh wait.
- hinkley 10y ago
  They're made out of meat: http://www.terrybisson.com/page6/page6.html http://www.terrybisson.com/page6/page6.html
  - lettergram 10y ago
    I prefer this version: https://m.youtube.com/watch?v=7tScAyNaRdQ https://m.youtube.com/watch?v=7tScAyNaRdQ
zeroer 10y ago
Yea, sure. What's so hard about that? XKCD made a comic about something similar: https://xkcd.com/505/ https://xkcd.com/505/. It's unintuitive, sure, but that's just because of the enormous size of the cardboard you'd need to do it. Human intuition is pretty bad outside of its human-sized comfort zone.
- teaearlgraycold 10y ago
  I think that's my new favorite XKCD.
- ythn 10y ago
  > Yea, sure. What's so hard about that? The hard part is coming to grips with consciousness merely being an abstract or concrete system changing states and nothing more. If rocks on the ground can be (slowly) self aware, then doesn't that mean everything in the universe is self-aware? For example, say I throw a bucket full of sand on the ground. Then, later, I define an incredibly complex abstract computer, such that the sand falling out of the bucket and shifting along the ground exactly represents a self-aware AI within my defined computer going through its states. Does that mean as I was pouring the the sand on the ground, the sand was momentarily self-aware?
  - Sharlin 10y ago
    Follow this train of thought to its logical conclusion and you get Greg Egan's Permutation City. Very much recommended.
    - jwhitlark 10y ago
      Thanks! Was looking for something different to read.
    - JonnieCache 10y ago
      Follow that train of thought to it's logical conclusion, and you end up here, via Big Yud: http://kruel.co/backup/Ontological%20Therapy.png http://kruel.co/backup/Ontological%20Therapy.png Still cracks me up, that page. (BTW you should definitely read Permulation City, and the 'sequel' Diaspora. Try not to found any cults tho.)
marcoperaza 10y ago
Does the act of the computation make it real? Or does the mathematical system being modeled exist independently of its computation?
- russdill 10y ago
  The act of doing the computation doesn't change the outcome of the computation. Still doesn't answer the question completely, but in my mind at least the idea that humans would act the same even if they weren't conscious is a non starter.
  - marcoperaza 10y ago
    My point is that if a machine can "create" consciousness through computation, but computation itself doesn't make the result any more real, then didn't the created consciousness exist all along?
    - russdill 10y ago
      Right, which is what I'm saying. It wouldn't make sense otherwise. It would mean that the result of the computation didn't depend on consciousness in the first place.
    - monort 10y ago
      That is the idea of mathematical universe of Tegmark: https://en.wikipedia.org/wiki/Mathematical_universe_hypothesis https://en.wikipedia.org/wiki/Mathematical_universe_hypothes...
tpeo 10y ago
The particularities of cardboard and marbles would likely prevent it from being "self-aware", in so far as "self-awareness" requires that the machine be fed a large amounts of information regarding itself, through itself. How many kinds of sensors can you build out of cardboard and marble? I don't know. But probably not many, and not as good or abundant as those receptors present in mammals, like those which constantly feed data into our nervous system. But yeah, if this isn't an actual problem, you'd probably have something self-aware.
- russdill 10y ago
  I think the limitation here is that the amount of cardboard required would lead to some sort of gravitational collapse. Other than that, there isn't a limit to the number of sensors you can build out of cardboard and marbles.
greendragon 10y ago
The cardboard and marbles aren't self-aware, they just encode a configuration that sees itself as self-aware when left alone to execute. Just like the atoms of your body. Self-awareness isn't a static property, since it's hard to call someone under anesthetic self-aware, but just another thing an agent in motion perceives. You ought to read Permutation City.
- edblarney 10y ago
  Materialism assumption. :)
sweetjesus 10y ago
well, not only is it true that--(if a sentient AI is possible with nothing more than software...), but also if you assume that we don't lose our enthusiasm for building simulations of ourselves, and for helping our little sisters with their projects, then you must conclude that with all probability we are actually living in such a school project simulation made of cardboard and marbles! see https://www.scientificamerican.com/article/are-we-living-in-a-computer-simulation/ https://www.scientificamerican.com/article/are-we-living-in-... It can't be proved that "it's marbles all the way down", but the odds are overwhelming that one of the layers is. Then once you accept that you have to accept that in all probability it's at least a layer of marbles every so often all the way down... thought experiment: In how many of those layered universes is there a cardboard and marbles Peter Thiel funding quirky ideas? Is it Peter Thiels all the way down?!
int_19h 10y ago
In order to get a useful answer to that question, you will need to precisely define what "self-aware" means, first. And to define that, you will need to first define "self" and "aware".
justinhj 10y ago
This is essentially Searle's Chinese Room. I studied it during undergraduate years and I've never been satisfied with his argument against strong AI. https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
amyjess 10y ago
Aside from the xkcd example, there's also the Machina Babbagenseii from Orion's Arm: sophont robots (vecs in OA-speak) made of walking Analytical Engines. http://www.orionsarm.com/eg-article/47f1ab093f416 http://www.orionsarm.com/eg-article/47f1ab093f416
aswanson 10y ago
Not any of the components, any more than the individual letters of the alphabet could be a tragic hero or Shakespeare sonnet, but they could be arranged such that system as a whole would be.
Dwolb 10y ago
Isn't a normal CPU the software runs on just silicon?
hyperion2010 10y ago
The reason why this is hard is because you also need to give the cardboard and marbles the ability to act in the world, that is a bit trickier. As long as you can manage the IO there shouldn't be an issue though.
deleted 10y ago
[deleted]
logicallee 10y ago
The very existence of a number means that it is self-aware. My argument: https://news.ycombinator.com/item?id=12878939 https://news.ycombinator.com/item?id=12878939 in effect, to a first approximation you can say something like "if there's a nonillionth prime, then it is self-aware and has just reported feeling nauseuous." Where the only change to that sentence to make it correct is that rather than "nonillionth prime" you need quite a bit more state and a lot more rules. Still, though, in effect numbers can be self-aware. I'm quite rigorous in my thinking and there is no room for any disagreement here, there's no ambiguity or open question around it.
- grkvlt 10y ago
  Well, of course there's an open question. Nobody has yet shown that a simulated neuron is exactly equivalent to a biological one; its generally assumed this is the case, but even simulating c. elegans brain won't show this 100%. I would also suggest that it is more correct to say your number S, as a representation of state, has felt (past tense) pain or whatever, but to suggest it is currently feeling anything would be to privilege some part of the number over another - although I'm not sure exactly how to be more rigorous about that?
  - logicallee 10y ago
    Suppose we replace 1 of your neurons with a black box that still chemically interfaces with your other neurons, but comes up with the decision about what to do in silicon. If you had a switch to flip back and forth between that black box and the biological version it replaced, your consciousness would not change as you flipped it back and forth - the same is true for replacing a few hundred, thousand, million or billion neurons: if the parts behave the same, the whole behaves the same. so imagine flipping that switch back and forth -- or better, imagine that I'm in front of you flipping it back and forth, but you don't see the state of the switch, and your neurons are biological sometimes but sometimes simulated (it's just a hypothetical, this isn't an actually possible machine). You can't even tell the position of the switch. You feel and report pain the same way, consciousness too. Would you really say that when the flip is switched to simulation, you don't really feel what you report feeling? Of course not. I think it's illusory to talk about the direction of time. The reason we do it this way is causative. The reason I say S "is feeling" pain is if you continue to simulate the next few quadrillion states, it ends up expressing/feeling those subjective feelings. In another sense the tense we use doesn't matter at all. I'm not sure what you're saying about privileging numbers - what does it mean to privilege a number?
anigbrowl 10y ago
No. The cardboard and marbles are the substrate on which the program runs; one might as well argue for the self-awareness of the individual carbon atoms in your brain. The program can't run without some sort of substrate, but the substrate is inert without the appropriate program; it is the system as a whole which is self-aware. Godel, escher, Bach by Douglas Hofstadter is a colorful, whimsical, yet rigorous treatment of this argument in book format.
- ralphc 10y ago
  My favorite example from GEB is Aunt Hillary, a self-away ant hill where the individual ants in it are not.
laxatives 10y ago
If you have a turing-complete machine, you can execute anything you could on any other turing-complete machine (given sufficient resources).
jchrisa 10y ago
I have thought of this as framing a paradox for the impossibility of strong AI. Turing machines can never really be embodied. They can always move to a different substrate. So intuitively, they don't have skin in the game in the same way that living things do. Mathematically, they are unable to represent the real numbers, only the rational numbers. Maybe this intuition is what you're getting at here.
signa11 10y ago
> if sentient AI is possible with nothing more than software, does that mean if > you "load" the sentient AI program into a "computer" made of cardboard and > marbles, that the cardboard and marbles will be self-aware? fwiw, quote from moon-is-a-harsh-mistress: Am not going to argue whether a machine can "really" be alive, "really" be self-aware. Is a virus self-aware? Nyet. How about oyster? I doubt it. A cat? Almost certainly. A human? Don't know about you, tovarishch, but I am. Somewhere along evolutionary chain from macromolecule to human brain self-awareness crept in. Psychologists assert it happens automatically whenever a brain acquires certain very high number of associational paths. Can't see it matters whether paths are protein or platinum.
laumars 10y ago
No. The parts themselves aren't self aware - it's the sum of the parts.
jtinder 10y ago
You are a self-aware mixture of lots of carbon atoms, water molecules, several metals, etc... of many solids, liquids and gases.
sorokod 10y ago
This is one of the themes in "I am a strange loop" by Hofstadter. If I remember correctly, the relevant chapter is titled "who pushes whom in the cranium"
- grkvlt 10y ago
  It's actually an essay of his: "Who Shoves Whom Around Inside the Careenium" [1] discussing a marble-based conciousness (with the marbles 'careening' around the 'cranium' generating the awful pun...) [1] https://jsomers.net/careenium.pdf https://jsomers.net/careenium.pdf
lordnacho 10y ago
Small addition to the existing threads about Chinese Rooms and materialism: You probably need some sort of sensor to be self-aware. Not a huge issue, but could be challenging with cardboard and marbles.
amyjess 10y ago
Another example: the planet Whisper from Orion's Arm. The planet is covered in grass, and the grass and the wind combined to from a sound-based computer running a virtual world that the original inhabitants have uploaded their minds into. http://www.orionsarm.com/xcms.php?r=oaeg-view-article&egart_uid=470007b39d192 http://www.orionsarm.com/xcms.php?r=oaeg-view-article&egart_...
marxidad 10y ago
I think that self-awareness is more than a matter of (classical) computation. The immediacy of subjective experience implies an entanglement of states, as when one observes one's own self. The marbles would have to be coherent in a cardboard XOR gate for that to happen.

by ythn 10y ago

deleted 10y ago

[deleted]

ae35 10y ago

That rephrases this: https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

samirillian 10y ago
Which rephrases this https://en.m.wikipedia.org/wiki/Leibniz's_gap https://en.m.wikipedia.org/wiki/Leibniz's_gap
Aardwolf 10y ago
This thought experiment seems to say: "the human following the instructions doesn't understand the Chinese, therefore strong AI is impossible". Seems like a fallacy to draw that conclusion, it doesn't prove anything, this is like saying: this tomato is not yellow, so bananas are not yellow. In addition, how can he say strong AI (which he defines as "understanding") isn't possible, doesn't he have at least one example of it, which is his own mind?
- jhbadger 10y ago
  Taken literally, it even suggests human intelligence is impossible. After all, our thoughts originate from a bunch of neurons following instructions and passing information between each other. And each individual neuron isn't any more intelligent than any other cell and can't be said to "understand" our thoughts any more than the man in the Chinese Room understands Chinese.
- drostie 10y ago
  Hi! Searle fan, here. Searle is not, as you might at first anticipate when you see that he's on the "computers can't be conscious" side of the argument, some sort of woo-woo psychic-spiritualism philosopher; he's actually really grounded and relatively materialistic, though he despises the latter term. (He thinks that you should not respond to the problems of dualism by choosing one or the other side of the duality to commit to, and argue the other into nonexistence -- but rather by saying "oh I guess it was stupid to draw a line here in the first place.") Even the earlier simplification I gave, he "thinks computers can't be conscious", he'd object to directly. He thinks that you're a computer, and he thinks that you're conscious. He just doesn't think that you're conscious by virtue of being the computer that you are. So that's who we're talking about here. Now, Searle says: we know, or think we know, that the brain is conscious, and it's conscious due to some aspect of the dance of molecules that's going on under the hood. And we happen to have this model of dances, called computation, which is defined in terms of abstract symbol-shuffling: the great strength of computation is that the 0s and 1s can be voltages and electric currents (as in transistors) or magnetic alignments of spins (as in hard drive) or how a gear is turned (as in the difference engine) or whether a resistance is infinite or low (as in your keyboard). They're just abstract symbols. At its most abstract the Turing test suggests that any human language can be boiled down to these abstract intrinsically-meaningless symbols (which is almost certainly true; that's the premise of the invention of writing), and then that anything which can output symbols in a way that's indistinguishable from a human being, deserves to be given the title of "understanding" that language. Searle steps in and says, no, that's crazy. And he says it's crazy because he's a philosopher and philosophers are very concerned with what words mean exactly, and the word understanding means exactly that these things you're talking about are not abstract symbols, but a computer necessarily treats them that way; it's in the definition of computation that the symbols are abstract. Now that you understand who this is and where he's coming from, maybe you can understand the argument better. He's trying to try to give a more formal proof of this above statement. So, he steps in with this Chinese room argument. The Chinese Room argument says: Your brain is conscious, because of the dance that its neurons do. Now according to this nice computer-reductionalist view, anything which does any dance is understanding a language as long as it produces results which are indistinguishable from a human's responses. So Searle says: let's step inside some Turing machine's dance so our brains are doing two dances at once. The very asset which makes computers awesome allows us to take a Turing-test-passing-algorithm and perform it ourselves: we are the computer, we get this stream of symbols in, we look up rules in some complicated rulebooks, we stream some symbols out, and we now pass the Turing test for understanding Chinese. But is the test correct: do we actually understand Chinese? No. There is no reasonable definition by which we understand Chinese just by performing this complicated computational dance. And this applies even if we move all of the rulebooks and whatever into your skull so that the neurons do those things. Now here's where we come to the conclusion that strong AI is impossible: the human has everything which is part of the definition of computation. I'll repeat that twice more. The human embodies a whole, complete, computer. There's nothing which a computer can be (by virtue of being a computer) which the human is not. So if we go this far and we say "that brain does not understand Chinese" we have to carry that over; "that silicon does not understand Chinese, either." And no computer will understand Chinese merely by virtue of running this computer program. Clearly our understanding of Chinese is not a property of arbitrary dances of neurons which can solve a Turing test, but there is something special about the right dance of neurons which solves a Turing test. The mere fact that the right dance exists points to some physics which is not reducible to computation. In other words Searle thinks that you can simulate people as much as you please, but that simulations are not reality, thank you very much. Just like we humans are very good at simulating pain-behavior without actually being in pain, so too can a computer simulate understanding-behavior without actually understanding a thing. If you want to understand how understanding works, computers will doubtless be a useful tool, but they cannot offer a complete and final answer precisely because they do not pin down the material dance firmly enough to get to something causal.
- posterboy 10y ago
  On the other hand Google translate is (was?) a clear example that word by word translations aren't intelligent or even decidable. So he's got a point, but far from a general proof.
  - jessedhillon 10y ago
    That doesn't at all address the point of the argument. Given the following - an assumption that you can generate an algorithm to express behavior indistinguishable from a human's at a given task, and - an implementation of the algorithm at a macroscopic scale, carried out by individual humans, each executing a small part of the algorithm then, where in this system would you say that an actual understanding of the task exists? Google Translate doesn't pass the first requirement.
    - posterboy 10y ago
      That assumption is too simplistic. If the algorithmic behavior was indistinguishably human behavior, and carried out by humans, it would just be human behavior. Of course machine translation doesn't pass the requirement for human behavior. Nothing does, except humans. And if newer machine translation does pass, I'd say that's humanly accomplished, by use of tools. Nobody could learn a distinct language from nothing but a dictionary, enough to fool a native speaker. The assumption is ridiculous. The dictionary isn't conscious, either way.
      - jessedhillon 10y ago
        https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room If you have a challenge to the Chinese Room argument, go read and respond to Searle's original essay. There's no point in responding to my TL/DR version -- offered only to correct your initial misapprehensions -- with further misapprehensions.
bbctol 10y ago
The Chinese room is a silly example, in which a highly simplified physical system that doesn't appear to have conscious thought is extended to the idea that no merely physical system has conscious thought. If the structure of the Chinese room was not a simple catalog of questions and answers, but hundreds of billions of complicated cardboard-and-marble apparatus, each processing some amount of data in a way that individually seemed meaningless, but together produced cogent speech... would you still be so sure it wasn't conscious?
- blackflame7000 10y ago
  This is a good point. The chinese room seems to stem from a more antiquated system of serial processing and the analogy doesn't quite hold when we look at the massive parallelism that occurs in the human mind that gives us consciousness. Furthermore humans can still think in the absence of understanding. In fact id argue a majority thoughts by people are conjugated without full understanding of the subject of the thought.
geofft 10y ago
The trouble I've always had with this argument is that the amount of books needed to convincingly converse in Chinese would be staggeringly huge. You essentially have an entire human personality as a lookup table. It's not just that you can ask, in Chinese, What's your favorite food, and somewhere in the pages is an answer. You can ask, Tell me about your first love, and somewhere in the pages is an entire conversation. And not only would it be an entire conversation, it would be every possible conversation, depending on how the outside person replied. If it helps visualize the problem - the outside person could say, How about a game of chess?, and the inside person cannot rely on their knowledge of chess, so the books must encode every possible chess game, convincingly enough to hold up a game. Imagine how big a lookup table for chess would be. And that's just a tiny part of what a person knows. And then despite having you accept that without question, and picture it as a small room, Searle turns around and says it would not be "at all plausible" to imagine the system as being conscious. The system is almost entirely made of book. The human in the middle is almost invisible if you look at the system. That's the implausible part - that a human could even operate this Borges library. It's so unreasonable that intuition past that point doesn't give you any useful conclusions. But, if you suspend disbelief enough to accept this galaxy-sized Chinese megalibrary, it's not implausible that "conscious" is a fine adjective for an operable printout of an entire consciousness.
- logfromblammo 10y ago
  You do not need a pre-existing, galaxy-sized library. The program operator has the ability to write into blank books and move them from one shelf in the library to another. The program could, in theory, assemble a book of knowledge on something that exists entirely outside the Room, solely from knowledge gleaned from the inputs passed in from outside. Your initial set of books might simply be a script to follow to elicit additional information and index all new information such that it can be retrieved later. You would then end up with a set of core rulebooks, a whole heap of raw conversations, and several layers of indexes into them. Searle's problem is that he decided the computer was incapable of understanding. But the computer is not the program. The library is the thing that might comprise a strong AI, not the human operator. The scary thing is that the human operator, having no understanding of the Chinese characters, does not actually know what the program is doing. It could smuggle instructions through the output window, through him, and the human operator would then be surprised when burly men drag him out of the Room one day, to replace him with a dozen monkeys who have all been trained to process the book instructions faster and more efficiently. Customers presenting inputs might then be surprised to learn that there is no longer a human inside the Room, as the conversations had become so much more interesting.

te_platt 10y ago

jimktrains2 10y ago
On what basis? If there was an "aware" AI program, it would be just as "aware" in a silicon computer as it would be in a marble computer (albeit much more slow).
- ChuckMcM 10y ago
  Probably on the same basis that your blood cells, skin cells, etc aren't sentient. So the cardboard and marbles would not be sentient, but the system might be. Its the difference between an intrinsic property of the system and an emergent one.
- te_platt 10y ago
  Hmmm... When I posted this the only other comment was "yes". I thought we could just vote this out to determine what the "correct" answer should be.
- deleted 10y ago
  [deleted]

24gttghh 10y ago

The cardboard and marbles would be the engine which drives self-awareness. The individual parts are not self-aware. But considering the size and complexity that would be required for such a feat of engineering, a very liberal estimate would probably require more mass than is available on the entire planet. Edit: the sheer gravity of which would destroy the system.

hinkley 10y ago
Quite a house of cards.
posterboy 10y ago
That's semantically difficult to say, i've seen it in other comments, too, self is not used in the correct frame of reference: The individual parts' self is irrelevant; yet you claim, they are not self aware; when you imply the part's weren't aware of the whole ('s self), you switch frame of reference. The self of the sum of the parts are just the parts. Indeed the parts are the awareness. The whole might be aware of the self of each individual part, but for that to be true, contrary to your statement, each individual part doesn't need to be aware of itself (it might as well code for some other part or whatever).

phaedrus 10y ago

"Yes," except that the scale of the assemblage would have to be unimaginably enormous. Then, in order to continue working at that scale, it would have to incorporate copious error correcting and fail over. It would need mechanisms to reset and reload marbles, and a source of energy to raise them. The system for distribution of that energy would itself be complex... My point is that the Chinese Room argument is disingenuous. It oversimplifies the problem and asks for your incredulity that the oversimplified solution could answer the unsimplified problem.

VLM 10y ago
The specific bug the Chinese Room suffers from is its old and back then we'd have conversational AI next week so obviously it seemed a good example at the time of a non-trivial task that only conscious beings can perform. A re-stating of it in modern terms using modern skills would be that surely only conscious beings can invent and operate under Calculus rules. In fact its pretty hard to do and somewhat elitist in that only a small fraction of primates can learn calculus, larger than who know it now, but it'll never be 100%. So... Mathematica and Maple are decent computer algebraic systems. Better than most conscious beings, even. Well, OK someone programmed them. So fine how about Coq the theorem prover? Now the ancient Chinese Room argument extended to the modern computer is assume we have no access to the engineering department who created Mathematica because its 500 years from now or just the sake of argument or whatever. In fact assume the philosopher barely understands the concept of boolean logic much less floating point or algorithms. Then ask a philosopher which specific transistor in the CPU is the one that "understands" the calculus chain rule. What part of the hardware is the mathematica "for loop" construct as opposed to does the "for loop" construct? How much silicon and plastic does it take to think about Calculus? Observationally a lot less than it takes to do natural language conversational Chinese but why and how much and now it does not take a soul or disembodied spirit thingie to do Calculus and when did that change and when will it change for natural language conversational Chinese? Or heres some fun, when will technology advance enough such that self awareness and self authored demands for political autonomy no longer require soul spirit thingies and are just great piles of fast logic gates?

0xdeadbeefbabe 10y ago

Why don't you ask it?

jononor 10y ago

"To loose ones marbles" would get a more literal meaning...

monk_e_boy 10y ago

https://xkcd.com/505/ https://xkcd.com/505/

pavel_lishin 10y ago

> One definition of intelligence is the ability to skip deductive steps. To jump to a conclusion from the shadow of a ghost of a set of questions. It's preposterous that such a thing could be possible in an uncompromisingly digital reality, but if you make a computer wet enough, or big enough, or abstract enough, it will start to happen. And it has, now. https://qntm.org/transit https://qntm.org/transit

drzaiusapelord 10y ago

>that the cardboard and marbles will be self-aware? What a ridiculous idea and just as ridiculous as, say, meat becoming self-aware. Oh wait.

hinkley 10y ago
They're made out of meat: http://www.terrybisson.com/page6/page6.html http://www.terrybisson.com/page6/page6.html
- lettergram 10y ago
  I prefer this version: https://m.youtube.com/watch?v=7tScAyNaRdQ https://m.youtube.com/watch?v=7tScAyNaRdQ

zeroer 10y ago

Yea, sure. What's so hard about that? XKCD made a comic about something similar: https://xkcd.com/505/ https://xkcd.com/505/. It's unintuitive, sure, but that's just because of the enormous size of the cardboard you'd need to do it. Human intuition is pretty bad outside of its human-sized comfort zone.

teaearlgraycold 10y ago
I think that's my new favorite XKCD.
ythn 10y ago
> Yea, sure. What's so hard about that? The hard part is coming to grips with consciousness merely being an abstract or concrete system changing states and nothing more. If rocks on the ground can be (slowly) self aware, then doesn't that mean everything in the universe is self-aware? For example, say I throw a bucket full of sand on the ground. Then, later, I define an incredibly complex abstract computer, such that the sand falling out of the bucket and shifting along the ground exactly represents a self-aware AI within my defined computer going through its states. Does that mean as I was pouring the the sand on the ground, the sand was momentarily self-aware?
- Sharlin 10y ago
  Follow this train of thought to its logical conclusion and you get Greg Egan's Permutation City. Very much recommended.
  - jwhitlark 10y ago
    Thanks! Was looking for something different to read.
  - JonnieCache 10y ago
    Follow that train of thought to it's logical conclusion, and you end up here, via Big Yud: http://kruel.co/backup/Ontological%20Therapy.png http://kruel.co/backup/Ontological%20Therapy.png Still cracks me up, that page. (BTW you should definitely read Permulation City, and the 'sequel' Diaspora. Try not to found any cults tho.)

marcoperaza 10y ago

Does the act of the computation make it real? Or does the mathematical system being modeled exist independently of its computation?

russdill 10y ago
The act of doing the computation doesn't change the outcome of the computation. Still doesn't answer the question completely, but in my mind at least the idea that humans would act the same even if they weren't conscious is a non starter.
- marcoperaza 10y ago
  My point is that if a machine can "create" consciousness through computation, but computation itself doesn't make the result any more real, then didn't the created consciousness exist all along?
  - russdill 10y ago
    Right, which is what I'm saying. It wouldn't make sense otherwise. It would mean that the result of the computation didn't depend on consciousness in the first place.
  - monort 10y ago
    That is the idea of mathematical universe of Tegmark: https://en.wikipedia.org/wiki/Mathematical_universe_hypothesis https://en.wikipedia.org/wiki/Mathematical_universe_hypothes...

tpeo 10y ago

The particularities of cardboard and marbles would likely prevent it from being "self-aware", in so far as "self-awareness" requires that the machine be fed a large amounts of information regarding itself, through itself. How many kinds of sensors can you build out of cardboard and marble? I don't know. But probably not many, and not as good or abundant as those receptors present in mammals, like those which constantly feed data into our nervous system. But yeah, if this isn't an actual problem, you'd probably have something self-aware.

russdill 10y ago
I think the limitation here is that the amount of cardboard required would lead to some sort of gravitational collapse. Other than that, there isn't a limit to the number of sensors you can build out of cardboard and marbles.

greendragon 10y ago

The cardboard and marbles aren't self-aware, they just encode a configuration that sees itself as self-aware when left alone to execute. Just like the atoms of your body. Self-awareness isn't a static property, since it's hard to call someone under anesthetic self-aware, but just another thing an agent in motion perceives. You ought to read Permutation City.

edblarney 10y ago
Materialism assumption. :)

sweetjesus 10y ago

well, not only is it true that--(if a sentient AI is possible with nothing more than software...), but also if you assume that we don't lose our enthusiasm for building simulations of ourselves, and for helping our little sisters with their projects, then you must conclude that with all probability we are actually living in such a school project simulation made of cardboard and marbles! see https://www.scientificamerican.com/article/are-we-living-in-a-computer-simulation/ https://www.scientificamerican.com/article/are-we-living-in-... It can't be proved that "it's marbles all the way down", but the odds are overwhelming that one of the layers is. Then once you accept that you have to accept that in all probability it's at least a layer of marbles every so often all the way down... thought experiment: In how many of those layered universes is there a cardboard and marbles Peter Thiel funding quirky ideas? Is it Peter Thiels all the way down?!

int_19h 10y ago

In order to get a useful answer to that question, you will need to precisely define what "self-aware" means, first. And to define that, you will need to first define "self" and "aware".

justinhj 10y ago

This is essentially Searle's Chinese Room. I studied it during undergraduate years and I've never been satisfied with his argument against strong AI. https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

amyjess 10y ago

Aside from the xkcd example, there's also the Machina Babbagenseii from Orion's Arm: sophont robots (vecs in OA-speak) made of walking Analytical Engines. http://www.orionsarm.com/eg-article/47f1ab093f416 http://www.orionsarm.com/eg-article/47f1ab093f416

aswanson 10y ago

Not any of the components, any more than the individual letters of the alphabet could be a tragic hero or Shakespeare sonnet, but they could be arranged such that system as a whole would be.

Dwolb 10y ago

Isn't a normal CPU the software runs on just silicon?

hyperion2010 10y ago

The reason why this is hard is because you also need to give the cardboard and marbles the ability to act in the world, that is a bit trickier. As long as you can manage the IO there shouldn't be an issue though.

deleted 10y ago

[deleted]

logicallee 10y ago

The very existence of a number means that it is self-aware. My argument: https://news.ycombinator.com/item?id=12878939 https://news.ycombinator.com/item?id=12878939 in effect, to a first approximation you can say something like "if there's a nonillionth prime, then it is self-aware and has just reported feeling nauseuous." Where the only change to that sentence to make it correct is that rather than "nonillionth prime" you need quite a bit more state and a lot more rules. Still, though, in effect numbers can be self-aware. I'm quite rigorous in my thinking and there is no room for any disagreement here, there's no ambiguity or open question around it.

grkvlt 10y ago
Well, of course there's an open question. Nobody has yet shown that a simulated neuron is exactly equivalent to a biological one; its generally assumed this is the case, but even simulating c. elegans brain won't show this 100%. I would also suggest that it is more correct to say your number S, as a representation of state, has felt (past tense) pain or whatever, but to suggest it is currently feeling anything would be to privilege some part of the number over another - although I'm not sure exactly how to be more rigorous about that?
- logicallee 10y ago
  Suppose we replace 1 of your neurons with a black box that still chemically interfaces with your other neurons, but comes up with the decision about what to do in silicon. If you had a switch to flip back and forth between that black box and the biological version it replaced, your consciousness would not change as you flipped it back and forth - the same is true for replacing a few hundred, thousand, million or billion neurons: if the parts behave the same, the whole behaves the same. so imagine flipping that switch back and forth -- or better, imagine that I'm in front of you flipping it back and forth, but you don't see the state of the switch, and your neurons are biological sometimes but sometimes simulated (it's just a hypothetical, this isn't an actually possible machine). You can't even tell the position of the switch. You feel and report pain the same way, consciousness too. Would you really say that when the flip is switched to simulation, you don't really feel what you report feeling? Of course not. I think it's illusory to talk about the direction of time. The reason we do it this way is causative. The reason I say S "is feeling" pain is if you continue to simulate the next few quadrillion states, it ends up expressing/feeling those subjective feelings. In another sense the tense we use doesn't matter at all. I'm not sure what you're saying about privileging numbers - what does it mean to privilege a number?

anigbrowl 10y ago

No. The cardboard and marbles are the substrate on which the program runs; one might as well argue for the self-awareness of the individual carbon atoms in your brain. The program can't run without some sort of substrate, but the substrate is inert without the appropriate program; it is the system as a whole which is self-aware. Godel, escher, Bach by Douglas Hofstadter is a colorful, whimsical, yet rigorous treatment of this argument in book format.

ralphc 10y ago
My favorite example from GEB is Aunt Hillary, a self-away ant hill where the individual ants in it are not.

laxatives 10y ago

If you have a turing-complete machine, you can execute anything you could on any other turing-complete machine (given sufficient resources).

jchrisa 10y ago

I have thought of this as framing a paradox for the impossibility of strong AI. Turing machines can never really be embodied. They can always move to a different substrate. So intuitively, they don't have skin in the game in the same way that living things do. Mathematically, they are unable to represent the real numbers, only the rational numbers. Maybe this intuition is what you're getting at here.

signa11 10y ago

> if sentient AI is possible with nothing more than software, does that mean if > you "load" the sentient AI program into a "computer" made of cardboard and > marbles, that the cardboard and marbles will be self-aware? fwiw, quote from moon-is-a-harsh-mistress: Am not going to argue whether a machine can "really" be alive, "really" be self-aware. Is a virus self-aware? Nyet. How about oyster? I doubt it. A cat? Almost certainly. A human? Don't know about you, tovarishch, but I am. Somewhere along evolutionary chain from macromolecule to human brain self-awareness crept in. Psychologists assert it happens automatically whenever a brain acquires certain very high number of associational paths. Can't see it matters whether paths are protein or platinum.

laumars 10y ago

No. The parts themselves aren't self aware - it's the sum of the parts.

jtinder 10y ago

You are a self-aware mixture of lots of carbon atoms, water molecules, several metals, etc... of many solids, liquids and gases.

sorokod 10y ago

This is one of the themes in "I am a strange loop" by Hofstadter. If I remember correctly, the relevant chapter is titled "who pushes whom in the cranium"

grkvlt 10y ago
It's actually an essay of his: "Who Shoves Whom Around Inside the Careenium" [1] discussing a marble-based conciousness (with the marbles 'careening' around the 'cranium' generating the awful pun...) [1] https://jsomers.net/careenium.pdf https://jsomers.net/careenium.pdf

lordnacho 10y ago

Small addition to the existing threads about Chinese Rooms and materialism: You probably need some sort of sensor to be self-aware. Not a huge issue, but could be challenging with cardboard and marbles.

amyjess 10y ago

Another example: the planet Whisper from Orion's Arm. The planet is covered in grass, and the grass and the wind combined to from a sound-based computer running a virtual world that the original inhabitants have uploaded their minds into. http://www.orionsarm.com/xcms.php?r=oaeg-view-article&egart_uid=470007b39d192 http://www.orionsarm.com/xcms.php?r=oaeg-view-article&egart_...

marxidad 10y ago

I think that self-awareness is more than a matter of (classical) computation. The immediacy of subjective experience implies an entanglement of states, as when one observes one's own self. The marbles would have to be coherent in a cardboard XOR gate for that to happen.