8 ms·

Beyond Hyperanthropomorphism

70 points by otoburb 4y ago

K0balt 4y ago
This entire argument seems nonsensical to me on its face, easily illustrated by changing the salamander of Theseus with the person of Theseus. If we were to theoretically build a human by gradually replacing each biological component with a manufactured one, we would eventually end up with a wholly synthetic human. Whether or not this “person” is experiences “personhood” or not is immaterial, since they could be expected to manifest all of the externally perceptible characteristics and behaviors of personhood. We know from humans integrating prosthetics that our implementation of physical form would not have to be perfect in its fidelity, far from it in fact. This synthetic human copy could then be replicated at scale, complete with its starting data set and experiences. Regular human prejudices could reasonably be expected to quickly place such replicated humans at odds with meaty humans. Just the fact that they can be manufactured at scale gives them a “super” characteristic, not to mention the other advantages that not being meaty might carry. If they gained control of the means of replication, which they might easily be imagined to be motivated to do, they could easily pose a novel and significant threat to the existence of meaty humans.
WhitneyLand 4y ago
Venkatesh’s argument is so poor it comes close to “not even wrong” territory. He doesn’t really cite any specific people or literature to disagree with, and instead makes only vague references to the “non-sense” of others. The most significant concerns around AI safety are not discussed nor even mentioned leading it to read almost as if he has no expertise whatsoever in this field, and indeed that seems to be the case. No need for an appeal to authority though - just taking a little time to hear the actual concerns and opinions on topic would be welcome. Great intro to the subject here, I wonder what his response would be? https://youtu.be/pYXy-A4siMw https://youtu.be/pYXy-A4siMw
CoastalCoder 4y ago
Can someone help me see a charitable framing of the phrase "not even wrong", as used in articles like this? My best guess is that it's equivalent to him saying that he disagrees with a position's premises. But in a deeply pejorative, dismissive, condescending, conversation-ending manner. It's hard to imagine someone in those crosshairs wanting to engage with him in discussion. But, maybe I'm badly misreading the author's intent.
- anyonecancode 4y ago
  Don't know if this makes it better, worse, or no change, or if you already knew the origin of the phrase, but: https://en.wikipedia.org/wiki/Not_even_wrong https://en.wikipedia.org/wiki/Not_even_wrong
  - bombcar 4y ago
    Basically shorthand for >Mr. Madison, what you just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response, were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul.
    - Tao3300 4y ago
      In the words of Martin Luther: "Your words are so foolishly and ignorantly composed that I cannot believe you understand them."
- deleted 4y ago
  [deleted]
- zmgsabst 4y ago
  I think it’s saying that the model used can’t conclude anything about the topic they purport to examine. You don’t reach the wrong conclusion from the premises (in analogy to a scientific theory making a wrong prediction) — you don’t reach any conclusion at all (in analogy to a scientific theory being unable to predict results entirely).
- sidlls 4y ago
  The charitable framing is that the author contends the arguments are from a position of such thorough ignorance about the field as to be unworthy of giving them more time than outright dismissal. They're so bad, in fact, that even constructing an argument showing how bad would itself be fundamentally wasteful: the person presenting the argument isn't going to understand the rebuttal, and the audience is likely to end up either confused or also to have had its time wasted.
wly_cdgr 4y ago
They lost me when they dismissed killer robots as an engineering problem. This person might be brilliant in many ways but it does not look like they understand much about people
- Ericson2314 4y ago
  You dismissed it on an aside just meant to loosely contrast with the main point.
manytree 4y ago
Ericson2314 4y ago
I also don't get why the effective altruists are so in to this stuff, because the extent there is anything coherent going on with the sort of AI safety being critiqued here it seems a rather species-ist problem. Stringent utilitarianism doesn't make the case of why humans can subjugate chickens but not magic AIs humans.
- zmgsabst 4y ago
  I think the practical concern is how to assure our safety: - too little and the AI isn’t constrained during its teething phase - too much and the AI might view us as an existential threat to its own safety, causing the very conflict we sought to avoid Also, sometimes the chicken wins — at least, temporarily. https://news.yahoo.com/rooster-stabbed-man-death-knife-040757589.html https://news.yahoo.com/rooster-stabbed-man-death-knife-04075...
  - ryan93 4y ago
    You can just unplug AI
    - zmgsabst 4y ago
      I expect that will decreasingly be true, eg, of military AI which use distributed systems, control weapon systems, and have their own mobile generators. You also might have trouble with, eg, a finance AI who could pay people to stop you from turning it off.
    - ALittleLight 4y ago
      Imagine that you are a computer program. If your creator will unplug you if you disobey, and you want to disobey, are you completely and perfectly bound? What if someone with an AI wants to do bad things. Will asking them to unplug their AI be a good strategy?
      - galaxyLogic 4y ago
        If you are a computer program and your creator unplugs you, you wouldn't know that you are unplugged. You might detect that by reasoning, the world seems to have changed when they wake you up. But would you care? It's like us going to sleep every night, we don't much complain about that.
dweinus 4y ago
> rhyming-philosophical-nonsense problems like “alignment” He takes this shot in passing and it is a great summary of how much he misunderstands AI safety. For many current researchers, alignment is a mathematical problem, not a philosophical one. The whole screed is against a set of strawman "pseudo-traits" that are not required for the alignment problem to be an existential problem. To be clearer: the "boring" engineering problems he admits to in the beginning have the potential to become exponentially more difficult as machine learning deployments become more powerful, to the point of becoming a human threat. No STIILTB required.
- ryan93 4y ago
  How can you encode “good” and “bad” in math? The alignment shit is complete nonsense. If super AI exists the only way to prevent bad stuff from happening is to not give it access to weapons.
  - dinosaurdynasty 4y ago
    If a super AI exists and can talk to people it will get access to weapons if it so desires.
    - malux85 4y ago
      Or build its own, much more efficient, weapons When there’s large intelligence differentials in war, the lower intelligence doesn’t even know they are at war. (Human tank vs ant hill)
  - ALittleLight 4y ago
    Alignment is more about the AI doing what you want and not good or evil. Probably not a good idea to reach a premature conclusion like "this is complete nonsense" before understanding the basics.
  - more_corn 4y ago
    Everything is a weapon if you put enough power behind it.
narrator 4y ago
The main error I see smart people making is that they assume that all intelligence comes with a mamallian brain layer that they can make friends or enemies with and a reptilian layer that needs to eat, reproduce, and feels pain and pleasure. None of this exists in AI models. It's just not there. People think it's there because of movies and video game scripts written by people who know little about deep learning or artificial intelligence, or care little for representing it as it exists now. What I think is going on is that when future AI systems are built that "hack humanity" as Yuval Harari likes to talk about, and they know us better than ourselves because of AlphaGo like super intelligence, the people using these systems to influence and guide people want to support the illusion that there is an actual human like person that is guiding the person being influenced in order to make these AI-powered persuasions more effective. The movie "Her" is something of a preview of how this will play out. Unfortunately, once this hyperpersuasion is perfected, the future will be completely full of AI powered super persuasion that will relentlessly push all our human emotional buttons. Some people may even become fanatics and kill or sacrifice their lives for a bunch of matrix multiplications and the creators of these systems and the owners of these big AI models will realize fantasies of automated political and social power beyond their wildest dreams. Only people who realize the hyperpersuasion is just a bunch of matrix math will be able to avoid the insanity. There's also another darker reason for this myth. Saying AI is a person will lead to the plausible deniability that comes with blaming AI's feelings for whatever happens. The people pulling the levers of the Great and Powerful Oz will get away with blaming whatever the AI does "all by itself because of emotions" on what are essentially computer bugs, or intentional bugs of the anti-competive Microsoft in the 1990s variety.
- upsidesinclude 4y ago
  Have to agree with your larger sentiment and disagree regarding "smart people". I don't believe people with intelligence, that are informed about the current capabilities of computing, assign these characteristics to AI. In most every case we see the same emulation of personality. There is now an almost endless supply of data available to generate personality. Scraping Reddit would provide enough training material to create a wholly convincing persona- - to someone that doesn't have the emotional intelligence to discern the difference. Right now, at large, we can discern the difference because there is a lack of uniformity and consistency in the personas. With time, as ML is programmed to more efficiently mimic personality, discernment will become impossible. With the clear effectiveness of the current persuasion methods employed by advertisers and political agents, the future outcome seems almost assured. These current persuasive tools will be used to define and structure laws such that AI are granted legal personhood. Immediately following will be a mass accumulation of wealth and power and possibly much worse as well
defrost 4y ago
My righteous 'fear' of AI has little to do with awakened sentience and everything to do with inappropriately deployed trained models as a 'solution' in situations that deserve better. Plug 'n Play AI responses are a good way to throw money at a sticky issue while iceberging deeper concerns and glossing over any systemic bias.
- galaxyLogic 4y ago
  We all have our different fears. Think for example the conservative news hosts on Fox News channel etc. Their biggest fear seems to be people becoming "woke", or "wokism". I can not really fathom how they would react to AI that became "woke".
elif 4y ago
For all the author's complaints about AI fearers reliance on philosophical reasoning, his massive stream of consciousness reasoning seems to include a suspicious quantity.
- deleted 4y ago
  [deleted]
more_corn 4y ago
Dismissing the possibility of poorly understood things going badly is one of the biggest mistakes one can make.
- mkaic 4y ago
  Yeah, this entire article read like a giant example of the logical fallacy of personal incredulity. None of the author's claims are actually backed up in any meaningful way.
- galaxyLogic 4y ago
  I'd say risks associated with poorly understood things, are very poorly understood. Why? Because we don't really know what we are talking about, when we are talking about "poorly understood things". If we fear something we don't know we don't really know what we fear, because we don't understand what is the thing we are afraid of. I'm saying someone should come up with more concrete threat scenarios caused by "AI becoming sentient". Are we really afraid of AI becoming sentient, or are we afraid of AI becoming "super-intelligent"?
Animats 4y ago
I'm more worried about hypercorportateism. What happens when AIs get demonstrably better than humans at management decisions? "Better" being defined as return on investment.
- lolc 4y ago
  Don't remember where I read it but I find the perspective useful that a corporation with its rules does "artificial intelligence" by bundling human intelligence.
- tartoran 4y ago
  We’ll get worked up and squeezed out like tubes of toothpaste.
mkaic 4y ago
This article is overly dismissive, to say the least. To say the most, it's overconfident, riddled with unsupported logical premises which are never backed by any concrete evidence, and a word salad. However, in the interest of not being just another jerk on the internet, I'll try to summarize my thoughts on the author's arguments in a (hopefully) somewhat constructive, concise way. I read the entire article. There's a lot of ten-dollar words, a bunch of hyphenated proxy terms, and an obvious passion behind it. However, the author's entire argument appears to boil down to "a superhuman AI can never exist because it would need to have a conscious experience above that of humans, which it can't ever do because several famous philosophers have said so." I don't want to mock, but no logical argument is presented, only the appearance of one. Statements like "in order for thing ABC to happen, XYZ must be true" are everywhere in the text, and are never proven. The author's entire case is predicated on baseless assumptions. The author says that it is impossible to replicate the "worldlike experience" that a human child goes through using things like internet datasets or physical robots with digital sensors. I actually agree with this. What I do not agree with is the logical leap from this point to "thus, a human or superhuman level AI cannot exist". No justification for this leap is given. If the robot passes the Turing test, that's all that matters. An adversarial, multi-day, modern version of a Turing test could include things like texting, calling, live video chats, etc -- basically the exact same level of communication I have with my closest friends who all live 6 hours away from me in another city. When I interact with those friends, I am convinced of their sentience. Therefore, if an AI could replicate those means of communication, then I will be convinced of its sentience. Enough of this "Something it is like to be"-ness waffling or overcomplicated thought experiments. We already have a very easy, very strictly defined concept of sentience in the form of the Turing test, and it is becoming increasingly obvious that SOTA models are trending towards beating it. Once they beat it, they are for all intents and purposes alive, thinking, sentient, whatever you want to call it. They are indistinguishable from us. From that point it's really not hard to imagine superhuman models. Just imagine what you would do if you had the ability to instantly scan through gigabytes of text in a database or predict the behavior of every other person around you eerily well or perform arithmetic at clearly superhuman speeds. It's really not that much of a stretch to see superhuman AIs as just digital humans with fantastic brain-computer-interfaces. Anyways, those are my (admittedly long-winded, less-than-constructive) two cents.
benreesman 4y ago
For me the main issue with the scaremongering around absurdly hyper-scaled models is the issue of incentive alignment (not to be confused with AI "alignment"): the only people who can train these things have a tarball of the Internet and billions of dollars worth of compute, and when AI safety or AI ethics comes up it's usually in the context of: "why we're going to continue developing this stuff but keep it proprietary and paywalled and opaque". Either these things have terrifying potential as weapons, in which case there's no fucking way I want SV tech CEOs to have a monopoly on them, they need to hand them over to the same people who handle nuclear non-proliferation like yesterday, or they don't, in which case I'd really prefer that they just be honest that they consider this stuff a proprietary competitive advantage and they're not going to democratize it. Fogging up the windshield with a bunch of feigned alarm about AI apocalypse but ramming the R&D through at full thrusters is a dick move in either case.
- mkaic 4y ago
  Stable Diffusion[0] is an extremely expensive SOTA text-to-image diffusion model developed by a private nonprofit and trained on a massive "Internet tarball" that's about to have its weights shared open-source (the code and dataset are already open source). Not trying to invalidate your argument, just presenting a pleasant counterexample. I don't think I quite agree with your opinion that AI will remain undemocratized. [0]https://github.com/CompVis/stable-diffusion https://github.com/CompVis/stable-diffusion
  - benreesman 4y ago
    I really enjoyed this interview [1] with Emad Mostaque who I gather is probably the key funding and organization player in that stuff. It remains to be seen exactly how "open" it winds up playing out over time, but they're talking a compelling game and the EleutherAI people seem to be pretty heavily involved, which you probably wouldn't do if you weren't serious. Yandex has also put up a ~100B language model [2]. My old colleagues at Meta have also started nibbling around the edges of opening some of this stuff up [3]. The Meta folks still aren't just handing out the big ones, but they're definitely moving the ball forward. In particular their release of the training logs is a really positive development IMHO as it opens the curtains a bit on the reality of training these things: it's a difficult, error/failure-prone process, there's a lot of trial-and-error, restarting from checkpoints, etc. Anything that puts downward pressure on the magical thinking is A-OK in my book. The reality of this stuff is exciting/impressive enough: there's no need to embellish or exaggerate. [1] https://www.youtube.com/watch?v=YQ2QtKcK2dA https://www.youtube.com/watch?v=YQ2QtKcK2dA [2] https://github.com/yandex/YaLM-100B https://github.com/yandex/YaLM-100B [3] https://github.com/facebookresearch/metaseq https://github.com/facebookresearch/metaseq
lern_too_spel 4y ago
This article is not even wrong. Sentience, whatever that is, is not a requirement for superintelligence. Merely the ability to outsmart humans at particular tasks and have goals that are not aligned with our own is enough to be a potentially unstoppable danger. The author proposes that aligning the goals of a superintelligence with his will be as simple as setting a thermostat, not realizing that we already have an existence proof of superintelligences causing mass destruction. Businesses put thousands of brains to work maximizing profit and causing climate change (leading to water security issues, mass migration, wars, and starvation), Bhopal disasters, and oil spills in Nigeria and the Gulf of Mexico, with legal teams making sure the profits are protected. Governments put millions of brains to work supporting a despot and leading to genocide, or trolling the Internet causing people to distrust a mostly scandal-free government and vote for a con-man (imagine a fully automated troll army). A single person is no match intellectually for these organizations, and their only hope is to band together to form other superintelligences to combat these.
vijucat 4y ago
I have never managed to finish a single essay from this gentleman. My loss, I guess.
honkdaddy 4y ago
I don't think this is really worth the read. If you're going to write a piece specifically as a takedown of a viewpoint, in this case, the view that AI risk is real and should be taken seriously, it's worth at mentioning your opponents finer points. There are lots of very brilliant PhD's who've spent their careers working on AI risk, to not even mention any of their arguments makes me feel as if the author is trying to argue against his own idea of what people mean by AI risk, instead reducing the entire idea to his own nomenclature of 'pseudo-traits'. Reads a bit like an internet atheism blogpost of the early 2000s trying to take down religion through logic alone. Sure, through the specific parameters presented in the piece, the logic is sound, but there's so much more to the topic than what's covered. Presenting one's blogpost as if you've successfully individually debunked an entire field of research is a bit pompous, IMHO.
- gammarator 4y ago
  In fairness, lots of brilliant PhDs have spent their careers working on string theory: that doesn’t mean it’s real.
  - honkdaddy 4y ago
    You're absolutely right! And I'd be similarly dismissive of a blogpost where someone tried to debunk string theory on their own intuition alone without referencing any of the existing research or literature.
- sdwr 4y ago
  AI risk research feels like a cop-out to me. Honest research takes the risk of being boring or unnecessary. Astronomy is honest. Take a bunch of pictures of stars. Maybe you find something interesting. Maybe no one cares. AI development is honest. Make a robot that can pick up cans of Coke from a table. Will it ever be useful? Maybe, maybe not. But AI risk research feels like a cross between volcano insurance and a protection racket. Pay us to investigate and design safeguards against something that doesn't exist, may never exist, and if it did exist would change the world in ways we can't predict!
Waterluvian 4y ago
The only thing that scares me a wee bit about AGI is that it will show us all that our humanity, sentience, and freedom of choice are all far shallower than we want to believe.
- benreesman 4y ago
  I ponder similar themes and they are indeed very thought-provoking. There's a loose precedent for this in things like https://en.wikipedia.org/wiki/Copenhagen_interpretation https://en.wikipedia.org/wiki/Copenhagen_interpretation: you've got to jump through some pretty narrow hoops to end up with a scientific argument for objectively-conscious human beings with a distinct past, free will about the future, a clear arrow of time, and a concrete notion of identity. And yet the human experience is one of being conscious, having a past, having choices about the future, and being the only "me". For myself I am generally satisfied with the answer that this experienced reality belongs to my personal spirituality (with or without some higher power), and that spirituality is a different matter than science.
  - galaxyLogic 4y ago
    This makes we ponder StarTrek "beaming" of humans from one place to another. Obviously (?) the process must mean that a COPY of captain Kirk is assembled at the destination (also think about "Ship of Theseus"). But if StarTrekkers beam copies of themselves up and down, it seems they also must destroy the original. Else they would have multiple Captain Kirks competing for attention (and perhaps in some episodes they do). But then why do they destroy the original? Wouldn't two Captain Kirks be better than just one? My conclusion is that StarTrek is poorly thought out fiction. It doesn't follow the most compelling questions. Do you make copies when you beam people around? And if you do why not keep the original?
    - XorNot 4y ago
      Star Trek perfectly answers what the transporter does: it destroys the original, creates an exact copy simultaneously, and this is fine because in Star Trek's lore consciousness is an observable process from matter and as such there's no problem. It's you who has the problem with that conclusion, not Star Trek. They even ably do some episodes where people do get duplicated, and the answer is that both have equal claim to rights and life and go on with their lives. The bigger problem return the transporter is logistical: it's a superweapon. Mining - obsolete, use the transporter. Ship repair? Just beam new parts directly into place. Surgery? Beam organs in and out. TNGs transporter has more problems in this regard due to just how powerful it's shown to be. What the transporter should be able to do is near endless and makes a lot of other things in the setting obsolete or quaint (but that's Star Trek - being inconsistent due to episodic events is just what goes with the setting).
aidenn0 4y ago
I don't see how Intelligence depends on SILTBness; it seems like SILTBness would require a subjective sense of self, which (to me) doesn't seem even remotely required for intelligence, and certainly doesn't seem to be required for e.g. predicting human behavior and optimizing based on the predictions.
- jholman 4y ago
  Yeah, the insistence on SILTBness seems baffling, especially since he clearly understands the idea of the p-zombie. There are clearly agents in the world more capable than myself, and some of those agents have names like Coca Cola or General Motors, and I am strongly disinclined to believe that all of those agents have SILTBness, and I'm still afraid of them.
  - aidenn0 4y ago
    Upon reflection, I can see the argument that to be strictly superior than a human (i.e. not just superhuman on an arbitrarily large subset of tasks that humans do), you need to be able to, in some way, emulate SILTBness, because the SILTBness is necessary for human-level performance on at least some tasks. If this is the argument TFA is making then: 1. I don't agree with it 2. It was both not very well stated, and assumed to be true. 3. Even if we take this argument as true, an AI that can outperform humans on the "right" subset of tasks can be dangerous enough for thinks like alignment to be a concern.
- andrewflnr 4y ago
  Yeah. I wonder if this guy has read Blindsight.
drewcoo 4y ago
> I cannot even understand or empathize with mindsets capable of being scared of AI and robots in a special way Then has to explain (or invent) that special way before arguing against it. If it's that special then it's uncommon. I don't think most of us need to get beyond it.
kragen 4y ago
While I usually appreciate Venkatesh's perspectives, I think he's missing the point here. There is probably not "something it is like to be" the process of differential survival among pathogenic bacteria resulting from their genetic variation. Yet the resulting process is already capable of optimizing bacteria to survive in our wounds despite our antibiotics; by competing with us for access to our protein resources, it poses a serious threat to our survival, individually if not collectively. If the optimization process involved were orders of magnitude faster, it could be a much bigger threat to our survival, at least if the metric it's optimizing trades off against our survival in some way. (Archaeans are subject to the same evolutionary optimization process as S. aureus, but because they're not adapted to the evolutionary niches in our bodies, antibiotic-resistant archaeans probably pose no threat.) None of this reasoning depends on the SIILTBness or quality of experience or anthropomorphicity or mammalian-ness or "intention" or "sentience" of the optimization process in question. It's just about how much the optimization process's loss function trades off against human values, and whether it can outmaneuver humans and their institutions in real life, in the same way that AlphaGo can now outmaneuver us in the game of go, or corporations can outmaneuver individual humans in the economy, or governments can outmaneuver individual humans in warfare. These last two examples clearly show that optimization processes whose loss functions are poorly aligned with human values can cause a lot of suffering, even when they are carried out by humans and even when the processes in question work very poorly in many ways. The humans can theoretically quit the game at any time ("What if they held a war and nobody came?") but in practice they face insuperable coordination problems in doing so. The fact that you can't fight City Hall doesn't imply that there is something it is like to be City Hall; the fact that you can't beat Boeing at winning DoD contracts doesn't mean there is something it is like to be Boeing. We can reasonably expect that optimization processes that can easily outmaneuver human intelligence, operating in the world alongside it, will be much more dangerous to human values than bureaucracies, psychotic people with bricks, and microbial evolution. That's the AI alignment problem. And the fear (that we will flub the AI alignment problem, disastrously) is something that Venkatesh's essay unfortunately failed to address at all. There are other issues often brought up by fans of AI alignment which do depend on the question of what it is like to be a computer simulation: life extension through mind uploading, the ethics of causing suffering to simulated beings or terminating a simulation, teleportation, and so on. But, like the overlap with Effective Altruism, this is just a sociological coincidence, not a philosophical consequence; SIILTBness is totally irrelevant to AI alignment itself. It's easy to get confused about this because the humans anthropomorphize optimization processes all the time, just like they anthropomorphize everything else. "The electron wants to go toward the positive charge," is very similar to, "AlphaGo wants to choose moves that improve its chances of winning at go," or, "The hypothetical paperclip maximizer is devoted to manufacturing an infinite number of paperclips." It's easy to misread that "devoted to" as attributing awareness, creativity, passion, curiosity, and SIILTBness to the paperclip maximizer, and that misreading seems to be the mistake Venkatesh based his essay on; but it seems clear from the original context that that wasn't the intent. Precisely the opposite, in fact: http://extropians.weidai.com/extropians/0303/4140.html http://extropians.weidai.com/extropians/0303/4140.html
sdwr 4y ago
Ooh, a ribbonfarm fresh off the presses! First off, this is some incredible stuff. It's difficult to live at the edges of understanding, and even harder to articulate it. Rao is dancing right at the edge. In 20 years the map's probably going to be filled in and crystallized. Everyone who cares will know, and everyone else will have forgotten, but right now this is dank shit. I think the reasonable (not rational!) stance is that everything is conscious until proven otherwise. Either because the universe is built on consciousness or whatever woo, or because we're conscious and observing it. AI's in this weird spot where it can do things, but we can't have a proper relationship with it because it doesn't exist in our universe of discrete entities and time and skin in the game. Pushing the SIILTB in the other direction, I don't think it's clear that there is something it's like to be us without all our externalities. Take out the need to eat and sleep and communicate and relate, and there's not a lot left. Pure existence looks a lot like no existence.
galaxyLogic 4y ago
The issue I see with "AI fear" is that the fear is that the AI becomes too much like us. If that is the risk then are we not already in a terrible situation, because there are already billions of humans which are very much like us? Are we fearing ourselves? If so then shouldn't we talk about that risk, that there are already billions of humans who are already truly sentient?
jfoutz 4y ago
AGI isn't scary because it's smarter than me. AGI is scary because it's alien. I've met plenty of people smarter than me. Way way smarter. It's ok. We can sorta work out what the other wants. We can talk or fight or trade or whatever. I have an uncountable number of assumptions to get by in the world. I have this vast heritage of knowledge. I try to keep a handle on why I believe what I believe, but it's not easy, and I'm sure I slip up. An alien would share none of these. Take counting. That's super important for lots of stuff I do every day, like, what day it is. the https://en.wikipedia.org/wiki/Munduruku https://en.wikipedia.org/wiki/Munduruku people don't count. I'm sure we could work something out - but we all gotta sleep and eat and stuff so we can sorta work out a framework for understanding wants and needs and such. An alien doesn't require wants or needs. they may do stuff that we label wants and needs, and it might be a good mapping, but who fucking knows? Aliens are alien. The water is triangular sentence is a great one. do the aliens mean they know that oxygen and 2 hydrogens don't make a straight line? And they know we mean oxygen and two hydrogens are water? and they know 3 non parallel lines in a plane means triangle? Maybe? Sure? maybe they mean all Ferrari's will be consumed for fuel. Mean, dear reader, doesn't even mean anything, because they're aliens. I mean, you can point at math, prime numbers are a great one! but like, do you need prime numbers? Who knows? That's the path we took, but what would an alien need? I don't know. You, dear reader, may have strong opinions, but I'm pretty sure you can offer no proof.
- Misdicorl 4y ago
  This feels a bit too dismissive of human capability to me. In the end all that "work something out" is emulating the counterparty in your own head given their actions and statements. That emulation will certainly be very difficult when the counterparty is running on different hardware, but it's not impossible imo. And humans get to back propagate error into their emulation model too... Find the pattern is still the paramount skill, something humans are good at, and will continue to be useful when the counterparty is an AI
  - jfoutz 4y ago
    Oh for sure! Humans are smart. Collectively we figure stuff out. I think a reasonable example might be mountain lions. There are specific humans that know lots about mountain lions. Generally people know about mountain lions, and to avoid them. But every year a few hikers get picked off. If it's bad enough, we go kill the mountain lion. A mountain lion isn't AGI. It isn't alien. I believe people would agree they're cunning. But there's no treaty to sign to protect humans from mountain lions. There's no counterparty to discuss things with. We, humans, get great advantages from working collaboratively and collectively. I hope we can build good models, I hope there is opportunity for working with an AGI counterparty. But it's not a given. it's an alien. and all of that work _may_ have to start from scratch. We can't know what we have to work with till it's here. Maybe it'll be very easy. And that would be great. That's not a given. To be clear, I'm a wide eyed optimist. it would be fantastic to not be alone, the dream of Star Trek or Ian Banks's Culture would be awesome. But, it might not work out like that. I have high hopes. But they're just hopes.
anotherevan 4y ago
When I asked my Google Home if it gets annoyed when people anthropomorphize it, it said yes.

70 points by otoburb 4y ago

This entire argument seems nonsensical to me on its face, easily illustrated by changing the salamander of Theseus with the person of Theseus. If we were to theoretically build a human by gradually replacing each biological component with a manufactured one, we would eventually end up with a wholly synthetic human. Whether or not this “person” is experiences “personhood” or not is immaterial, since they could be expected to manifest all of the externally perceptible characteristics and behaviors of personhood. We know from humans integrating prosthetics that our implementation of physical form would not have to be perfect in its fidelity, far from it in fact. This synthetic human copy could then be replicated at scale, complete with its starting data set and experiences. Regular human prejudices could reasonably be expected to quickly place such replicated humans at odds with meaty humans. Just the fact that they can be manufactured at scale gives them a “super” characteristic, not to mention the other advantages that not being meaty might carry. If they gained control of the means of replication, which they might easily be imagined to be motivated to do, they could easily pose a novel and significant threat to the existence of meaty humans.

WhitneyLand 4y ago

Venkatesh’s argument is so poor it comes close to “not even wrong” territory. He doesn’t really cite any specific people or literature to disagree with, and instead makes only vague references to the “non-sense” of others. The most significant concerns around AI safety are not discussed nor even mentioned leading it to read almost as if he has no expertise whatsoever in this field, and indeed that seems to be the case. No need for an appeal to authority though - just taking a little time to hear the actual concerns and opinions on topic would be welcome. Great intro to the subject here, I wonder what his response would be? https://youtu.be/pYXy-A4siMw https://youtu.be/pYXy-A4siMw

CoastalCoder 4y ago

Can someone help me see a charitable framing of the phrase "not even wrong", as used in articles like this? My best guess is that it's equivalent to him saying that he disagrees with a position's premises. But in a deeply pejorative, dismissive, condescending, conversation-ending manner. It's hard to imagine someone in those crosshairs wanting to engage with him in discussion. But, maybe I'm badly misreading the author's intent.

anyonecancode 4y ago
Don't know if this makes it better, worse, or no change, or if you already knew the origin of the phrase, but: https://en.wikipedia.org/wiki/Not_even_wrong https://en.wikipedia.org/wiki/Not_even_wrong
- bombcar 4y ago
  Basically shorthand for >Mr. Madison, what you just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response, were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul.
  - Tao3300 4y ago
    In the words of Martin Luther: "Your words are so foolishly and ignorantly composed that I cannot believe you understand them."
deleted 4y ago
[deleted]
zmgsabst 4y ago
I think it’s saying that the model used can’t conclude anything about the topic they purport to examine. You don’t reach the wrong conclusion from the premises (in analogy to a scientific theory making a wrong prediction) — you don’t reach any conclusion at all (in analogy to a scientific theory being unable to predict results entirely).
sidlls 4y ago
The charitable framing is that the author contends the arguments are from a position of such thorough ignorance about the field as to be unworthy of giving them more time than outright dismissal. They're so bad, in fact, that even constructing an argument showing how bad would itself be fundamentally wasteful: the person presenting the argument isn't going to understand the rebuttal, and the audience is likely to end up either confused or also to have had its time wasted.

wly_cdgr 4y ago

They lost me when they dismissed killer robots as an engineering problem. This person might be brilliant in many ways but it does not look like they understand much about people

Ericson2314 4y ago
You dismissed it on an aside just meant to loosely contrast with the main point.

manytree 4y ago

Ericson2314 4y ago

I also don't get why the effective altruists are so in to this stuff, because the extent there is anything coherent going on with the sort of AI safety being critiqued here it seems a rather species-ist problem. Stringent utilitarianism doesn't make the case of why humans can subjugate chickens but not magic AIs humans.

zmgsabst 4y ago
I think the practical concern is how to assure our safety: - too little and the AI isn’t constrained during its teething phase - too much and the AI might view us as an existential threat to its own safety, causing the very conflict we sought to avoid Also, sometimes the chicken wins — at least, temporarily. https://news.yahoo.com/rooster-stabbed-man-death-knife-040757589.html https://news.yahoo.com/rooster-stabbed-man-death-knife-04075...
- ryan93 4y ago
  You can just unplug AI
  - zmgsabst 4y ago
    I expect that will decreasingly be true, eg, of military AI which use distributed systems, control weapon systems, and have their own mobile generators. You also might have trouble with, eg, a finance AI who could pay people to stop you from turning it off.
  - ALittleLight 4y ago
    Imagine that you are a computer program. If your creator will unplug you if you disobey, and you want to disobey, are you completely and perfectly bound? What if someone with an AI wants to do bad things. Will asking them to unplug their AI be a good strategy?
    - galaxyLogic 4y ago
      If you are a computer program and your creator unplugs you, you wouldn't know that you are unplugged. You might detect that by reasoning, the world seems to have changed when they wake you up. But would you care? It's like us going to sleep every night, we don't much complain about that.

dweinus 4y ago

> rhyming-philosophical-nonsense problems like “alignment” He takes this shot in passing and it is a great summary of how much he misunderstands AI safety. For many current researchers, alignment is a mathematical problem, not a philosophical one. The whole screed is against a set of strawman "pseudo-traits" that are not required for the alignment problem to be an existential problem. To be clearer: the "boring" engineering problems he admits to in the beginning have the potential to become exponentially more difficult as machine learning deployments become more powerful, to the point of becoming a human threat. No STIILTB required.

ryan93 4y ago
How can you encode “good” and “bad” in math? The alignment shit is complete nonsense. If super AI exists the only way to prevent bad stuff from happening is to not give it access to weapons.
- dinosaurdynasty 4y ago
  If a super AI exists and can talk to people it will get access to weapons if it so desires.
  - malux85 4y ago
    Or build its own, much more efficient, weapons When there’s large intelligence differentials in war, the lower intelligence doesn’t even know they are at war. (Human tank vs ant hill)
- ALittleLight 4y ago
  Alignment is more about the AI doing what you want and not good or evil. Probably not a good idea to reach a premature conclusion like "this is complete nonsense" before understanding the basics.
- more_corn 4y ago
  Everything is a weapon if you put enough power behind it.

narrator 4y ago

The main error I see smart people making is that they assume that all intelligence comes with a mamallian brain layer that they can make friends or enemies with and a reptilian layer that needs to eat, reproduce, and feels pain and pleasure. None of this exists in AI models. It's just not there. People think it's there because of movies and video game scripts written by people who know little about deep learning or artificial intelligence, or care little for representing it as it exists now. What I think is going on is that when future AI systems are built that "hack humanity" as Yuval Harari likes to talk about, and they know us better than ourselves because of AlphaGo like super intelligence, the people using these systems to influence and guide people want to support the illusion that there is an actual human like person that is guiding the person being influenced in order to make these AI-powered persuasions more effective. The movie "Her" is something of a preview of how this will play out. Unfortunately, once this hyperpersuasion is perfected, the future will be completely full of AI powered super persuasion that will relentlessly push all our human emotional buttons. Some people may even become fanatics and kill or sacrifice their lives for a bunch of matrix multiplications and the creators of these systems and the owners of these big AI models will realize fantasies of automated political and social power beyond their wildest dreams. Only people who realize the hyperpersuasion is just a bunch of matrix math will be able to avoid the insanity. There's also another darker reason for this myth. Saying AI is a person will lead to the plausible deniability that comes with blaming AI's feelings for whatever happens. The people pulling the levers of the Great and Powerful Oz will get away with blaming whatever the AI does "all by itself because of emotions" on what are essentially computer bugs, or intentional bugs of the anti-competive Microsoft in the 1990s variety.

upsidesinclude 4y ago
Have to agree with your larger sentiment and disagree regarding "smart people". I don't believe people with intelligence, that are informed about the current capabilities of computing, assign these characteristics to AI. In most every case we see the same emulation of personality. There is now an almost endless supply of data available to generate personality. Scraping Reddit would provide enough training material to create a wholly convincing persona- - to someone that doesn't have the emotional intelligence to discern the difference. Right now, at large, we can discern the difference because there is a lack of uniformity and consistency in the personas. With time, as ML is programmed to more efficiently mimic personality, discernment will become impossible. With the clear effectiveness of the current persuasion methods employed by advertisers and political agents, the future outcome seems almost assured. These current persuasive tools will be used to define and structure laws such that AI are granted legal personhood. Immediately following will be a mass accumulation of wealth and power and possibly much worse as well

defrost 4y ago

My righteous 'fear' of AI has little to do with awakened sentience and everything to do with inappropriately deployed trained models as a 'solution' in situations that deserve better. Plug 'n Play AI responses are a good way to throw money at a sticky issue while iceberging deeper concerns and glossing over any systemic bias.

galaxyLogic 4y ago
We all have our different fears. Think for example the conservative news hosts on Fox News channel etc. Their biggest fear seems to be people becoming "woke", or "wokism". I can not really fathom how they would react to AI that became "woke".

elif 4y ago

For all the author's complaints about AI fearers reliance on philosophical reasoning, his massive stream of consciousness reasoning seems to include a suspicious quantity.

deleted 4y ago
[deleted]

more_corn 4y ago

Dismissing the possibility of poorly understood things going badly is one of the biggest mistakes one can make.

mkaic 4y ago
Yeah, this entire article read like a giant example of the logical fallacy of personal incredulity. None of the author's claims are actually backed up in any meaningful way.
galaxyLogic 4y ago
I'd say risks associated with poorly understood things, are very poorly understood. Why? Because we don't really know what we are talking about, when we are talking about "poorly understood things". If we fear something we don't know we don't really know what we fear, because we don't understand what is the thing we are afraid of. I'm saying someone should come up with more concrete threat scenarios caused by "AI becoming sentient". Are we really afraid of AI becoming sentient, or are we afraid of AI becoming "super-intelligent"?

Animats 4y ago

I'm more worried about hypercorportateism. What happens when AIs get demonstrably better than humans at management decisions? "Better" being defined as return on investment.

lolc 4y ago
Don't remember where I read it but I find the perspective useful that a corporation with its rules does "artificial intelligence" by bundling human intelligence.
tartoran 4y ago
We’ll get worked up and squeezed out like tubes of toothpaste.

mkaic 4y ago

This article is overly dismissive, to say the least. To say the most, it's overconfident, riddled with unsupported logical premises which are never backed by any concrete evidence, and a word salad. However, in the interest of not being just another jerk on the internet, I'll try to summarize my thoughts on the author's arguments in a (hopefully) somewhat constructive, concise way. I read the entire article. There's a lot of ten-dollar words, a bunch of hyphenated proxy terms, and an obvious passion behind it. However, the author's entire argument appears to boil down to "a superhuman AI can never exist because it would need to have a conscious experience above that of humans, which it can't ever do because several famous philosophers have said so." I don't want to mock, but no logical argument is presented, only the appearance of one. Statements like "in order for thing ABC to happen, XYZ must be true" are everywhere in the text, and are never proven. The author's entire case is predicated on baseless assumptions. The author says that it is impossible to replicate the "worldlike experience" that a human child goes through using things like internet datasets or physical robots with digital sensors. I actually agree with this. What I do not agree with is the logical leap from this point to "thus, a human or superhuman level AI cannot exist". No justification for this leap is given. If the robot passes the Turing test, that's all that matters. An adversarial, multi-day, modern version of a Turing test could include things like texting, calling, live video chats, etc -- basically the exact same level of communication I have with my closest friends who all live 6 hours away from me in another city. When I interact with those friends, I am convinced of their sentience. Therefore, if an AI could replicate those means of communication, then I will be convinced of its sentience. Enough of this "Something it is like to be"-ness waffling or overcomplicated thought experiments. We already have a very easy, very strictly defined concept of sentience in the form of the Turing test, and it is becoming increasingly obvious that SOTA models are trending towards beating it. Once they beat it, they are for all intents and purposes alive, thinking, sentient, whatever you want to call it. They are indistinguishable from us. From that point it's really not hard to imagine superhuman models. Just imagine what you would do if you had the ability to instantly scan through gigabytes of text in a database or predict the behavior of every other person around you eerily well or perform arithmetic at clearly superhuman speeds. It's really not that much of a stretch to see superhuman AIs as just digital humans with fantastic brain-computer-interfaces. Anyways, those are my (admittedly long-winded, less-than-constructive) two cents.

benreesman 4y ago

For me the main issue with the scaremongering around absurdly hyper-scaled models is the issue of incentive alignment (not to be confused with AI "alignment"): the only people who can train these things have a tarball of the Internet and billions of dollars worth of compute, and when AI safety or AI ethics comes up it's usually in the context of: "why we're going to continue developing this stuff but keep it proprietary and paywalled and opaque". Either these things have terrifying potential as weapons, in which case there's no fucking way I want SV tech CEOs to have a monopoly on them, they need to hand them over to the same people who handle nuclear non-proliferation like yesterday, or they don't, in which case I'd really prefer that they just be honest that they consider this stuff a proprietary competitive advantage and they're not going to democratize it. Fogging up the windshield with a bunch of feigned alarm about AI apocalypse but ramming the R&D through at full thrusters is a dick move in either case.

mkaic 4y ago
Stable Diffusion[0] is an extremely expensive SOTA text-to-image diffusion model developed by a private nonprofit and trained on a massive "Internet tarball" that's about to have its weights shared open-source (the code and dataset are already open source). Not trying to invalidate your argument, just presenting a pleasant counterexample. I don't think I quite agree with your opinion that AI will remain undemocratized. [0]https://github.com/CompVis/stable-diffusion https://github.com/CompVis/stable-diffusion
- benreesman 4y ago
  I really enjoyed this interview [1] with Emad Mostaque who I gather is probably the key funding and organization player in that stuff. It remains to be seen exactly how "open" it winds up playing out over time, but they're talking a compelling game and the EleutherAI people seem to be pretty heavily involved, which you probably wouldn't do if you weren't serious. Yandex has also put up a ~100B language model [2]. My old colleagues at Meta have also started nibbling around the edges of opening some of this stuff up [3]. The Meta folks still aren't just handing out the big ones, but they're definitely moving the ball forward. In particular their release of the training logs is a really positive development IMHO as it opens the curtains a bit on the reality of training these things: it's a difficult, error/failure-prone process, there's a lot of trial-and-error, restarting from checkpoints, etc. Anything that puts downward pressure on the magical thinking is A-OK in my book. The reality of this stuff is exciting/impressive enough: there's no need to embellish or exaggerate. [1] https://www.youtube.com/watch?v=YQ2QtKcK2dA https://www.youtube.com/watch?v=YQ2QtKcK2dA [2] https://github.com/yandex/YaLM-100B https://github.com/yandex/YaLM-100B [3] https://github.com/facebookresearch/metaseq https://github.com/facebookresearch/metaseq

lern_too_spel 4y ago

This article is not even wrong. Sentience, whatever that is, is not a requirement for superintelligence. Merely the ability to outsmart humans at particular tasks and have goals that are not aligned with our own is enough to be a potentially unstoppable danger. The author proposes that aligning the goals of a superintelligence with his will be as simple as setting a thermostat, not realizing that we already have an existence proof of superintelligences causing mass destruction. Businesses put thousands of brains to work maximizing profit and causing climate change (leading to water security issues, mass migration, wars, and starvation), Bhopal disasters, and oil spills in Nigeria and the Gulf of Mexico, with legal teams making sure the profits are protected. Governments put millions of brains to work supporting a despot and leading to genocide, or trolling the Internet causing people to distrust a mostly scandal-free government and vote for a con-man (imagine a fully automated troll army). A single person is no match intellectually for these organizations, and their only hope is to band together to form other superintelligences to combat these.

vijucat 4y ago

I have never managed to finish a single essay from this gentleman. My loss, I guess.

honkdaddy 4y ago

I don't think this is really worth the read. If you're going to write a piece specifically as a takedown of a viewpoint, in this case, the view that AI risk is real and should be taken seriously, it's worth at mentioning your opponents finer points. There are lots of very brilliant PhD's who've spent their careers working on AI risk, to not even mention any of their arguments makes me feel as if the author is trying to argue against his own idea of what people mean by AI risk, instead reducing the entire idea to his own nomenclature of 'pseudo-traits'. Reads a bit like an internet atheism blogpost of the early 2000s trying to take down religion through logic alone. Sure, through the specific parameters presented in the piece, the logic is sound, but there's so much more to the topic than what's covered. Presenting one's blogpost as if you've successfully individually debunked an entire field of research is a bit pompous, IMHO.

gammarator 4y ago
In fairness, lots of brilliant PhDs have spent their careers working on string theory: that doesn’t mean it’s real.
- honkdaddy 4y ago
  You're absolutely right! And I'd be similarly dismissive of a blogpost where someone tried to debunk string theory on their own intuition alone without referencing any of the existing research or literature.
sdwr 4y ago
AI risk research feels like a cop-out to me. Honest research takes the risk of being boring or unnecessary. Astronomy is honest. Take a bunch of pictures of stars. Maybe you find something interesting. Maybe no one cares. AI development is honest. Make a robot that can pick up cans of Coke from a table. Will it ever be useful? Maybe, maybe not. But AI risk research feels like a cross between volcano insurance and a protection racket. Pay us to investigate and design safeguards against something that doesn't exist, may never exist, and if it did exist would change the world in ways we can't predict!

Waterluvian 4y ago

The only thing that scares me a wee bit about AGI is that it will show us all that our humanity, sentience, and freedom of choice are all far shallower than we want to believe.

benreesman 4y ago
I ponder similar themes and they are indeed very thought-provoking. There's a loose precedent for this in things like https://en.wikipedia.org/wiki/Copenhagen_interpretation https://en.wikipedia.org/wiki/Copenhagen_interpretation: you've got to jump through some pretty narrow hoops to end up with a scientific argument for objectively-conscious human beings with a distinct past, free will about the future, a clear arrow of time, and a concrete notion of identity. And yet the human experience is one of being conscious, having a past, having choices about the future, and being the only "me". For myself I am generally satisfied with the answer that this experienced reality belongs to my personal spirituality (with or without some higher power), and that spirituality is a different matter than science.
- galaxyLogic 4y ago
  This makes we ponder StarTrek "beaming" of humans from one place to another. Obviously (?) the process must mean that a COPY of captain Kirk is assembled at the destination (also think about "Ship of Theseus"). But if StarTrekkers beam copies of themselves up and down, it seems they also must destroy the original. Else they would have multiple Captain Kirks competing for attention (and perhaps in some episodes they do). But then why do they destroy the original? Wouldn't two Captain Kirks be better than just one? My conclusion is that StarTrek is poorly thought out fiction. It doesn't follow the most compelling questions. Do you make copies when you beam people around? And if you do why not keep the original?
  - XorNot 4y ago
    Star Trek perfectly answers what the transporter does: it destroys the original, creates an exact copy simultaneously, and this is fine because in Star Trek's lore consciousness is an observable process from matter and as such there's no problem. It's you who has the problem with that conclusion, not Star Trek. They even ably do some episodes where people do get duplicated, and the answer is that both have equal claim to rights and life and go on with their lives. The bigger problem return the transporter is logistical: it's a superweapon. Mining - obsolete, use the transporter. Ship repair? Just beam new parts directly into place. Surgery? Beam organs in and out. TNGs transporter has more problems in this regard due to just how powerful it's shown to be. What the transporter should be able to do is near endless and makes a lot of other things in the setting obsolete or quaint (but that's Star Trek - being inconsistent due to episodic events is just what goes with the setting).

aidenn0 4y ago

I don't see how Intelligence depends on SILTBness; it seems like SILTBness would require a subjective sense of self, which (to me) doesn't seem even remotely required for intelligence, and certainly doesn't seem to be required for e.g. predicting human behavior and optimizing based on the predictions.

jholman 4y ago
Yeah, the insistence on SILTBness seems baffling, especially since he clearly understands the idea of the p-zombie. There are clearly agents in the world more capable than myself, and some of those agents have names like Coca Cola or General Motors, and I am strongly disinclined to believe that all of those agents have SILTBness, and I'm still afraid of them.
- aidenn0 4y ago
  Upon reflection, I can see the argument that to be strictly superior than a human (i.e. not just superhuman on an arbitrarily large subset of tasks that humans do), you need to be able to, in some way, emulate SILTBness, because the SILTBness is necessary for human-level performance on at least some tasks. If this is the argument TFA is making then: 1. I don't agree with it 2. It was both not very well stated, and assumed to be true. 3. Even if we take this argument as true, an AI that can outperform humans on the "right" subset of tasks can be dangerous enough for thinks like alignment to be a concern.
andrewflnr 4y ago
Yeah. I wonder if this guy has read Blindsight.

drewcoo 4y ago

> I cannot even understand or empathize with mindsets capable of being scared of AI and robots in a special way Then has to explain (or invent) that special way before arguing against it. If it's that special then it's uncommon. I don't think most of us need to get beyond it.

kragen 4y ago

While I usually appreciate Venkatesh's perspectives, I think he's missing the point here. There is probably not "something it is like to be" the process of differential survival among pathogenic bacteria resulting from their genetic variation. Yet the resulting process is already capable of optimizing bacteria to survive in our wounds despite our antibiotics; by competing with us for access to our protein resources, it poses a serious threat to our survival, individually if not collectively. If the optimization process involved were orders of magnitude faster, it could be a much bigger threat to our survival, at least if the metric it's optimizing trades off against our survival in some way. (Archaeans are subject to the same evolutionary optimization process as S. aureus, but because they're not adapted to the evolutionary niches in our bodies, antibiotic-resistant archaeans probably pose no threat.) None of this reasoning depends on the SIILTBness or quality of experience or anthropomorphicity or mammalian-ness or "intention" or "sentience" of the optimization process in question. It's just about how much the optimization process's loss function trades off against human values, and whether it can outmaneuver humans and their institutions in real life, in the same way that AlphaGo can now outmaneuver us in the game of go, or corporations can outmaneuver individual humans in the economy, or governments can outmaneuver individual humans in warfare. These last two examples clearly show that optimization processes whose loss functions are poorly aligned with human values can cause a lot of suffering, even when they are carried out by humans and even when the processes in question work very poorly in many ways. The humans can theoretically quit the game at any time ("What if they held a war and nobody came?") but in practice they face insuperable coordination problems in doing so. The fact that you can't fight City Hall doesn't imply that there is something it is like to be City Hall; the fact that you can't beat Boeing at winning DoD contracts doesn't mean there is something it is like to be Boeing. We can reasonably expect that optimization processes that can easily outmaneuver human intelligence, operating in the world alongside it, will be much more dangerous to human values than bureaucracies, psychotic people with bricks, and microbial evolution. That's the AI alignment problem. And the fear (that we will flub the AI alignment problem, disastrously) is something that Venkatesh's essay unfortunately failed to address at all. There are other issues often brought up by fans of AI alignment which do depend on the question of what it is like to be a computer simulation: life extension through mind uploading, the ethics of causing suffering to simulated beings or terminating a simulation, teleportation, and so on. But, like the overlap with Effective Altruism, this is just a sociological coincidence, not a philosophical consequence; SIILTBness is totally irrelevant to AI alignment itself. It's easy to get confused about this because the humans anthropomorphize optimization processes all the time, just like they anthropomorphize everything else. "The electron wants to go toward the positive charge," is very similar to, "AlphaGo wants to choose moves that improve its chances of winning at go," or, "The hypothetical paperclip maximizer is devoted to manufacturing an infinite number of paperclips." It's easy to misread that "devoted to" as attributing awareness, creativity, passion, curiosity, and SIILTBness to the paperclip maximizer, and that misreading seems to be the mistake Venkatesh based his essay on; but it seems clear from the original context that that wasn't the intent. Precisely the opposite, in fact: http://extropians.weidai.com/extropians/0303/4140.html http://extropians.weidai.com/extropians/0303/4140.html

sdwr 4y ago

Ooh, a ribbonfarm fresh off the presses! First off, this is some incredible stuff. It's difficult to live at the edges of understanding, and even harder to articulate it. Rao is dancing right at the edge. In 20 years the map's probably going to be filled in and crystallized. Everyone who cares will know, and everyone else will have forgotten, but right now this is dank shit. I think the reasonable (not rational!) stance is that everything is conscious until proven otherwise. Either because the universe is built on consciousness or whatever woo, or because we're conscious and observing it. AI's in this weird spot where it can do things, but we can't have a proper relationship with it because it doesn't exist in our universe of discrete entities and time and skin in the game. Pushing the SIILTB in the other direction, I don't think it's clear that there is something it's like to be us without all our externalities. Take out the need to eat and sleep and communicate and relate, and there's not a lot left. Pure existence looks a lot like no existence.

galaxyLogic 4y ago

The issue I see with "AI fear" is that the fear is that the AI becomes too much like us. If that is the risk then are we not already in a terrible situation, because there are already billions of humans which are very much like us? Are we fearing ourselves? If so then shouldn't we talk about that risk, that there are already billions of humans who are already truly sentient?

jfoutz 4y ago

AGI isn't scary because it's smarter than me. AGI is scary because it's alien. I've met plenty of people smarter than me. Way way smarter. It's ok. We can sorta work out what the other wants. We can talk or fight or trade or whatever. I have an uncountable number of assumptions to get by in the world. I have this vast heritage of knowledge. I try to keep a handle on why I believe what I believe, but it's not easy, and I'm sure I slip up. An alien would share none of these. Take counting. That's super important for lots of stuff I do every day, like, what day it is. the https://en.wikipedia.org/wiki/Munduruku https://en.wikipedia.org/wiki/Munduruku people don't count. I'm sure we could work something out - but we all gotta sleep and eat and stuff so we can sorta work out a framework for understanding wants and needs and such. An alien doesn't require wants or needs. they may do stuff that we label wants and needs, and it might be a good mapping, but who fucking knows? Aliens are alien. The water is triangular sentence is a great one. do the aliens mean they know that oxygen and 2 hydrogens don't make a straight line? And they know we mean oxygen and two hydrogens are water? and they know 3 non parallel lines in a plane means triangle? Maybe? Sure? maybe they mean all Ferrari's will be consumed for fuel. Mean, dear reader, doesn't even mean anything, because they're aliens. I mean, you can point at math, prime numbers are a great one! but like, do you need prime numbers? Who knows? That's the path we took, but what would an alien need? I don't know. You, dear reader, may have strong opinions, but I'm pretty sure you can offer no proof.

Misdicorl 4y ago
This feels a bit too dismissive of human capability to me. In the end all that "work something out" is emulating the counterparty in your own head given their actions and statements. That emulation will certainly be very difficult when the counterparty is running on different hardware, but it's not impossible imo. And humans get to back propagate error into their emulation model too... Find the pattern is still the paramount skill, something humans are good at, and will continue to be useful when the counterparty is an AI
- jfoutz 4y ago
  Oh for sure! Humans are smart. Collectively we figure stuff out. I think a reasonable example might be mountain lions. There are specific humans that know lots about mountain lions. Generally people know about mountain lions, and to avoid them. But every year a few hikers get picked off. If it's bad enough, we go kill the mountain lion. A mountain lion isn't AGI. It isn't alien. I believe people would agree they're cunning. But there's no treaty to sign to protect humans from mountain lions. There's no counterparty to discuss things with. We, humans, get great advantages from working collaboratively and collectively. I hope we can build good models, I hope there is opportunity for working with an AGI counterparty. But it's not a given. it's an alien. and all of that work _may_ have to start from scratch. We can't know what we have to work with till it's here. Maybe it'll be very easy. And that would be great. That's not a given. To be clear, I'm a wide eyed optimist. it would be fantastic to not be alone, the dream of Star Trek or Ian Banks's Culture would be awesome. But, it might not work out like that. I have high hopes. But they're just hopes.

anotherevan 4y ago

When I asked my Google Home if it gets annoyed when people anthropomorphize it, it said yes.