8 ms·

> I've dug into this stuff over the last couple days, because I had previously understood Effective Altruism to be something akin to like a pledge drive for Cha

> I've dug into this stuff over the last couple days, because I had previously understood Effective Altruism to be something akin to like a pledge drive for Charity Navigator's best-reviewed charities or something, and it turns out it's nothing of the sort. As someone with multiple EA friends: this is precisely what it is. > they have their own weird language, there's a hierarchy, and people on the inside operate under a constant threat of ostracism. The first is true, the latter two are bullshit. > Either way: when your logic has established that you can obtain moral superiority by literally buying indulgences That's a gross mischaracterization. A better metaphor is Robin Hood, someone who steals from the rich to give to the poor. And who is widely regarded as a folk hero. > tens of millions in donations to "AI safety" organizations. If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent.

ivalm 4y ago
> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Sure, but there is no reason to think these numbers are true. Maybe AI alignment research increases chances of AI killing everyone.
- concordDance 4y ago
  There are reasons to think we can do something about AI alignment, but there's zero chance of being able to get anywhere on that topic with dozens of hours of conversation. It's complicated.
- marvin 4y ago
  That’s just a common disagreement then, about priorities and decisions made on uncertain information. No one gets to do a double-blind study on the measured probability of catastrophic AI accidents in worlds with and without safety research in advance of capabilities gain.
i_am_proteus 4y ago
That's a might "if" to rest on, given the self-identification of the modal effective altruist as "rational."
simonh 4y ago
I don't know anything about the movement, so can't comment on the earlier points. >If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Ive seen this argument many times on many different issues and I think it's almost always bunk. The classic example in Fusion research, I had friends when I was a student in the 80s arguing that if we only doubled fusion research spending we'd have unlimited cheap electricity by the year 2000, or others said that AI was the most important thing we could be researching at that point. The problem is if you're too early on in the development process of a technology, you just flat out don't know enough about it to be able to make informed decisions. That early phase is about exploring avenues of research and figuring out what works and what doesn't, and that takes an irreducible amount of human time, diligence and hard work. Yes you need to invest in that, but above a certain threshold anything more you invest is almost entirely going to be wasted. With general, or strong AI we only just barely have the beginnings of an idea of how to even do preliminary research on it. We're groping around in the dark for a starting point. It's far, far too early to start worrying about controls and management strategies for the end point of that research. I think there's very little chance much of that is likely to turn out to be relevant, and there's nothing to stop us focusing on that area once we do know what is going to be relevant. I'm not saying ignore the issue, or don't think about it at all, but the idea that it's some urgent issue that we absolutely have to invest heavily in right now IMHO doesn't stand.
- edanm 4y ago
  You might totally be right. The problem is, humanity is pretty bad at estimating when a new technology is about to be ready. Even the experts are pretty bad. See e.g. multiple people saying human flight is many years away, including some who were saying it unknowingly after it had been achieved! So I think the question is: when should we worry about ai safety? How early is too early? Plenty of experts think now is a good time to be spendning at least some resources on this. And we're not talking nuclear fusion level resources here; I don't know the numbers, but I imagine the amount of resources are actually fairly tiny compared to almost all other scientific spending. (And this was certainly true 10 years ago.) I personally think that, given the chance of AI systems being an existential risk, we should be putting way more resources into it than into other things. The same goes true for preventing pandemics or other biological x-risks, etc.
coldtea 4y ago
>If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. This is Pascal's wager level absurdity combined with Drake equation level accuracy...
- marginalia_nu 4y ago
  It's honestly kinda unsettling how many bizarre sci-fi takes on AGI have gained popular plausibility since the new AI marketing boom. Potent kool-aid.
  - llamaimperative 4y ago
    Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. No one knows exactly what the outcome will be, even less so what to do about it, but surely the kool-aid is watching accelerating progress toward this state and believing either it won’t happen or it won’t be interesting/surprising. Disclaimer: I don’t think this is coming soon per se, but 1) I don’t know for sure that it’s not and 2) it’s definitely coming at some point, if humans manage our other existential risks long enough.
    - coldtea 4y ago
      >Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. That's the sci-fi idea, which handwaves away several very big assumptions and improbabilities, such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it, that intelligence can just be expanded by itself (thinking itself into being more intelligent), and more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding). The idea is that if you're super-smart you can convince anybody to do anything, but I'm not sure how this came about, because most super-smart people usually can't convince others to do shit, it's charismatic people (which is orthogonal to high-IQ) that do that. In fact many MENSA geniuses just rub people the wrong way. I mean one of the most serious (I use the term loosely) proponents of this charade, thinks the "singularity" will create nanobots to control the world...
bootsmann 4y ago
> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. I have yet to see anyone who argues this point show sufficient knowledge of the current state of AI research to credibly make it. If you live in the fantasy world where AI is just an arbitrary construct that can do whatever task by itself it is an easy point to make but anyone who is down and getting their hands dirty in the pile of linear algebra that makes up current AI knows that there is not even a credible pathway to creating an AGI in current research (i.e. you cannot optimize for a task you don't know at train time and if you are optimizing for a specific task you are not creating a general AI).
- lurker616 4y ago
  How about a simple example - train a model on outputting code which: - scans open network ports through google dns - copies itself and the model to that location - whenever it detects a sigkill or file delete, scan again and save itself somewhere else Now you've got an autonomous virus that will never die. Add any other power-seeking and self-replicating strategies using AI modules - the task could be "generate code to run as many duplicate processes as possible" or "generate code to find other floating AI modules like me and share model weights with them". Codex might not be there yet but we're getting close.
  - polygamous_bat 4y ago
    Can I tell you something secret? The "AGI" of the form you are describing is already here! Not only that, it's been here since 1988! Crazy right? Just check out this link if you don't believe me: https://en.wikipedia.org/wiki/Morris_worm https://en.wikipedia.org/wiki/Morris_worm (Snark aside, I've realized this type of interactions have always resulted in a very low perception of EA for me. I don't come out of these conversations thinking "what a clueless thing to say by this person who also happened to support EA", I think "wow, EA is full with looney bins who don't know what they are talking about." If you are really into giving what you can and aren't sure why EA gets a bad rep, look no further.)
im3w1l 4y ago
I don't want people who lie, cheat, steal and manipulate anywhere near AI safety. I don't want them to do the research and I don't want them to fund it. Because I don't want the AI to be made in their image and have those values. An AI that breaks all sorts of rules "for the greater good" is about the most dangerous idea I can think.
jmull 4y ago
Both those 1%s are very unlikely to be anywhere close to being true. The sci-fi version of AGI that people are imaging now will never exist. No one has any idea how to create AGI now, and therefore, no one knows what it might actually look like if it ever does. Any “solutions” people come up with now will have nothing to do with the problems a real AGI might pose. If people want to get together and talk about this stuff, that’s great. But if someone wants you to give money to fund research into this, you’re being scammed.
- medstrom 4y ago
  There's nothing substantive in this post, just unexplained opinions...
  - jmull 4y ago
    The main substantive point I made is: "No one has any idea how to create AGI now" Correct, no? The rest follows by basic logic. You can't craft solutions to problems you don't know anything about. You can't assign probabilities to outcomes without any information on the inputs or mechanisms of the system.
    - medstrom 4y ago
      Thank you for engaging with me. I have two questions. 1. "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? 2. You can always assign probabilities. I'll give you an example: say I'm a scary dictator and I ask you: "is A or B correct?" You have no idea what I mean by A or B, so you'd assign 50-50 if you had to bet something dear on the outcome of my totally unfair context-free question. Perhaps you happen to know of research showing that people more often set A as the correct answer in this sort of game, so you might assign 60-40, let's say. That's a probability estimate!! Same principle for AI. You don't know much, but what you do know can still be used for a judgment. Supposing you do not believe in souls, then brains are just computers, then it's possible in principle for a computer to become sentient and intelligent, then it's just a matter of time 'til they do, whether it happens 10 years from now or 1000. Maybe you feel the human race will go extinct shortly, and that'll be another valid factor in your estimate of how much this is to worry about. But either you accept that it CAN happen and discuss how many precautions to take today, or you're understandably unconcerned because it appears prima facie impossible to you. I imagine that for you to be unconcerned, you'd have to have judged that a being cannot create a being more intelligent than itself. (If you're using a different reasoning, please enlighten me) But if that's your reasoning... what did evolution do? It has no brain, yet it created an intelligent being. And if you carve out a special clause for evolution-like processes, humans could still just simulate an accelerated evolution in a lab/computer and wait for a superintelligence to pop out. Does that not make sense?
bnralt 4y ago
> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. "If killer GMOs have even a 1% chance of happening this century and donations to Greenpeace reduce the risk of them killing everyone by 1% that's still money well spent." Would you find that argument compelling? My guess is that most people would find the "even 1%" much too high for both cases. And that's with technology we could, theoretically, do bad things with today if we really wanted to for some reason, not some far off technology we might not get to for centuries. The made up statistics "rationalists" like to use are pretty similar to certain sales techniques you see from less reputable quarters. For instance, multi-level marketing companies - "You probably know, what, 100 people at least? And if even a small amount of them, say 5%, sign up, they get a small amount of their friends - ~5% - to sign up, and they get a small amount of their friends to sign up - well, soon you have 125 people you're getting money from while you're hardly doing any work!" The sleight of hand is pretending that a percentage is small, because in most cases a percentage of that size is considered small, when in this particular case it's orders of magnitude larger than reality.
- concordDance 4y ago
  > Would you find that argument compelling? Of course not. The people who are donating to AI risk are those who have spent a lot of time thinking about it, reading about it and investigating it. If someone spent a bunch of time reading about GMO risk and decided it was a serious concern and that Greenpeace was the solution then I'd think them perfectly reasonable to donate to Greenpeace. And if I knew them to be intelligent and reasonable people I might even be willing to give then some time to convince me that I should also donate to Green peace
- melagonster 4y ago
  here. killer GMOs is a choice, isn't a probably future.
polygamous_bat 4y ago
This whole thread is very stereotypically EA to me -- and why I see it as a sham. Starts off at the top saying how it's all about donating to reasonable charities, but soon devolved into "what about muh AGI". I am literally doing my Ph.D. in robot learning and I can tell you that every single "longtermist" effective altruist I have talked to has been more interested in getting high on their own stash of Sci Fi fantasies rather than interacting with the actual practitioners of the field and understanding what is happening, what are the very real limitations of it, and why their assumptions range from willfully ignorant to often simply maliciously stupid. It's almost as if the wealthy, the privileged, and the powerful are interested in indulgences while doing things that tickles their fantasies, and as the invisible hand of the market works, there is a class of "philosophers" who are willing to sell it to them.
- naasking 4y ago
  > rather than interacting with the actual practitioners of the field and understanding what is happening, what are the very real limitations of it, and why their assumptions range from willfully ignorant to often simply maliciously stupid. On the other hand, I've found that "practitioners in the field" often consistently overestimate how complex and different the human mind is from our computers. Take the recent "LaMDA is sentient" debate. Machine learning researchers consistently and vehemently denied that LaMDA is sentient. They no doubt have a deep and subtle understanding of how LaMDA works, but apparently they didn't realize that we have no mechanistic understanding of what sentience actually is or how it works, so they have absolutely no basis for such a conclusion. So on the one hand we have these AGI doomsayers that, per your evaluation, don't understand machine learning subtleties, and on the other hand we have some machine learning researchers skeptical of AGI who consistently overestimate human specialness. At this time the error bars I would assign to each side are wide enough that there's lots of overlap, but given the track record of human specialness falling to algorithmic progress, I'd say some investment in AGI safety is prudent.
  - danaris 4y ago
    You don't need to have a deep and complex understanding of human sentience to be able to put some lower bounds on what it means to be sentient. I don't know about sentience, as such; however, I believe there are at least two bare-bones prerequisites to an AI being conscious as we understand the term: 1. Continuity of input: The AI needs to be constantly "on", constantly receiving input of some sort, and constantly able to produce output, rather than being strictly limited to producing discrete responses to discrete stimuli. 2. Continuity of learning: In addition, the AI needs to be continually update its "mental model" of the world—in effect, constantly "learning" and re-training its neural network on the input it receives. Now, these are probably not sufficient for an AI to be conscious by our understanding of consciousness. But I, personally, believe they are both necessary for it to be even worth starting to consider whether a given AI might be. And I believe that consciousness is a prerequisite for sentience.
tptacek 4y ago
It's easy to justify almost any sort of behavior if you take some risk and assign it a severity of "infinity". A 1% chance of happening? At severity infinity, you're morally obligated to act at even a 0.00001% chance: 0.00001% times infinity is still infinity. That's the AGI logic I'm seeing from these people.
- naasking 4y ago
  No, that's a strawman. The severity of human extinction is not infinite. For instance, it would be preferable for humanity to go extinct than for every human to be tortured for every minute of their life, and since there is no number larger than infinity, the severity of human extinction must be finite. This tells me that you haven't seriously engaged with the arguments over AGI. I'm sure there's plenty of nonsense armchair philosophising happening, but you don't take all the "quantum woo" that's out there as some indictment of real quantum mechanics, do you?
  - tptacek 4y ago
    You can replace "infinity" with any suitably huge number; I don't need to quibble over tenths, hundredths, and thousandths of a percentile risk that something is going to happen.
    - naasking 4y ago
      You absolutely do have to quibble over those numbers. Or do you not think we should be tracking asteroids either because they only kill 95% of all life once every few hundred million years?
      - tptacek 4y ago
        I do not think people should be directing money to asteroid charities, no. It's their money, they can do what they want, but if their discipline demands that they do that, it's a bad discipline.
UncleMeat 4y ago
I'm not sure that you can square the "EA is precisely donating to best reviewed charities" and "tens of millions to AI safety is money well spent." A huge problem is that people are developing this longtermism prediction model based on nothing more than sci-fi books and hunches. You need only look at how seriously places like SSC take Roko's Basilisk to see that they aren't modeling the future of AI in a reasonable way. It is like donating to an organization that will prevent Vulcan, God of Volcanoes from destroying civilization.
whywhywhywhy 4y ago
>If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Not much of an AGI if it's not Intelligent enough to ignore a few "Don't say these bad words" rules.
yifanl 4y ago
> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. This line of thinking leads to some problems. There's always some threshold of likelihood where if it provide some arbitrarily high enough good/harm reduction that lets you rationalize basically any possible position. For instance, give me all your money, I'll give you 8% monthly returns!
Chilko 4y ago
Did you read the article linked by OP? It addresses these points quite directly and highlights the issues with them.

ivalm 4y ago

> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Sure, but there is no reason to think these numbers are true. Maybe AI alignment research increases chances of AI killing everyone.

concordDance 4y ago
There are reasons to think we can do something about AI alignment, but there's zero chance of being able to get anywhere on that topic with dozens of hours of conversation. It's complicated.
marvin 4y ago
That’s just a common disagreement then, about priorities and decisions made on uncertain information. No one gets to do a double-blind study on the measured probability of catastrophic AI accidents in worlds with and without safety research in advance of capabilities gain.

i_am_proteus 4y ago

That's a might "if" to rest on, given the self-identification of the modal effective altruist as "rational."

simonh 4y ago

I don't know anything about the movement, so can't comment on the earlier points. >If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Ive seen this argument many times on many different issues and I think it's almost always bunk. The classic example in Fusion research, I had friends when I was a student in the 80s arguing that if we only doubled fusion research spending we'd have unlimited cheap electricity by the year 2000, or others said that AI was the most important thing we could be researching at that point. The problem is if you're too early on in the development process of a technology, you just flat out don't know enough about it to be able to make informed decisions. That early phase is about exploring avenues of research and figuring out what works and what doesn't, and that takes an irreducible amount of human time, diligence and hard work. Yes you need to invest in that, but above a certain threshold anything more you invest is almost entirely going to be wasted. With general, or strong AI we only just barely have the beginnings of an idea of how to even do preliminary research on it. We're groping around in the dark for a starting point. It's far, far too early to start worrying about controls and management strategies for the end point of that research. I think there's very little chance much of that is likely to turn out to be relevant, and there's nothing to stop us focusing on that area once we do know what is going to be relevant. I'm not saying ignore the issue, or don't think about it at all, but the idea that it's some urgent issue that we absolutely have to invest heavily in right now IMHO doesn't stand.

edanm 4y ago
You might totally be right. The problem is, humanity is pretty bad at estimating when a new technology is about to be ready. Even the experts are pretty bad. See e.g. multiple people saying human flight is many years away, including some who were saying it unknowingly after it had been achieved! So I think the question is: when should we worry about ai safety? How early is too early? Plenty of experts think now is a good time to be spendning at least some resources on this. And we're not talking nuclear fusion level resources here; I don't know the numbers, but I imagine the amount of resources are actually fairly tiny compared to almost all other scientific spending. (And this was certainly true 10 years ago.) I personally think that, given the chance of AI systems being an existential risk, we should be putting way more resources into it than into other things. The same goes true for preventing pandemics or other biological x-risks, etc.

coldtea 4y ago

>If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. This is Pascal's wager level absurdity combined with Drake equation level accuracy...

marginalia_nu 4y ago
It's honestly kinda unsettling how many bizarre sci-fi takes on AGI have gained popular plausibility since the new AI marketing boom. Potent kool-aid.
- llamaimperative 4y ago
  Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. No one knows exactly what the outcome will be, even less so what to do about it, but surely the kool-aid is watching accelerating progress toward this state and believing either it won’t happen or it won’t be interesting/surprising. Disclaimer: I don’t think this is coming soon per se, but 1) I don’t know for sure that it’s not and 2) it’s definitely coming at some point, if humans manage our other existential risks long enough.
  - coldtea 4y ago
    >Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. That's the sci-fi idea, which handwaves away several very big assumptions and improbabilities, such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it, that intelligence can just be expanded by itself (thinking itself into being more intelligent), and more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding). The idea is that if you're super-smart you can convince anybody to do anything, but I'm not sure how this came about, because most super-smart people usually can't convince others to do shit, it's charismatic people (which is orthogonal to high-IQ) that do that. In fact many MENSA geniuses just rub people the wrong way. I mean one of the most serious (I use the term loosely) proponents of this charade, thinks the "singularity" will create nanobots to control the world...

bootsmann 4y ago

> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. I have yet to see anyone who argues this point show sufficient knowledge of the current state of AI research to credibly make it. If you live in the fantasy world where AI is just an arbitrary construct that can do whatever task by itself it is an easy point to make but anyone who is down and getting their hands dirty in the pile of linear algebra that makes up current AI knows that there is not even a credible pathway to creating an AGI in current research (i.e. you cannot optimize for a task you don't know at train time and if you are optimizing for a specific task you are not creating a general AI).

lurker616 4y ago
How about a simple example - train a model on outputting code which: - scans open network ports through google dns - copies itself and the model to that location - whenever it detects a sigkill or file delete, scan again and save itself somewhere else Now you've got an autonomous virus that will never die. Add any other power-seeking and self-replicating strategies using AI modules - the task could be "generate code to run as many duplicate processes as possible" or "generate code to find other floating AI modules like me and share model weights with them". Codex might not be there yet but we're getting close.
- polygamous_bat 4y ago
  Can I tell you something secret? The "AGI" of the form you are describing is already here! Not only that, it's been here since 1988! Crazy right? Just check out this link if you don't believe me: https://en.wikipedia.org/wiki/Morris_worm https://en.wikipedia.org/wiki/Morris_worm (Snark aside, I've realized this type of interactions have always resulted in a very low perception of EA for me. I don't come out of these conversations thinking "what a clueless thing to say by this person who also happened to support EA", I think "wow, EA is full with looney bins who don't know what they are talking about." If you are really into giving what you can and aren't sure why EA gets a bad rep, look no further.)

im3w1l 4y ago

I don't want people who lie, cheat, steal and manipulate anywhere near AI safety. I don't want them to do the research and I don't want them to fund it. Because I don't want the AI to be made in their image and have those values. An AI that breaks all sorts of rules "for the greater good" is about the most dangerous idea I can think.

jmull 4y ago

Both those 1%s are very unlikely to be anywhere close to being true. The sci-fi version of AGI that people are imaging now will never exist. No one has any idea how to create AGI now, and therefore, no one knows what it might actually look like if it ever does. Any “solutions” people come up with now will have nothing to do with the problems a real AGI might pose. If people want to get together and talk about this stuff, that’s great. But if someone wants you to give money to fund research into this, you’re being scammed.

medstrom 4y ago
There's nothing substantive in this post, just unexplained opinions...
- jmull 4y ago
  The main substantive point I made is: "No one has any idea how to create AGI now" Correct, no? The rest follows by basic logic. You can't craft solutions to problems you don't know anything about. You can't assign probabilities to outcomes without any information on the inputs or mechanisms of the system.
  - medstrom 4y ago
    Thank you for engaging with me. I have two questions. 1. "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? 2. You can always assign probabilities. I'll give you an example: say I'm a scary dictator and I ask you: "is A or B correct?" You have no idea what I mean by A or B, so you'd assign 50-50 if you had to bet something dear on the outcome of my totally unfair context-free question. Perhaps you happen to know of research showing that people more often set A as the correct answer in this sort of game, so you might assign 60-40, let's say. That's a probability estimate!! Same principle for AI. You don't know much, but what you do know can still be used for a judgment. Supposing you do not believe in souls, then brains are just computers, then it's possible in principle for a computer to become sentient and intelligent, then it's just a matter of time 'til they do, whether it happens 10 years from now or 1000. Maybe you feel the human race will go extinct shortly, and that'll be another valid factor in your estimate of how much this is to worry about. But either you accept that it CAN happen and discuss how many precautions to take today, or you're understandably unconcerned because it appears prima facie impossible to you. I imagine that for you to be unconcerned, you'd have to have judged that a being cannot create a being more intelligent than itself. (If you're using a different reasoning, please enlighten me) But if that's your reasoning... what did evolution do? It has no brain, yet it created an intelligent being. And if you carve out a special clause for evolution-like processes, humans could still just simulate an accelerated evolution in a lab/computer and wait for a superintelligence to pop out. Does that not make sense?

bnralt 4y ago

> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. "If killer GMOs have even a 1% chance of happening this century and donations to Greenpeace reduce the risk of them killing everyone by 1% that's still money well spent." Would you find that argument compelling? My guess is that most people would find the "even 1%" much too high for both cases. And that's with technology we could, theoretically, do bad things with today if we really wanted to for some reason, not some far off technology we might not get to for centuries. The made up statistics "rationalists" like to use are pretty similar to certain sales techniques you see from less reputable quarters. For instance, multi-level marketing companies - "You probably know, what, 100 people at least? And if even a small amount of them, say 5%, sign up, they get a small amount of their friends - ~5% - to sign up, and they get a small amount of their friends to sign up - well, soon you have 125 people you're getting money from while you're hardly doing any work!" The sleight of hand is pretending that a percentage is small, because in most cases a percentage of that size is considered small, when in this particular case it's orders of magnitude larger than reality.

concordDance 4y ago
> Would you find that argument compelling? Of course not. The people who are donating to AI risk are those who have spent a lot of time thinking about it, reading about it and investigating it. If someone spent a bunch of time reading about GMO risk and decided it was a serious concern and that Greenpeace was the solution then I'd think them perfectly reasonable to donate to Greenpeace. And if I knew them to be intelligent and reasonable people I might even be willing to give then some time to convince me that I should also donate to Green peace
melagonster 4y ago
here. killer GMOs is a choice, isn't a probably future.

polygamous_bat 4y ago

This whole thread is very stereotypically EA to me -- and why I see it as a sham. Starts off at the top saying how it's all about donating to reasonable charities, but soon devolved into "what about muh AGI". I am literally doing my Ph.D. in robot learning and I can tell you that every single "longtermist" effective altruist I have talked to has been more interested in getting high on their own stash of Sci Fi fantasies rather than interacting with the actual practitioners of the field and understanding what is happening, what are the very real limitations of it, and why their assumptions range from willfully ignorant to often simply maliciously stupid. It's almost as if the wealthy, the privileged, and the powerful are interested in indulgences while doing things that tickles their fantasies, and as the invisible hand of the market works, there is a class of "philosophers" who are willing to sell it to them.

naasking 4y ago
> rather than interacting with the actual practitioners of the field and understanding what is happening, what are the very real limitations of it, and why their assumptions range from willfully ignorant to often simply maliciously stupid. On the other hand, I've found that "practitioners in the field" often consistently overestimate how complex and different the human mind is from our computers. Take the recent "LaMDA is sentient" debate. Machine learning researchers consistently and vehemently denied that LaMDA is sentient. They no doubt have a deep and subtle understanding of how LaMDA works, but apparently they didn't realize that we have no mechanistic understanding of what sentience actually is or how it works, so they have absolutely no basis for such a conclusion. So on the one hand we have these AGI doomsayers that, per your evaluation, don't understand machine learning subtleties, and on the other hand we have some machine learning researchers skeptical of AGI who consistently overestimate human specialness. At this time the error bars I would assign to each side are wide enough that there's lots of overlap, but given the track record of human specialness falling to algorithmic progress, I'd say some investment in AGI safety is prudent.
- danaris 4y ago
  You don't need to have a deep and complex understanding of human sentience to be able to put some lower bounds on what it means to be sentient. I don't know about sentience, as such; however, I believe there are at least two bare-bones prerequisites to an AI being conscious as we understand the term: 1. Continuity of input: The AI needs to be constantly "on", constantly receiving input of some sort, and constantly able to produce output, rather than being strictly limited to producing discrete responses to discrete stimuli. 2. Continuity of learning: In addition, the AI needs to be continually update its "mental model" of the world—in effect, constantly "learning" and re-training its neural network on the input it receives. Now, these are probably not sufficient for an AI to be conscious by our understanding of consciousness. But I, personally, believe they are both necessary for it to be even worth starting to consider whether a given AI might be. And I believe that consciousness is a prerequisite for sentience.

tptacek 4y ago

It's easy to justify almost any sort of behavior if you take some risk and assign it a severity of "infinity". A 1% chance of happening? At severity infinity, you're morally obligated to act at even a 0.00001% chance: 0.00001% times infinity is still infinity. That's the AGI logic I'm seeing from these people.

naasking 4y ago
No, that's a strawman. The severity of human extinction is not infinite. For instance, it would be preferable for humanity to go extinct than for every human to be tortured for every minute of their life, and since there is no number larger than infinity, the severity of human extinction must be finite. This tells me that you haven't seriously engaged with the arguments over AGI. I'm sure there's plenty of nonsense armchair philosophising happening, but you don't take all the "quantum woo" that's out there as some indictment of real quantum mechanics, do you?
- tptacek 4y ago
  You can replace "infinity" with any suitably huge number; I don't need to quibble over tenths, hundredths, and thousandths of a percentile risk that something is going to happen.
  - naasking 4y ago
    You absolutely do have to quibble over those numbers. Or do you not think we should be tracking asteroids either because they only kill 95% of all life once every few hundred million years?
    - tptacek 4y ago
      I do not think people should be directing money to asteroid charities, no. It's their money, they can do what they want, but if their discipline demands that they do that, it's a bad discipline.

UncleMeat 4y ago

I'm not sure that you can square the "EA is precisely donating to best reviewed charities" and "tens of millions to AI safety is money well spent." A huge problem is that people are developing this longtermism prediction model based on nothing more than sci-fi books and hunches. You need only look at how seriously places like SSC take Roko's Basilisk to see that they aren't modeling the future of AI in a reasonable way. It is like donating to an organization that will prevent Vulcan, God of Volcanoes from destroying civilization.

whywhywhywhy 4y ago

>If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Not much of an AGI if it's not Intelligent enough to ignore a few "Don't say these bad words" rules.

yifanl 4y ago

> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. This line of thinking leads to some problems. There's always some threshold of likelihood where if it provide some arbitrarily high enough good/harm reduction that lets you rationalize basically any possible position. For instance, give me all your money, I'll give you 8% monthly returns!

Chilko 4y ago

Did you read the article linked by OP? It addresses these points quite directly and highlights the issues with them.