5 ms·

People in EA donate lots of money to causes that help people today. AI is just one of the more interesting subjects they often talk about. AI safety isn't abou

People in EA donate lots of money to causes that help people today. AI is just one of the more interesting subjects they often talk about. AI safety isn't about Roko's basilisk, or the idea that AI will take revenge on us for our moral choices. It's about the risk that an AI smarter than us will be made and built to pursue some goal without caring about our well-being.

PartiallyTyped 4y ago
> It's about the risk that an AI smarter than us It's also about the risk that AI is dumber than what we expect it to be and fails to properly solve the issue. A sufficiently smart AI should be able to understand our intentions and - assuming it is aligned with us - attempt to fulfill that purpose. Dumb "AI" will fail in ways that don't actually solve the task, or solve it only on the surface / do reward hacking.
- AgentME 4y ago
  >assuming it is aligned with us If we don't figure out alignment, we can't make this assumption. This is the part that AI safety people are concerned with making reality.
  - PartiallyTyped 4y ago
    I think I was misunderstood. My point is that not only do we need to consider the cases where the AI is smarter than us, but also the cases where it is dumber than we expect it. An aligned agent will not fall into stupid errors, thus solving the lower bound of performance is a necessity for complete alignment.
    - AgentME 4y ago
      Ohh, right on. Definitely agree now that I've re-read it right. I was a little confused by the first sentence of your other post and thought you were putting forward a disagreement in the second sentence.
  - pharmakom 4y ago
    I don't see how alignment is a different problem from making an AI do what we want.
    - PartiallyTyped 4y ago
      It's more about the unintended consequences, the implied/contextual information and getting it to do what we want in a way actually fulfills our goals instead of merely satisfying a condition. A task such as "bring me a cup of tea" has lots of implied information such as not making it too hot to burn the person, not breaking anything in the process, not harming other people in the process and so on.
    - AgentME 4y ago
      I think that's a fair way of putting alignment. The concern is that we might make an AI that's capable of outsmarting us before we're good at making it care about doing what we want instead of some other goal that disregards what we really care about like our safety.
AnIdiotOnTheNet 4y ago
> It's about the risk that an AI smarter than us will be made and built to pursue some goal without caring about our well-being. We already have those and they're called corporations. They've done significant real damage to the world already and they are still working hard to do more damage. It makes little sense to me to focus on this potential future problem when we haven't even agreed to deal with the ones that we already have.
- AgentME 4y ago
  I'm not sure these issues are in conflict. Corporations cause lots of harm despite being held back by their dependence on people who control them, who can occasionally put the brakes on some of their worst excesses, and by their inability to recursively self-improve their intelligence. Removing these two handicaps on corporations would allow them to do significantly more damage, so preventing this from happening is important.
  - AnIdiotOnTheNet 4y ago
    > inability recursively self-improve their intelligence It is not for lack of trying, corporations modify themselves all the time. That they are quite often unsuccessful at making meaningful improvement should be taken as a prior against the idea that an AI will inherently be better at it.

PartiallyTyped 4y ago

> It's about the risk that an AI smarter than us It's also about the risk that AI is dumber than what we expect it to be and fails to properly solve the issue. A sufficiently smart AI should be able to understand our intentions and - assuming it is aligned with us - attempt to fulfill that purpose. Dumb "AI" will fail in ways that don't actually solve the task, or solve it only on the surface / do reward hacking.

AgentME 4y ago
>assuming it is aligned with us If we don't figure out alignment, we can't make this assumption. This is the part that AI safety people are concerned with making reality.
- PartiallyTyped 4y ago
  I think I was misunderstood. My point is that not only do we need to consider the cases where the AI is smarter than us, but also the cases where it is dumber than we expect it. An aligned agent will not fall into stupid errors, thus solving the lower bound of performance is a necessity for complete alignment.
  - AgentME 4y ago
    Ohh, right on. Definitely agree now that I've re-read it right. I was a little confused by the first sentence of your other post and thought you were putting forward a disagreement in the second sentence.
- pharmakom 4y ago
  I don't see how alignment is a different problem from making an AI do what we want.
  - PartiallyTyped 4y ago
    It's more about the unintended consequences, the implied/contextual information and getting it to do what we want in a way actually fulfills our goals instead of merely satisfying a condition. A task such as "bring me a cup of tea" has lots of implied information such as not making it too hot to burn the person, not breaking anything in the process, not harming other people in the process and so on.
  - AgentME 4y ago
    I think that's a fair way of putting alignment. The concern is that we might make an AI that's capable of outsmarting us before we're good at making it care about doing what we want instead of some other goal that disregards what we really care about like our safety.

AnIdiotOnTheNet 4y ago

> It's about the risk that an AI smarter than us will be made and built to pursue some goal without caring about our well-being. We already have those and they're called corporations. They've done significant real damage to the world already and they are still working hard to do more damage. It makes little sense to me to focus on this potential future problem when we haven't even agreed to deal with the ones that we already have.

AgentME 4y ago
I'm not sure these issues are in conflict. Corporations cause lots of harm despite being held back by their dependence on people who control them, who can occasionally put the brakes on some of their worst excesses, and by their inability to recursively self-improve their intelligence. Removing these two handicaps on corporations would allow them to do significantly more damage, so preventing this from happening is important.
- AnIdiotOnTheNet 4y ago
  > inability recursively self-improve their intelligence It is not for lack of trying, corporations modify themselves all the time. That they are quite often unsuccessful at making meaningful improvement should be taken as a prior against the idea that an AI will inherently be better at it.