6 ms·

You're putting a weirdly large amount of trust into, functionally, some dude who posted on lesswrong. Sure he has a PhD and is smart, but so is basically everyo

You're putting a weirdly large amount of trust into, functionally, some dude who posted on lesswrong. Sure he has a PhD and is smart, but so is basically everyone else in the field, not just in alignment, and the median person in the field thinks the risk of "doom" is 2-5% (and that's conditioned on the supposed existence of a high level machine intelligence that the median expert believes might exist in 40 years). That still might be higher than you'd like, but it's not actually a huge worry in the grand scheme of things. Like, if I told you that in 40 years, there was a 50% chance of something existing that had a 2% chance of causing extreme harm to the human population, I'm actually not sure that thing should be the biggest priority. Other issues may have more than a 1% chance of leading to terrible outcomes sooner.

comp_throw7 3y ago
The median person in field thinks 5-10%, not 2-5%, and median timelines are shorter than 40 years. But this is all a distraction, since unaligned ASI is the default case absent significant efforts (that we aren't currently making), and trying to evaluate risk by averaging out the views of people who haven't actually explored the arguments very carefully (vs. just evaluating the arguments yourself) is a doomed effort.
- joshuamorton 3y ago
  > The median person in field thinks 5-10%, not 2-5% The median person in the study here, under a particular definition was 5-10%, other comparable studies have found 2%, and similar questions using arguably better definitions in the same study found lower percentages. > median timelines are shorter than 40 years. The median person suggested a 50% chance in 39 years. > since unaligned ASI is the default case I challenge this assertion. Many relatively smart scholars who are more involved in the alignment space than, presumably either you or I, have put forth cogent arguments that alignment-by-default is perfectly reasonable. Dismissing those out of hand seems naive.
  - deleted 3y ago
    [deleted]
  - comp_throw7 3y ago
    I work in the space (doing tech stuff that isn't direct research). The best argument I've seen for alignment by default is something like "morality comes from training data, and therefore the fact that LLM training data sets contain human moral intuitions will mean that an ASI stemming from such a training regime will share enough human values" (Quintin Pope believes something like this, as far as I can tell), which is deeply unconvincing, since it contradicts the evidence we _do_ have from human & animal value formation. Happy to entertain other arguments that alignment-by-default is reasonable; most arguments I've seen are much worse than that one. I haven't seen many people make an active case for alignment-by-default, so much as leave open a whole bunch of swath of uncertainty for unknown unknowns.

comp_throw7 3y ago

The median person in field thinks 5-10%, not 2-5%, and median timelines are shorter than 40 years. But this is all a distraction, since unaligned ASI is the default case absent significant efforts (that we aren't currently making), and trying to evaluate risk by averaging out the views of people who haven't actually explored the arguments very carefully (vs. just evaluating the arguments yourself) is a doomed effort.

joshuamorton 3y ago
> The median person in field thinks 5-10%, not 2-5% The median person in the study here, under a particular definition was 5-10%, other comparable studies have found 2%, and similar questions using arguably better definitions in the same study found lower percentages. > median timelines are shorter than 40 years. The median person suggested a 50% chance in 39 years. > since unaligned ASI is the default case I challenge this assertion. Many relatively smart scholars who are more involved in the alignment space than, presumably either you or I, have put forth cogent arguments that alignment-by-default is perfectly reasonable. Dismissing those out of hand seems naive.
- deleted 3y ago
  [deleted]
- comp_throw7 3y ago
  I work in the space (doing tech stuff that isn't direct research). The best argument I've seen for alignment by default is something like "morality comes from training data, and therefore the fact that LLM training data sets contain human moral intuitions will mean that an ASI stemming from such a training regime will share enough human values" (Quintin Pope believes something like this, as far as I can tell), which is deeply unconvincing, since it contradicts the evidence we _do_ have from human & animal value formation. Happy to entertain other arguments that alignment-by-default is reasonable; most arguments I've seen are much worse than that one. I haven't seen many people make an active case for alignment-by-default, so much as leave open a whole bunch of swath of uncertainty for unknown unknowns.