3 ms·

The CEO of that company that sold rides on an unsafe submersible to view the wreck of the Titanic (namely Stockton Rush, CEO of OceanGate, which killed 5 people

The CEO of that company that sold rides on an unsafe submersible to view the wreck of the Titanic (namely Stockton Rush, CEO of OceanGate, which killed 5 people when the submersible imploded) responded to concerns about the safety of his operation by claiming that the critics were motivated by a desire to protect the established players in the underwater-tourism industry from competition. The point is that some companies are actually reckless (and also that some users of powerful technology are reckless).

Terr_ 2y ago
> claiming that the critics were motivated by a desire to protect the established players in the underwater-tourism industry from competition. At this point I suspect a great amount of reasonable engineering criticism has come from people who can't even name any of those "established players in the underwater tourism industry", let alone have a favorable bias towards them.
jahewson 2y ago
But he was deluded and believed that his sub was safe. Not sure what your point is.
- hollerith 2y ago
  My point is that spending $50 billion training an AI is a reckless thing to do, so OpenAI and its competitors need to be stopped. It's reckless because the AI might turn out to be better at planning and better at reality than our most capable institutions (e.g., the FBI and the military) so there would be no way to stop it from doing whatever it wants, and there is no plan that anyone has published or proposed that might prevent the AI from wanting something incompatible with continued human survival. And most things are incompatible with continued human survival if enough optimization pressure is applied to bringing the thing about, so it is unlikely we will get lucky and the AI ends up wanting something compatible with continued human survival. If someone somewhere comes up with a viable plan before the end, then the labs could probably be persuaded to follow the viable plan (so we would be saved) because the leaders of the labs understand at some level that what they are doing is very dangerous, and they don't want to be killed, but it is unlikely that anyone anywhere is going to come up with a viable plan in time, so the labs are going to stick with the clearly inadequate plans they have now. The reason I think it is unlikely that anyone anywhere is going to come up with a viable plan is that MIRI (then called the Singularity Institute for Artificial Intelligence) starting the search for a viable plan in about 2002 and the 2 people at MIRI (Yudkowsky and Soares) with the most experience in this search both say it is unlikely that anyone is going to come up with a plan before the end unless there is a multi-decade moratorium on AI research and on very large training runs.

Terr_ 2y ago

> claiming that the critics were motivated by a desire to protect the established players in the underwater-tourism industry from competition. At this point I suspect a great amount of reasonable engineering criticism has come from people who can't even name any of those "established players in the underwater tourism industry", let alone have a favorable bias towards them.

jahewson 2y ago

But he was deluded and believed that his sub was safe. Not sure what your point is.

hollerith 2y ago
My point is that spending $50 billion training an AI is a reckless thing to do, so OpenAI and its competitors need to be stopped. It's reckless because the AI might turn out to be better at planning and better at reality than our most capable institutions (e.g., the FBI and the military) so there would be no way to stop it from doing whatever it wants, and there is no plan that anyone has published or proposed that might prevent the AI from wanting something incompatible with continued human survival. And most things are incompatible with continued human survival if enough optimization pressure is applied to bringing the thing about, so it is unlikely we will get lucky and the AI ends up wanting something compatible with continued human survival. If someone somewhere comes up with a viable plan before the end, then the labs could probably be persuaded to follow the viable plan (so we would be saved) because the leaders of the labs understand at some level that what they are doing is very dangerous, and they don't want to be killed, but it is unlikely that anyone anywhere is going to come up with a viable plan in time, so the labs are going to stick with the clearly inadequate plans they have now. The reason I think it is unlikely that anyone anywhere is going to come up with a viable plan is that MIRI (then called the Singularity Institute for Artificial Intelligence) starting the search for a viable plan in about 2002 and the 2 people at MIRI (Yudkowsky and Soares) with the most experience in this search both say it is unlikely that anyone is going to come up with a plan before the end unless there is a multi-decade moratorium on AI research and on very large training runs.