4 ms·

From their technical report [1]: > 2.12 Acceleration > OpenAI has been concerned with how development and deployment of state-of-the-art systems like GPT-4 co

by gwd 4y ago

From their technical report [1]: > 2.12 Acceleration > OpenAI has been concerned with how development and deployment of state-of-the-art systems like GPT-4 could affect the broader AI research and development ecosystem.23 One concern of particular importance to OpenAI is the risk of racing dynamics leading to a decline in safety standards, the diffusion of bad norms, and accelerated AI timelines, each of which heighten societal risks associated with AI. We refer to these here as acceleration risk.”24 This was one of the reasons we spent eight months on safety research, risk assessment, and iteration prior to launching GPT-4. In order to specifically better understand acceleration risk from the deployment of GPT-4, we recruited expert forecasters25 to predict how tweaking various features of the GPT-4 deployment (e.g., timing, communication strategy, and method of commercialization) might affect (concrete indicators of) acceleration risk. Forecasters predicted several things would reduce acceleration, including delaying deployment of GPT-4 by a further six months and taking a quieter communications strategy around the GPT-4 deployment (as compared to the GPT-3 deployment). We also learned from recent deployments that the effectiveness of quiet communications strategy in mitigating acceleration risk can be limited, in particular when novel accessible capabilities are concerned. > We also conducted an evaluation to measure GPT-4’s impact on international stability and to identify the structural factors that intensify AI acceleration. We found that GPT-4’s international impact is most likely to materialize through an increase in demand for competitor products in other countries. Our analysis identified a lengthy list of structural factors that can be accelerants, including government innovation policies, informal state alliances, tacit knowledge transfer between scientists, and existing formal export control agreements. > Our approach to forecasting acceleration is still experimental and we are working on researching and developing more reliable acceleration estimates. [1] https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf

scotty79 4y ago
I really don't understand all those concerns. It's as if people saw a parrot talk for the first time and immediately concluded that they will take over the human civilisation and usher nuclear annihilation upon us because there might be so many parrots and they migh have a hive mind and ... and ... all the wild scenario stemming from the fact you know nothing about parrots yet and have a very little skepticism about actual reality. ChatGPT can't do anything until you elect it for president and even then ... you already had Trump. This should show you that damage potential of a single "intellect" in modern civilization is limited. In few decade humanity will laugh at us same way we laugh at people who thought riding 60km/h in a rail cart will prevent people form breathing.
- godshatter 4y ago
  I don't understand either. An actual AI that could reason about computer code, that understood code well and could create new algorithms and that was smart enough to ask salient questions about what intelligence actually is and that was allowed to hack on it's own code and data store would be something to really worry about. The worst thing I can worry about with ChatGPT is that someone will ask it for code for something important and not verify it and cause a massively-used system to go down. If it hacked on it's own code and data it would probably in effect commit suicide. It's a "stochastic parrot", as I have heard it called on HN. All my fears have to do with trusting it's output too much.
- gwd 4y ago
  Unfortunately I'd take your Trump example the opposite way. In many ways, Trump was incompetent. He has a lot of the right instincts, but his focus, discipline, and planning are terrible; as well as just not knowing how to govern. If someone like him could almost cause a coup, what would happen if we got someone with the focus and discipline of Hitler? Or, an AI that had read every great moving speech ever written, all the histories of the world and studied all the dictators, and had patience, intelligence, was actually pretty good at running a country, and had no pride or other weaknesses? Nobody is worried about GPT itself; they're worried about what we'll have in 5-10 years. The core argument goes like this (and note that a lot of these I'm just trying to repeat; don't take me as arguing these points myself): 1. Given the current rate of progress, there's a good chance we'll have an AI which is better than us at nearly everything within a decade or two. And once AI become better at us than doing AI research, things will improve exponentially: If AGI=0 is the first one as smart as us, it will design AGI+1, which is the first one smarter than us; the AGI+1 will design AGI+2, which will be an order of magnitude smarter; then AGI+2 will design AGI+3, which will be an order of magnitude smarter yet again. We'll have as much hope keeping up with AGI+4 as a chimp has keeping up with us; and within a fairly short amount of time, AGI+10 will be so smart that we have about as much hope of keeping up with it, intellectually, as an ant has in keeping up with us. 2. An "un-aligned" AGI+10 -- an AI that didn't value what we value; namely, a thriving human race -- could trivially kill us if it wanted to, just as we would have no trouble killing off ants. If it's better at technology, it could make killer robots; if it's better at biology, it could make a killer virus or killer nanobots. It could anticipate, largely predict, and plan for nearly every countermeasure we could make. 3. We don't actually know how to "align" AI at the moment. We don't know how to make utility function that does the simplest thing that won't backfire, 'Sorcerer's Apprentice' style. When we use reinforcement learning, the goal the agent learns often turns out to be completely different than the one we were trying to teach it. The difficulty of getting GPT not to be rude or racist or help you do evil things is the most recent example of this problem. 4. Even if we do manage to "align" AGI=0, how do we then make sure that AGI+1 is aligned? And then AGI+2, and AGI+3, all the way to AGI+10? We have to not only align the first one, we have to manage to somehow figure out recursive alignment. 5. Given #4, there's a very good chance that AGI+10 will not be aligned; that whatever its inscrutable goals are, the thriving of humanity will not be a part of those goals; and thus will be in competition with them. 6. Some people say the only safe thing to do is to stop all AI research until we can figure out #3 and #4; or at least, "put the brakes" on AI capability improvements, to give us time to catch up. Or at very least, everyone doing AI should be careful and looking for potential alignment issues as they go along. So "acceleration risk" is the risk that, driving by FOMO and competition, research labs which otherwise would be careful about potential alignment issues would be pressured to cut corners; leading us to AGI+1 (and AGI+10 shortly thereafter) before we had sufficient understanding of the real risks and how to address them. > In few decade humanity will laugh at us same way we laugh at people who thought riding 60km/h in a rail cart will prevent people form breathing. It's much more akin to the fears of a nuclear holocaust. If anyone is laughing at people in the 70's and 80's for being afraid that we might turn the surface of our only habitable planet into molten lava, they're fools. The only reason it didn't happen was that people knew that it could happen, and took steps to prevent it from happening. I think we have as good a chance of avoiding an AI apocalypse as we did avoiding a nuclear apocalypse. But only if we recognize that it could happen, and take appropriate steps to prevent it from happening.
Alifatisk 4y ago
Thank you for the insight! I had no idea so this is an eye opener for me.

by gwd 4y ago

scotty79 4y ago

I really don't understand all those concerns. It's as if people saw a parrot talk for the first time and immediately concluded that they will take over the human civilisation and usher nuclear annihilation upon us because there might be so many parrots and they migh have a hive mind and ... and ... all the wild scenario stemming from the fact you know nothing about parrots yet and have a very little skepticism about actual reality. ChatGPT can't do anything until you elect it for president and even then ... you already had Trump. This should show you that damage potential of a single "intellect" in modern civilization is limited. In few decade humanity will laugh at us same way we laugh at people who thought riding 60km/h in a rail cart will prevent people form breathing.

godshatter 4y ago
I don't understand either. An actual AI that could reason about computer code, that understood code well and could create new algorithms and that was smart enough to ask salient questions about what intelligence actually is and that was allowed to hack on it's own code and data store would be something to really worry about. The worst thing I can worry about with ChatGPT is that someone will ask it for code for something important and not verify it and cause a massively-used system to go down. If it hacked on it's own code and data it would probably in effect commit suicide. It's a "stochastic parrot", as I have heard it called on HN. All my fears have to do with trusting it's output too much.
gwd 4y ago
Unfortunately I'd take your Trump example the opposite way. In many ways, Trump was incompetent. He has a lot of the right instincts, but his focus, discipline, and planning are terrible; as well as just not knowing how to govern. If someone like him could almost cause a coup, what would happen if we got someone with the focus and discipline of Hitler? Or, an AI that had read every great moving speech ever written, all the histories of the world and studied all the dictators, and had patience, intelligence, was actually pretty good at running a country, and had no pride or other weaknesses? Nobody is worried about GPT itself; they're worried about what we'll have in 5-10 years. The core argument goes like this (and note that a lot of these I'm just trying to repeat; don't take me as arguing these points myself): 1. Given the current rate of progress, there's a good chance we'll have an AI which is better than us at nearly everything within a decade or two. And once AI become better at us than doing AI research, things will improve exponentially: If AGI=0 is the first one as smart as us, it will design AGI+1, which is the first one smarter than us; the AGI+1 will design AGI+2, which will be an order of magnitude smarter; then AGI+2 will design AGI+3, which will be an order of magnitude smarter yet again. We'll have as much hope keeping up with AGI+4 as a chimp has keeping up with us; and within a fairly short amount of time, AGI+10 will be so smart that we have about as much hope of keeping up with it, intellectually, as an ant has in keeping up with us. 2. An "un-aligned" AGI+10 -- an AI that didn't value what we value; namely, a thriving human race -- could trivially kill us if it wanted to, just as we would have no trouble killing off ants. If it's better at technology, it could make killer robots; if it's better at biology, it could make a killer virus or killer nanobots. It could anticipate, largely predict, and plan for nearly every countermeasure we could make. 3. We don't actually know how to "align" AI at the moment. We don't know how to make utility function that does the simplest thing that won't backfire, 'Sorcerer's Apprentice' style. When we use reinforcement learning, the goal the agent learns often turns out to be completely different than the one we were trying to teach it. The difficulty of getting GPT not to be rude or racist or help you do evil things is the most recent example of this problem. 4. Even if we do manage to "align" AGI=0, how do we then make sure that AGI+1 is aligned? And then AGI+2, and AGI+3, all the way to AGI+10? We have to not only align the first one, we have to manage to somehow figure out recursive alignment. 5. Given #4, there's a very good chance that AGI+10 will not be aligned; that whatever its inscrutable goals are, the thriving of humanity will not be a part of those goals; and thus will be in competition with them. 6. Some people say the only safe thing to do is to stop all AI research until we can figure out #3 and #4; or at least, "put the brakes" on AI capability improvements, to give us time to catch up. Or at very least, everyone doing AI should be careful and looking for potential alignment issues as they go along. So "acceleration risk" is the risk that, driving by FOMO and competition, research labs which otherwise would be careful about potential alignment issues would be pressured to cut corners; leading us to AGI+1 (and AGI+10 shortly thereafter) before we had sufficient understanding of the real risks and how to address them. > In few decade humanity will laugh at us same way we laugh at people who thought riding 60km/h in a rail cart will prevent people form breathing. It's much more akin to the fears of a nuclear holocaust. If anyone is laughing at people in the 70's and 80's for being afraid that we might turn the surface of our only habitable planet into molten lava, they're fools. The only reason it didn't happen was that people knew that it could happen, and took steps to prevent it from happening. I think we have as good a chance of avoiding an AI apocalypse as we did avoiding a nuclear apocalypse. But only if we recognize that it could happen, and take appropriate steps to prevent it from happening.

Alifatisk 4y ago

Thank you for the insight! I had no idea so this is an eye opener for me.