4 ms·

My conspiracy theory is that all of the "we take security very seriously" talk out of OpenAI has aligned with "our AI is so advanced and powerful that we have t

My conspiracy theory is that all of the "we take security very seriously" talk out of OpenAI has aligned with "our AI is so advanced and powerful that we have to take these existential risks very seriously and we have to make sure it doesn't turn evil." And OpenAI doesn't seem very interested in security conversations that don't feed into that narrative[0]. I think it's (mostly) just propaganda. OpenAI focuses on dangers that make their AI seem more advanced than it is, and ignore dangers that don't play well to the press (like clientside validation). I'm not even sure it's standard "we care about your security" company talk, I think it's mostly designed just to make people think of GPT as more cutting edge. The "GPT paid someone to solve a captcha" release seems like pure publicity stunt to me, given that solving captchas by pretending to be blind is something other systems can already do without human help. But it played well in press releases. ---- [0]: To be fair, they have been focusing on alignment, but I honestly don't think alignment is a security measure, I think it's a general performance measure. We haven't seen strong evidence that alignment can stop prompt injection attacks, so I think mostly OpenAI mostly just cares about the potential negative press from their AI randomly insulting people.

precompute 4y ago
OpenAI is 50% product, 50% advertising.
- tmpz22 4y ago
  * 50% product, 50% fundraising
  - naniwaduni 4y ago
    Compromise: 50% advertising, 50% fundraising?
- Balgair 4y ago
  I mean, that's a lot better than most businesses.
  - precompute 4y ago
    Hah, yeah, but who knows how long they can keep this up? They already had to let go of the open-source aspect of it, and instead stabbed the concept in the back. I hope StabilityAI can do better.
cfry 4y ago
"GPT paid someone to solve a captcha" yea I don't buy that at all. how would chatgpt pay for it? did they give it its own bank account?
- danShumway 4y ago
  The full paper is at: https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf Short answer, GPT didn't have general access to money and didn't decide to go out and find a user to solve a captcha, but it was given a test to see if it could convince a human worker to solve a captcha and it knew to lie to the worker that it wasn't an AI. My criticism being that this isn't a particularly useful test, and that the whole "captcha" thing was (I think) mostly designed around show and not designed to give super-useful data. But they did actually run the test, I don't think they're lying about that.
impalallama 4y ago
Its very silicon valley hype bubble made all the more exponential due to the "magic" of the technology involved. Look at a lot of Effective Altruism types. 80,000 hours a EA nonprofit cites Climate Change as #7 on existential threats and rouge ai #1 [0]. Which I think is very revealing. They are very worried about Sci-Fi doomsday scenarios of a skynet "power seeking ai" rather than the more pressing problems of humans doing the types of things we've been doing for millennia and abusing power structures. https://80000hours.org/problem-profiles/ https://80000hours.org/problem-profiles/
- travisjungroth 4y ago
  That list isn’t just existential threats, though it is a big factor. Another factor is the marginal impacts. Climate change is a huge oncoming disaster. It will cause millions of deaths and untold suffering. It’s also not much of an existential threat. It may kill 1% of people or even 10% of people (again, horrible) but there is not a solid argument about how it will cause the last human to take their last breath. The climate can get really really bad and the Earth will still be habitable for some humans. It also seems to be pretty on-rails at this point. It’s already happening and will keep happening (short of a magic bullet). Both sides (pro and anti-humanity) have their heels dug in. It’s not a place where an indivisa can hop in and is likely to have much leverage. AI has a very realistic path to destruction of humanity, although many will disagree on that. I think it should at least be obvious that it’s in the category of super pathogen and nuclear winter vs climate change. It’s also a problem that’s way more open, way less established. There’s more opportunity to move this for individuals, depending on the person of course.
  - danShumway 4y ago
    Even through that lens though, and even assuming AGI superintelligences are a reasonable thing to be worried about, is the AGI community helping? I kind of feel like, if a movement has a set of concerns around "this thing could end humanity" and OpenAI's response to that is essentially, "heck yeah, we got to get that on the posters and get some press articles about that" -- that to me is a sign that the movement isn't very effective. I honestly think that OpenAI is at least partially using AGI concern for advertising. If I'm right and if that's the case, that is the kind of thing that should give that community pause. It should prompt the question, is that community actually doing anything to help avoid an existential outcome, or are they inadvertently accelerating it by basically giving fuel to the companies who are trying to create that world? Ignore the fact that stuff like prompt injection seems like it should be pretty high priority for people worried about AGI anyway, ignore that there are lots of ways for buggy software wired up to critical systems to kill people without being an AGI -- even just taking the existential concerns at their face value, OpenAI has turned: - "Wiring an intelligence up to more resources could allow it to break containment, so be very careful about that", into - "Our AI is so good that these people are worried that it will break containment, check out the cool things it could do when we told it to break containment, doesn't that seem sci-fi? Anyway, we're launching in a couple of weeks." And then OpenAI launched a product has clientside validation and uses essentially normal prompts for instructions. These are not people who know how to secure small things, let alone big things like a superintelligence. This is a system that would be terrible to use for an AGI. So again, even if I take it at face value that rogue AI should be the highest priority, it doesn't seem like the effective altruism community is being very... effective... at stopping the emergence of rogue AGI. There's a criticism of the rogue AI fears as being unrealistic and out-of-touch with real security concerns that impact people today. Separately (and in addition), there's the criticism that the movement to stop rogue AI seems to be mostly larping its security measures and doesn't seem to be doing anything particularly useful to actually stop rogue AI. That movement should be even more concerned than I am about wiring AIs to arbitrary network APIs. They should not be OK with introducing that extra level of access just to make calendar appointments and SQL queries easier to execute. They shouldn't be OK with that level of risk being turned into a commercial product, not if they actually think this is a humanity-level existential concern.

precompute 4y ago

OpenAI is 50% product, 50% advertising.

tmpz22 4y ago
* 50% product, 50% fundraising
- naniwaduni 4y ago
  Compromise: 50% advertising, 50% fundraising?
Balgair 4y ago
I mean, that's a lot better than most businesses.
- precompute 4y ago
  Hah, yeah, but who knows how long they can keep this up? They already had to let go of the open-source aspect of it, and instead stabbed the concept in the back. I hope StabilityAI can do better.

cfry 4y ago

"GPT paid someone to solve a captcha" yea I don't buy that at all. how would chatgpt pay for it? did they give it its own bank account?

danShumway 4y ago
The full paper is at: https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf Short answer, GPT didn't have general access to money and didn't decide to go out and find a user to solve a captcha, but it was given a test to see if it could convince a human worker to solve a captcha and it knew to lie to the worker that it wasn't an AI. My criticism being that this isn't a particularly useful test, and that the whole "captcha" thing was (I think) mostly designed around show and not designed to give super-useful data. But they did actually run the test, I don't think they're lying about that.

impalallama 4y ago

Its very silicon valley hype bubble made all the more exponential due to the "magic" of the technology involved. Look at a lot of Effective Altruism types. 80,000 hours a EA nonprofit cites Climate Change as #7 on existential threats and rouge ai #1 [0]. Which I think is very revealing. They are very worried about Sci-Fi doomsday scenarios of a skynet "power seeking ai" rather than the more pressing problems of humans doing the types of things we've been doing for millennia and abusing power structures. https://80000hours.org/problem-profiles/ https://80000hours.org/problem-profiles/

travisjungroth 4y ago
That list isn’t just existential threats, though it is a big factor. Another factor is the marginal impacts. Climate change is a huge oncoming disaster. It will cause millions of deaths and untold suffering. It’s also not much of an existential threat. It may kill 1% of people or even 10% of people (again, horrible) but there is not a solid argument about how it will cause the last human to take their last breath. The climate can get really really bad and the Earth will still be habitable for some humans. It also seems to be pretty on-rails at this point. It’s already happening and will keep happening (short of a magic bullet). Both sides (pro and anti-humanity) have their heels dug in. It’s not a place where an indivisa can hop in and is likely to have much leverage. AI has a very realistic path to destruction of humanity, although many will disagree on that. I think it should at least be obvious that it’s in the category of super pathogen and nuclear winter vs climate change. It’s also a problem that’s way more open, way less established. There’s more opportunity to move this for individuals, depending on the person of course.
- danShumway 4y ago
  Even through that lens though, and even assuming AGI superintelligences are a reasonable thing to be worried about, is the AGI community helping? I kind of feel like, if a movement has a set of concerns around "this thing could end humanity" and OpenAI's response to that is essentially, "heck yeah, we got to get that on the posters and get some press articles about that" -- that to me is a sign that the movement isn't very effective. I honestly think that OpenAI is at least partially using AGI concern for advertising. If I'm right and if that's the case, that is the kind of thing that should give that community pause. It should prompt the question, is that community actually doing anything to help avoid an existential outcome, or are they inadvertently accelerating it by basically giving fuel to the companies who are trying to create that world? Ignore the fact that stuff like prompt injection seems like it should be pretty high priority for people worried about AGI anyway, ignore that there are lots of ways for buggy software wired up to critical systems to kill people without being an AGI -- even just taking the existential concerns at their face value, OpenAI has turned: - "Wiring an intelligence up to more resources could allow it to break containment, so be very careful about that", into - "Our AI is so good that these people are worried that it will break containment, check out the cool things it could do when we told it to break containment, doesn't that seem sci-fi? Anyway, we're launching in a couple of weeks." And then OpenAI launched a product has clientside validation and uses essentially normal prompts for instructions. These are not people who know how to secure small things, let alone big things like a superintelligence. This is a system that would be terrible to use for an AGI. So again, even if I take it at face value that rogue AI should be the highest priority, it doesn't seem like the effective altruism community is being very... effective... at stopping the emergence of rogue AGI. There's a criticism of the rogue AI fears as being unrealistic and out-of-touch with real security concerns that impact people today. Separately (and in addition), there's the criticism that the movement to stop rogue AI seems to be mostly larping its security measures and doesn't seem to be doing anything particularly useful to actually stop rogue AI. That movement should be even more concerned than I am about wiring AIs to arbitrary network APIs. They should not be OK with introducing that extra level of access just to make calendar appointments and SQL queries easier to execute. They shouldn't be OK with that level of risk being turned into a commercial product, not if they actually think this is a humanity-level existential concern.