4 ms·

So there’s a middling sci-fi novel from maybe a decade ago trying to tell the AI Takeoff story called Avogadro Corp. It’s not exactly Heinlein but it’s a fun li

So there’s a middling sci-fi novel from maybe a decade ago trying to tell the AI Takeoff story called Avogadro Corp. It’s not exactly Heinlein but it’s a fun little yarn. The basic premise is that Google pardon me Avogadro Corp is doing NLP email copy-editing, but it takes too many compute resources so in desperation the TL hand wave sets it loose to secure its own budget, and being good at writing convincing emails, well, one thing leads to another. Now this is very hand-wavy and involves a lot of magical thinking and suspension of disbelief. But that’s ok, it’s a pulpy sci-fi novel for fun. Now this fictional scenario is dramatically more specific, more detailed, and frankly more believable than anything anyone has linked on this thread. And I have no issue with folks wanting to LARP this sort of thing in a capital-R Rationalist forum somewhere: it’s a premise that has fascinated sci-fi fans since Asimov wrote his 3 Laws and probably before that. But when it starts hitting the mainstream press, K Street, freaking out everyday people, and being used to turbocharge fear-driven land-grabs around intellectual property precedent? Yeah, it’s time to do better than “But what if I Have No Mouth and I Must Scream Was Real man?!?!”

lmm 3y ago
Scorn is not an argument. I really don't see which part you think would actually be hard for this hypothetical AI to do, if there even is such a part. The whole premise is that it has better-than-human intelligence, so it can do anything that e.g. a physically disabled human employee could do, and such people are already able to improve AIs. Maybe put more effort into defining and explaining where you think that barrier is rather than expounding in detail how you think anyone who disagrees with you is the kind of person you enjoy bullying.
- benreesman 3y ago
  I apologize for coming off scornful, the last thing I read before replying was “WTF kind of logic is this” and I kinda said, ok, that’s the tone on this sub-thread. I’ll watch my snark but I’ll also admonish you to only take the vibe places you want to go. My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence. The burden is not on skeptics of paperclip optimizers in the near term to prove it can’t happen. That which is not prohibited by the laws of physics is admitted by the laws of physics. Of course it’s possible in the abstract. The burden on anyone advocating positions being taken seriously by the public, by government, by industry, etc. to show that these arguments are in the public welfare, and I contend that they are not.
  - lmm 3y ago
    > My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence. Obviously if it can't improve its code then it can't improve its code. But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve. And it's not at all obvious (at least to me) what that barrier is. We can already throw ChatGPT or equivalent at a codebase, ask it to suggest improvements to the code, and it does; the hit rate isn't great, but it's already good enough that people are incorporating it into their workflows. As far as I can see, that's enough. Yes, people who are worried about this don't spend a lot of time going through the details, just as people who are worried about the military staging a coup don't spend a lot of time going into how they will get weapons or coordinate their movements or what have you. Those things are pretty complex, but they're ultimately routine; we are right to gloss over them, and if someone thinks that those things make a coup impossible, the burden would be on them to explain why (maybe there's some specific protocol that means they can't get access to weapons without authorisation from the legitimate authorities - but you'd need to explain what that protocol actually is and justify why you expect it to be wartertight). Your argument seemed to be that babysitting the kind of complex code that we use for AI is not merely complex, but so complex that it requires some unique human ability that we would not expect a ChatGPT-like AI to possess (implicitly, that it requires something a lot more difficult than making improvements to code). And I just don't see that. Put it this way: we would never expect a human worker at these companies who was smart enough to make improvements to the core AI code to get stuck when it comes to actually running it. So why would it be an issue for the AI? I do think the burden is on you here given that the whole premise is that the AI is smart enough to improve its own code (you're welcome to argue that it's impossible for an AI to ever get smart enough to improve its own code - but in that case all this talk about operational complexity is beside the point).
    - benreesman 3y ago
      I’m looking for an argument in any technical detail about how we go from beam-searching: P(next_bpe_token|previous_bpe_tokens,rlhf_orderings,internet) to Cyberdyne Systems Model T-100. That’s a bit of a cheeky oversimplification of a modern decoder, but the LLaMA2 code is there for anyone to read, it’s kinda that. It’s not GPT-4, there’s clearly some scale or some special sauce or both on the way from here to there, but it’s damned close. LLMs aren’t even a particularly great example of “adversarial” AI: AlphaZero is a way better example of AI kicking the shit out of humans in a zero-sum scenario. If I wanted to scare people I’d be talking about the DeepMind stuff, not the OpenAI stuff. But the real crux of my argument is that bad-faith arguments have a tendency to escape the control of those making them, and if a short-term advantage in the copyright treatment of model weights resulting from the commons is achieved by Yud posting about GPT and nuclear weapons in Time Magazine and scaring the living hell out of the lay public about a capricious digital deity in the process, that’s unlikely to be a genie that goes back in the bottle. So I’d really like it if someone injected some: here’s how this could actually happen into the dialog. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

lmm 3y ago

Scorn is not an argument. I really don't see which part you think would actually be hard for this hypothetical AI to do, if there even is such a part. The whole premise is that it has better-than-human intelligence, so it can do anything that e.g. a physically disabled human employee could do, and such people are already able to improve AIs. Maybe put more effort into defining and explaining where you think that barrier is rather than expounding in detail how you think anyone who disagrees with you is the kind of person you enjoy bullying.

benreesman 3y ago
I apologize for coming off scornful, the last thing I read before replying was “WTF kind of logic is this” and I kinda said, ok, that’s the tone on this sub-thread. I’ll watch my snark but I’ll also admonish you to only take the vibe places you want to go. My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence. The burden is not on skeptics of paperclip optimizers in the near term to prove it can’t happen. That which is not prohibited by the laws of physics is admitted by the laws of physics. Of course it’s possible in the abstract. The burden on anyone advocating positions being taken seriously by the public, by government, by industry, etc. to show that these arguments are in the public welfare, and I contend that they are not.
- lmm 3y ago
  > My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence. Obviously if it can't improve its code then it can't improve its code. But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve. And it's not at all obvious (at least to me) what that barrier is. We can already throw ChatGPT or equivalent at a codebase, ask it to suggest improvements to the code, and it does; the hit rate isn't great, but it's already good enough that people are incorporating it into their workflows. As far as I can see, that's enough. Yes, people who are worried about this don't spend a lot of time going through the details, just as people who are worried about the military staging a coup don't spend a lot of time going into how they will get weapons or coordinate their movements or what have you. Those things are pretty complex, but they're ultimately routine; we are right to gloss over them, and if someone thinks that those things make a coup impossible, the burden would be on them to explain why (maybe there's some specific protocol that means they can't get access to weapons without authorisation from the legitimate authorities - but you'd need to explain what that protocol actually is and justify why you expect it to be wartertight). Your argument seemed to be that babysitting the kind of complex code that we use for AI is not merely complex, but so complex that it requires some unique human ability that we would not expect a ChatGPT-like AI to possess (implicitly, that it requires something a lot more difficult than making improvements to code). And I just don't see that. Put it this way: we would never expect a human worker at these companies who was smart enough to make improvements to the core AI code to get stuck when it comes to actually running it. So why would it be an issue for the AI? I do think the burden is on you here given that the whole premise is that the AI is smart enough to improve its own code (you're welcome to argue that it's impossible for an AI to ever get smart enough to improve its own code - but in that case all this talk about operational complexity is beside the point).
  - benreesman 3y ago
    I’m looking for an argument in any technical detail about how we go from beam-searching: P(next_bpe_token|previous_bpe_tokens,rlhf_orderings,internet) to Cyberdyne Systems Model T-100. That’s a bit of a cheeky oversimplification of a modern decoder, but the LLaMA2 code is there for anyone to read, it’s kinda that. It’s not GPT-4, there’s clearly some scale or some special sauce or both on the way from here to there, but it’s damned close. LLMs aren’t even a particularly great example of “adversarial” AI: AlphaZero is a way better example of AI kicking the shit out of humans in a zero-sum scenario. If I wanted to scare people I’d be talking about the DeepMind stuff, not the OpenAI stuff. But the real crux of my argument is that bad-faith arguments have a tendency to escape the control of those making them, and if a short-term advantage in the copyright treatment of model weights resulting from the commons is achieved by Yud posting about GPT and nuclear weapons in Time Magazine and scaring the living hell out of the lay public about a capricious digital deity in the process, that’s unlikely to be a genie that goes back in the bottle. So I’d really like it if someone injected some: here’s how this could actually happen into the dialog. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...