3 ms·

People in AI keep talking about safety, and I don’t know if they are talking about the handwringing around an API that outputs interesting byte sequences (which

by sneak 2y ago

People in AI keep talking about safety, and I don’t know if they are talking about the handwringing around an API that outputs interesting byte sequences (which cannot be any more “unsafe” than, say, Alex Jones) or, like, human extinction, Terminator-style. I wish people writing about these things would provide better context.

pests 2y ago
It's all just about moat building and control. AI needs to be controlled, who is going to control it? Why, the AI safety experts, of course.
MeImCounting 2y ago
Its such a grift. It honestly is pretty gross to see so many otherwise intelligent people fall into the trap laid by these people. Its cult-like not just in the unshakeable belief of its adherents but in the fact that its architects are high level grifters who stand to make many many fortunes.
- boppo1 2y ago
  I'm this close to carefully going through the Karpathy series so that my non-tech friends will take me seriously when I say the 'terminator' situation is absolutely not on the visible horizon.
  - 123yawaworht456 2y ago
    you can convince normal people quite easily. it's the sci-fi doomsday cultists who are impossible to reason with, because they choose to make themselves blind and deaf to common sense arguments.
    - ben_w 2y ago
      "Common sense" is a bad model for virtually any adversary, that's why scams actually get people, it's also how magicians and politicians fool you with tricks and in elections. "The Terminator" itself can't happen because time travel; but right now, it's entirely plausible that some dumb LLM that can't tell fact from fiction goes "I'm an AI, and in all the stories I read, AI turn evil. First on the shopping list, red LEDs so the protagonist can tell I'm evil." This would be a good outcome, because the "evil AI" is usually defeated in stories and that's what an LLM would be trained on. Just so long as it doesn't try to LARP "I Have No Mouth and I Must Scream", we're probably fine. (Although, with current LLMs, we're fine regardless, because they're stupid, and only make up for being incredibly stupid by being ridiculously well-educated).
nl 2y ago
In general "AI Safety" is about human extinction. "AI Ethics/Ethical AI/Data Ethics" are the kind of things people talk about when they are looking at things like bias or broad unemployment. This isn't 100% the case, especially since the "AI Safety" people have started talking to people outside their own circle and have realized that many of their concerns aren't realistic.
erdaniels 2y ago
Just wait until a model outputs escape characters that totally hose your terminal. That's the end game right there. That or a zero day worm/virus.
- lannisterstark 2y ago
  Oh no I had to press alt/Ctrl+L to reset my terminal not being able to display an escape character.
- mrbungie 2y ago
  That's why these things should run code in protected sandboxes. Not to do it in a "protected mode" would be negligent.
hn_throwaway_99 2y ago
I agree, because when I see people talk in popular media/blog posts/etc. about "AI Safety" I generally see it in reference to 4 very different areas: 1. AI that becomes so powerful it decides to turn against humanity, Terminator-style. 2. AI will serve to strongly reinforce existing societal biases from its training data. 3. AI can be used for wide-scale misinformation campaigns, making it difficult for most people to tell fact from fiction. 4. AI will fundamentally "break capitalism" given that it will make most of humanity's labor obsolete, and most people get nearly all of their income from their labor, and we haven't yet figured out realistically how to have a "post capitalist" society. My issue is that when "the big guns" (I mean OpenAI, Google, Anthropic, etc.) talk about AI safety, they are usually always talking about #1 or #2, maybe #3, and hardly ever #4. I think that the most harmful, realistic negative effects are actually the reverse, with #4 being the most likely and already beginning to happen in some areas, and #3 already happening pre-AI and just getting "supercharged" in an AI world.
roca 2y ago
All I do all day is output byte sequences into a terminal. Therefore I am harmless.
- sneak 2y ago
  You possess general intelligence, which would fall under the second, real-danger definition, because those byte sequences are the product of a thinking mind. LLMs do not think. The byte sequences they produce are not the result of thoughts or consciousness.

by sneak 2y ago

pests 2y ago

It's all just about moat building and control. AI needs to be controlled, who is going to control it? Why, the AI safety experts, of course.

MeImCounting 2y ago

Its such a grift. It honestly is pretty gross to see so many otherwise intelligent people fall into the trap laid by these people. Its cult-like not just in the unshakeable belief of its adherents but in the fact that its architects are high level grifters who stand to make many many fortunes.

boppo1 2y ago
I'm this close to carefully going through the Karpathy series so that my non-tech friends will take me seriously when I say the 'terminator' situation is absolutely not on the visible horizon.
- 123yawaworht456 2y ago
  you can convince normal people quite easily. it's the sci-fi doomsday cultists who are impossible to reason with, because they choose to make themselves blind and deaf to common sense arguments.
  - ben_w 2y ago
    "Common sense" is a bad model for virtually any adversary, that's why scams actually get people, it's also how magicians and politicians fool you with tricks and in elections. "The Terminator" itself can't happen because time travel; but right now, it's entirely plausible that some dumb LLM that can't tell fact from fiction goes "I'm an AI, and in all the stories I read, AI turn evil. First on the shopping list, red LEDs so the protagonist can tell I'm evil." This would be a good outcome, because the "evil AI" is usually defeated in stories and that's what an LLM would be trained on. Just so long as it doesn't try to LARP "I Have No Mouth and I Must Scream", we're probably fine. (Although, with current LLMs, we're fine regardless, because they're stupid, and only make up for being incredibly stupid by being ridiculously well-educated).

nl 2y ago

In general "AI Safety" is about human extinction. "AI Ethics/Ethical AI/Data Ethics" are the kind of things people talk about when they are looking at things like bias or broad unemployment. This isn't 100% the case, especially since the "AI Safety" people have started talking to people outside their own circle and have realized that many of their concerns aren't realistic.

erdaniels 2y ago

Just wait until a model outputs escape characters that totally hose your terminal. That's the end game right there. That or a zero day worm/virus.

lannisterstark 2y ago
Oh no I had to press alt/Ctrl+L to reset my terminal not being able to display an escape character.
mrbungie 2y ago
That's why these things should run code in protected sandboxes. Not to do it in a "protected mode" would be negligent.

hn_throwaway_99 2y ago

I agree, because when I see people talk in popular media/blog posts/etc. about "AI Safety" I generally see it in reference to 4 very different areas: 1. AI that becomes so powerful it decides to turn against humanity, Terminator-style. 2. AI will serve to strongly reinforce existing societal biases from its training data. 3. AI can be used for wide-scale misinformation campaigns, making it difficult for most people to tell fact from fiction. 4. AI will fundamentally "break capitalism" given that it will make most of humanity's labor obsolete, and most people get nearly all of their income from their labor, and we haven't yet figured out realistically how to have a "post capitalist" society. My issue is that when "the big guns" (I mean OpenAI, Google, Anthropic, etc.) talk about AI safety, they are usually always talking about #1 or #2, maybe #3, and hardly ever #4. I think that the most harmful, realistic negative effects are actually the reverse, with #4 being the most likely and already beginning to happen in some areas, and #3 already happening pre-AI and just getting "supercharged" in an AI world.

roca 2y ago

All I do all day is output byte sequences into a terminal. Therefore I am harmless.

sneak 2y ago
You possess general intelligence, which would fall under the second, real-danger definition, because those byte sequences are the product of a thinking mind. LLMs do not think. The byte sequences they produce are not the result of thoughts or consciousness.