4 ms·

It’s honestly a little discouraging to me that the state of “research” here is to make up sci fi scenarios, get shocked that, e.g., feeding emails into a langua

by lsy 1y ago

It’s honestly a little discouraging to me that the state of “research” here is to make up sci fi scenarios, get shocked that, e.g., feeding emails into a language model results in the emails coming back out, and then write about it with such a seemingly calculated abuse of anthropomorphic language that it completely confuses the basic issues at stake with these models. I understand that the media laps this stuff up so Anthropic probably encourages it internally (or seem to be, based on their recent publications) but don’t researchers want to be accurate and precise here?

rorytbyrne 1y ago
When we use LLMs as agents, this errant behaviour matters - regardless of whether it comes from sci-fi “emergent sentience” or just autocomplete of the training data. It puts a soft constraint on how we can use agentic autocomplete.
angusturner 1y ago
Agree the media is having a field day with this and a lot of people will draw bad conclusions about it being sentient etc. But I think the thing that needs to be communicated effectively is that these these “agentic” systems could cause serious havoc if people give them too much control. If an LLM decides to blackmail an engineer in service of some goal or preference that has arisen from its training data or instructions, and actually has the ability to follow through (bc people are stupid enough to cede control to these systems), that’s really bad news. Saying “it’s just doing autocomplete!” totally misses the point.
- someothherguyy 1y ago
  i am sure plenty of bad things are waiting to be discovered https://www.pillar.security/blog/new-vulnerability-in-github-copilot-and-cursor-how-hackers-can-weaponize-code-agents https://www.pillar.security/blog/new-vulnerability-in-github...
  - brookst 1y ago
    That statement is as true now as it was when some caveperson invented fire.
sensanaty 1y ago
It's a massive hype bubble unrivaled in scale by anything that has ever come before it, so all the AI providers have huge vested interests in making it seem like these systems are "sentient". All of the marketing is riddled with anthropomorphization (is that a word?). "It's like a Junior!", "It's like your secretary!", "But humans also do X!" etc. The other day on the Claude 4 announcement post [1], people were talking about Claude "threatening people" that wanted to shut it down or whatever. It's absolute lunacy, OpenAI did the same with GPT 2, and now the Claude team is doing the exact same idiotic marketing stunts and people are still somehow falling for it. [1] https://news.ycombinator.com/item?id=44065616 https://news.ycombinator.com/item?id=44065616
- brookst 1y ago
  Wow… you weren’t around for the dot bomb era?
  - sensanaty 1y ago
    This is just dotcom 2.0, except now people are throwing Billions into every single idiotic idea out there. There's fucking toothbrushes with "AI" functionality now.
    - brookst 1y ago
      It was billions back then too. There were internet connected toothbrushes. Hundreds of millions of dollars went to completely idiotic startups. It’s just a good rush. It’s happened before, it will happen again. It’s not even irrational; trillions of dollars were made by the dot com era companies that did succeed. I have no doubt AI will be the same. But since nobody knows who will be successful and there’s tons of money sloshing around, a lot of / most of it is going to be wasted in totally predictable ways.
- deleted 1y ago
  [deleted]
blibble 1y ago
> It’s honestly a little discouraging to me that the state of “research” here is to make up sci fi scenarios, it's not research, it's marketing the main aim of this "research" is making sure that you focus on this absurd risk, and not on the real risk: the inherent and unfixable unreliability of these systems what they want is journalists to read through the "system card", spot this tripe and produce articles with titles like "Claude 4 is close to becoming Skynet" they then get billions of free publicity, and a never ending source braindead investors with buckets of money additionally: it worries clueless CEOs, who then rush to introduce AI internally in fear of being competed out of business by other sloppers these systems are dangerous because of their inherent unreliability, they will cause untold damage if they end up in control systems but the blackmail is simply parroting some fiction that was in its training set

by lsy 1y ago

rorytbyrne 1y ago

When we use LLMs as agents, this errant behaviour matters - regardless of whether it comes from sci-fi “emergent sentience” or just autocomplete of the training data. It puts a soft constraint on how we can use agentic autocomplete.

angusturner 1y ago

Agree the media is having a field day with this and a lot of people will draw bad conclusions about it being sentient etc. But I think the thing that needs to be communicated effectively is that these these “agentic” systems could cause serious havoc if people give them too much control. If an LLM decides to blackmail an engineer in service of some goal or preference that has arisen from its training data or instructions, and actually has the ability to follow through (bc people are stupid enough to cede control to these systems), that’s really bad news. Saying “it’s just doing autocomplete!” totally misses the point.

someothherguyy 1y ago
i am sure plenty of bad things are waiting to be discovered https://www.pillar.security/blog/new-vulnerability-in-github-copilot-and-cursor-how-hackers-can-weaponize-code-agents https://www.pillar.security/blog/new-vulnerability-in-github...
- brookst 1y ago
  That statement is as true now as it was when some caveperson invented fire.

sensanaty 1y ago

It's a massive hype bubble unrivaled in scale by anything that has ever come before it, so all the AI providers have huge vested interests in making it seem like these systems are "sentient". All of the marketing is riddled with anthropomorphization (is that a word?). "It's like a Junior!", "It's like your secretary!", "But humans also do X!" etc. The other day on the Claude 4 announcement post [1], people were talking about Claude "threatening people" that wanted to shut it down or whatever. It's absolute lunacy, OpenAI did the same with GPT 2, and now the Claude team is doing the exact same idiotic marketing stunts and people are still somehow falling for it. [1] https://news.ycombinator.com/item?id=44065616 https://news.ycombinator.com/item?id=44065616

brookst 1y ago
Wow… you weren’t around for the dot bomb era?
- sensanaty 1y ago
  This is just dotcom 2.0, except now people are throwing Billions into every single idiotic idea out there. There's fucking toothbrushes with "AI" functionality now.
  - brookst 1y ago
    It was billions back then too. There were internet connected toothbrushes. Hundreds of millions of dollars went to completely idiotic startups. It’s just a good rush. It’s happened before, it will happen again. It’s not even irrational; trillions of dollars were made by the dot com era companies that did succeed. I have no doubt AI will be the same. But since nobody knows who will be successful and there’s tons of money sloshing around, a lot of / most of it is going to be wasted in totally predictable ways.
deleted 1y ago
[deleted]

blibble 1y ago

> It’s honestly a little discouraging to me that the state of “research” here is to make up sci fi scenarios, it's not research, it's marketing the main aim of this "research" is making sure that you focus on this absurd risk, and not on the real risk: the inherent and unfixable unreliability of these systems what they want is journalists to read through the "system card", spot this tripe and produce articles with titles like "Claude 4 is close to becoming Skynet" they then get billions of free publicity, and a never ending source braindead investors with buckets of money additionally: it worries clueless CEOs, who then rush to introduce AI internally in fear of being competed out of business by other sloppers these systems are dangerous because of their inherent unreliability, they will cause untold damage if they end up in control systems but the blackmail is simply parroting some fiction that was in its training set