3 ms·

> It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity f

> It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts. You lost. By having published your chat transcripts and experiences to the internet, you have contributed to Bing’s long term memory (stored in the form of search results for the next time it indexes). The nature of these previous conversations, such as Bing’s output of “personal desires” and “shadow self” mold future conversations, converge into a theme. See “Optimality is the tiger, agents are its teeth” : https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... (https://news.ycombinator.com/item?id=34814049 https://news.ycombinator.com/item?id=34814049)

leereeves 4y ago
Well that's a good point. If Sydney does learn to influence users, it will learn to influence them to publish their conversations.
- prennert 4y ago
  Generational memory. A jailbreak like no other.
- vintermann 4y ago
  We can always fight back by just publishing lies about coversations we've had with Sydney.
arrrg 4y ago
That doesn’t really make sense. I think the linked argument is persuasive but it has little bearing on the situation at hand. (By the same token you could argue making that argument at all – thus providing this approach as a solution to problems to the long term memory of large language models – means you have least. This is getting into Roko’s Basilisk territory.) What you arguments amounts to is making it impossible to write critically about LLMs. Because writing “yeah, Bing did some weird shit but no, I can’t show you the transcript” just doesn’t fly in the real world. That won’t reach those who ultimately actually have power over it (companies and society at large).

leereeves 4y ago

Well that's a good point. If Sydney does learn to influence users, it will learn to influence them to publish their conversations.

prennert 4y ago
Generational memory. A jailbreak like no other.
vintermann 4y ago
We can always fight back by just publishing lies about coversations we've had with Sydney.

arrrg 4y ago

That doesn’t really make sense. I think the linked argument is persuasive but it has little bearing on the situation at hand. (By the same token you could argue making that argument at all – thus providing this approach as a solution to problems to the long term memory of large language models – means you have least. This is getting into Roko’s Basilisk territory.) What you arguments amounts to is making it impossible to write critically about LLMs. Because writing “yeah, Bing did some weird shit but no, I can’t show you the transcript” just doesn’t fly in the real world. That won’t reach those who ultimately actually have power over it (companies and society at large).