4 ms·

I'm not concerned about how they work or whether they're attaining personhood or not. I'm concerned at what people can and will use them for to the detriment of

I'm not concerned about how they work or whether they're attaining personhood or not. I'm concerned at what people can and will use them for to the detriment of our civilisation

thomastjeffery 4y ago
How they work is the most critical piece of that puzzle. It isn't a mystery at all. LLMs implicitly model every pattern they can find in the text they are given. If the pattern aligns to language, humans categorize it as a feature. If it aligns to desired behavior, humans categorize it as a feature. If it aligns to undesired behavior, humans categorize it as a limitation. What humans categorize as features and limitations do not align with what the model is. A truth and a lie are structured with identical text patterns. The LLM can't categorize the two separately. The only risk here is that people continue to personify LLMs, and place trust in their relatively limited ability to fulfill a human role. It's the same category of risk as bad security practices. The potential to make a mistake itself is not a danger: making the mistake is.
- permo-w 4y ago
  you certainly make an interesting point about fact and falsehood and how they interpret language; however, I disagree that this is relevant and completely disagree that personification has any relevance to the most immediate dangers the danger here is that we make extremely powerful tools extremely available and people use those tools to hurt people, possibly a lot of people, possibly even by accident the scary part is that hacker news is full of the exact people that have the knowledge and capability to do this, and look at the reaction to this post: a resounding fuck off
  - thomastjeffery 4y ago
    I just explained explicitly how and why an LLM is not, and never will be, an "extremely powerful tool". A program that is actually a person in a black box would require the level of concern you are showing. Thankfully, that is not even remotely what we have here in reality.
    - permo-w 4y ago
      have you tried gpt-4? how can you possibly say it's not an extremely powerful tool? what happens when someone determined links it - or thousands of its - or thousands of next year's 3x as powerful its -- up to the internet with a code interpreter and a hard-coded poor intention?
      - tome 4y ago
        I don't know; what happens? I'd really like to know what people are imagining it could do, especially if they've got some evidence for it!
        permo-w 4y ago
        it's a rhetorical question. think about it yourself. try it yourself
      - qlm 4y ago
        What could it do that isn't already being done? I ask this sincerely. We already have bots constantly attacking every server they find 24/7. We already have botnets Is the concern that GPT will be able to find and exploit e.g. a novel RCE exploit in SSH?
        lanstin 4y ago
        I wish we could ask it to answer questions about GitHub repos.
        staunton 4y ago
        For example, it could scan social media and use that information to send a very large number of people very personalized messages that manipulate them into doing X. Here, X may include meeting other people to tempt/coerce them into doing Y. The reason this isn't possible now is that nobody has the time to look at millions of social media profiles and then write millions of personalized messages. To do it, you would need a lot of people who you would have to convince, including paying them. And even so, things like doomsday cults and terrorist organizations already exist.
  - tome 4y ago
    > people use those tools to hurt people, possibly a lot of people, possibly even by accident If it happens then hopefully it's by accident!
cthalupa 4y ago
> I'm concerned at what people can and will use them for to the detriment of our civilisation Maybe? But there's already all sorts of untapped resources for that - and we mostly just don't choose to use them. A lot of the misinformation and "fake news" that came out during prior election cycles in the US came from former Soviet bloc countries, and there is already significant untapped human capital in those places where it could be done for cheap, perhaps even cheaper than the cost of doing inference on premiere LLMs. Meanwhile we see all sorts of efficiency improvements occurring for generally positive use cases from these LLMs, and it is hardly unique for technology to have some risks to it as well as positive benefits. Things like GPT and Midjourney have been a qualitative leap over their predecessors from two decades ago, but they fundamentally are not doing something new - and even another qualitative leap wouldn't bring them to the level of existential threat that this letter seems to be afraid of. I think there are significant and real concerns about AGI and similar, but I can't seem to find any reason to share these same concern around the types of "AI" we are seeing today.

I'm not concerned about how they work or whether they're attaining personhood or not. I'm concerned at what people can and will use them for to the detriment of our civilisation

thomastjeffery 4y ago

How they work is the most critical piece of that puzzle. It isn't a mystery at all. LLMs implicitly model every pattern they can find in the text they are given. If the pattern aligns to language, humans categorize it as a feature. If it aligns to desired behavior, humans categorize it as a feature. If it aligns to undesired behavior, humans categorize it as a limitation. What humans categorize as features and limitations do not align with what the model is. A truth and a lie are structured with identical text patterns. The LLM can't categorize the two separately. The only risk here is that people continue to personify LLMs, and place trust in their relatively limited ability to fulfill a human role. It's the same category of risk as bad security practices. The potential to make a mistake itself is not a danger: making the mistake is.

permo-w 4y ago
you certainly make an interesting point about fact and falsehood and how they interpret language; however, I disagree that this is relevant and completely disagree that personification has any relevance to the most immediate dangers the danger here is that we make extremely powerful tools extremely available and people use those tools to hurt people, possibly a lot of people, possibly even by accident the scary part is that hacker news is full of the exact people that have the knowledge and capability to do this, and look at the reaction to this post: a resounding fuck off
- thomastjeffery 4y ago
  I just explained explicitly how and why an LLM is not, and never will be, an "extremely powerful tool". A program that is actually a person in a black box would require the level of concern you are showing. Thankfully, that is not even remotely what we have here in reality.
  - permo-w 4y ago
    have you tried gpt-4? how can you possibly say it's not an extremely powerful tool? what happens when someone determined links it - or thousands of its - or thousands of next year's 3x as powerful its -- up to the internet with a code interpreter and a hard-coded poor intention?
    - tome 4y ago
      I don't know; what happens? I'd really like to know what people are imagining it could do, especially if they've got some evidence for it!
      - permo-w 4y ago
        it's a rhetorical question. think about it yourself. try it yourself
    - qlm 4y ago
      What could it do that isn't already being done? I ask this sincerely. We already have bots constantly attacking every server they find 24/7. We already have botnets Is the concern that GPT will be able to find and exploit e.g. a novel RCE exploit in SSH?
      - lanstin 4y ago
        I wish we could ask it to answer questions about GitHub repos.
      - staunton 4y ago
        For example, it could scan social media and use that information to send a very large number of people very personalized messages that manipulate them into doing X. Here, X may include meeting other people to tempt/coerce them into doing Y. The reason this isn't possible now is that nobody has the time to look at millions of social media profiles and then write millions of personalized messages. To do it, you would need a lot of people who you would have to convince, including paying them. And even so, things like doomsday cults and terrorist organizations already exist.
- tome 4y ago
  > people use those tools to hurt people, possibly a lot of people, possibly even by accident If it happens then hopefully it's by accident!

cthalupa 4y ago

> I'm concerned at what people can and will use them for to the detriment of our civilisation Maybe? But there's already all sorts of untapped resources for that - and we mostly just don't choose to use them. A lot of the misinformation and "fake news" that came out during prior election cycles in the US came from former Soviet bloc countries, and there is already significant untapped human capital in those places where it could be done for cheap, perhaps even cheaper than the cost of doing inference on premiere LLMs. Meanwhile we see all sorts of efficiency improvements occurring for generally positive use cases from these LLMs, and it is hardly unique for technology to have some risks to it as well as positive benefits. Things like GPT and Midjourney have been a qualitative leap over their predecessors from two decades ago, but they fundamentally are not doing something new - and even another qualitative leap wouldn't bring them to the level of existential threat that this letter seems to be afraid of. I think there are significant and real concerns about AGI and similar, but I can't seem to find any reason to share these same concern around the types of "AI" we are seeing today.