3 ms·

I mean people can get through your front door using any number of attacks that either exploit weaknesses in the locking mechanism, or circumvent your locks via

I mean people can get through your front door using any number of attacks that either exploit weaknesses in the locking mechanism, or circumvent your locks via a carefully placed brick through a window or something like that. Even if you have alarms and other measures, that doesn't stop someone from doing a quick smash and grab, etc. All of these security flaws doesn't mean that you should leave your door open and unlocked all the time. Imperfect security isn't useless security. And the purpose of things like bad-word filters is to make a best effort at blocking stuff which violates the platform TOC and makes plausible deniability much less likely when someone is deliberately circumventing the filters. The existence of false positives and false negatives is considered acceptable in an imperfect world. The filters themselves also only block the action to change a username or whatever and don't punish the user or deny use of the platform entirely (they're much less punitive than the AI abuse algorithms that auto-ban people off of Google/GitHub/etc).

advael 3y ago
Your analogy is also bad. I agree that perfect security is impossible and that is completely irrelevant here. What platforms like this do by not publishing their policies is more akin to insisting you use their special locks on your door that they claim protect you better because no one knows how they work. Maybe they're operated by an AI working with a Ring camera or something? Very fancy stuff. With this kind of tech, you may be locked out of your home for reasons you don't understand. An independent locksmith might have a hard time figuring out what's going wrong with the door if it fails. You have no idea if some burglars are authorized to enter your home trivially by a deal with the company. If the company decides you are in the wrong in any context, they have the unilateral power to deny you access with no clear recourse. They may get you arrested for trying to get into your own home
- lamontcg 3y ago
  > They may get you arrested for trying to get into your own home The AI algorithms that ban people from platforms like Google and GitHub do that, which I explicitly called out as needing more oversight. That is different from algorithms which just prevent you from doing something on a platform like using n-bombs in your username, or the LLM guardrails that just give you mangled answers or tell you that they can't do that. That isn't analogous to getting arrested. And in these cases the analogy really falls apart because it isn't your home, and it isn't critical for your life.
  - advael 3y ago
    Friend, it was your analogy
    - lamontcg 3y ago
      I was intending to make a point about security through obscurity, not to make an analogy with those systems.
      - advael 3y ago
        Yes, and the point about security through obscurity is what I was addressing in my response. Security through obscurity is, especially by itself, bad security. By bad I partially mean ineffective: Attackers (e.g., to continue using your examples, the person who really want to signal that they really do not like black people to fellow racists, or the person who prompt-engineers GPT to replicate bomb-building instructions from the internet) can still get what they want done with some extra effort, and the threat model isn't actually such that subjecting them to inconvenience in doing this really matters a lot. Also, independent auditors can't adversarially improve the system, which is a staple of how robust security systems work and have worked for decades, despite the protestations of big tech companies who stand to profit from additional secrecy and, often, plausibly deniable security failures. By bad I also mean detrimental, because not having clear policies creates Kafkaesque situations for users and removes accountability for the choices made in those policies. Even if this somehow helped security - which again it doesn't - these negative effects of that secrecy would be a tradeoff
        lamontcg 3y ago
        So again, the point of most security systems isn't to be impenetrable against dedicated attackers. They're supposed to stop relatively casual attackers and to establish that measures had been taken to secure the system. In the case of doorlocks having a simple Kwickset that the Lockpicking Lawyer could get through in 20 seconds is really sufficient security for most people. That keeps out the people rattling doorknobs looking for the person who left their house unlocked and a laptop on the coffee table. It also establishes that attempts had been made to secure the system. Similarly, the systems that block keywords in usernames on games don't have to be perfect either, some rate of both positive and negative failures are acceptable. The system works to block casual abuse, while users who go out of their way to circumvent the systems really establish the fact that they've actively worked around those systems, which makes the justification for punishment easier. And we pretty much know that in the case of prompt engineering that by disclosing the prompts that were used to secure the system would defeat the system and people would immediately publish how to work around the prompts. There isn't any use in independent auditing, because its a never ending cat and mouse game. And the failures of the system ARE NOT as significant as the failures in doorlocks or even keyword banning. False positives mean that you can't get what you want out of the system, which is just a failure in usability. It isn't like being locked out of your house or having your stuff stolen. And false negatives just means that the company has to work to improve the systems and whatever embarrassing content was constructed can be handled by PR. Since the company worked to prevent casual abuse and avoided a racist-tay-chatbot situation most people understand that going out of your way to hack prompts doesn't indicate that the company was negligent. And I don't see the parallels with the Kafkaesque systems that kick you out of systems which have turned into economic necessities like locking you out of your Google or GitHub or Apple accounts. All that is at stake here is that the prompt you wanted answered didn't work. That's just a usability problem.

advael 3y ago

Your analogy is also bad. I agree that perfect security is impossible and that is completely irrelevant here. What platforms like this do by not publishing their policies is more akin to insisting you use their special locks on your door that they claim protect you better because no one knows how they work. Maybe they're operated by an AI working with a Ring camera or something? Very fancy stuff. With this kind of tech, you may be locked out of your home for reasons you don't understand. An independent locksmith might have a hard time figuring out what's going wrong with the door if it fails. You have no idea if some burglars are authorized to enter your home trivially by a deal with the company. If the company decides you are in the wrong in any context, they have the unilateral power to deny you access with no clear recourse. They may get you arrested for trying to get into your own home

lamontcg 3y ago
> They may get you arrested for trying to get into your own home The AI algorithms that ban people from platforms like Google and GitHub do that, which I explicitly called out as needing more oversight. That is different from algorithms which just prevent you from doing something on a platform like using n-bombs in your username, or the LLM guardrails that just give you mangled answers or tell you that they can't do that. That isn't analogous to getting arrested. And in these cases the analogy really falls apart because it isn't your home, and it isn't critical for your life.
- advael 3y ago
  Friend, it was your analogy
  - lamontcg 3y ago
    I was intending to make a point about security through obscurity, not to make an analogy with those systems.
    - advael 3y ago
      Yes, and the point about security through obscurity is what I was addressing in my response. Security through obscurity is, especially by itself, bad security. By bad I partially mean ineffective: Attackers (e.g., to continue using your examples, the person who really want to signal that they really do not like black people to fellow racists, or the person who prompt-engineers GPT to replicate bomb-building instructions from the internet) can still get what they want done with some extra effort, and the threat model isn't actually such that subjecting them to inconvenience in doing this really matters a lot. Also, independent auditors can't adversarially improve the system, which is a staple of how robust security systems work and have worked for decades, despite the protestations of big tech companies who stand to profit from additional secrecy and, often, plausibly deniable security failures. By bad I also mean detrimental, because not having clear policies creates Kafkaesque situations for users and removes accountability for the choices made in those policies. Even if this somehow helped security - which again it doesn't - these negative effects of that secrecy would be a tradeoff
      - lamontcg 3y ago
        So again, the point of most security systems isn't to be impenetrable against dedicated attackers. They're supposed to stop relatively casual attackers and to establish that measures had been taken to secure the system. In the case of doorlocks having a simple Kwickset that the Lockpicking Lawyer could get through in 20 seconds is really sufficient security for most people. That keeps out the people rattling doorknobs looking for the person who left their house unlocked and a laptop on the coffee table. It also establishes that attempts had been made to secure the system. Similarly, the systems that block keywords in usernames on games don't have to be perfect either, some rate of both positive and negative failures are acceptable. The system works to block casual abuse, while users who go out of their way to circumvent the systems really establish the fact that they've actively worked around those systems, which makes the justification for punishment easier. And we pretty much know that in the case of prompt engineering that by disclosing the prompts that were used to secure the system would defeat the system and people would immediately publish how to work around the prompts. There isn't any use in independent auditing, because its a never ending cat and mouse game. And the failures of the system ARE NOT as significant as the failures in doorlocks or even keyword banning. False positives mean that you can't get what you want out of the system, which is just a failure in usability. It isn't like being locked out of your house or having your stuff stolen. And false negatives just means that the company has to work to improve the systems and whatever embarrassing content was constructed can be handled by PR. Since the company worked to prevent casual abuse and avoided a racist-tay-chatbot situation most people understand that going out of your way to hack prompts doesn't indicate that the company was negligent. And I don't see the parallels with the Kafkaesque systems that kick you out of systems which have turned into economic necessities like locking you out of your Google or GitHub or Apple accounts. All that is at stake here is that the prompt you wanted answered didn't work. That's just a usability problem.