3 ms·

I know this subject is a combination of subjects that get people riled up, hence the comments here, but I think people are missing the forest for the trees: 1.

by dagmx 2y ago

I know this subject is a combination of subjects that get people riled up, hence the comments here, but I think people are missing the forest for the trees: 1. This is a failure in the spam blocking system. This would have gotten through prior to the AI additions. 2. Perhaps this is whataboutism, but competing products like GMail constantly let through spam that is much more obvious than this. I’m constantly flagging stuff as spam and it is terrible at learning what is spam or not. And with both Apple Mail And GMail, I have many legitimate mails going to spam as well. 3. I haven’t seen a solution posited on how to detect this better? The only tell here is the domain of the email imho. Otherwise the email looks legitimate. The poor quality of spam filters has been a thing for years, and not a job for a local LLM which is designed for summarizing and priority detection. No matter how you feel about AI, this is a failure at another step in the system. The AI itself is a red herring.

acdha 2y ago
> No matter how you feel about AI, this is a failure at another step in the system. The AI itself is a red herring. I agree that it’s an earlier failure but it highlights a major limitation for LLMs: there is currently no known way to safely use them in adversarial contexts. You have to design a product like this with the expectation that attackers can send it somewhat arbitrary inputs and that means you have to think about things like whether your prioritization system removes other cues which could help a user recognize phishing.
- dagmx 2y ago
  Somewhat agreed, but I don’t think this demonstrates that kind of failure . If the assumption is that the earlier system is responsible for rejecting spam, then I think it’s reasonable for the AI part to trust the email. To your point, it should perhaps protect against text that would abuse the user. But in this case the text is very similar to official emails so wouldn’t be distinguishable to an LLM. I think you’d need a more complex failure case to show the AI bit was failing or succeeding.
  - acdha 2y ago
    I don’t think it’s reasonable to trust the mail server to have perfect filters. Systems like this have to be designed assuming realistic error rates and failures, and part of that has to be asking questions like whether the resulting UI will lend an attacker more credibility than they would have or whether the tool should be surfacing information which people miss such as telling you that you’ve never received emails from that domain before and your normal M365 interactions use Microsoft.com.

by dagmx 2y ago

acdha 2y ago

> No matter how you feel about AI, this is a failure at another step in the system. The AI itself is a red herring. I agree that it’s an earlier failure but it highlights a major limitation for LLMs: there is currently no known way to safely use them in adversarial contexts. You have to design a product like this with the expectation that attackers can send it somewhat arbitrary inputs and that means you have to think about things like whether your prioritization system removes other cues which could help a user recognize phishing.

dagmx 2y ago
Somewhat agreed, but I don’t think this demonstrates that kind of failure . If the assumption is that the earlier system is responsible for rejecting spam, then I think it’s reasonable for the AI part to trust the email. To your point, it should perhaps protect against text that would abuse the user. But in this case the text is very similar to official emails so wouldn’t be distinguishable to an LLM. I think you’d need a more complex failure case to show the AI bit was failing or succeeding.
- acdha 2y ago
  I don’t think it’s reasonable to trust the mail server to have perfect filters. Systems like this have to be designed assuming realistic error rates and failures, and part of that has to be asking questions like whether the resulting UI will lend an attacker more credibility than they would have or whether the tool should be surfacing information which people miss such as telling you that you’ve never received emails from that domain before and your normal M365 interactions use Microsoft.com.