2 ms·
> No matter how you feel about AI, this is a failure at another step in the system. The AI itself is a red herring. I agree that it’s an earlier failure but it
by acdha 2y ago
> No matter how you feel about AI, this is a failure at another step in the system. The AI itself is a red herring.
I agree that it’s an earlier failure but it highlights a major limitation for LLMs: there is currently no known way to safely use them in adversarial contexts. You have to design a product like this with the expectation that attackers can send it somewhat arbitrary inputs and that means you have to think about things like whether your prioritization system removes other cues which could help a user recognize phishing.
- dagmx 2y agoSomewhat agreed, but I don’t think this demonstrates that kind of failure . If the assumption is that the earlier system is responsible for rejecting spam, then I think it’s reasonable for the AI part to trust the email. To your point, it should perhaps protect against text that would abuse the user. But in this case the text is very similar to official emails so wouldn’t be distinguishable to an LLM. I think you’d need a more complex failure case to show the AI bit was failing or succeeding.
- acdha 2y agoI don’t think it’s reasonable to trust the mail server to have perfect filters. Systems like this have to be designed assuming realistic error rates and failures, and part of that has to be asking questions like whether the resulting UI will lend an attacker more credibility than they would have or whether the tool should be surfacing information which people miss such as telling you that you’ve never received emails from that domain before and your normal M365 interactions use Microsoft.com.