5 ms·

Isn’t this an issue with the spam filter more than the AI? Why should the AI be doing spam filtering? They’re two different things (The spam filter might use s

by davedx 2y ago

Isn’t this an issue with the spam filter more than the AI? Why should the AI be doing spam filtering? They’re two different things (The spam filter might use statistical or ML methods of course but it’s a different software?)

latexr 2y ago
The issue isn’t merely that the message wasn’t filtered, but that it was given active prominence. In other words, it made it more likely that someone would get phished.
Sakos 2y ago
Really need more details on what happened. Are they replacing whatever spam filter they had with Apple Intelligence? Is the usual spam filter disabled? Is there usually a spam filter at all? What client is this, Mail?
- latexr 2y ago
  > What client is this, Mail? Yes. As to the other questions, the person reporting this is one of the founders of Panic¹, who are trusted developers who have been making Mac apps for decades. So you can be reasonably sure there’s at least a modicum of due diligence in the report. ¹ https://panic.com https://panic.com
  - Sakos 2y ago
    I'm not questioning the Twitter post. I just need more details in order to get an accurate picture of what's actually happening. The tweet is too vague for me to really understand, regardless of how trusted the person is.
    - latexr 2y ago
      In that case, going back: > Are they replacing whatever spam filter they had with Apple Intelligence? I’m not sure. I don’t think so, but I also don’t know if we know for certain. > Is there usually a spam filter at all? Yes. In addition to what may be marked as spam on the server, Mail can also do its own filtering. https://support.apple.com/en-gb/guide/mail/mlhlp1065/mac https://support.apple.com/en-gb/guide/mail/mlhlp1065/mac > Is the usual spam filter disabled? Can’t say, as I’m not the reporter. But again, email can be marked as junk from Mail, the server, both, or neither.
- dwighttk 2y ago
  I read there is a new mailbox that just has these messages that are high priority, which sounds like it was just added alongside whatever is going on already… I bet it’s just a question of do you want ai to try to find important messages the spam filter accidentally flagged or not. (Me? No)
chaoz__ 2y ago
> Machine learning is a subset of artificial intelligence that automatically enables a machine or system to learn and improve from experience. I got your point, but according to most definitions ML \subset AI. Also, LLMs are not as explainable as classic ML algos, but they might certainly have its place. The real problem is that it was not combined nicely for nice user-experience and (probably) False Positive Rate is higher than what people expected from trendy "AI".
throwaway290 2y ago
There is no such thing as "AI". It is all ML, both what spam filters been using for decades and what chatbots use now. I assure you they don't use chatgpt for spam filtering if that's what you call "AI".
slightwinder 2y ago
The question is, did this mail really pass the spam filter, or did the AI high jacked the mail before it went to the spam filter. Or even worse, the AI found it in the spam-folder and moved it back to the inbox...
alistairSH 2y ago
How is Apple AI even involved here? Assuming this is the Apple-native Mail app, and an external (not iCloud) email service, isn't the spam filtering done on the server (not the client)? If that's true, did Apple AI on the client pick something out of spam and move it back into the Inbox? If that's the case, that's shady AF and definitely a bad thing. But, there aren't enough details to know.
- acdha 2y ago
  The spam filter failed to catch it, which is expected - nothing gets 100% – but then the Mail app interpreted the text of the message in the inbox as legitimate and placed it in the priority section with an excerpt which sounds legit and none of the suspicious parts displayed. This is basically the Achilles heel of LLMs: they’re gullible, and in a context like spam there are many people with a financial incentive to figuring out how to exploit that. The rush to deploy them will lead to more of these problems as people start using them on untrusted inputs at scale and I imagine a ton of money is going to flood to people who say they can limit this.
  - alistairSH 2y ago
    Ah, that makes so much more sense!
  - zombiwoof 2y ago
    Yes and wait till the spammers figure out how to trick LLMs
whywhywhywhy 2y ago
The fact either let an msnbilling.co.in email through is embarassing. Although I don't think iOS Mail app has spam filtering like the desktop Mail.app if you're using your own server, at least mines never worked if it does.
- benhurmarcel 2y ago
  This is also made much worse by the UI of the iOS mail app, which has no way of displaying the email address of the sender. It only shows the name the sender has set for himself.
bluedino 2y ago
Could have been stopped at so many levels. We had a 'Please review this invoice' email get through. Generic email message with a PDF that had an exploit and a web link to some low-quality impersonating site. Infected the users computer, sent a copy to everyone on their address book, got another person inside the company, sent a copy to everyone in their address book... IT staff had to manually intervene with those two users. Disable their accounts, rebuild their machines, change their passwords, etc. Who's job was it to stop that email? Was it Microsoft? Our email is hosted through them. Exchange Online boasts: Data loss prevention capabilities prevent users from mistakenly sending sensitive information to unauthorized people. Globally redundant servers, premier disaster recovery capabilities, and a team of security experts monitoring Exchange Online around the clock safeguard your data. What about Outlook itself? Advanced data, device, and file security Maybe our AV/EDR software should have caught it? AI-powered prevention, detection, response, and threat hunting across user endpoints, containers, cloud workloads, and IoT devices. Enabling modern enterprises to defend faster, at greater scale, and with higher accuracy across their entire attack surface, we empower the world to run securely. Maybe our firewalls should have caught it. Packet inspection, URL ratings, lots of things should have triggered something. And then our SIEM...I guess we had it all logged, even though we never had any warnings or messages from them. So much for millions of community submitted icidents and threat intelligence and whatever else they sell to make people sleep at night.

by davedx 2y ago

latexr 2y ago

The issue isn’t merely that the message wasn’t filtered, but that it was given active prominence. In other words, it made it more likely that someone would get phished.

Sakos 2y ago

Really need more details on what happened. Are they replacing whatever spam filter they had with Apple Intelligence? Is the usual spam filter disabled? Is there usually a spam filter at all? What client is this, Mail?

latexr 2y ago
> What client is this, Mail? Yes. As to the other questions, the person reporting this is one of the founders of Panic¹, who are trusted developers who have been making Mac apps for decades. So you can be reasonably sure there’s at least a modicum of due diligence in the report. ¹ https://panic.com https://panic.com
- Sakos 2y ago
  I'm not questioning the Twitter post. I just need more details in order to get an accurate picture of what's actually happening. The tweet is too vague for me to really understand, regardless of how trusted the person is.
  - latexr 2y ago
    In that case, going back: > Are they replacing whatever spam filter they had with Apple Intelligence? I’m not sure. I don’t think so, but I also don’t know if we know for certain. > Is there usually a spam filter at all? Yes. In addition to what may be marked as spam on the server, Mail can also do its own filtering. https://support.apple.com/en-gb/guide/mail/mlhlp1065/mac https://support.apple.com/en-gb/guide/mail/mlhlp1065/mac > Is the usual spam filter disabled? Can’t say, as I’m not the reporter. But again, email can be marked as junk from Mail, the server, both, or neither.
dwighttk 2y ago
I read there is a new mailbox that just has these messages that are high priority, which sounds like it was just added alongside whatever is going on already… I bet it’s just a question of do you want ai to try to find important messages the spam filter accidentally flagged or not. (Me? No)

chaoz__ 2y ago

> Machine learning is a subset of artificial intelligence that automatically enables a machine or system to learn and improve from experience. I got your point, but according to most definitions ML \subset AI. Also, LLMs are not as explainable as classic ML algos, but they might certainly have its place. The real problem is that it was not combined nicely for nice user-experience and (probably) False Positive Rate is higher than what people expected from trendy "AI".

throwaway290 2y ago

There is no such thing as "AI". It is all ML, both what spam filters been using for decades and what chatbots use now. I assure you they don't use chatgpt for spam filtering if that's what you call "AI".

slightwinder 2y ago

The question is, did this mail really pass the spam filter, or did the AI high jacked the mail before it went to the spam filter. Or even worse, the AI found it in the spam-folder and moved it back to the inbox...

alistairSH 2y ago

How is Apple AI even involved here? Assuming this is the Apple-native Mail app, and an external (not iCloud) email service, isn't the spam filtering done on the server (not the client)? If that's true, did Apple AI on the client pick something out of spam and move it back into the Inbox? If that's the case, that's shady AF and definitely a bad thing. But, there aren't enough details to know.

acdha 2y ago
The spam filter failed to catch it, which is expected - nothing gets 100% – but then the Mail app interpreted the text of the message in the inbox as legitimate and placed it in the priority section with an excerpt which sounds legit and none of the suspicious parts displayed. This is basically the Achilles heel of LLMs: they’re gullible, and in a context like spam there are many people with a financial incentive to figuring out how to exploit that. The rush to deploy them will lead to more of these problems as people start using them on untrusted inputs at scale and I imagine a ton of money is going to flood to people who say they can limit this.
- alistairSH 2y ago
  Ah, that makes so much more sense!
- zombiwoof 2y ago
  Yes and wait till the spammers figure out how to trick LLMs

whywhywhywhy 2y ago

The fact either let an msnbilling.co.in email through is embarassing. Although I don't think iOS Mail app has spam filtering like the desktop Mail.app if you're using your own server, at least mines never worked if it does.

benhurmarcel 2y ago
This is also made much worse by the UI of the iOS mail app, which has no way of displaying the email address of the sender. It only shows the name the sender has set for himself.

bluedino 2y ago

Could have been stopped at so many levels. We had a 'Please review this invoice' email get through. Generic email message with a PDF that had an exploit and a web link to some low-quality impersonating site. Infected the users computer, sent a copy to everyone on their address book, got another person inside the company, sent a copy to everyone in their address book... IT staff had to manually intervene with those two users. Disable their accounts, rebuild their machines, change their passwords, etc. Who's job was it to stop that email? Was it Microsoft? Our email is hosted through them. Exchange Online boasts: Data loss prevention capabilities prevent users from mistakenly sending sensitive information to unauthorized people. Globally redundant servers, premier disaster recovery capabilities, and a team of security experts monitoring Exchange Online around the clock safeguard your data. What about Outlook itself? Advanced data, device, and file security Maybe our AV/EDR software should have caught it? AI-powered prevention, detection, response, and threat hunting across user endpoints, containers, cloud workloads, and IoT devices. Enabling modern enterprises to defend faster, at greater scale, and with higher accuracy across their entire attack surface, we empower the world to run securely. Maybe our firewalls should have caught it. Packet inspection, URL ratings, lots of things should have triggered something. And then our SIEM...I guess we had it all logged, even though we never had any warnings or messages from them. So much for millions of community submitted icidents and threat intelligence and whatever else they sell to make people sleep at night.