7 ms·

Some of those highlighted rules, such as using CC or having the string “can help” being used to decide if something is spam or not is so absurd I’ll make sure t

Some of those highlighted rules, such as using CC or having the string “can help” being used to decide if something is spam or not is so absurd I’ll make sure to never use SpamAssasin.

jeffbee 3y ago
It is not and has never been a good classifier. If open AI fans want to contribute something of value to society, they would train a spam classifier on a large, manually-labeled corpus of mail, where the features include envelope data. That would get open source maybe 10% of the way to Gmail quality, or 100x better than SA.
- CamperBob2 3y ago
  Does SpamBayes still work?
  - dugite-code 3y ago
    The Python project that's clent side? Best of my knowledge it hasn't been updated in years, spamassasin's in built Bayesian classifier works just fine.
    - jepler 3y ago
      I still used spambayes up until I basically abandoned my self-hosted e-mail setup (in favor of an @gmail). It occurred to me recently that LLM-style tokenization + bayesian classification would be a sweet upgrade for spambayes, which always struggled with ad-hoc tokenization rules. (I don't think of it as "client side"; it was integrated with my system via procmail on what I'd call the "server side". You could use it in other ways, including as an Outlook plugin, way back in the day. Or it could connect to an IMAP mailbox and filter messages it found, etc. Really versatile tool for its time)
- acidburnNSA 3y ago
  FWIW I've been using SpamAssassin for over a decade personally (partly to avoid Google dependence), and it's been pretty darn good once I ran the Bayesian learning thing a few times many years ago. I get like 3-5 spams per week in my inbox. Do others really consider SA that bad?
  - brokensegue 3y ago
    On Gmail I get maybe 1 spam a month max in my inbox (and it blocks many per day)
    - acidburnNSA 3y ago
      That's pretty good. I have no doubt that Gmail spam protection is better than my self-hosted SA protection. For me, independence from a somewhat suspicious large company for something as important as e-mail is worth it.
      - dugite-code 3y ago
        I have a gmail honeypot where I fetchmail junk email straight to my junk folder and have a scheduled sa-learn cronjob. Ever since I started this I essentially stopped getting junk email in my selfhosted inbox. I also have dovecot set to learn Ham every time I file an email from the inbox to a folder for good measure.
    - kjs3 3y ago
      So...statistically insignificant difference from SA for most mail users.
      - brokensegue 3y ago
        3-5 spams per week vs. 1 per month is a big difference
        kjs3 3y ago
- jerf 3y ago
  We'll get there eventually, but it will be a bit. Spam classification at scale is already a compute-bound, or at least compute-starved, operation. Spam classification systems already do what they can to avoid so much as invoking a virus scanner if they can avoid it, because at scale it's so expensive. LLM-based spam classification is another order of magnitude more expensive and would require hardware that current spam systems do not have. But that's a problem that will resolve itself over time, in a variety of ways. And the spam systems can play the same tricks with only invoking it on a fraction of emails too, of course. It's just at current expense levels, that would be a very small fraction indeed. I'd hazard that trying to use modern AI on spam classification at scale could easily consume 10x-100x of all current AI hardware and still make less of a dent than you'd hope.
  - jeffbee 3y ago
    It doesn't need to be computationally costly because, as you seem to imply, there are tiers of cost tradeoffs. You can invoke a very cheap classifier at SMTP time, that is biased to have few false positives, that will temporarily reject all that which is highly likely to be spam. You can do this without even glancing at the body. Of course, having signals about peer reputation is the strong suit of Gmail or Microsoft, and the distributed, open community would need to solve the problem of promptly updating and distributing such reputation signals. And by "promptly" I mean within seconds of the leading edge of an attack. Then there are increasing tiers of cost that you would only run after it becomes likely that the message is acceptable. As you say, you would only run an antivirus on a message on the verge of delivery, because decoding the attachment and running the AV (in an expensive sandbox) is so costly.
- dugite-code 3y ago
  Really any of the open source language models might work well enough for the job. If you could manage to get a classifier that runs with tensorflow to take advantage of a coral tpu it would certainly be a major step up with managable performance.
- thesuitonym 3y ago
  I hope against hope that AI spam detection never becomes a thing. At least with today's methods, I can tell a person why their message was marked as spam. If AI detection becomes the norm, all I can do is shrug and say, "Sorry, it's the algorithm."
  - potatoman22 3y ago
    How do you think SpamAssassin/gmail/outlook created their spam rules?
  - jeffbee 3y ago
    Gmail has used machine learning to classify spam since its creation. https://workspace.google.com/blog/identity-and-security/an-overview-of-gmails-spam-filters https://workspace.google.com/blog/identity-and-security/an-o...
    - Mailtemi 3y ago
      Gmail is successful because it naturally is the biggest honeypot. Most antispam API filters are like accumulators. When a trend is detected, the rest are protected. But overall, it's about scale.
  - layer8 3y ago
    SpamAssassin has a Bayes filter that you can train with ham and spam. This has basically been a thing since forever.
    - andrewf 3y ago
      "A plan for Spam" (2002) - https://paulgraham.com/spam.html https://paulgraham.com/spam.html
- El_RIDO 3y ago
  Correct, Spamassassin will not classify emails be default. But it does include a Bayesian classifier and tooling that can be used to train it with curated ham and spam emails. It does require extra steps to set this up and feed it selected maildirs (for example, exclude inbox, spam and trash for training ham). https://spamassassin.apache.org/full/3.0.x/dist/doc/sa-learn.html#description https://spamassassin.apache.org/full/3.0.x/dist/doc/sa-learn... https://cwiki.apache.org/confluence/display/spamassassin/BayesInSpamAssassin https://cwiki.apache.org/confluence/display/spamassassin/Bay...
- creeble 3y ago
  I find it far more likely that AI will be used (indeed, is used already) to generate spam, rather than filter it.
throw0101d 3y ago
> being used to decide if something is spam or not Each rule has a score associated with it. By default a message needs to reach 5.0 to be marked as "spam": * https://spamassassin.apache.org/full/3.0.x/dist/doc/Mail_SpamAssassin_Conf.html#item_nn https://spamassassin.apache.org/full/3.0.x/dist/doc/Mail_Spa... The threshold is configurable. An header is added post-processing, e.g.: X-Spam-Status: Yes, score=21.6 required=4.0 […] * https://cwiki.apache.org/confluence/display/SPAMASSASSIN/X+Spam+Status https://cwiki.apache.org/confluence/display/SPAMASSASSIN/X+S... One can then choose what do to with this information (via procmail or Sieve). There is another header as well: > X-Spam-Level: This displays your spam level with asterisks, with one asterisk displayed per point, rounded down. For example, if your overall SpamAssassin score is 4.3, it will display ****. If you score less than 1, for example, 0.5, it will display nothing. * https://www.mailercheck.com/articles/spamassassin-score https://www.mailercheck.com/articles/spamassassin-score

Some of those highlighted rules, such as using CC or having the string “can help” being used to decide if something is spam or not is so absurd I’ll make sure to never use SpamAssasin.

jeffbee 3y ago

It is not and has never been a good classifier. If open AI fans want to contribute something of value to society, they would train a spam classifier on a large, manually-labeled corpus of mail, where the features include envelope data. That would get open source maybe 10% of the way to Gmail quality, or 100x better than SA.

CamperBob2 3y ago
Does SpamBayes still work?
- dugite-code 3y ago
  The Python project that's clent side? Best of my knowledge it hasn't been updated in years, spamassasin's in built Bayesian classifier works just fine.
  - jepler 3y ago
    I still used spambayes up until I basically abandoned my self-hosted e-mail setup (in favor of an @gmail). It occurred to me recently that LLM-style tokenization + bayesian classification would be a sweet upgrade for spambayes, which always struggled with ad-hoc tokenization rules. (I don't think of it as "client side"; it was integrated with my system via procmail on what I'd call the "server side". You could use it in other ways, including as an Outlook plugin, way back in the day. Or it could connect to an IMAP mailbox and filter messages it found, etc. Really versatile tool for its time)
acidburnNSA 3y ago
FWIW I've been using SpamAssassin for over a decade personally (partly to avoid Google dependence), and it's been pretty darn good once I ran the Bayesian learning thing a few times many years ago. I get like 3-5 spams per week in my inbox. Do others really consider SA that bad?
- brokensegue 3y ago
  On Gmail I get maybe 1 spam a month max in my inbox (and it blocks many per day)
  - acidburnNSA 3y ago
    That's pretty good. I have no doubt that Gmail spam protection is better than my self-hosted SA protection. For me, independence from a somewhat suspicious large company for something as important as e-mail is worth it.
    - dugite-code 3y ago
      I have a gmail honeypot where I fetchmail junk email straight to my junk folder and have a scheduled sa-learn cronjob. Ever since I started this I essentially stopped getting junk email in my selfhosted inbox. I also have dovecot set to learn Ham every time I file an email from the inbox to a folder for good measure.
  - kjs3 3y ago
    So...statistically insignificant difference from SA for most mail users.
    - brokensegue 3y ago
      3-5 spams per week vs. 1 per month is a big difference
      - kjs3 3y ago
jerf 3y ago
We'll get there eventually, but it will be a bit. Spam classification at scale is already a compute-bound, or at least compute-starved, operation. Spam classification systems already do what they can to avoid so much as invoking a virus scanner if they can avoid it, because at scale it's so expensive. LLM-based spam classification is another order of magnitude more expensive and would require hardware that current spam systems do not have. But that's a problem that will resolve itself over time, in a variety of ways. And the spam systems can play the same tricks with only invoking it on a fraction of emails too, of course. It's just at current expense levels, that would be a very small fraction indeed. I'd hazard that trying to use modern AI on spam classification at scale could easily consume 10x-100x of all current AI hardware and still make less of a dent than you'd hope.
- jeffbee 3y ago
  It doesn't need to be computationally costly because, as you seem to imply, there are tiers of cost tradeoffs. You can invoke a very cheap classifier at SMTP time, that is biased to have few false positives, that will temporarily reject all that which is highly likely to be spam. You can do this without even glancing at the body. Of course, having signals about peer reputation is the strong suit of Gmail or Microsoft, and the distributed, open community would need to solve the problem of promptly updating and distributing such reputation signals. And by "promptly" I mean within seconds of the leading edge of an attack. Then there are increasing tiers of cost that you would only run after it becomes likely that the message is acceptable. As you say, you would only run an antivirus on a message on the verge of delivery, because decoding the attachment and running the AV (in an expensive sandbox) is so costly.
dugite-code 3y ago
Really any of the open source language models might work well enough for the job. If you could manage to get a classifier that runs with tensorflow to take advantage of a coral tpu it would certainly be a major step up with managable performance.
thesuitonym 3y ago
I hope against hope that AI spam detection never becomes a thing. At least with today's methods, I can tell a person why their message was marked as spam. If AI detection becomes the norm, all I can do is shrug and say, "Sorry, it's the algorithm."
- potatoman22 3y ago
  How do you think SpamAssassin/gmail/outlook created their spam rules?
- jeffbee 3y ago
  Gmail has used machine learning to classify spam since its creation. https://workspace.google.com/blog/identity-and-security/an-overview-of-gmails-spam-filters https://workspace.google.com/blog/identity-and-security/an-o...
  - Mailtemi 3y ago
    Gmail is successful because it naturally is the biggest honeypot. Most antispam API filters are like accumulators. When a trend is detected, the rest are protected. But overall, it's about scale.
- layer8 3y ago
  SpamAssassin has a Bayes filter that you can train with ham and spam. This has basically been a thing since forever.
  - andrewf 3y ago
    "A plan for Spam" (2002) - https://paulgraham.com/spam.html https://paulgraham.com/spam.html
El_RIDO 3y ago
Correct, Spamassassin will not classify emails be default. But it does include a Bayesian classifier and tooling that can be used to train it with curated ham and spam emails. It does require extra steps to set this up and feed it selected maildirs (for example, exclude inbox, spam and trash for training ham). https://spamassassin.apache.org/full/3.0.x/dist/doc/sa-learn.html#description https://spamassassin.apache.org/full/3.0.x/dist/doc/sa-learn... https://cwiki.apache.org/confluence/display/spamassassin/BayesInSpamAssassin https://cwiki.apache.org/confluence/display/spamassassin/Bay...
creeble 3y ago
I find it far more likely that AI will be used (indeed, is used already) to generate spam, rather than filter it.

throw0101d 3y ago

> being used to decide if something is spam or not Each rule has a score associated with it. By default a message needs to reach 5.0 to be marked as "spam": * https://spamassassin.apache.org/full/3.0.x/dist/doc/Mail_SpamAssassin_Conf.html#item_nn https://spamassassin.apache.org/full/3.0.x/dist/doc/Mail_Spa... The threshold is configurable. An header is added post-processing, e.g.: X-Spam-Status: Yes, score=21.6 required=4.0 […] * https://cwiki.apache.org/confluence/display/SPAMASSASSIN/X+Spam+Status https://cwiki.apache.org/confluence/display/SPAMASSASSIN/X+S... One can then choose what do to with this information (via procmail or Sieve). There is another header as well: > X-Spam-Level: This displays your spam level with asterisks, with one asterisk displayed per point, rounded down. For example, if your overall SpamAssassin score is 4.3, it will display ****. If you score less than 1, for example, 0.5, it will display nothing. * https://www.mailercheck.com/articles/spamassassin-score https://www.mailercheck.com/articles/spamassassin-score