6 ms·

Sieve: An Email Filtering Language (RFC 5228)

48 points by stefankuehnel 3y ago

isaacisaac 3y ago
I work at Nylas (Gmail/MS/IMAP email & calendar connectivity APIs) on the product side and I really wish something like this received adoption. Between the insufficiencies of IMAP for server-side use and the disparities between the "winners" of the space, the lowest common denominator of what's supported for email is abysmally small.
grawlinson 3y ago
I don’t know what I would do without sieve, I use it to manage my mailing list subscriptions, eg move to folders, mark emails from specific chatty users as read, etc. It automates a lot of manual cruft and I’m endlessly thankful for it. I do wish it was more widely supported, and I try to do my part to encourage adoption (I maintain several sieve related packages for Arch Linux).
- ts4z 3y ago
  This is very gratifying to read. I had a lot to do with the original spec, and I'm glad you find it useful.
koito17 3y ago
In case people are curious on using Sieve, the official(?) site[0] offers a list of libraries implementing the language in C, Java, Ruby, etc. There is also a list of e-mail clients[1] that seem to support Sieve as a filtering language. (I didn't know Roundcube supported this!) --- [0] http://sieve.info/libraries http://sieve.info/libraries [1] http://sieve.info/clients http://sieve.info/clients
- tredre3 3y ago
  I don't think roundcube implements sieve filtering, it's just a client to managesieve. A pretty good one!
Tempest1981 3y ago
Has anyone written a tool to scan your email headers, and maybe show UI to help craft Sieve rules? Sort by top senders or mailing-lists, and create/extend rules to file them appropriately.
fotta 3y ago
Fastmail supports Sieve rules which is really nice because I can do all my filtering server-side https://www.fastmail.help/hc/en-us/articles/360060591373-How-to-use-Sieve https://www.fastmail.help/hc/en-us/articles/360060591373-How...
- sliken 3y ago
  As does Dovecot's delivery agent.
- aidenn0 3y ago
  I'll have to look into that; right now I use isync and run maildrop on everything in my inbox; upside is it works for all my imap accounts; downside is that there is a delay, causing other clients to sometimes see unfiltered mail.
- inkyoto 3y ago
  Except Sieve rules in Fastmail either work or do not, and it is non-deterministic. A few notoriously persistent problems I have been facing with them for years: – A notorious spammer that has been spamming me for over a decade that the Fastmail spam filter does not trap[*]. They are easy to screen out via Sieve rules using the "From" header. Except, for 1 out 5 spam emails from them the Sieve rules do not fire, and the spam gets into the inbox. – Simple email filing into different folders using From/Subject headers. It mostly works except for just a handful senders whose emails Fastmail Sieve do not fire upon no matter how I modify them. They just. do. not. work. I am talking about the most basic From/Subject filing rules. [*] Spam filtering in Fastmail has been another pet peeve of mine. It is average at best, and mediocre on average. Some spam emails that slip through need to be sent into the spam folder (that has the training enabled) for more than 10x times before*
  - nmjenkins 3y ago
    (I work for Fastmail). That sounds very surprising. Please email support@fastmail.com with the details and we'll be happy to look into it for you. I don't believe we've ever had an instance of Sieve not working correctly. (We have had plenty of reports that boiled down to people making errors in their Sieve scripts; we recommend most people use our UI instead to make their rules, to allow you to preview results and help ensure the syntax is correct.)
mynegation 3y ago
I am Fastmail customer so I have access to their Sieve functionality but decided against it. For me the fundamental limitation is that Sieve looks at email only once, when email arrives. I wanted to do the things like: archive email after 3 days being in my inbox, or older than my most recent read email (which would mean I scanned the inbox and the summary was enough) etc. I ended up implementing a Python script that manages my inbox connecting to IMAP server on cron schedule. It also sends me telegram notifications from follow ups on starred threads.
- Groxx 3y ago
  Yeah, "keep sales newsletters for two weeks, then delete forever" has been a thing I've wanted for a long time. Nothing seems to make that easy. I've been leaning towards local processing on an mbox folder... that would also let me build up multiple Bayesian filters to try categorizing my other mail - why does almost everyone only auto-categorize spam? Google kinda broke out of that with their "social" and "important" semi-categories, but it'd be pretty easy to build in local clients, yet I've never seen it.
  - mjl- 3y ago
    Interesting indeed. What kind of classification would you have in mind? Separate out different types of senders, e.g. friend groups, or professional clients? Or more the personal email, transactional email, newsletter? For bayesian junk filtering to work, you train with messages marked either as junk or nonjunk. I suppose you could train based on positive tag, and assume the message has the opposite signal if it didn't get that tag. My historic email is only classified for (non)junk. If I would start more classifications, I would have to ignore the existing messages. Fyi, I'm working on https://github.com/mjl-/mox https://github.com/mjl-/mox, which also includes a bayesian filter. This sounds implementable.
    - Groxx 3y ago
      Gmail's categories are a pretty reasonable starting place: primary, social, updates, forums, promotions. All reasonably identifiable by word choices, and often at extremely different urgency / priority levels. But I would personally also have an "interesting" category for newsletters and whatnot that are more likely to get a full read (newsblur has a weak version of this for my RSS, it's nice). And an "urgent" category that notifies more visibly than others. "Recruiters" is also periodically useful - a bunch of cold emails filled with buzzwords, should be identifiable a mile away. But underneath it all it's mostly that it confuses me that there's so little ability to experiment here. Shared always-updating spam filters makes sense, but there's a lot of similar things people could try unique to their needs that are just totally unsupported. I don't want better global categorization (gmail), I want personalization. Email is weirdly non-personal in the vast majority of systems, and I suspect it's part of why people burn out on it and switch to a never-ending churn of apps that are a better fit for one piece or another.
a-dub 3y ago
this was part of cyrus imap from cmu, which iirc was the first big imap server to support server side indexes. (i think they were bdbs, iirc) believe it or not, but many hugely popular applications from 20+ years ago didn't even implement indexing. in this case if you had too much mail your imap client would just time out. well through the mid-2000s you even had to use a third party plugin for outlook called "lookout" if you wanted searches to not take minutes.
- ts4z 3y ago
  FWIW they were not Berkley DBs. IIRC the only db was the duplicate suppression feature. Folders (mailboxes in proper IMAP lingo) had hand-built indexes. Good stuff. Credit to jgm, the original author.
  - a-dub 3y ago
    that sounds right. and yes, very good stuff! i remember thinking at the time "hah. this is built with real computer science!". and well, at the time, it was the only thing that worked for large mail spools. i also think it was the first support for sasl(?) encryption upgrades for legacy text/tcp mail protocols also, fun sidebar: indices vs. indexes, both are apparently valid english... but it seems computer people have adopted the latter almost exclusively. never noticed it before...
    - ts4z 3y ago
      SASL came out as a generalization of the IMAP AUTHENTICATE mechanism. CMU wanted Kerberos to work and it had been done as something of a one-off in telnet, and initially in IMAP. There were a couple companion protocols proposed for IMAP (stuff like contacts) that the same group was working on, and they wanted to leverage the same mechanism. From there, might as well do the same thing for POP and SMTP, etc. So they started working on a library (which Cyrus IMAP didn’t use for at least a while, sorry my fault). Kind of funny that SASL is the most durable piece of the effort. (I doubt this is entirely accurate. I wasn’t there for a lot of it.)
nirui 3y ago
I always want to have a programmable mailserver, but not just for filtering. I hope it can also do things like calling API then react according to the respond etc. It's like a eBPF program but for mailservers. I'm not sure Sieve is a language that enables such usage. I know maybe it's a domain-specific language, but since it's a language capable of programming the server, I would rather expect more from it than "it's just yet another way to configure your mail filter".
- mjl- 3y ago
  Sieve seems not powerful enough. I had this though experiment a few days ago: What if you could run code in a sandbox on the server (javascript, webassembly?) with api's (provided by the server) to access mailboxes/messages and to take actions (move to folder, change message flags), and perhaps some networking (e.g. dns blocklist)?
mediumsmart 3y ago
I use procmail, fetchmail and nmh via mh-e. Never used imap and the only email service account I have is with lavabit. Each email is one file so the only filtering is which folder the file goes to. Procmail does that and mairix finds them. I just have to write and read, maybe answer or delete.
jmprspret 3y ago
Proton mail supports multiple filters through a GUI only if you pay them - otherwise you only get a single email rule. What you can do though is edit your rule as a seive filter and just add multiple rules in there and it works fine.

48 points by stefankuehnel 3y ago

I work at Nylas (Gmail/MS/IMAP email & calendar connectivity APIs) on the product side and I really wish something like this received adoption. Between the insufficiencies of IMAP for server-side use and the disparities between the "winners" of the space, the lowest common denominator of what's supported for email is abysmally small.

grawlinson 3y ago

I don’t know what I would do without sieve, I use it to manage my mailing list subscriptions, eg move to folders, mark emails from specific chatty users as read, etc. It automates a lot of manual cruft and I’m endlessly thankful for it. I do wish it was more widely supported, and I try to do my part to encourage adoption (I maintain several sieve related packages for Arch Linux).

ts4z 3y ago
This is very gratifying to read. I had a lot to do with the original spec, and I'm glad you find it useful.

koito17 3y ago

In case people are curious on using Sieve, the official(?) site[0] offers a list of libraries implementing the language in C, Java, Ruby, etc. There is also a list of e-mail clients[1] that seem to support Sieve as a filtering language. (I didn't know Roundcube supported this!) --- [0] http://sieve.info/libraries http://sieve.info/libraries [1] http://sieve.info/clients http://sieve.info/clients

tredre3 3y ago
I don't think roundcube implements sieve filtering, it's just a client to managesieve. A pretty good one!

Tempest1981 3y ago

Has anyone written a tool to scan your email headers, and maybe show UI to help craft Sieve rules? Sort by top senders or mailing-lists, and create/extend rules to file them appropriately.

fotta 3y ago

Fastmail supports Sieve rules which is really nice because I can do all my filtering server-side https://www.fastmail.help/hc/en-us/articles/360060591373-How-to-use-Sieve https://www.fastmail.help/hc/en-us/articles/360060591373-How...

sliken 3y ago
As does Dovecot's delivery agent.
aidenn0 3y ago
I'll have to look into that; right now I use isync and run maildrop on everything in my inbox; upside is it works for all my imap accounts; downside is that there is a delay, causing other clients to sometimes see unfiltered mail.
inkyoto 3y ago
Except Sieve rules in Fastmail either work or do not, and it is non-deterministic. A few notoriously persistent problems I have been facing with them for years: – A notorious spammer that has been spamming me for over a decade that the Fastmail spam filter does not trap[*]. They are easy to screen out via Sieve rules using the "From" header. Except, for 1 out 5 spam emails from them the Sieve rules do not fire, and the spam gets into the inbox. – Simple email filing into different folders using From/Subject headers. It mostly works except for just a handful senders whose emails Fastmail Sieve do not fire upon no matter how I modify them. They just. do. not. work. I am talking about the most basic From/Subject filing rules. [*] Spam filtering in Fastmail has been another pet peeve of mine. It is average at best, and mediocre on average. Some spam emails that slip through need to be sent into the spam folder (that has the training enabled) for more than 10x times before*
- nmjenkins 3y ago
  (I work for Fastmail). That sounds very surprising. Please email support@fastmail.com with the details and we'll be happy to look into it for you. I don't believe we've ever had an instance of Sieve not working correctly. (We have had plenty of reports that boiled down to people making errors in their Sieve scripts; we recommend most people use our UI instead to make their rules, to allow you to preview results and help ensure the syntax is correct.)

mynegation 3y ago

I am Fastmail customer so I have access to their Sieve functionality but decided against it. For me the fundamental limitation is that Sieve looks at email only once, when email arrives. I wanted to do the things like: archive email after 3 days being in my inbox, or older than my most recent read email (which would mean I scanned the inbox and the summary was enough) etc. I ended up implementing a Python script that manages my inbox connecting to IMAP server on cron schedule. It also sends me telegram notifications from follow ups on starred threads.

Groxx 3y ago
Yeah, "keep sales newsletters for two weeks, then delete forever" has been a thing I've wanted for a long time. Nothing seems to make that easy. I've been leaning towards local processing on an mbox folder... that would also let me build up multiple Bayesian filters to try categorizing my other mail - why does almost everyone only auto-categorize spam? Google kinda broke out of that with their "social" and "important" semi-categories, but it'd be pretty easy to build in local clients, yet I've never seen it.
- mjl- 3y ago
  Interesting indeed. What kind of classification would you have in mind? Separate out different types of senders, e.g. friend groups, or professional clients? Or more the personal email, transactional email, newsletter? For bayesian junk filtering to work, you train with messages marked either as junk or nonjunk. I suppose you could train based on positive tag, and assume the message has the opposite signal if it didn't get that tag. My historic email is only classified for (non)junk. If I would start more classifications, I would have to ignore the existing messages. Fyi, I'm working on https://github.com/mjl-/mox https://github.com/mjl-/mox, which also includes a bayesian filter. This sounds implementable.
  - Groxx 3y ago
    Gmail's categories are a pretty reasonable starting place: primary, social, updates, forums, promotions. All reasonably identifiable by word choices, and often at extremely different urgency / priority levels. But I would personally also have an "interesting" category for newsletters and whatnot that are more likely to get a full read (newsblur has a weak version of this for my RSS, it's nice). And an "urgent" category that notifies more visibly than others. "Recruiters" is also periodically useful - a bunch of cold emails filled with buzzwords, should be identifiable a mile away. But underneath it all it's mostly that it confuses me that there's so little ability to experiment here. Shared always-updating spam filters makes sense, but there's a lot of similar things people could try unique to their needs that are just totally unsupported. I don't want better global categorization (gmail), I want personalization. Email is weirdly non-personal in the vast majority of systems, and I suspect it's part of why people burn out on it and switch to a never-ending churn of apps that are a better fit for one piece or another.

a-dub 3y ago

this was part of cyrus imap from cmu, which iirc was the first big imap server to support server side indexes. (i think they were bdbs, iirc) believe it or not, but many hugely popular applications from 20+ years ago didn't even implement indexing. in this case if you had too much mail your imap client would just time out. well through the mid-2000s you even had to use a third party plugin for outlook called "lookout" if you wanted searches to not take minutes.

ts4z 3y ago
FWIW they were not Berkley DBs. IIRC the only db was the duplicate suppression feature. Folders (mailboxes in proper IMAP lingo) had hand-built indexes. Good stuff. Credit to jgm, the original author.
- a-dub 3y ago
  that sounds right. and yes, very good stuff! i remember thinking at the time "hah. this is built with real computer science!". and well, at the time, it was the only thing that worked for large mail spools. i also think it was the first support for sasl(?) encryption upgrades for legacy text/tcp mail protocols also, fun sidebar: indices vs. indexes, both are apparently valid english... but it seems computer people have adopted the latter almost exclusively. never noticed it before...
  - ts4z 3y ago
    SASL came out as a generalization of the IMAP AUTHENTICATE mechanism. CMU wanted Kerberos to work and it had been done as something of a one-off in telnet, and initially in IMAP. There were a couple companion protocols proposed for IMAP (stuff like contacts) that the same group was working on, and they wanted to leverage the same mechanism. From there, might as well do the same thing for POP and SMTP, etc. So they started working on a library (which Cyrus IMAP didn’t use for at least a while, sorry my fault). Kind of funny that SASL is the most durable piece of the effort. (I doubt this is entirely accurate. I wasn’t there for a lot of it.)

nirui 3y ago

I always want to have a programmable mailserver, but not just for filtering. I hope it can also do things like calling API then react according to the respond etc. It's like a eBPF program but for mailservers. I'm not sure Sieve is a language that enables such usage. I know maybe it's a domain-specific language, but since it's a language capable of programming the server, I would rather expect more from it than "it's just yet another way to configure your mail filter".

mjl- 3y ago
Sieve seems not powerful enough. I had this though experiment a few days ago: What if you could run code in a sandbox on the server (javascript, webassembly?) with api's (provided by the server) to access mailboxes/messages and to take actions (move to folder, change message flags), and perhaps some networking (e.g. dns blocklist)?

mediumsmart 3y ago

I use procmail, fetchmail and nmh via mh-e. Never used imap and the only email service account I have is with lavabit. Each email is one file so the only filtering is which folder the file goes to. Procmail does that and mairix finds them. I just have to write and read, maybe answer or delete.

jmprspret 3y ago

Proton mail supports multiple filters through a GUI only if you pay them - otherwise you only get a single email rule. What you can do though is edit your rule as a seive filter and just add multiple rules in there and it works fine.