5 ms·

Gmailctl – Declarative Configuration for Gmail Filters

155 points by tbodt 7y ago

woodruffw 7y ago
This is interesting; what benefit/advantage does it have over IMAP-level email management tools like imapfilter[1]? I don't ask that to be confrontational; I've successfully managed several email inboxes on several different providers for years with plain old IMAP rules, so I'd like to know if there's anything I'm missing. [1]: https://github.com/lefcha/imapfilter https://github.com/lefcha/imapfilter
- kccqzy 7y ago
  You offload all the processing to the gmail backend? No compute needed on your part?
mehrdadn 7y ago
I made something like this a while back for myself, and it worked all right, but I ended up never even using it myself. The main problems I had were: - You end up with multiple sources of truth for what's happening to your email. And/or you won't be able to easily edit your filters from multiple machines. - Either the source code or the generated code ends up being hard to read. - There's imperative (and sometimes counterintuitive) behavior in Gmail regarding labels, and that can interfere with declarative reasoning. - Filters have a size limit! So you might need to take that into account if you have complicated filters.
- saulrh 7y ago
  `gmailctl` resolves a few of these, actually. The big one is that there's a command to diff your config file with what's in gmail, so you can capture any accidental changes that happen in gmail-land. This also helps with version-controlling your configuration, since it shows you when you just forgot to upload your config. The imperative behavior and filter size limits are still annoying, granted. You can deal with it by staging through multiple labels, but it's not great.
  - mehrdadn 7y ago
    Oh cool! Version control is useful. I might give it a shot sometime. I didn't have anything of the sort. If you're curious, the tool I came up with had a DSL like this: $excom = from:example.com $exorg = from:(example.org OR example.net) if ($excom AND $exorg) AND -(to:me OR in:spam) archive if cc:person label "Person" end end and it'd auto-translate to this: <?xml version='1.0' encoding='UTF-8'?> <feed xmlns='http://www.w3.org/2005/Atom' xmlns:apps='http://schemas.google.com/apps/2006'> <entry> <category term='filter'></category> <apps:property name='hasTheWord' value='(from:example.com from:{example.org example.net}) -{to:me in:spam}'/> <apps:property name='shouldArchive' value='true'/> </entry> <entry> <category term='filter'></category> <apps:property name='hasTheWord' value='(from:example.com from:{example.org example.net}) -{to:me in:spam} cc:person'/> <apps:property name='label' value='Person'/> </entry> </feed> which it would translate back to this: if (from:example.com from:{example.org example.net}) -{to:me in:spam} archive end if (from:example.com from:{example.org example.net}) -{to:me in:spam} cc:person label "Person" end It didn't treat the filter as a blob (it actually parsed it and removed quotes etc. if appropriate) but it also didn't really change the underlying language either. But it had variable and did let you nest conditions which made it easier to write some things. (Or so I hoped. In the end I didn't feel it was worth it...)
    - tbodt 7y ago
      Another advantage of gmailctl is it uploads the filters using the API, so you don't have open Gmail settings and manually import an XML file.
- inakarmacoma 7y ago
  You can technically get around the filter limit by switching from filters to Appsv Script. Setup a staging inbox, then run the query and action rules against that stagig area every 1 to 5 minutes.
  - mehrdadn 7y ago
    I've considered that for a long time, but the trouble is it's not as robust, not to mention there's also a script time limit which I would hit as a result of having other scripts...
stevekemp 7y ago
That's a pretty awesome project! I wrote something to automate label manipulation too, but I used a simple scripting language. I migrated from self-hosted mail where I had the power of procmail, to gsuite, where I had less flexibility. Making simple filters via the google UI is easy, but it gets complicated when you want to look at both the subject and the sender - to file "Facebook comment replies", "Facebook messages", and "Facebook event invitations" under different labels for example. It's a pain to authenticate, but once that's done I like my solution. Downside is you need to run it under cron: https://github.com/skx/labeller/ https://github.com/skx/labeller/ The appeal of this gmailctl is that it keeps everything native, and doesn't require a server to run a cronjob. Nice work.
rv-de 7y ago
F### GMail.
tex0 7y ago
Nice tool. Pretty popular at my workplace.
toomuchtodo 7y ago
If the author comes along, are there any plans to support FastMail for those of us who don’t use Gmail? I’d be willing to pay a bounty.
- mikelward 7y ago
  Fastmail already supports Sieve, so I'm not sure you'd need something like this. https://www.fastmail.com/help/technical/sieve-howto.html https://www.fastmail.com/help/technical/sieve-howto.html
  - toomuchtodo 7y ago
    TIL! Thank you kind stranger.
mikelward 7y ago
I use something like this that a Googler shared with me. It's super helpful to get around the way Gmail doesn't really let you order rules. This helps you generate mutually exclusive rules, so the ordering doesn't matter. https://github.com/mbrt/gmailctl/blob/master/README.md#chain-filtering https://github.com/mbrt/gmailctl/blob/master/README.md#chain...
techntoke 7y ago
The real benefit Gmail has over most other email solutions, including Outlook, is labels for emails which are like virtual mailboxes. Almost all IMAP-based solutions I've seen don't attempt to solve this problem, which is why people keep using Gmail. If something like this was automated through Dovecot, people would be much more likely to host there own email.
mwarkentin 7y ago
I wonder if a terraform provider for gmail would make sense.
- Znafon 7y ago
  I have one that I've been using for some time now: https://github.com/remilapeyre/terraform-provider-gmail https://github.com/remilapeyre/terraform-provider-gmail It should work without issues, I need to document it though. Let me know if you have some troubles using it.
  - ndrpnt 7y ago
    Oh well, I pretty much did the same thing at the same time! Albeit with some tests/docs :D https://github.com/ndrpnt/terraform-provider-gmail https://github.com/ndrpnt/terraform-provider-gmail
xg15 7y ago
Huge respect to the author for ealizing this - however, I'm honestly sceptical of including yet another turing-complete configuration DSL without sufficient debugging capabilities. Seems to me, now you've added several more sources of complexity: - You have to get the JSONnet syntax right and learn about th various functionalities and expressions. - You have to ensure Jsonnet is correctly translated into the GMail filter chain. - You still have to ensure the GMail filter chain works correctly - this is one of the original problems this project set out to solve, but I don't understand what additional tools the project gives you to archieve this.
- mbrt 7y ago
  Author here. I understand your concerns, and in fact one of the last things I added is unit tests: https://github.com/mbrt/gmailctl/blob/master/README.md#tests https://github.com/mbrt/gmailctl/blob/master/README.md#tests Getting things right is not easy, especially in complex languages like jsonnet. In my own experience though, every time you limit the expressivity of config languages you end up inventing even more complex workarounds in the end (e.g. external templating and things like that).

155 points by tbodt 7y ago

This is interesting; what benefit/advantage does it have over IMAP-level email management tools like imapfilter[1]? I don't ask that to be confrontational; I've successfully managed several email inboxes on several different providers for years with plain old IMAP rules, so I'd like to know if there's anything I'm missing. [1]: https://github.com/lefcha/imapfilter https://github.com/lefcha/imapfilter

kccqzy 7y ago
You offload all the processing to the gmail backend? No compute needed on your part?

mehrdadn 7y ago

I made something like this a while back for myself, and it worked all right, but I ended up never even using it myself. The main problems I had were: - You end up with multiple sources of truth for what's happening to your email. And/or you won't be able to easily edit your filters from multiple machines. - Either the source code or the generated code ends up being hard to read. - There's imperative (and sometimes counterintuitive) behavior in Gmail regarding labels, and that can interfere with declarative reasoning. - Filters have a size limit! So you might need to take that into account if you have complicated filters.

saulrh 7y ago
`gmailctl` resolves a few of these, actually. The big one is that there's a command to diff your config file with what's in gmail, so you can capture any accidental changes that happen in gmail-land. This also helps with version-controlling your configuration, since it shows you when you just forgot to upload your config. The imperative behavior and filter size limits are still annoying, granted. You can deal with it by staging through multiple labels, but it's not great.
- mehrdadn 7y ago
  Oh cool! Version control is useful. I might give it a shot sometime. I didn't have anything of the sort. If you're curious, the tool I came up with had a DSL like this: $excom = from:example.com $exorg = from:(example.org OR example.net) if ($excom AND $exorg) AND -(to:me OR in:spam) archive if cc:person label "Person" end end and it'd auto-translate to this: <?xml version='1.0' encoding='UTF-8'?> <feed xmlns='http://www.w3.org/2005/Atom' xmlns:apps='http://schemas.google.com/apps/2006'> <entry> <category term='filter'></category> <apps:property name='hasTheWord' value='(from:example.com from:{example.org example.net}) -{to:me in:spam}'/> <apps:property name='shouldArchive' value='true'/> </entry> <entry> <category term='filter'></category> <apps:property name='hasTheWord' value='(from:example.com from:{example.org example.net}) -{to:me in:spam} cc:person'/> <apps:property name='label' value='Person'/> </entry> </feed> which it would translate back to this: if (from:example.com from:{example.org example.net}) -{to:me in:spam} archive end if (from:example.com from:{example.org example.net}) -{to:me in:spam} cc:person label "Person" end It didn't treat the filter as a blob (it actually parsed it and removed quotes etc. if appropriate) but it also didn't really change the underlying language either. But it had variable and did let you nest conditions which made it easier to write some things. (Or so I hoped. In the end I didn't feel it was worth it...)
  - tbodt 7y ago
    Another advantage of gmailctl is it uploads the filters using the API, so you don't have open Gmail settings and manually import an XML file.
inakarmacoma 7y ago
You can technically get around the filter limit by switching from filters to Appsv Script. Setup a staging inbox, then run the query and action rules against that stagig area every 1 to 5 minutes.
- mehrdadn 7y ago
  I've considered that for a long time, but the trouble is it's not as robust, not to mention there's also a script time limit which I would hit as a result of having other scripts...

stevekemp 7y ago

That's a pretty awesome project! I wrote something to automate label manipulation too, but I used a simple scripting language. I migrated from self-hosted mail where I had the power of procmail, to gsuite, where I had less flexibility. Making simple filters via the google UI is easy, but it gets complicated when you want to look at both the subject and the sender - to file "Facebook comment replies", "Facebook messages", and "Facebook event invitations" under different labels for example. It's a pain to authenticate, but once that's done I like my solution. Downside is you need to run it under cron: https://github.com/skx/labeller/ https://github.com/skx/labeller/ The appeal of this gmailctl is that it keeps everything native, and doesn't require a server to run a cronjob. Nice work.

rv-de 7y ago

F### GMail.

tex0 7y ago

Nice tool. Pretty popular at my workplace.

toomuchtodo 7y ago

If the author comes along, are there any plans to support FastMail for those of us who don’t use Gmail? I’d be willing to pay a bounty.

mikelward 7y ago
Fastmail already supports Sieve, so I'm not sure you'd need something like this. https://www.fastmail.com/help/technical/sieve-howto.html https://www.fastmail.com/help/technical/sieve-howto.html
- toomuchtodo 7y ago
  TIL! Thank you kind stranger.

mikelward 7y ago

I use something like this that a Googler shared with me. It's super helpful to get around the way Gmail doesn't really let you order rules. This helps you generate mutually exclusive rules, so the ordering doesn't matter. https://github.com/mbrt/gmailctl/blob/master/README.md#chain-filtering https://github.com/mbrt/gmailctl/blob/master/README.md#chain...

techntoke 7y ago

The real benefit Gmail has over most other email solutions, including Outlook, is labels for emails which are like virtual mailboxes. Almost all IMAP-based solutions I've seen don't attempt to solve this problem, which is why people keep using Gmail. If something like this was automated through Dovecot, people would be much more likely to host there own email.

mwarkentin 7y ago

I wonder if a terraform provider for gmail would make sense.

Znafon 7y ago
I have one that I've been using for some time now: https://github.com/remilapeyre/terraform-provider-gmail https://github.com/remilapeyre/terraform-provider-gmail It should work without issues, I need to document it though. Let me know if you have some troubles using it.
- ndrpnt 7y ago
  Oh well, I pretty much did the same thing at the same time! Albeit with some tests/docs :D https://github.com/ndrpnt/terraform-provider-gmail https://github.com/ndrpnt/terraform-provider-gmail

xg15 7y ago

Huge respect to the author for ealizing this - however, I'm honestly sceptical of including yet another turing-complete configuration DSL without sufficient debugging capabilities. Seems to me, now you've added several more sources of complexity: - You have to get the JSONnet syntax right and learn about th various functionalities and expressions. - You have to ensure Jsonnet is correctly translated into the GMail filter chain. - You still have to ensure the GMail filter chain works correctly - this is one of the original problems this project set out to solve, but I don't understand what additional tools the project gives you to archieve this.

mbrt 7y ago
Author here. I understand your concerns, and in fact one of the last things I added is unit tests: https://github.com/mbrt/gmailctl/blob/master/README.md#tests https://github.com/mbrt/gmailctl/blob/master/README.md#tests Getting things right is not easy, especially in complex languages like jsonnet. In my own experience though, every time you limit the expressivity of config languages you end up inventing even more complex workarounds in the end (e.g. external templating and things like that).