5 ms·

Google Apps Script for Gmail to filter and sort email from Gitlab

63 points by mattst88 5y ago

rrdharan 5y ago
See also https://github.com/ranmocy/gmail-automata https://github.com/ranmocy/gmail-automata for a more generalized variant.
- mattst88 5y ago
  Thanks, that looks impressive. It doesn't look like it supports filtering on headers -- at least I don't see any calls to GmailMessage.getHeader(), and the only call to GmailMessage.getRawContent() appears to be in retrieving the ListID header (which Gmail's search natively supports). Looks like there's an open feature request for generalized header matching: https://github.com/ranmocy/gmail-automata/issues/36 https://github.com/ranmocy/gmail-automata/issues/36 I noticed that in the article I referenced (https://mjasion.pl/label-gitlab-notifications/ https://mjasion.pl/label-gitlab-notifications/) they also don't mention using getHeader and instead use getRawContent. It makes me wonder if the getHeader API was added more recently than this article and the project you mention.
- stevekemp 5y ago
  That's pretty cool, thanks for sharing. I wrote a simple scripting utility to do similar labeling actions, bacause I came to gsuite with a large procmail configuration file previously. It was useful for a while, but gradually I stopped updating the filters: https://github.com/skx/labeller https://github.com/skx/labeller Nice to see there are other people doing similar things though, I'll definitely take a look at this more carefully over the next few days.
agentdrtran 5y ago
Apps Script is amazing and a sorely under-appreciated feature in Workspace.
- Edman274 5y ago
  I built a Google Apps Script Gmail add-on (with a UI and card view for a selected email) for archiving email contents as a PDF to an internal enterprise document system using GAS and AWS API Gateway. Here are two extremely annoying problems that I really can't fix using this platform: 1. Calls to UrlFetchApp.fetchAll() will fail with an "Unexpected Error" exception about every 3 to 8 percent of calls. The way my code works is that I'll make one GET request to an AWS API Gateway endpoint to request a pre-signed S3 URL, and then I'll use the pre-signed S3 url to PUT a document. The unauthorized first request will fail sometimes, and issuing a PUT to the pre-signed S3 url will fail sometimes too. I have found no reason that these requests should fail, so instead of torturing myself by trying to figure out the root cause, I've decided to wrap both fetchAll() calls in a loop that retries the exact same request three times before giving up. If there was anything wrong with the request itself, one would expect a retry loop to be pointless, but it's significantly reduced the number of add-on failures. 2. If an email has invalid SPF alignment, then the add-on won't run at all. It's been three years and there's still no workaround for this. I have had to explain to dozens of people over the past three years that there's nothing I can do about it. https://issuetracker.google.com/issues/112064778 https://issuetracker.google.com/issues/112064778 . I feel like Google Apps Script - especially Gmail add-ons - are sorely neglected and are treated as an annoying legacy product that will probably be discontinued very soon with no replacement.
  - saagarjha 5y ago
    Yeah, I found that UrlFetchApp.fetch had similarly weird issues–I got a lot of sporadic “Address unavailable” errors out of it.
  - agentdrtran 5y ago
    Yes, it's incredibly annoying how little attention it seems to get from google. Bugs sit in the bug tracker for years and while I like the redesign it broke a function I used (to get the current URL so create multi-page apps). Apps Script just got a new UX update not long ago which I took as a good sign but App Maker had a roadmap going out a year when it got killed, so.
- 5y ago
dataflow 5y ago
More interesting than GmailApp, DriveApp, etc. are the Gmail, Drive etc. classes. They're under the "advanced" services, and far more powerful (and faster). Highly recommend them for more serious uses of Google Apps Script.
- mattst88 5y ago
  Interesting, thanks. I'll check that out.
  - dataflow 5y ago
    Yup, np. More tips: - For more real-time filtering you might also want to look into integrating with Google's Pub/Sub service (see Gmail.newWatchRequest() and Gmail.Users.watch()). It's kind of a pain to set up but it can help you avoid having to wait 10 minutes for everything to run, though you'll still want to have a polling mechanism as a fallback, and you might need to periodically call the watch methods to ensure you keep getting notifications pushed to you. - If you need even better performance you can try manually batching up the HTTP requests via googleapis.com/batch (you can use ScriptApp.getOAuthToken() to get the auth token). This will let you factor out some of the server overhead across messages.
gbrindisi 5y ago
I've used https://github.com/mbrt/gmailctl/ https://github.com/mbrt/gmailctl/ to tame my inbox with some degrees of success, but gmail filters are just so flaky. Next step in my things-to-try list is to just download the inbox via imap and sort it out locally.
- dataflow 5y ago
  > gmail filters are just so flaky Are you referring to actual Gmail filters, or Google Apps Scripts that attempt to mimic Gmail filters? I don't recall seeing the former be flaky, but the latter is another story.
- mkdirp 5y ago
  It isn't entirely clear to me how gmailctl authenticates against the gmail API, could you shed some light on it?
known 5y ago
anshumankmr 5y ago
Very cool. I have been using AppScript at my work as well to sync data from a sheet to Firebase and it works quite well.
londons_explore 5y ago
As far as I'm aware, there is no way to trigger an apps script script to run when an email arrives. That leads to polling, which ends up with a lackluster user experience. I don't want a random 30 second delay added to all my mail delivery.
- heresathinf99 5y ago
navbaker 5y ago
I was placed on some sort of spam list immediately following a PayPal dispute with a seller and found this thread that helped me tame the spam bombing I’ve experienced ever since: https://webapps.stackexchange.com/questions/89119/how-to-filter-messages-with-animated-emoji-in-the-subject-in-gmail https://webapps.stackexchange.com/questions/89119/how-to-fil... The majority of the spam came in with emojis in the from or subject line, so this was immediately helpful for that, then I just add a rule whenever I notice certain phrases popping up more often. I’d be delighted if someone can tell me I’m wrong, but it seems like Gmail filters don’t have built in options for stuff like emojis or regular expressions. Very disappointing that I have to brute force script my own filters.
- mateo1 5y ago
  Beware that even "serious" companies are now starting to include emojis in the subject line. I had a flight cancellation notification email come up with plane emojis in the title... I almost overlooked it as some kind of quasi-spam marketing email (the titles was something like "[plane emojis] about your flight [plane emojis]"). Perhaps it was the intention since it included information about EU regulated refunds.

63 points by mattst88 5y ago

See also https://github.com/ranmocy/gmail-automata https://github.com/ranmocy/gmail-automata for a more generalized variant.

mattst88 5y ago
Thanks, that looks impressive. It doesn't look like it supports filtering on headers -- at least I don't see any calls to GmailMessage.getHeader(), and the only call to GmailMessage.getRawContent() appears to be in retrieving the ListID header (which Gmail's search natively supports). Looks like there's an open feature request for generalized header matching: https://github.com/ranmocy/gmail-automata/issues/36 https://github.com/ranmocy/gmail-automata/issues/36 I noticed that in the article I referenced (https://mjasion.pl/label-gitlab-notifications/ https://mjasion.pl/label-gitlab-notifications/) they also don't mention using getHeader and instead use getRawContent. It makes me wonder if the getHeader API was added more recently than this article and the project you mention.
stevekemp 5y ago
That's pretty cool, thanks for sharing. I wrote a simple scripting utility to do similar labeling actions, bacause I came to gsuite with a large procmail configuration file previously. It was useful for a while, but gradually I stopped updating the filters: https://github.com/skx/labeller https://github.com/skx/labeller Nice to see there are other people doing similar things though, I'll definitely take a look at this more carefully over the next few days.

agentdrtran 5y ago

Apps Script is amazing and a sorely under-appreciated feature in Workspace.

Edman274 5y ago
I built a Google Apps Script Gmail add-on (with a UI and card view for a selected email) for archiving email contents as a PDF to an internal enterprise document system using GAS and AWS API Gateway. Here are two extremely annoying problems that I really can't fix using this platform: 1. Calls to UrlFetchApp.fetchAll() will fail with an "Unexpected Error" exception about every 3 to 8 percent of calls. The way my code works is that I'll make one GET request to an AWS API Gateway endpoint to request a pre-signed S3 URL, and then I'll use the pre-signed S3 url to PUT a document. The unauthorized first request will fail sometimes, and issuing a PUT to the pre-signed S3 url will fail sometimes too. I have found no reason that these requests should fail, so instead of torturing myself by trying to figure out the root cause, I've decided to wrap both fetchAll() calls in a loop that retries the exact same request three times before giving up. If there was anything wrong with the request itself, one would expect a retry loop to be pointless, but it's significantly reduced the number of add-on failures. 2. If an email has invalid SPF alignment, then the add-on won't run at all. It's been three years and there's still no workaround for this. I have had to explain to dozens of people over the past three years that there's nothing I can do about it. https://issuetracker.google.com/issues/112064778 https://issuetracker.google.com/issues/112064778 . I feel like Google Apps Script - especially Gmail add-ons - are sorely neglected and are treated as an annoying legacy product that will probably be discontinued very soon with no replacement.
- saagarjha 5y ago
  Yeah, I found that UrlFetchApp.fetch had similarly weird issues–I got a lot of sporadic “Address unavailable” errors out of it.
- agentdrtran 5y ago
  Yes, it's incredibly annoying how little attention it seems to get from google. Bugs sit in the bug tracker for years and while I like the redesign it broke a function I used (to get the current URL so create multi-page apps). Apps Script just got a new UX update not long ago which I took as a good sign but App Maker had a roadmap going out a year when it got killed, so.
5y ago

dataflow 5y ago

More interesting than GmailApp, DriveApp, etc. are the Gmail, Drive etc. classes. They're under the "advanced" services, and far more powerful (and faster). Highly recommend them for more serious uses of Google Apps Script.

mattst88 5y ago
Interesting, thanks. I'll check that out.
- dataflow 5y ago
  Yup, np. More tips: - For more real-time filtering you might also want to look into integrating with Google's Pub/Sub service (see Gmail.newWatchRequest() and Gmail.Users.watch()). It's kind of a pain to set up but it can help you avoid having to wait 10 minutes for everything to run, though you'll still want to have a polling mechanism as a fallback, and you might need to periodically call the watch methods to ensure you keep getting notifications pushed to you. - If you need even better performance you can try manually batching up the HTTP requests via googleapis.com/batch (you can use ScriptApp.getOAuthToken() to get the auth token). This will let you factor out some of the server overhead across messages.

gbrindisi 5y ago

I've used https://github.com/mbrt/gmailctl/ https://github.com/mbrt/gmailctl/ to tame my inbox with some degrees of success, but gmail filters are just so flaky. Next step in my things-to-try list is to just download the inbox via imap and sort it out locally.

dataflow 5y ago
> gmail filters are just so flaky Are you referring to actual Gmail filters, or Google Apps Scripts that attempt to mimic Gmail filters? I don't recall seeing the former be flaky, but the latter is another story.
mkdirp 5y ago
It isn't entirely clear to me how gmailctl authenticates against the gmail API, could you shed some light on it?

known 5y ago

anshumankmr 5y ago

Very cool. I have been using AppScript at my work as well to sync data from a sheet to Firebase and it works quite well.

londons_explore 5y ago

As far as I'm aware, there is no way to trigger an apps script script to run when an email arrives. That leads to polling, which ends up with a lackluster user experience. I don't want a random 30 second delay added to all my mail delivery.

heresathinf99 5y ago

navbaker 5y ago

I was placed on some sort of spam list immediately following a PayPal dispute with a seller and found this thread that helped me tame the spam bombing I’ve experienced ever since: https://webapps.stackexchange.com/questions/89119/how-to-filter-messages-with-animated-emoji-in-the-subject-in-gmail https://webapps.stackexchange.com/questions/89119/how-to-fil... The majority of the spam came in with emojis in the from or subject line, so this was immediately helpful for that, then I just add a rule whenever I notice certain phrases popping up more often. I’d be delighted if someone can tell me I’m wrong, but it seems like Gmail filters don’t have built in options for stuff like emojis or regular expressions. Very disappointing that I have to brute force script my own filters.

mateo1 5y ago
Beware that even "serious" companies are now starting to include emojis in the subject line. I had a flight cancellation notification email come up with plane emojis in the title... I almost overlooked it as some kind of quasi-spam marketing email (the titles was something like "[plane emojis] about your flight [plane emojis]"). Perhaps it was the intention since it included information about EU regulated refunds.