7 ms·
Sending Emails to Myself
- efazati 5y agoI think emails is not really the best way to manage errors. Maybe something like https://sentry.io/ https://sentry.io/ works better
- akersten 5y agoThis is one dudes side projects, not an Enterprise K8n deployment - I think email is a great option. Of course when you go full "web scale" you should start using products like Sentry, but we're talking alternatives to "dump an error to stdout" here
- Minor49er 5y agoDevelopers can carry the practice into a company if they aren't careful. At my last job, some of the senior/architect developers had baked in code that would email them with various application statuses. They pointed the logs at their main email addresses for maximum visibility. But what ended up happening is that they let their inboxes get so flooded with email that they simply never bothered to check their email again, including any work-related messages. So to solve that problem, a couple of them just set up keyword filters to auto-forward inbox messages to yet another service that they would check on. Even now, there's some legacy system that emails all of the developers with some error messages. I think only two out of our 20+ development team even knows what they're for. Further, if you flood your email server, you can miss logs. And if you hand your project off to someone else, you'd have to figure out if you also want to hand over your email account, or if you want to point the logs to their email account. tl;dr: email is the wrong solution for logging
- voussoir 5y agoThanks for your comment and your experience. I agree that at a large scale it would be silly to receive individual emails for error messages. It would make more sense to have a dashboard with an aggregated view and statistics and everything. Piecing together a story or determining long-term performance by email would be no good. I would also dread the idea of multiple people logging into a single email account and triaging things without knowing who read what, or everyone getting their own copy of everything and not knowing what needs doing. But to know that my monthly backups are working or having trouble, this is working well for me so far!
- josephg 5y ago> Thanks for your comment and your experience. I agree that at a large scale it would be silly to receive individual emails for error messages. At small to medium scale, having a mailing list for the dev team which gets emailed when issues come up can be quite handy. It can't be your whole process - someone still needs to take responsibility for actually fixing problems. And you might need to aggressively rate limit it when errors happen. But for the occasional email it can work quite well. Its much easier than building a dashboard. Eg "[ops] Monthly backup process FAILED", "[ops] Warning: prod4 at 95% RAM usage"
- Minor49er 5y agoYou could just log to a file and then use logrotate to help manage the old logs, which should already be available on your system. Hopefully your application doesn't get so big that you need to rethink your logging solution.
- hnick 5y agoI had to turn Outlook rules off due to an issue temporarily. I was getting over 1500 notification emails a day that I usually just auto-delete. They just email entire departments and have us setup rules for things we don't care about, rather than manage proper lists so we can subscribe to things we do care about.
- jamesmishra 5y agoSentry is very easy to self-host[1] on a single node with Docker Compose. They make it a little tricky to self-host with multiple nodes, but I assume their SaaS product can scale indefinitely. I can't think of anything easier for error-tracking than Sentry, given its ability to automatically intercept exceptions in languages like Python. Sentry also has some automatic handling for stack traces, recording the state of Redis clusters and similar bits of infra, and redacting information that appears to be sensitive (e.g. such as database passwords). https://github.com/getsentry/onpremise https://github.com/getsentry/onpremise
- mr337 5y agoI was thinking the same thing! Any logging tool will help take the errors and group them or some kind of housekeeping to help vs 200 emails slamming an inbox.
- voussoir 5y agoIn my "my_operatornotify" file, which isn't published, I did test a batching mechanism that used a background thread to wait 60 seconds before sending the email, in case more with the same subject line came along. It worked but I decided to stop using it. Most of my programs send their complete logs after they finish, only the long-running daemon programs send individual warning emails right away. I got a laugh out of the YCDL explosion and fixed the issue, for now anyway. Thanks for the comments!
- 2bluesc 5y agoI wrote similar onfailure script[0] for failing systemd services that could be invoked by the `Onfailure` unit option[1]. [0] https://github.com/kylemanna/systemd-utils/tree/master/onfailure https://github.com/kylemanna/systemd-utils/tree/master/onfai... [1] https://www.freedesktop.org/software/systemd/man/systemd.unit.html https://www.freedesktop.org/software/systemd/man/systemd.uni...
- kodisha 5y agohttps://archive.is/uZVPV https://archive.is/uZVPV
- hanlec 5y agoIs the email notification sent synchronously? What happens if there's an error sending the email?
- voussoir 5y agoSynchronous vs asynchronous is a choice for the author of the my_operatornotify file. This module just hands the subject + body to your code. I did some experimenting with async sending and even batching in mine, but decided to remove that and stick with synchronous. The SMTP connection is really fast and most programs send the email at the end of runtime, so it doesn't matter anyway. I prefer the simplicity. I certainly did not want to get into managing an outbound spool, waiting for the server to come back, running a separate daemon for this... In my_operatornotify I use tenacity to do a couple of retries, then write the message and the fatal exception to a txt file on my desktop or ~/ if there is an error sending the email. I'll notice it eventually. The stuff I'm doing is not important enough to demand more retrying. Thanks for reading!
- Topgamer7 5y agoUsing loggers in python can be quite handy for a number of reasons. But similar to this you can log ship the data to DD, NR, ES, Loki, etc. And all you need to do is write a class to handle the deets.
- duiker101 5y agoVery nice write-up! I did something similar, but with Telegram instead of emails. Their API is so simple and easy. https://blog.hackertyper.net/post/creating-a-personal-notification-system-for-free/ https://blog.hackertyper.net/post/creating-a-personal-notifi...
- voiper1 5y agoI have things sending emails to myself, but sometimes I just want to mute them for a while. With this, I'd create a new bot for each project. Then I could manage the mute on each project individually with ease. Cool!
- newhotelowner 5y agoWow! this is nice. Took me less than 5 mins to create a bot and start sending message/notification. Thank you!
- avnigo 5y agoFor similar use cases, I go with Pushover [0], since its API [1] is pretty straightforward and built specifically for notifications/messages. [0]: https://pushover.net/ https://pushover.net/ [1]: https://pushover.net/api https://pushover.net/api
- jmartrican 5y agoI am going to copy this in my code. Great post. Since I do Java code, my guess is to use Spring AOP and create an aspect for this that has a point-cut on an annotation called something like @EmailOnError.
- EvanAnderson 5y agoI deal with a number of devices and applications that don't support SNMP traps for notification but do support email. The couple times I've been "mail-bombed" by a device sending hundreds, thousands, or more notifications in a short period of time have been frustrating. Has anybody heard of a rate-limiting SMTP proxy or MTA? I've thought about writing such a thing, but I'd gladly use something already-developed. I'm thinking of a simple token-bucket that uses to/from (or perhaps to/from/subject) to limit duplicate notifications and/or collapse duplicates in a manner similar to a syslog daemon logging "Last message repeated xxx times".
- Karrot_Kream 5y agoMost MTAs have rate-limiting out of the box. Postfix, Courier, and OpenSMTPD have rate-limiting in-built, and though I'm less familiar with Qmail, there's probably patches to make Qmail work with rate limiting.
- denton-scratch 5y agoYou don't need a proxy; Postfix can rate-limit natively. Plugins, "milters" and stuff inherited from Sendmail; and socket-level filters like Spamassassin, provide pretty-much comprehensive control. I agree that Postfix config is confusing. It beats the bejabers out of Sendmail config, but that's whataboutism. Dovecot config isn't that hard, unless you're trying to do hard stuff. Even then, your site config is an overlay on the default config, which should be what you get with your distro. The site-specific stuff is usually ony a couple of dozen lines. I expect you could also rig Postfix as a rate-limiting front-end for some inferior MTA :-) [Edit: oh - and Postfix/Dovecot doesn't need a machine to itself, if you're dealing with less than 20-or-so accounts, and you haven't set up a bot to spam the MTA. Actually, I think there are exactly zero circumstances in which a mailserver shouldn't be kicked-off the machine, in favour of a higher-priority job - email is a best-efforts, store-and-forward system]
- EvanAnderson 5y agoIt looks like PostFWD would probably do what I want. I haven't had occasion to configure Postfix for much more than a small office plain vanilla MTA w/ a smart host for outbound in thr last few years. I really need to spend some quality time doing oddball things with it.
- conor_f 5y agoThis seems like an ideal thing to integrate with Apprise https://github.com/caronc/apprise https://github.com/caronc/apprise
- l2g 5y agoThanks for the shoutout. As the dev of Apprise, I can tell you that it already supports emails. It also supports attachments too (https://github.com/caronc/apprise/wiki/Notify_email https://github.com/caronc/apprise/wiki/Notify_email). :) But none the less, it's still a great effort by OP. It's awesome when you solve a problem and even more rewarding when you can share your efforts!
- indymike 5y agoI once got hired to fix an application where the original dev heard "message queue" and implemented in using email boxes because... wait for it... because email has queues full of messages. So, instead of reaching for an actual message queue, everything was queued via emails, and worker tasks started by fetching an email message. About all I can say, is, it worked... sometimes slowly.. but it did work, and had the benefit of allowing you to inject stuff into the queue by sending in a properly formatted email.
- jiggawatts 5y agoYou laugh, but Active Directory supports SMTP as one of its transport options, because it allows occasionally-connected sites to participate in replication.
- voussoir 5y agoThis is something I can definitely see myself doing, even if I might regret it later. Now that I've got code for sending emails, it's only logical that I write code to pull them back down again and...
- kinow 5y agoI have seen that too in a previous job! It worked until there was a restart in the e-mail relay server, or something in the message (I think it was Java, sending multipart messages with some custom message format).
- bob1029 5y agoThinking strategically about the most reliable systems in our organization today, email is probably at the top of the list. I feel like there is a semi-auto magic thing email is still missing for a lot of businesses. Many things can be managed within the latency constraints of email. All of the hardest & most complex things almost certainly can be. For instance, on-boarding a new employee could be totally driven by some back-end service that simply sends & interprets emails all day to ensure that all of the required activities are executed. Now I am wondering if I can rewrite our B2B product interface to just be email. The more I think about this the more it starts to make a ridiculous amount of sense... User=>Server (Email 1): Subject: Get Workflow List Server=>User (Email 2): Subject: Workflow List [hash code] Body: <all workflows by name and id> User=>Server (Email 3): Subject: Start Workflow #21 Server=>User (Email 4): Subject: Workflow #12345532 Body: <First page of onboarding questions the user needs to fill out> User=>Server (Email 4 - reply): Subject: Re: Workflow #12345532 Body: <First page of questions with responses> Server=>User (Email 4 - reply): Subject: Workflow #12345532 Body: <Second page of questions, or error regarding first page responses> ... Server=>User (Email 4 - reply): Subject: Workflow #12345532 [COMPLETED] Body: <Completion summary with whatever final artifacts attached> I know our users would totally reject this proposal if it were made to be the only way to interact with our product, but if it were an additional way to interact, they might grow accustomed over time. Think about the technical advantages of the above. If you keep it plaintext, anyone with a computer made within the last 3 decades could access your business system unconditionally. Developer productivity is likely excellent. I can't imagine you would be able to get very distracted with plain text emails.
- tdeck 5y agoVery useful project! The great thing about email is that it's proactively pushed to all your devices, and the client keeps track of what you've already looked at. You're given powerful tools like labels and folders to organize the alerts and filter them as you see fit. Years ago I built a little REST API for sending emails to myself. The main advantage is that you don't have to bother setting up a mail server. I still use it constantly for personal projects. It's just the quickest way for me to make my webapp tell me about new user registrations, or for some random Google Apps Script I wrote to let me know about an unexpected condition, or for me to create a bash script that tells me when my VPS is running out of disk space. I've even written scripts to monitor election results or check websites for vaccine availability. I never did much to promote it, but anyone's welcome to use it as well (website: varmail.me). While I have had almost no downtime, I wouldn't rely on it for something that needs a particular SLA :). Edit: I forgot to mention another tiny feature that I use all the time. The API lets you specify an idempotence key that will prevent the email from being sent if it's already been used. It's very useful to be able to only send an email the first time a user takes a certain action, or to set it to time()/86400 and only get one disk space reminder email per day. I've found a lot of uses for that.
- tweetle_beetle 5y ago> I never did much to promote it, but anyone's welcome to use it as well (website: varmail.me). While I have had almost no downtime, I wouldn't rely on it for something that needs a particular SLA :). https://www.varmail.me https://www.varmail.me seems to be down? The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
- tdeck 5y agoThanks for pointing that out! It looks like the www. prefix wasn't set up correctly. It should be fixed now.
- akshayshah 5y agoThis is _such_ a great idea, especially for personal projects that don't need something as full-featured as Sentry. One thing to note is that it's easy to exceed your email provider's quotas with a system like this. For example, Gmail allows a maximum of 3600 emails per hour [1] before incoming mail bounces. [1]: https://support.google.com/a/answer/1366776?hl=en https://support.google.com/a/answer/1366776?hl=en
- funnybeam 5y agoNice work. In a similar vein I've considered using twtxt for this kind of thing, having each server/service generate a stream of messages that you can selectively subscribe to. Obviously there a other (and better) ways to do this but twtxt seems like a really easy option that could be just be used by default so I've always got something and if I decide I need better logging/monitoring for a particular thing then I can put that in alongside twtxt [https://pypi.org/project/twtxt/ https://pypi.org/project/twtxt/] not affiliated just thought it was a cool idea with potential for a lot more than just 'microblogging'
- mlk 5y agoI've done this multiple times, it is very useful but more than once an unexpected condition triggered an infinite loop of emails. Hundreds per second. If you rely on third party email service that can get you banned.