7 ms·
SMTP protocol basics from scratch in Go: receiving email from Gmail
- mraza007 5y agoInteresting article. I would love to implement something similar in python
- nonrandomstring 5y ago+1 Has anyone got a Python take on a bare bones SMTP mail server/relay and client because that would be super useful for teaching email concepts in a sandbox where full-blown Exim/Postfix etc us too clunky.
- svnpenn 5y ago> I've heard no few times how hard it is to send mail from a self-hosted server (because of spam filters). This is the second sentence. What does this even mean?
- vonwoodson 5y agoIt means that if you send email that isn’t from a gmail.com account, Google flags it as spam. (Alternatively if you send spam from a gmail account, it’s called “marketing” and Google will prioritize it for a small fee.)
- nojs 5y ago> Google will prioritize it for a small fee What do you mean by this?
- throwaway892238 5y agoPay for Workspace, your spam won't be flagged as spam (by Google) and you get raised sending limits. Limits are based on licenses, so the more you pay Google, the more you can spam, with a max of 4.6M unique addresses per 24H. If you send spam through Google, they will send your administrator an e-mail telling them that they are sending spam. If you send "a large volume" of this spam, they will stop relaying it until you promise not to send spam (or follow their guide to make your spam not look like spam).
- andris9 5y agoThe email user sending spam will get automatically suspended. The account admin can then un-suspend it though. From the TOS it seems to be an option for Google to permanently suspend such an email user. Not sure when that would happen though.
- HillRat 5y ago“Cold” IPs have lower trust scores, are more likely to end up on a blacklist, and are harder to get off blacklists, plus good luck finding a colo that will let you send bulk email in the first place. You can get away with sending low-volume programmatic, but that’s about it.
- klabb3 5y agoSo where do spammers send email from, since they clearly make it to my Gmail spam folder - sometimes even inbox? More generally: How does this weird trust system work? Is it just an inofficial, loosely connected club of large actors who trust each other enough to whitelist?
- throwawayboise 5y agoDepends on how you define spam. Services like Mailchimp will send mail that you and I may consider spam, but it gets delivered, because they have teams of people making sure they are following all the rules to the letter and they probably negotiated agreements with Google, Yahoo, Hotmail, etc to whitelist their servers.
- klabb3 5y agoThanks for the reply. What rules?
- mfincham 5y agoIt's basically like you described - an informally specified set of rules the various "big players" in e-mail enforce. There's some consensus about the things you "have" to do to participate in e-mail but it's generally not a black and white "do these things and you'll be fine" situation.
- CobrastanJorji 5y agoAs an example, mailing lists are likely to have a header like "list-unsubscribe: <mailto:list@host.com?subject=unsubscribe>", which enables email clients like offer easy unsubscribe buttons.
- brightly-salty 5y agoIt’s an odd construction but replacing “no few” with “not a few” (i.e. many) makes a lot more sense.
- patmorgan23 5y agoYeah it's just a clunky phrase. Saying "many" or "more than a few" is much clearer and easier to parse. "No few" is like ok do you mean one to none or many?
- don-code 5y agoMail self-hoster here. There's an additional wrinkle, which luckily you may never encounter in the public cloud space: in addition to typically being unable to _send_ mail from residential Internet connections, it's many times also not possible to _receive_ mail on a residential Internet connection. On the outbound side, many accepters of mail will deny mail from SMTP servers on a residential IP address, as provided by the Spamhaus Policy Block List (https://www.spamhaus.org/pbl/ https://www.spamhaus.org/pbl/) or equivalent. Unfortunately, there's not much one can do to get off this list - ISPs self-publish their own residential IP space onto it. In order to self-host from home, business-class service or higher is in order. Or, again, use the public cloud, and hope your compute instance's IP isn't on some other block list. On the inbound side, though, many ISPs block inbound port 25. Since that's the agreed-upon port for mail receipt, there's no real way around this. Several times while I self-hosted off residential, I was able to call Comcast support and ask for the ban to be lifted, but not before fighting through several layers of level one support asking me to configure my mail client differently. The real kicker, though, is any time my modem was rebooted (I owned my modem, too!), I had to call in and request it _again_. My (very small local) ISP in upstate New York also performed this blocking, but _would not_, in the face of any complaining, lift the block.
- throwawayboise 5y agoIn the ISP's defense, 99% of the time an SMTP sender in their residential address space is going to be a compromised device sending spam. It's pretty understandable that they would just decide that pissing off the relatively very few customers who want to run their own mail servers is a cost they are willing to accept.
- koolba 5y agoI bet it’s more than 99% too. Even amongst the tech savvy, there’s special level of dedication to run your own mail infrastructure. It’s like one in 10-25K.
- oefrha 5y agoThere’s running your own mail, then there’s running your own mail from residential. The dozens of us who do the former mostly don’t do the latter, thanks to the obvious vicious cycle.
- tedunangst 5y agonet/textproto has readers and writers that deal with crlf, and status codes, and dot encoding even.
- jeffbee 5y agoI guess it's "from scratch" though! Which I applaud for educational purposes. If you wanted more or less scratch-ness there are standard libraries for dot stuffing and for handling RFC 822 header parsing which is actually obscenely complicated. This program responds 250 OK to absolutely every command other than data which doesn't exactly make sense, so I assume it's a toy example. You don't send 250 to QUIT, for example. Nor to STARTTLS.
- hsbauauvhabzb 5y agoI’ve been super curious about how modern SMTP handles encryption - my understanding is SMTPS was dropped in favour of STARTTLS, with a HSTS-like directive hosted in DNS. Does the latter actually happen? Do smtp clients actively check this? What’s to stop some middleware performing a tls downgrade?
- jeffbee 5y agoYeah STARTTLS is fundamentally flawed that way. That's why Google and other SMTP interchange services allow paying customers (Google Workspace) to specify policy around TLS compliance. https://support.google.com/a/answer/2520500 https://support.google.com/a/answer/2520500
- hsbauauvhabzb 5y agoInteresting, any insight as to whether o365, and popular non saas servers do similar?
- 0x0 5y agoI wouldn't say STARTTLS is fundamentally flawed. Given a mechanism for declaring a policy of TLS being required, a middlebox downgrade attempt to deny STARTTLS should merely cause the sender to abort the SMTP transaction. This is no different from a middlebox hijacking a pure TLS tcp connection. In both cases it's on the sender to abort if a secure connection cannot be established.
- hsbauauvhabzb 5y agoDo real world smtp clients and servers do this? HSTS solves sslstrip type attacks, but I don’t think mail gets the same scrutiny..
- Symbiote 5y agoA couple of years ago, someone with a .gov address told me they couldn't subscribe to a mailing list I manage, since the mail server didn't support TLS. Adding the appropriate option in Postfix fixed the problem, and I think there were further options to require TLS.
- b112 5y agoThen there's this problem: http://139.177.194.177/bah.png http://139.177.194.177/bah.png
- xanaxagoras 5y ago> 177.194.177.139.in-addr.arpa domain name pointer systemd-the-porno.com. lol
- Animats 5y agoOne of my back burner projects was to write a mail forwarder which did not store messages. When someone opens an SMTP connection to it, it opens an SMTP connection to the destination and passes through the message, while holding the input connection open. When the message has been delivered, then the sender gets a completion status back. No need to return bounce messages. Any problem at any stage results in an immediate reject to the sender. You don't invisibly lose messages in transit. Outbound ISP SMTP servers should work that way. This makes email behave much like instant messaging. Messages with many recipients may need to take a store and forward path. They can be redirected to a slow bulk mail server. Historically, mail servers are store and forward because the destination might be offline, or on some store and forward UUCP system or something. Today, that's very rare. We don't have to emulate Sendmail forever.
- Geezus-42 5y agoWithout traditional NDRs, how do you tell which sever had a problem accepting the mail?
- cm2187 5y agoHow is that different from some sort of load balancer/reverse proxy?
- Jaruzel 5y ago> One of my back burner projects was to write a mail forwarder which did not store messages. I wrote one of these years ago, as a front-end to my Exchange server. At the time Exchange had this thing where it ALWAYS accepted messages as long as the domain part was an accepted domain, and then it would do a fuzzy match on the mailbox names and dump the mail anywhere it felt matched. So I had to have this SMTP front-end forwarder to weed out mis-addressed mail... which I eventually added SpamAssassin to as well. It ran for over 10 years without issue. What I never solved/wrote was the TLS stuff, as it always seemed to complicated to code from scratch.
- Animats 5y agoRight, that's the sort of thing I had in mind. Web site has addresses such as "employeename@example.com", and they forward to some mail server that actually knows if "employeename" is valid. Bogus names result in a status code, not an outgoing bounce message.
- Gys 5y agoWhat makes this specific for receiving from gmail? Why would this not receive email from any other email provider?
- capableweb 5y agoThe only gmail specific part seems to be about the body parsing. As different clients/services sends bodies in different formats, they are all using multipart, but some use multipart/alternative, some use multipart/mixed and so on. Seems author skipped that part ("In any case this looks like multipart bodies in HTTP. I don't want to deal with that so I'm just going to stop here."), so they tested receiving emails from just one specific client (gmail). Anything else would probably not render correctly, especially if they use other encodings which would make the message unreadable without extra parsing (think koi8-ru)
- jwilk 5y agoContent-Type is defined in RFC 2045 ("Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies"): https://datatracker.ietf.org/doc/html/rfc2045#section-5 https://datatracker.ietf.org/doc/html/rfc2045#section-5
- amelius 5y agoQuestion: is it possible to read an e-mail message while making the sender think you haven't received it?
- kevincox 5y agoSure, after you receive the message you can return an error. This is commonly done for messages that are immediately blocked as spam. When you return an error to the sender the expectation is that the message didn't end up in a mailbox and the sender will probably send the sending user a bounce notification.
- jamal-kumar 5y agoI've been trying to do this myself and it's surprisingly involved. Even for sending it isn't nearly as simple as it used to be like 5 years ago. You need to pass DKIM and MTA-STS specifically as two things most tutorials on 'sending email in golang' didn't tell me. I'm considering writing my own article on sending -deliverable- emails, because it's probably pretty important your stuff doesn't end up in spam.
- xanaxagoras 5y ago> I don't understand what zones are here. Cool article. I setup a wireguard tunnel back to a mail server hosted at home once, gotta have that PTR and my ISP isn't giving me one. I cobbled the instructions together from 3 or 4 different blogs to get exactly what I needed in my particular situation. There was all this stuff about source NATing which was like a foreign language to me and I was in "get it to work" mode so I just kind of blew past it. It seemed interesting and I got the gist of it but in the end I had a script full of iptables commands and I'm not really sure what they did, other than that they did what I wanted. I love that he mentions something like that here. There's a real temptation to sweep that kind of thing under the rug to puff yourself up and look more knowledgeable, and it certainly would have gone unnoticed by me if he chose to do that. Props.
- Anthony-G 5y agoThat sentence also jumped out for me. I also like the frankness of that sentence but what I really admire is the author’s gumption (courage) to continue with the project and not let such obstacles hinder progress. My problem is the opposite: I would have stopped and not continued until I knew what zones were – and any other part of the technology stack that might potentially have security consequences. With my (overly) cautions/conservative approach, I might still learn a lot but I don’t get the same sense of excitement – or accomplishment – as the author.
- mfashby 5y agoSomewhat related: https://maddy.email https://maddy.email and the libraries it's built on such as go-smtp https://github.com/emersion/go-smtp https://github.com/emersion/go-smtp
- koffiezet 5y agoAlso https://github.com/mailhog/MailHog https://github.com/mailhog/MailHog - although it hasn't been updated in a while. But great for testing mail functionality in applications you're developing.
- sylware 5y agoand it is well known you can create an account with noscript/basic (x)html browsers and use those very browsers to perform the periodic "re-authentication" of this account. (irony)