8 ms·
Giving away the secrets of 99.3% email delivery
- suking 15y agoHN is becoming 37Signals News.
- hello_moto 15y agoMarketing before the release of Basecamp.Next.
- effu 15y agoRegardless of whether HN has too many 37Signals posts, this is a valid point. HN is worth a thousand uniques an hour and startups would do well to study the success of 37Signals in writing popular content. I say this as someone who finds 37Signals very off putting!
- sakopov 15y agoI too find it rather annoying. I understand that the OP is one of their employees, but this is all he/she has been posting.
- omonra 15y agoCan anybody recommend a service that would let me check whether an email (sent via my or the service provider server) was picked up? They discuss this functionality in their system but I'm looking for someone I can outsource it to.
- themcgruff 15y agoIt's not worth outsourcing. We just add a tiny gif that has a unique hash. When that file is requested record the hash and mark the identified message as read.
- omonra 15y agoThanks - is there a library I can use? Wouldn't want to reinvent the wheel
- themcgruff 15y agoI'll talk with Noah about open sourcing it. Right now it would take a little bit of effort, but it's probably worth it for the good of the community.
- archangel_one 15y agoSeems like most e-mail clients these days don't load remote images automatically - do you find that significantly affects this technique?
- jeffclark 15y agoThe hashed image is the same method the big boys would use to record open rate.
- archangel_one 15y agoOk, but surely they have the same problem then?
- justincormack 15y agoDo not rely on it. As the other message says it is trivial to do, but it will underrecord by a lot. Measure by actions not opens.
- old-gregg 15y agoHaving a great delivery rates is actually a lot easier when you are big and famous, have your mail being signed by (and contain links to) a domain with excellent pagerank, have substantiation email volume, long emailing history and so on. But it gets much harder when you are running with a recently purchased domain on cold IPs or with the spammy subnet neighbors, your subnets are blocked by "know-it-all" small ESP admins. It is very similar to having a great credit history: you're getting approved for much nicer interest rates, there's no secret. So I don't believe an average person will get 37signals results simply by following Noah's advice. I work at Mailgun and we help startups and established companies get "37signals level" deliverability :) We also offer quite powerful parsing of incoming email into your app, so check out http://mailgun.net http://mailgun.net
- villagefool 15y agohttp://67signals.com/ http://67signals.com/
- guan 15y agoI’m sure their solution of tailing and parsing Postfix log files works well, but I believe the more typical and elegant solution to tracking bounces is to send every email with a with a unique envelope sender address that identifies that particular email, which makes it easier to collect bounces without relying on correct parsing of an MTA log file.
- MrEnigma 15y agoOther than the talk about tailing the logs, and using the three servers, there isn't much revealed to why their delivery rate is so high.
- pkteison 15y agoI'm disappointed with the hand-waving dismissal of "why should we pay someone tens of thousands of dollars to do it?" The list immediately after that of some of the headaches that getting e-mail delivered entails (monitoring and responding to blacklists, various configuration, feedback loops, etc.) is a very good argument for paying somebody to learn and handle the details. I don't see a convincing argument that getting 1% better delivery is worth spending time on instead of doing something else; indeed, they make the argument that improving validation and reporting on the app side is a much better use of time than fighting for that extra 1% on delivery.
- wlll 15y ago"why should we pay someone tens of thousands of dollars to do it?" Because it costs us less and we have better deliverability. Also, 1% of 50 million emails a month is a lot of undelivered mail.
- jonknee 15y agoDo you have better deliverability though? You said you use Campaign Monitor for newsletters and those are a different type of mail than transactional emails from an application (much more likely to be flagged as spam too!). Seems like two different things and would be hard to compare. Campaign Monitor isn't what you'd use for your apps, you want something like SendGrid.
- getsat 15y agoSendgrid + their whitelabel stuff (SPF/DKIM) = better than 99% delivery for over 100k emails/day. SG is probably the most pleasant email-related experience I've ever had.
- potyl 15y agoWe use SendGrid too at my company. It's really great and easy. Integrating with their SMTP server is really simple.
- 15y ago
- amix 15y agoGetting whitelisted and managing your mail server isn't easy and usually costs a lot. I think it's only something that should be optimized when you are sending millions of emails pr. month (like they are). For everybody else, you can save a lot of time and money by going for Amazon SES, Google App Engine, SendGrid or Postmark (etc.) A lot of these services also include analytics and monitoring and will be cheaper than rolling out your own customized solution (in terms of time and money). Even for them they would only pay around $6000-$7000 pr. month by using Amazon SES.
- themcgruff 15y agoFWIW we haven't done any "paid" whitelisting. That's too mafia-esque for us. SES requires each sending address to be verified upfront. For us that's an issue.
- suking 15y agoYou don't have to pay to get whitelisted with an ISP. You have to have good IP reputation then apply.
- culturestate 15y agoMany ISPs are turning to returnpath to manage their white lists, and that is indeed a daunting financial outlay.
- alphamale3000 15y agoCool. Then if you use their findings published in this blog post, they will publicly shame you and your investors.
- pork 15y agoWhile we all love services like Postmark and Mailgun on HN (and they are great services), does anyone know of any easy-to-setup open source projects that offer similar functionality? Traditional MTAs are a pain to setup and outdated to boot for webapps (mail parsing, signature and quote removal, UTF-8 transcoding, automatically updating spam detection, linkage to arbitrary storage handlers like MongoDB, and an HTTP+SSL JSON API should be a minimum).
- deleted 15y ago[deleted]
- heretohelp 15y agoI don't know if it provides precisely the functionality you're seeking, but Lamson was designed to address the need for an easier to setup, but still programmable and powerful mail system. It's designed to be able to use ORMs, databases, and data stores instead of text files as well. http://lamsonproject.org/ http://lamsonproject.org/ Example list of supported backends: Django's ORM web.py's simple database library Tokyo Tyrant Raw SQLite3 SQLObject CouchDB Mongo DB SQLAlchemy Part of the problem with this sort of thing is that in most minds, the modern MTA is qmail or postfix which is in my view just sendmail++. Let me know if this helps you, if not, tell me why so I can try to figure something out.
- pork 15y agoFantastic, thanks so much! It looks like it's almost there, and definitely something I will check out in more detail. Admittedly, my use case is limited, but as services like Mailgun and Sendgrid are demonstrating -- it is not an insignificant use case. Lamson looks like it's trying to achieve a bit more than that single case. So a couple of things I noticed that might be missing from Lamson: (1) Spam blocking auto-updates -- does this tap into Spamhaus, etc.? (2) UTF-8 transcoding? (3) Signature, quote stripping (like Mailgun). (4) DKIM signing? (5) Simple client libraries for sending mail (i.e., I don't want to build a "Lamson application", I just want to talk to a Lamson server on a privileged port using JSON). Of course, all these concerns might have answers that I'm missing. A bit more about my (fairly common, I'm guessing) use case: you have a VPS and a MongoDB instance. I want my MTA to take a minimal set of config parameters, say: -- MongoDB database name -- domain -> MongoDB table mapping -- privileged API sending port Now I want the MTA to receive mail for me, perform spam blocking, transcode to UTF-8, parse out signature and quotes in a separate field, and dump the whole object to a MongoDB document with the index field = the recipient's email address. I want it to auto-update it's spam blocking rules using whatever external services and internal analysis necessary. I want it to periodically dump some usage statistics into a separate MongoDB table (maybe a circular connection). Now for sending mail, all I should have to do is connect to the privileged port from a process on the same machine (it's firewalled to the outside world), and submit an HTTP POST request that specifies recipients, message body, attachments, etc. The MTA should accept the request, perform whatever queuing, rate-limiting, and retrying is necessary to send the message. All failures and diagnostic information are dumped to separate MongoDB tables. The applications deal with the database/storage system directly, to keep the MTA simple. I'm glossing over a lot, naturally, but I hope that helps.
- mhartl 15y agoBy far the biggest cause of failed email delivery we see is due to bad email addresses that were entered in to the system—problems like ‘joe@gmal.com’ or ‘sue@yahooo.com’. I wonder if you could hack up an 80% solution by comparing the submitted domain to common email domains, and giving a warning if it looks like a misspelling. That way, 'joe@gmal.com' would see a second step in the signup process asking him to double-check his email address. Any idea what percentage of misspellings that would catch?
- alexknowshtml 15y agoCheck out how we use Postmark bounce hooks (http://developer.postmarkapp.com/developer-bounces.html http://developer.postmarkapp.com/developer-bounces.html) to catch these problems in Beanstalk: http://blog.beanstalkapp.com/post/758755557/handling-email-delivery-issues-with-postmark http://blog.beanstalkapp.com/post/758755557/handling-email-d...
- mike-cardwell 15y agoFWIW, gmal.com has neither MX, nor A records so a couple of quick DNS lookups could have confirmed that the email address was incorrect.
- alexchamberlain 15y agoWould some sort of distance score be more efficient though?
- DanBC 15y agoWhat happens if the user makes a typo but enters a valid email address at some other domain?
- mike-cardwell 15y agoThe email goes to that address at that other domain
- ralph 15y ago
- datums 15y agoNo secrets here, these are basics. Since your mail is mostly transactional your volume is significantly low compared to people who specialize in mailing. Transactional email is less likely to produce spam complaints. Misspelled addresses are detected on first sends, which is what you want. You're going to get an error at the smtp level or a bounce. Here's a secret - Monitor the ip to domain ratios , usually gmail will allow 1k of mail from the same ip per hour.
- rogerbinns 15y agoAlso not mentioned is monitoring if your users actually read the emails. A web bug will work (some of the time) as will checks to see if users ever respond (eg if there is a link to click for full details) or login to the site after email receipt. LinkedIn noticed that I never read one of the group messages I was getting and so switched to a far more infrequent digest of highlights. I think they may even have unsubcribed me completely from one group.
- joering1 15y agore: Linkedin: I found it hard to believe. I receive the same emails from them for years and most do not open /straight delete. and they never stop.
- rogerbinns 15y agoThey are indeed inconsistent. They did the throttling for me for a few groups while others I never read keep on going. At some point I need to figure out how to unsubscribe, but it is currently quicker to hit delete (unread).
- whalesalad 15y agoObviously when you're at the scale of 37Signals... you're gonna roll this stuff yourself. But! I don't want people here to get discouraged about using 3rd party services. I too am guilty of "NIH syndrome" or the need to reinvent the wheel all the time. I've setup mail infrastructure. It's not pretty, and it's actually not necessary any more. I've been using Postmark for a while now and it's been fantastic. They provide a great admin interface with insight into what is being sent and why mail might not be getting delivered. The setup process is very simple (none of the downsides 37signals mentions... ) you essentially add some SPF and DKIM DNS entries and you're off to the races.
- alexknowshtml 15y agoSo glad you're happy with Postmark and thanks for the kudos - the entire team really appreciates it. 37S is doing a great job with all of their techniques. Of course, they also have an incredible sending volume on their side that helps improve the impact of any of these techniques. One of the biggest benefits of our approach at Postmark is that you get those same benefits regardless of your send volume: http://blog.postmarkapp.com/post/14127210172/the-false-promises-of-dedicated-ips http://blog.postmarkapp.com/post/14127210172/the-false-promi...
- SteveBrewer 15y agoPostmark is magical. I've used them for several projects - emails just work. Really excited to try out their new in-bound features.
- jbarham 15y agoPostmark's inbound email mail parsing feature is very nice. I used it last week to set up automated order shipped confirmation emails for an e-commerce site we built. I just set up an auto forward to the Postmark inbound address, and they turn around and pass a nicely parsed JSON representation of the email to a URL on our web server. See http://developer.postmarkapp.com/developer-inbound.html http://developer.postmarkapp.com/developer-inbound.html for details.
- 15y ago
- leeoniya 15y agoi recently finished a script that ran our 35k customer records through DNS/MX/SMTP servers and flagged all invalid addresses. sadly any properties that use yahoo servers or sympatico.ca, bell.ca, always return "OK" and you physically need to bounce an email to verify it. many of our customers have @yahoo addresses so we still have about 25% that simply cannot be validated without a "probe" message :(
- kooshball 15y ago> sadly any properties that use yahoo servers or sympatico.ca, bell.ca, always return "OK" and you physically need to bounce an email to verify it This is standard technique used to prevent spammer from mining the whole userbase.
- leeoniya 15y agoyes, i realize this. still doesn't help the situation much :(
- gokhan 15y agoWhat are those options for feedback loops? Hotmail got one [1], but Gmail and Yahoo don't, AFAIK. [1] https://support.msn.com/eform.aspx?productKey=edfsjmrpp&ct=eformts&st=1&wfxredirect=1 https://support.msn.com/eform.aspx?productKey=edfsjmrpp&...
- moe 15y agoAs anyone with half a clue about e-mail delivery will tell you: SMTP accept != delivery. Almost every spamfilter in the world will accept your e-mail and then silently discard it. The 0.7% instant rejects that you see are merely the tip of the ice-berg. Thus, the only secret you gave away is that you don't seem to understand how spamfilters work.
- snowwrestler 15y agoAmen. Without a robust delivery testing program, there's no way to know whether emails are actually landing in the inbox of actual users. 37signals has an easier time of it because the vast, vast majority of their emails are transactional, so users will pipe up and complain if they are not getting them. Marketing emails are a lot harder to troubleshoot. And before the masses get up in arms about spam, when I say "marketing" I am talking about even the most compliant, user-friendly, double-opt-in email campaigns. Even if people happily sign up for your emails, that doesn't mean they will tell you if they stop receiving them.
- ceejayoz 15y ago> Even if people happily sign up for your emails, that doesn't mean they will tell you if they stop receiving them. Plus, when they decide they don't want it six months later, they'll use "Report Spam" to stop them from showing up in the inbox.
- themcgruff 15y agoYou comment reminds me that we left out a biggie: We include a "stop sending me these messages" link in almost every email we send. The link actually works too.
- ceol 15y agoI think ceejayoz was saying some users are lazy or don't remember signing up for your service, so they'll click the "Report as Spam" button as a quick way to "unsubscribe".