11 ms·
Mining my mailbox for top email service providers
- simias 7y agoI must admit that I was a bit shocked when I realized that the author had used javascript for this task. At first I assumed that he was scripting some webmail interface but then I saw that he was using 'emailjs-imap-client'. I understand that people use what they're familiar with and there seems to be a vibrant ecosystem for JS outside of the browser but it'll never not be weird seeing it take over what would once have been written in Perl or Python, especially since for a long time all I heard about Javascript was "well yeah it sucks but it's not like I have a choice if I want to script in the browser". Software is eating the world, and the web is eating software.
- volgar1x 7y agowho cares, it's just a 100-line long script that probably won't be maintained or used anymore
- jaynetics 7y agoIt would be more readable in almost any other widespread language, and I'm saying that as someone who does a lot of JS.
- BossingAround 7y agoThis is quickly changing. Anecdotal evidence, of course, but I seem to come across a lot more scripts online in Node for some reason. And by 'scripts' I mean reproducers to various issues, such as triggering error state in an app I was working on. At first, I thought it was crazy. It kinda makes sense though, JS seems to be growing extremely fast.
- AnIdiotOnTheNet 7y agoJust because more people are using it does not mean it is more readable. Footguns about in JS and it is remarkably easy to read the same code over and over again and not see them because of it. Readable languages are the kinds where not only do you understand what they're trying to do at a quick glance, but also what they actually do.
- kerkeslager 7y agoAs someone who has worked in Javascript basically since Javascript was invented, it's been "quickly changing" that entire time, and the majority of the changes have made things worse, not better. It used to be that you could trace your code through 15 layers of callback hell and eventually find the problem. Now there's 10 different syntactic sugars around callbacks, so you get to save a few characters of typing and then trace your bugs through 50 layers of callback hell, most of which are in some 0.0.1-alpha-versioned promise library which may actually just be a XSS attack. Improvement? I think not. Just because more people are doing it, doesn't mean it's improving. I'll believe things are going in a good direction when `{}.foo` throws an exception in all major JS implementations. But I'm not optimistic about that ever happening. My only hope for JavaScript right now is that it might die because WebAssembly lets a reasonable language achieve dominance.
- crooked-v 7y ago> I'll believe things are going in a good direction when `{}.foo` throws an exception in all major JS implementations. But I'm not optimistic about that ever happening. It does. That's a SyntaxError.
- kerkeslager 7y agoOkay, please make a little effort to understand meaning rather than taking things completely literally. Try one of the following: > ({}).foo undefined > var a = {}; undefined > a.foo undefined Now imagine hundreds of lines of code and a bunch of asynchronous callbacks occur between the last two lines, and imagine what that does for your debugging experience. Or watch Wat[1] and see if any of that is what you want your language to do when it happens. The fact that JavaScript throws an exception on `{ foo: 'bar' }.foo` and not `({}).foo` isn't exactly a defense of the language. [1] https://www.destroyallsoftware.com/talks/wat https://www.destroyallsoftware.com/talks/wat
- fireattack 7y agoThe biggest problem I have with JS in term of readability is IIFE. I know what it does, but it just looks ugly when you have hundreds of them in a program. The "semicolon in front" is even worse, needless to say. There gotta be a better way to design syntax for an async language.. right?
- layoutIfNeeded 7y agoWait, why do you need IIFE in post-ES6 JS? Just use let/const.
- fireattack 7y agoWell, OP's example is using it, but I think it's for top level async/await (not a JS dev here)?
- kerkeslager 7y agoI'm not sure I could count the number of 100-line long throwaway scripts that I wrote that are still being used in production systems before I realized how dangerous that idea is and stopped treating any code as throwaway code. If I show my code to another human, there's a chance it will end up in production. The Pragmatic Programmer says to throw away prototypes, but in practice it always seems easier to click the merge button than to rewrite working code.
- lonelappde 7y agoReverse-survivor bias. You aren't seeing the non-existence all the well engineered stuff you never had time to build "properly".
- kerkeslager 7y agoSo? If only 1 in 10 throwaway scripts survives, that can easily cost more in the long run than writing all 10 scripts in a sustainable way. The open secret is that you almost always have time to do it properly. Doing it properly doesn't take that much more time, and most deadlines are self-imposed and artificial. EDIT: Time isn't actually the most important cost. Not my code, but a coworker's throwaway script where he thought, "We'll just log this to a file for now" filled up a production DB server with log entries and took down a high-traffic e-commerce site for 6 hours. That easily cost more money than an entire career's worth of "doing it properly". There were obviously multiple mistakes that allowed that to happen, but removing the "we'll do this properly later" mistake would have prevented it.
- an_ko 7y agoIt sounds like the problem was really about failing to document technical debt. > Doing it properly doesn't take that much more time If you don't have a cutting board for a small thing, surely you wouldn't put your cooking on hold to drive out to the kitchen supply shop to buy a cutting board. You'd just use a stack of cardboard or something. (And make a note to buy a proper cutting board later, of course.)
- andrewzah 7y agoTemporary code is permanent code in many, many cases.
- quonn 7y agoModern JavaScript/TypeScript is fine. Especially when compared to Perl, of all languages.
- AnIdiotOnTheNet 7y agoI disagree. JavaScript is one of those languages who's default failure mode seems to be to pretend everything worked and just produce nonsensical results.
- pxndxx 7y agoYou're thinking of PHP. Modern Javascript has good error handling and is a pretty nice language to use.
- mfer 7y agoModern PHP has good error handling, too. There are differences in style and views on things from the way they were long ago that float around. These impact a lot of opinions.
- tjoff 7y agoI know that PHP gets a lot of flak but come on. Compared to javascript PHP is a marvel. I recommend you to try it.
- patrec 7y agoAs opposed to perl? :D
- lonelappde 7y agoModern Perl is fine too
- bloopernova 7y agoI haven't used Perl in about 12 years, has it changed much since then? Do you have any links to good examples/comparisons?
- sashavingardt2 7y agoAs as a hardcore Perler and JS eng of 20 years - yes, Perl or Python would've been the best tool for the job. Perl in particular due to its regex readability.
- clSTophEjUdRanu 7y agoIsn't JavaScript much faster than Python these days?
- speedgoose 7y agoWas Python faster than javascript a long time ago? I'm not super old and javascript has always been way faster in my mind, but it's fine. Python has many great libraries for doing heavy computations very quickly.
- speedgoose 7y agoJavascript is quite neat today. I play a lot with map/reduce and async/await and it's great.
- isatty 7y agoIt always annoys me when some general purpose utility or script is written in JavaScript because I refuse to install node on my computer.
- 4cao 7y agoWhat would have been a shell script in the 1980s, a Perl script in the 1990s, a Python script in the 2000s, is now a JS bundled with an outdated Chromium build aka an Electron app.
- jethro_tell 7y agowant to parse the plain text 'From: ' email headers on your laptop? Install this 500MB bundle.
- pedrogpimenta 7y agoI'm curious, why do you refuse to install node on your computer?
- layoutIfNeeded 7y agoMaybe he hates bloat?
- JosephRedfern 7y agoI started (but quickly got bored of) writing a Python version here: https://gist.github.com/JosephRedfern/79314fdf1875166e08489746029851c2 https://gist.github.com/JosephRedfern/79314fdf1875166e084897... if anyone wants to pick it up.
- deleted 7y ago[deleted]
- bfishadow 7y agoThat’s really nice analysis. Mailchimp is the biggest one as I guessed. Don’t know if you count the spam folder since some marketing emails could be automatically sent there.
- frereubu 7y agoNot quite sure how you got that from the graphs. Top three: Sendgrid: 30.2% SES: 16.8% Mailchimp: 16.8%
- dane-pgp 7y agoSeeing Amazon's SES with such a large share is no surprise, but it is a little disheartening knowing about this limitation: "Amazon SES only supports open tracking over HTTP domains." https://docs.aws.amazon.com/ses/latest/DeveloperGuide/configure-custom-open-click-domains.html https://docs.aws.amazon.com/ses/latest/DeveloperGuide/config... Yes, big webmail providers fetch the tracking pixels for you and serve them to your browser over HTTPS, but not every mail provider does that, sometimes for good reason, leading to problems like this: https://protonmail.com/support/knowledge-base/connection-not-secure-warning/ https://protonmail.com/support/knowledge-base/connection-not...
- amirathi 7y agoDeliverability is a big marketing/selling point for all these providers & important selection criteria for their customers. So it'd be very interesting to see what % of emails for each provider landed in Inbox vs Promotions/Spam/Junk folder.
- michaelbuckbee 7y agoAs someone currently dealing with this on multiple fronts: it's complicated. ESP's block out different ranges for different customers, dedicated vs shared IPs, etc. really hard to generalize.
- npsomaratna 7y agoShameless plug: we had the same issue with our deliverability rates. We ended up creating a tool to automate the process of checking spam % vs. inbox % for various email subject/body combinations: https://vetter.monsoonyeti.com/ https://vetter.monsoonyeti.com/
- thedance 7y agoInteresting, but it’s hard to see the big utility. Most spam signals are based on behavior and reputation rather that content. Your email can look clean at first, but if a handful of gmail accounts mark it as spam then your whole network will be on the bozo list within seconds. Gmail in particular is capable of reclassifying messages, if it gets delivered but then your reputation falls, it will get moved to spam if the user hasn’t seen it yet, so initial deliverability is not final. Maybe the other big operators are more content-based?
- npsomaratna 7y agoFrom repeated testing using multiple sender accounts and domains, we've noticed that Outlook seems to pay significant weight to the content of the email - possibly as much as or even more than the email/domain of the sender. What you say certainly rings true about Gmail - email subjects/bodies that set off warning bells in Outlook seemingly pass through Gmail without a hitch if your account has a good sender reputation - and vice-versa. However, we've also noticed that even if you have a good sender reputation, Gmail can and will (automatically) spam your content if the wording is too spammy/phishy. Yahoo seems to be midway between Outlook and Gmail, at least from what I've observed. Anyway, do give the tool a spin and try out various types of emails. I'm sure you'll find the results interesting! [EDIT] Oh, BTW - you can set the tool to use your own service as well! Obviously, this will be more accurate than using the default (which is an SES account we use).
- bognition 7y agoVery interesting thanks for sharing. initially I was surprised that HubSpot didn’t make the list, until I realized that there is a nice intersection here with what you observed and the target businesses of the Email services. For example MailChimp Targets small B2C companies while Hubspot targets small B2B business. I’m guessing the op isn’t a small B2B business hence the lack of hubspot in the list.
- TacticalTable 7y agoIt's interesting that this is missing a lot of major enterprise marketing email providers. The author clearly isn't subscribing to things like Best Buy's weekly deal email, or Target cart reminders.
- runlevel1 7y agoA few years back, one of our (Twilio SendGrid's) developer evangelists created an open source Chrome extension to show the icon of the ESP who sent a message in Gmail. The last big Gmail UI update broke it, but most of the ESP identification strategies probably still work: https://github.com/nquinlan/Email-Intelligence/blob/master/content.js https://github.com/nquinlan/Email-Intelligence/blob/master/c...
- kehers 7y agoThis is nice. Shows a different approach to getting the ESP from the mail header.
- pingec 7y agoThis is a brilliant idea, if only it were updated for the latest gmail UI
- mbrameld 7y agohttps://www.inboxsdk.com/ https://www.inboxsdk.com/ lets you pretty easily manipulate the Gmail UI if you want to have a crack at it.
- alooPotato 7y agoWe (Streak) made the InboxSDK. Great for projects like this because we autoupdate so you can write the extension against our high level api and not have to worry about gmail changes going forward. Let me know if we can help.
- nquinlan 7y ago
- thih9 7y agoNitpicking: in the final chart with merged providers, the 'mandrillapp.com' section and 'mailchimp' section are still shown as separate. Is this intentional? I assumed the opposite, given that: > Mandrillapp.com is the transactional email service for Mailchimp. It used to be a standalone service until it became deeply integrated into Mailchimp.
- dredmorbius 7y agoThis website presents as a 23-chacter-wide skinny column in mobile vew.
- lettergram 7y agoThere’s some really interesting things you can pull out of your email. Having done some analysis myself it’s actually fairly easy to export data from Gmail to do an analysis. This is from a few years ago, but I bet it’ll still do the trick: https://austingwalters.com/analyzing-email-data/ https://austingwalters.com/analyzing-email-data/ Everything is in python: https://github.com/lettergram/Email_Analysis https://github.com/lettergram/Email_Analysis Regardless, I encourage everyone to do an analysis of their emails. Cut out the junk, set reminders on when to check, etc. it’s both enlightening and potentially horrifying to discover trends.
- saagarjha 7y agoSemi-related: emails can often give away more information than you might intend. At one of my previous employers, it was quite easy to get a general idea of how internal development was progressing “from the outside” by keeping track of who was testing which version of a prerelease OS via proxy of the information it would leave in the emails they sent.
- technion 7y agoI've identified companies as connected to each other by enumerating domains on an azure tenancy.
- hellofunk 7y agoWow those pie charts are almost impossible to read! I've never seen such a small font in a chart that readers are expected to understand easily.
- hombre_fatal 7y ago[Nitpick] Instead of this: ;(async function () { try { // All your code } catch (e) { console.log(e) } })() Just do this: ;(async function () { // All your code })().catch(console.error) Overuse of try/catch is probably the #1 mistake I see when people use async/await.
- throwanem 7y agoThe point of async/await is to enable asynchronous code to participate in the same patterns synchronous code does, instead of being coerced into a promise-style interface. Using try/catch is part of that, and it's good practice here. If you're not going to take advantage of it, why bother with async/await at all, especially when using it at top level requires a workaround anyway?
- mrweasel 7y agoEither way, I find the syntax of both to be extremely ugly. Arguably this may simply be me not getting JavaScript. Starting a line with ; is just plain weird and the last bit })() seems like German, where all the stuff that you technically not need to understand the sentence is at the end. Maybe someone can explain with JavaScript developers love anonymous functions so much. Why not just name your function and make the whole thing a bit more readable.
- akamaozu 7y agoNot all JavaScript devs. Love the language, hate the popularity of anonymous functions and non-descriptive variable names. My production code as a sample. http://designbymobi.us/wp-content/uploads/2020/02/rabbitmq-customer-rpc-api.jpg http://designbymobi.us/wp-content/uploads/2020/02/rabbitmq-c...
- mrweasel 7y agoYour code sort of have the same issue. Why not define a function: function someName(p1, p2) {} and then when you need it for something, like a handler, just say: handler: someName
- ksahin 7y agoI know the results are biased given it's a sample from OP inbox but I'm surprised not so see https://postmarkapp.com/ https://postmarkapp.com/ here. I thought they were pretty popular.
- opan 7y agoI've never heard of them before this comment. Maybe they're popular only in certain regions.
- jamestomasino 7y agoThey're just for transactional emails, not for marketing ones. They wouldn't show very much based on how he's measuring.
- ksahin 7y agoIt seems this post is about transactional emails, not marketing ones. CF the first paragraph!
- wjossey 7y agoI had actually just tweeted about my woes dealing with email going to spam for no good reason from my GSuite account. https://twitter.com/dustywes/status/1228733404826333185?s=21 https://twitter.com/dustywes/status/1228733404826333185?s=21 I’m planning on doing a more thorough write-up on my solution (it’s convoluted), but my tl;dr is that email seems woefully broken at the moment. Hope my experience can be helpful for others dealing with the same issue.
- londons_explore 7y agoI'm wondering the legality of this... The emails in your mailbox aren't yours to do with as you please - they're all still owned by their original sender, who has given you an implicit license to read them, and store them for later re-reading. You don't really have rights to datamine, aggregate, train AI on, post stats about, etc. the data in those emails. Its a violation of the rights of the original sender to do it without permission.
- layoutIfNeeded 7y agoSo if I send an email to a gmail address, it is illegal for Google to datamine it? Even though I’d love this to be true, somehow I doubt it really is the case.
- scambier 7y agoAre you serious? Unless those emails contain work protected by copyright, they're not "owned" by their senders. And I doubt that metadata falls in this category.
- icedchai 7y agoI disagree. They relinquished all rights when they hit send.
- roseway4 7y agoThe methodology used excludes a number of services using SparkPost. For e.g. the excerpt below of an Intercom header. It's a challenging problem to solve given the complexity of email delivery and appreciate this was a fun project to take on. Return-Path: <bounces+XXXX@XXXX.intercom-mail.com> Received: from mta-216-35.sparkpostmail.com (mta-216-35.sparkpostmail.com. [147.253.216.35]) by mx.google.com with ESMTPS id t13si7000210pgg.534.2020.02.03.08.37.30 for <XXXXXX> (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Feb 2020 08:37:30 -0800 (PST)