6 ms·

How we send 22k emails every hour

114 points by jitbit 7y ago

zxcvbn4038 7y ago
22k e-mails per hour averages just six per second — even the stated peak of 30 per second isn’t anything to write home about. If you go strictly by the book you are writing email to disk and flushing the buffers when the smtp server receives the email for relaying, and only acking reception when that has completed - and that gets into iops, channels, spindles, and other fun considerations that are often obscured by the cloud these days. But an easy optimization for a bulk sender is to forego that and use a less resource intensive method of tracking what hasn’t sent successfully and needs to be retried. But whatever method is used I don’t think it would really get interesting until you started sending thousands per second or higher.
- symfoniq 7y ago
  I have to agree. I built a custom email solution for a small business similar to the one presented in the article, with a bespoke newsletter creation/mailing list management/engagement reporting platform on top of it. It handles far more than 22K emails per hour, which is a trivial volume for even very modest modern hardware. However, I do agree with the article that building a system like this can be highly cost-effective. At one point, my aforementioned client was spending nearly $75K per year to outsource this. They've been using the system I built for more than half a decade now, with the only fixed monthly costs being a few virtual private servers.
  - zxcvbn4038 7y ago
    I think the main value add with the outsourced solutions is not having to deal with all the RBL issues, complaints, staying on top of the latest smtp requirements from Google etc. That does seem to be a full time job and the 75k is probably cheaper then staffing that directly. It’s not always about just the raw infrastructure costs.
    - jorangreef 7y ago
      I think you would be surprised at how many outsourced solutions don't deal with all the RBL issues, complaints, RFC compliance etc.
  - sova 7y ago
    May I ask what the main purpose of the e-mailing is? Notifications for a web-app, or is it marketing outreach, some combo thereof? Thanks
- marsdepinski 7y ago
  sums it up 100%
- mktmkr 7y ago
  This is the hazard of putting concrete numbers in a blog post like this. Among your readers will be people with experience sending this amount of traffic every second.
- Swenrekcah 7y ago
  Then don’t read this article and write about what you think is interesting.
- projektfu 7y ago
  So it makes it interesting that you can either buy service, which seems expensive according to his experience, or spend engineering time (salary money) to build it once and pay almost nothing to keep it online. I think a lot of us would build it just for the experience, but many shops have a habit of buying things that are not their core competency.
  - tatersolid 7y ago
    > or spend engineering time (salary money) to build it once and pay almost nothing to keep it online Every time an engineer says this, somewhere a puppy dies. Especially with an email service, keeping up with the spammers, scammers, blacklists, AV false positives, delivery failures at large companies, etc. cost far more per year than building the thing. Operations is never free.
- mtnGoat 7y ago
  i concur, sending 22k an hour isn't very interesting nor difficult. I think the highest i've tuned a system to was about 1m per hour sustained for days and this was back in 02-03 era. we definitely burned out a few parts and upset the hands in the datacenter a few times. would be a fun project to do with modern hardware, but i have no reason to send that quantity of email anymore. removing atime was probably the biggest single speed gain, along with understanding the MTA very well.
polote 7y ago
Very good article, that proves the common belief that self hosting emails is not possible. I do the same that the person in the article, sending 20k emails per day with the same setup and same result: it works
- koolba 7y ago
  s/proves/disproves/ right?
numlock86 7y ago
Article summary: To send large amounts of emails get a VPS and use Postfix. Watch out for IP reputation. Seriously, not to rant or anything, but isn't this "email server 101"?
- fock 7y ago
  but still there's probably enough companies out there paying half a remote sysadmin in fees for such a service ;).
- spiderfarmer 7y ago
  For you it is. For me it is. But for a lot of people the first thing that would come to mind is: lets see what (insert: SAAS) can do for us. And that's a good thing, because a lot of people just shouldn't be running their own servers.
disiplus 7y ago
how do you deal with spam ? do you have multiple "prepped" ip addresses that you switch if some of yours end on the spam list. at your scale it probably makes sense to take care of it, but i had problems with office365 simply sending ours to spam folder, with mxmonitor showing everything in green. for me it was important that every email is delivered (i have 99.9% now)
noego 7y ago
> We looked at paid PaaS providers like Mailgun, Sendgrid, Amazon SES... And all of them would cost several thousand a month at this scale. And since we're not a sexy funded startup from California, but a boring customer-funded business, we can't afford this bill I'm a little curious about their reasoning for buy-vs-build. They say they "can't afford several thousand a month", but their list of clients includes Adobe, VMWare, GE and many other impressive names. I find it hard to believe that with a list of major enterprise clients, they can't afford several thousand dollars a month? It seems more plausible that they can indeed afford it, but figured building it in-house would be cheaper and produce more profits. Developer costs in USA run about ~$5-10k/month, and that's assuming you're not trying to compete with companies like Google. Would be curious to hear about the number of dev-weeks they have spent learning all the stuff they mentioned, building it, troubleshooting any issues, and keeping the system running and up-to-date. I'm also curious about the effect building this project has had on their bus factor.
- miken123 7y ago
  > I find it hard to believe that with a list of major enterprise clients, they can't afford several thousand dollars a month? They can probably afford it, but if every piece of your stack needs that kind of money you run out of money fairly quickly. > Developer costs in USA run about ~$5-10k/month They are based in the UK. > Would be curious to hear about the number of dev-weeks they have spent learning all the stuff they mentioned, building it, troubleshooting any issues, and keeping the system running and up-to-date. It's just an SMTP server, not rocket science.
- jitbit 7y ago
  Hi, OP here. Well, to be embarrassingly honest... We suck at pricing. We were offering "unlimited" plans to everyone until recently. And the "impressive names" like you mention, well, they mostly pay us around $250 a month - which used to be our "Enterprise" pricing plan with unlimited everything (users, storage, agents etc.) So I guess the real answer is - we suck at positioning and we suck at marketing. As the result - profits were REALLY low (Lesson learned - don't compete on pricing). P.S. Couple of years ago I met Thomas from "FE International" at some conference, really experienced guy, who told me "dude, this is crazy, dump the unlimited plan like right now" so we did. So I guess technically we can afford a PaaS now...
crispyporkbites 7y ago
I am literally about to migrate from the mailgun API to my own email server, so this is a really helpful overview. Thanks!
pjc50 7y ago
This is interesting to compare with this 2004 paper on managing a large email for the University of Cambridge: http://people.ds.cam.ac.uk/fanf2/hermes/doc/talks/2004-02-ukuug/paper.html http://people.ds.cam.ac.uk/fanf2/hermes/doc/talks/2004-02-uk... managing 300k messages per day. > The system’s back end Cyrus message store comprises 16 single-CPU PCs, each with 3GB RAM and 7 72GB SCSI disks providing 350GB of available storage after RAID and filesystem overheads. The front end redirector ppsw is 6 dual-CPU PCs each with 1GB RAM and 4 18GB SCSI disks providing 50GB of available storage. The webmail system is another PC with the same configuration. The terminal-based Menu System is still running on the old Hermes systems at the moment, until migration to the new system is complete. It will then move onto a PC with plenty of RAM and CPU. The backup system is attached to a disk shelf containing 16 250GB SATA disks providing 3.2TB of usable storage, and a tape robot containing a 17 cartridge magazine and two LTO drives These days each of those would be a "small" VPS. Edit: just found an even better example, https://github.com/Exim/exim/wiki/Q1002 https://github.com/Exim/exim/wiki/Q1002 "On a PII 400 with 128M of RAM running Linux 2.2.5, I have achieved 36656 messages per hour (outgoing unique messages and recipients)"
- NKCSS 7y ago
  It's not really the same... You are talking about hosting big mailboxes; the article talks about sending and receiving e-mail. Sent e-mails are probably just discarded and incoming mails go into a processing pipeline, which is handled externally, which removes the need for a web interface to browse messages, search them and maintain backups.
- C1sc0cat 7y ago
  For reference Back in 1984 Huge was a cluster of 16 or so prime 750 super mini's over two Data centres - this was Telecom Gold. Wish I had saved some of the printouts of the billing monitor when it was running (the monthly map reduce billing system) as it listed the DASD and memory for each member of the cluster.
NKCSS 7y ago
Haha, this made me chuckle :) > Here's a cute completely irrelevant story, feel free to skip. The phrase "Because Australians!" has actually become an inside joke/catchphrase in our company. See, Aussies and Kiwis (New Zealanders) wake up before everyone else (even before Japan). Basically Australians are the reason SaaS businesses don't have an outage/maintenance window. And a couple of times when we were about to put a server down for maintenance on a late Sunday evening (EU time), someone would shout "Wait! Don't do that!" - "What... Why?" - "Because Australians!" So after 4 or 5 times this phrase is now used as an ultimate "42" answer why everything has to be up and running 24/7. I even have a tiny map of Australia in my office. I (nor anyone from the team) have never been to Australia or New Zealand, but hey, we're always thinking about you mates [wink]
- sova 7y ago
  Australia is a good, almost-habitable place and the time difference is quite a trip. Sometimes talking over video chat with my buddy it's daylight in both the US and Australia and we both say "huh?"
  - lostsock 7y ago
    FYI there is generally 3 hours overlap of working hours between the East coast of Australia and the West coast of USA. 8am our Melbourne time is 3pm San Fran time (the day before). https://www.timeanddate.com/worldclock/meetingtime.html?iso=20190730&p1=152&p2=224 https://www.timeanddate.com/worldclock/meetingtime.html?iso=...
forinti 7y ago
I once had to send out personalised emails for a large client in Portugal. We couldn't handle sending them out one by one, but we found out that there was a 50% chance for at least two people to have the same name and surname. So we bundled together people with the same names and then managed to overcome the bottleneck.
marsdepinski 7y ago
Not sure if this article is serious or a joke. 22k/h is quite small and most large enterprises handle this volume easily. Very very small scale. I personally, built a system that routinely would send between 10-20 million daily and doing about 3k/s. I know of systems that send 10x that ie. 100M/day.
- Yajirobe 7y ago
  > I know of systems that send 10x that ie. 100M/day Is that a spam system?
  - marsdepinski 7y ago
    No, e-commerce transactional and marketing.
    - pbhjpbhj 7y ago
      >> Is that spam? > No, [...] marketing. Excuse me while I lol. (It might be completely wanted marketing, just amused me; almost all spam is marketing for something.)
      - asark 7y ago
        I doubt more than 1% of marketing emails are desired. They may as well be spam. Forgetting to check a box, or to uncheck a box, or accidentally checking a box, doesn't make them wanted. Not wanting them anymore but also not bothering to go disable them doesn't make them wanted. It's fair to class them with spam. [EDIT] I mean 1% of marketing emails where there's any relationship between the sender and the recipient whatsoever, of course. All others are spam, clearly.
    - finnthehuman 7y ago
      So still spam, but falls outside the legal classification of impermissible spam?
- aytekin 7y ago
joekrill 7y ago
> Setting it all up is a one-time expense...but this is knowledge that stays with you (and the company) forever. That sort of depends on how well documented the entire system is, I'd think. And even then, I'm not sure this is quite as good as it sounds. Presumably this system is going to be humming along smoothly for quite some time. But when something breaks or goes wrong, is it going to be easy to go find the problem and fix it? Probably not. Whoever built it - if they are still with the company - will likely have forgotten a lot of the little details. Even if it's documented well, that documentation needs to be reviewed, understood, and (re)learned. So yeah, it's pretty low maintenance, sure - until something goes wrong. And it inevitably will. I'm not saying this is the wrong approach. Thousands of dollars a month for a PaaS is nothing to shake a stick at. So this very well could be a great approach under the circumstances. But there will be hidden costs down the road. It's just a matter of when and how much trouble they cause. (Will it cause your systems to go down for days while a bug is investigated, for example).
wefarrell 7y ago
But we had to learn everything about email. Every damn bit. SMTP, relays, MX-records, SPF verification, DKIM signatures, spam filtering, reverse-DNS, blacklisting, "Internet Message Format", throttling, MIME-encoding, email-antiviruses, bounces, headers, log-parsing, rate-limiting, greylisting, RFC standards (and that no one follows them)... And of course - about IP address reputation (more on that later). Are there any open source tools to tackle some of these? Given the maturity of email marketing I would expect some pretty robust free tooling available.
- atonse 7y ago
  I'd love to see if they did the math of the engineer time spent to learn all this vs paying another service. (Although outsourcing this doesn't necessarily mean you don't have to learn a lot of these things). There must be an inflection point where it becomes more economical to host your own, but I doubt 22k an hour is that point? Maybe it is.
  - allana 7y ago
    Sendgrid is pretty spendy, going by their public pricing and assuming the aveage usage is half the peak usage, the Sendgrid bill could be around ~$2500 usage.
- nodesocket 7y ago
  So, instead of spending “several thousand a month at this scale” which isn’t a lot of money for a business, you wasted how many engineering hours or worse founder hours that could have utilized building features or signing-up new customers. Developer; I have to pay money, nahhhh I’ll just build it complex.
  - wefarrell 7y ago
    It looks like they're helpdesk software. Email would pretty integral to their business model so I don't blame them for wanting to do it themselves, although it wouldn't be my first choice unless I needed some feature missing from all of the ESPs.
jorangreef 7y ago
"before that the inbound email is scanned for viruses (ClamAV)" We moved away from ClamAV because it detected less than a tenth of virus samples we tested in practice. Simple things like EXEs, ISOs, VBAs and JS attachments. VirusTotal uploads showed the same single digit detection rate for ClamAV. I thought ClamAV was a better scanner, have things stagnated?
- wielebny 7y ago
  Did you use anything else than stock rules?
  - jorangreef 7y ago
    Nothing other than stock settings and the latest signatures, same as VirusTotal's ClamAV I think?
- amaccuish 7y ago
  I use the free version of sophos for linux plugged into rspamd. Hopefully it has a better detection rate since Sophos. I did it this way since I read about the poor detection rate of ClamAV. Tbh we've blanket banned exes and their ilk so I'm not even sure if we need it anymore.
CodeWriter23 7y ago
@jitbit - you might want to add “Feedback Loops” (aka FBLs) to your list of “Everything” you needed to learn.

114 points by jitbit 7y ago

22k e-mails per hour averages just six per second — even the stated peak of 30 per second isn’t anything to write home about. If you go strictly by the book you are writing email to disk and flushing the buffers when the smtp server receives the email for relaying, and only acking reception when that has completed - and that gets into iops, channels, spindles, and other fun considerations that are often obscured by the cloud these days. But an easy optimization for a bulk sender is to forego that and use a less resource intensive method of tracking what hasn’t sent successfully and needs to be retried. But whatever method is used I don’t think it would really get interesting until you started sending thousands per second or higher.

symfoniq 7y ago
I have to agree. I built a custom email solution for a small business similar to the one presented in the article, with a bespoke newsletter creation/mailing list management/engagement reporting platform on top of it. It handles far more than 22K emails per hour, which is a trivial volume for even very modest modern hardware. However, I do agree with the article that building a system like this can be highly cost-effective. At one point, my aforementioned client was spending nearly $75K per year to outsource this. They've been using the system I built for more than half a decade now, with the only fixed monthly costs being a few virtual private servers.
- zxcvbn4038 7y ago
  I think the main value add with the outsourced solutions is not having to deal with all the RBL issues, complaints, staying on top of the latest smtp requirements from Google etc. That does seem to be a full time job and the 75k is probably cheaper then staffing that directly. It’s not always about just the raw infrastructure costs.
  - jorangreef 7y ago
    I think you would be surprised at how many outsourced solutions don't deal with all the RBL issues, complaints, RFC compliance etc.
- sova 7y ago
  May I ask what the main purpose of the e-mailing is? Notifications for a web-app, or is it marketing outreach, some combo thereof? Thanks
marsdepinski 7y ago
sums it up 100%
mktmkr 7y ago
This is the hazard of putting concrete numbers in a blog post like this. Among your readers will be people with experience sending this amount of traffic every second.
Swenrekcah 7y ago
Then don’t read this article and write about what you think is interesting.
projektfu 7y ago
So it makes it interesting that you can either buy service, which seems expensive according to his experience, or spend engineering time (salary money) to build it once and pay almost nothing to keep it online. I think a lot of us would build it just for the experience, but many shops have a habit of buying things that are not their core competency.
- tatersolid 7y ago
  > or spend engineering time (salary money) to build it once and pay almost nothing to keep it online Every time an engineer says this, somewhere a puppy dies. Especially with an email service, keeping up with the spammers, scammers, blacklists, AV false positives, delivery failures at large companies, etc. cost far more per year than building the thing. Operations is never free.
mtnGoat 7y ago
i concur, sending 22k an hour isn't very interesting nor difficult. I think the highest i've tuned a system to was about 1m per hour sustained for days and this was back in 02-03 era. we definitely burned out a few parts and upset the hands in the datacenter a few times. would be a fun project to do with modern hardware, but i have no reason to send that quantity of email anymore. removing atime was probably the biggest single speed gain, along with understanding the MTA very well.

polote 7y ago

Very good article, that proves the common belief that self hosting emails is not possible. I do the same that the person in the article, sending 20k emails per day with the same setup and same result: it works

koolba 7y ago
s/proves/disproves/ right?

numlock86 7y ago

Article summary: To send large amounts of emails get a VPS and use Postfix. Watch out for IP reputation. Seriously, not to rant or anything, but isn't this "email server 101"?

fock 7y ago
but still there's probably enough companies out there paying half a remote sysadmin in fees for such a service ;).
spiderfarmer 7y ago
For you it is. For me it is. But for a lot of people the first thing that would come to mind is: lets see what (insert: SAAS) can do for us. And that's a good thing, because a lot of people just shouldn't be running their own servers.

disiplus 7y ago

how do you deal with spam ? do you have multiple "prepped" ip addresses that you switch if some of yours end on the spam list. at your scale it probably makes sense to take care of it, but i had problems with office365 simply sending ours to spam folder, with mxmonitor showing everything in green. for me it was important that every email is delivered (i have 99.9% now)

noego 7y ago

> We looked at paid PaaS providers like Mailgun, Sendgrid, Amazon SES... And all of them would cost several thousand a month at this scale. And since we're not a sexy funded startup from California, but a boring customer-funded business, we can't afford this bill I'm a little curious about their reasoning for buy-vs-build. They say they "can't afford several thousand a month", but their list of clients includes Adobe, VMWare, GE and many other impressive names. I find it hard to believe that with a list of major enterprise clients, they can't afford several thousand dollars a month? It seems more plausible that they can indeed afford it, but figured building it in-house would be cheaper and produce more profits. Developer costs in USA run about ~$5-10k/month, and that's assuming you're not trying to compete with companies like Google. Would be curious to hear about the number of dev-weeks they have spent learning all the stuff they mentioned, building it, troubleshooting any issues, and keeping the system running and up-to-date. I'm also curious about the effect building this project has had on their bus factor.

miken123 7y ago
> I find it hard to believe that with a list of major enterprise clients, they can't afford several thousand dollars a month? They can probably afford it, but if every piece of your stack needs that kind of money you run out of money fairly quickly. > Developer costs in USA run about ~$5-10k/month They are based in the UK. > Would be curious to hear about the number of dev-weeks they have spent learning all the stuff they mentioned, building it, troubleshooting any issues, and keeping the system running and up-to-date. It's just an SMTP server, not rocket science.
jitbit 7y ago
Hi, OP here. Well, to be embarrassingly honest... We suck at pricing. We were offering "unlimited" plans to everyone until recently. And the "impressive names" like you mention, well, they mostly pay us around $250 a month - which used to be our "Enterprise" pricing plan with unlimited everything (users, storage, agents etc.) So I guess the real answer is - we suck at positioning and we suck at marketing. As the result - profits were REALLY low (Lesson learned - don't compete on pricing). P.S. Couple of years ago I met Thomas from "FE International" at some conference, really experienced guy, who told me "dude, this is crazy, dump the unlimited plan like right now" so we did. So I guess technically we can afford a PaaS now...

crispyporkbites 7y ago

I am literally about to migrate from the mailgun API to my own email server, so this is a really helpful overview. Thanks!

pjc50 7y ago

This is interesting to compare with this 2004 paper on managing a large email for the University of Cambridge: http://people.ds.cam.ac.uk/fanf2/hermes/doc/talks/2004-02-ukuug/paper.html http://people.ds.cam.ac.uk/fanf2/hermes/doc/talks/2004-02-uk... managing 300k messages per day. > The system’s back end Cyrus message store comprises 16 single-CPU PCs, each with 3GB RAM and 7 72GB SCSI disks providing 350GB of available storage after RAID and filesystem overheads. The front end redirector ppsw is 6 dual-CPU PCs each with 1GB RAM and 4 18GB SCSI disks providing 50GB of available storage. The webmail system is another PC with the same configuration. The terminal-based Menu System is still running on the old Hermes systems at the moment, until migration to the new system is complete. It will then move onto a PC with plenty of RAM and CPU. The backup system is attached to a disk shelf containing 16 250GB SATA disks providing 3.2TB of usable storage, and a tape robot containing a 17 cartridge magazine and two LTO drives These days each of those would be a "small" VPS. Edit: just found an even better example, https://github.com/Exim/exim/wiki/Q1002 https://github.com/Exim/exim/wiki/Q1002 "On a PII 400 with 128M of RAM running Linux 2.2.5, I have achieved 36656 messages per hour (outgoing unique messages and recipients)"

NKCSS 7y ago
It's not really the same... You are talking about hosting big mailboxes; the article talks about sending and receiving e-mail. Sent e-mails are probably just discarded and incoming mails go into a processing pipeline, which is handled externally, which removes the need for a web interface to browse messages, search them and maintain backups.
C1sc0cat 7y ago
For reference Back in 1984 Huge was a cluster of 16 or so prime 750 super mini's over two Data centres - this was Telecom Gold. Wish I had saved some of the printouts of the billing monitor when it was running (the monthly map reduce billing system) as it listed the DASD and memory for each member of the cluster.

NKCSS 7y ago

Haha, this made me chuckle :) > Here's a cute completely irrelevant story, feel free to skip. The phrase "Because Australians!" has actually become an inside joke/catchphrase in our company. See, Aussies and Kiwis (New Zealanders) wake up before everyone else (even before Japan). Basically Australians are the reason SaaS businesses don't have an outage/maintenance window. And a couple of times when we were about to put a server down for maintenance on a late Sunday evening (EU time), someone would shout "Wait! Don't do that!" - "What... Why?" - "Because Australians!" So after 4 or 5 times this phrase is now used as an ultimate "42" answer why everything has to be up and running 24/7. I even have a tiny map of Australia in my office. I (nor anyone from the team) have never been to Australia or New Zealand, but hey, we're always thinking about you mates [wink]

sova 7y ago
Australia is a good, almost-habitable place and the time difference is quite a trip. Sometimes talking over video chat with my buddy it's daylight in both the US and Australia and we both say "huh?"
- lostsock 7y ago
  FYI there is generally 3 hours overlap of working hours between the East coast of Australia and the West coast of USA. 8am our Melbourne time is 3pm San Fran time (the day before). https://www.timeanddate.com/worldclock/meetingtime.html?iso=20190730&p1=152&p2=224 https://www.timeanddate.com/worldclock/meetingtime.html?iso=...

forinti 7y ago

I once had to send out personalised emails for a large client in Portugal. We couldn't handle sending them out one by one, but we found out that there was a 50% chance for at least two people to have the same name and surname. So we bundled together people with the same names and then managed to overcome the bottleneck.

marsdepinski 7y ago

Not sure if this article is serious or a joke. 22k/h is quite small and most large enterprises handle this volume easily. Very very small scale. I personally, built a system that routinely would send between 10-20 million daily and doing about 3k/s. I know of systems that send 10x that ie. 100M/day.

Yajirobe 7y ago
> I know of systems that send 10x that ie. 100M/day Is that a spam system?
- marsdepinski 7y ago
  No, e-commerce transactional and marketing.
  - pbhjpbhj 7y ago
    >> Is that spam? > No, [...] marketing. Excuse me while I lol. (It might be completely wanted marketing, just amused me; almost all spam is marketing for something.)
    - asark 7y ago
      I doubt more than 1% of marketing emails are desired. They may as well be spam. Forgetting to check a box, or to uncheck a box, or accidentally checking a box, doesn't make them wanted. Not wanting them anymore but also not bothering to go disable them doesn't make them wanted. It's fair to class them with spam. [EDIT] I mean 1% of marketing emails where there's any relationship between the sender and the recipient whatsoever, of course. All others are spam, clearly.
  - finnthehuman 7y ago
    So still spam, but falls outside the legal classification of impermissible spam?
aytekin 7y ago

joekrill 7y ago

> Setting it all up is a one-time expense...but this is knowledge that stays with you (and the company) forever. That sort of depends on how well documented the entire system is, I'd think. And even then, I'm not sure this is quite as good as it sounds. Presumably this system is going to be humming along smoothly for quite some time. But when something breaks or goes wrong, is it going to be easy to go find the problem and fix it? Probably not. Whoever built it - if they are still with the company - will likely have forgotten a lot of the little details. Even if it's documented well, that documentation needs to be reviewed, understood, and (re)learned. So yeah, it's pretty low maintenance, sure - until something goes wrong. And it inevitably will. I'm not saying this is the wrong approach. Thousands of dollars a month for a PaaS is nothing to shake a stick at. So this very well could be a great approach under the circumstances. But there will be hidden costs down the road. It's just a matter of when and how much trouble they cause. (Will it cause your systems to go down for days while a bug is investigated, for example).

wefarrell 7y ago

But we had to learn everything about email. Every damn bit. SMTP, relays, MX-records, SPF verification, DKIM signatures, spam filtering, reverse-DNS, blacklisting, "Internet Message Format", throttling, MIME-encoding, email-antiviruses, bounces, headers, log-parsing, rate-limiting, greylisting, RFC standards (and that no one follows them)... And of course - about IP address reputation (more on that later). Are there any open source tools to tackle some of these? Given the maturity of email marketing I would expect some pretty robust free tooling available.

atonse 7y ago
I'd love to see if they did the math of the engineer time spent to learn all this vs paying another service. (Although outsourcing this doesn't necessarily mean you don't have to learn a lot of these things). There must be an inflection point where it becomes more economical to host your own, but I doubt 22k an hour is that point? Maybe it is.
- allana 7y ago
  Sendgrid is pretty spendy, going by their public pricing and assuming the aveage usage is half the peak usage, the Sendgrid bill could be around ~$2500 usage.
nodesocket 7y ago
So, instead of spending “several thousand a month at this scale” which isn’t a lot of money for a business, you wasted how many engineering hours or worse founder hours that could have utilized building features or signing-up new customers. Developer; I have to pay money, nahhhh I’ll just build it complex.
- wefarrell 7y ago
  It looks like they're helpdesk software. Email would pretty integral to their business model so I don't blame them for wanting to do it themselves, although it wouldn't be my first choice unless I needed some feature missing from all of the ESPs.

jorangreef 7y ago

"before that the inbound email is scanned for viruses (ClamAV)" We moved away from ClamAV because it detected less than a tenth of virus samples we tested in practice. Simple things like EXEs, ISOs, VBAs and JS attachments. VirusTotal uploads showed the same single digit detection rate for ClamAV. I thought ClamAV was a better scanner, have things stagnated?

wielebny 7y ago
Did you use anything else than stock rules?
- jorangreef 7y ago
  Nothing other than stock settings and the latest signatures, same as VirusTotal's ClamAV I think?
amaccuish 7y ago
I use the free version of sophos for linux plugged into rspamd. Hopefully it has a better detection rate since Sophos. I did it this way since I read about the poor detection rate of ClamAV. Tbh we've blanket banned exes and their ilk so I'm not even sure if we need it anymore.

CodeWriter23 7y ago

@jitbit - you might want to add “Feedback Loops” (aka FBLs) to your list of “Everything” you needed to learn.