17 ms·

A serverless email server on AWS using S3 and SES

406 points by st_goliath 5y ago

alfiedotwtf 5y ago
> This stack was created out of frustration due to the fact that to this day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages. Looking at diagram, I'm not sure Postfix + Cyrus is more complicated than what you've got here. Congrats though, I'm sure this will help a lot of people wanting to manage mail themselves. Just not sure how you're going to get mail accepted by others from constantly changing (and possibly blacklisted) Amazon IPs without using trusted smart hosts.
- goodpoint 5y ago
  You spotted the issue: maintaining a mailer requires constant monitoring of blocklists and contacting the list owners to be unblocked. It required a fixed ipaddr and takes months to gain trust. Using random AWS IP addresses is a lost cause.
- xenophonf 5y ago
  > Just not sure how you're going to get mail accepted by others from constantly changing (and possibly blacklisted) Amazon IPs without using trusted smart hosts. Are you thinking about EC2 public IP addresses here? That service uses a different IP range than SES, which you can verify by comparing https://ip-ranges.amazonaws.com/ip-ranges.json https://ip-ranges.amazonaws.com/ip-ranges.json to `dig amazonses.com TXT`. In my experience, we haven't had any issues with SES emails getting blocked, but you must set up DMARC properly and respond to bounces quickly.
eljimmy 5y ago
Am I reading this correctly, you have to upload a JSON file to an S3 bucket to send an email? That’s not really email then, is it?
- fnord77 5y ago
  it wouldn't be hard to have a JS ui that presents an email form, but converts that to JSON and posts it to S3.
  - deleted 5y ago
    [deleted]
  - bdcravens 5y ago
    Sure, but pretty much every other email option out there has protocol support. Without that, I think it's a stretch to call this an email server. (Though there are a few that are only accessible via their client, such as Hey)
- ydant 5y ago
  SES supports using SMTP to send emails - so I don't know why this particular setup would prohibit that. https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp.html https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-e...
  - eljimmy 5y ago
    > This flow was designed to take advantage of the S3 trigger system and break each action into a small Lambda. The email flows from S3 => Lambda => SES so the underlying design seems to prohibit sending of emails directly from SES. It sounds like SES could still be used directly, but you then need to add in authorization and user management if you're accessing it externally. It's a cool project and I can envision it being used for internal tooling for automated email notifications or something of that nature. Just doesn't seem practical for users.
    - weego 5y ago
      AFAIK Amazon provides no guarantees that s3 file triggers will actually run, though they 'probably always will'
habibur 5y ago
Reading it I get that AWS now by default restricts email by - 200 mails per day max. 1 per second. - Can only send mail to verified addresses. Not sure how they verify though. - Can only send mail from the domain you own and verified. All these are good steps. Guess that will gradually take AWS IPs out of all those mail blacklists. And while we are on the topic, I find IP based mail blacklist services such as BRBL obsolete and possibly harmful. Now that we have domain verification services like DKIM SPF etc. Blacklist the domain. Not ip addresses as most institutions don't own their IPs like in the past. Most of all are hosted, and that IP might get assigned to someone else a few days later.
- ydant 5y ago
  Verifying emails just involves SES sending an opt-in / verification link to the recipient email address in question.
- deleted 5y ago
  [deleted]
- cutthegrass2 5y ago
  SES is pretty good at monitoring your email usage, such that it maintains a "reputation" score calculated by monitoring the hard bounce rate generated by your account. The premise being that high bounce rates (AWS define this as >5%) strongly indicate you're not managing your email recipient list properly i.e. sending untargeted SPAM.
- manishsharan 5y ago
  >> I find IP based mail server blacklist services such as BRBL as obsolete and possibly harmful. Now that we have domain verification services like DKIM SPF etc. Blacklist the domain. Not the ip Agree with you 100%. I want to run a mail server on my AWS server and not use Yahoo so that Verizon doesnt snoop on my mail and insert creepy ads that seem like emails. But this is impossible as my mail would get dropped into a blackhole.
- acdha 5y ago
  > Guess that will gradually take AWS IPs out of all those mail blacklists. Are you thinking of EC2? SES has never had a problem with this in my experience. Regarding verification, you can verify any email address where you can receive a link long enough to click on it. Domain validation is only needed if you want to send mail from arbitrary addresses without that step.
joana035 5y ago
One micro instance with Postfix is more than enough and can beat this setup not only in performance but in simplicity. What a year to be alive ...
- ungrateful-dead 5y ago
  In cost? Running postfix.org mail sever on a VM must have a nonzero cost, with the additional need to maintain the underlying OS.
  - caymanjim 5y ago
    EC2 micro instances are free.
    - beagle3 5y ago
      but are often blacklisted as SMTP sources, unfortunately.
      - joana035 5y ago
        Not really, only if it the IP was used to delivery spam. But IP reputation can be established with a bit of warm up and human interaction.
fareesh 5y ago
I feel like the current methods for spam filtration are getting so many false positives that it makes email unnecessarily difficult for users with good intentions. I also don't know how much of this is unethical behaviour by the big email providers, forcing you to sign up for their cloud email solution instead of doing your own. It sure would be nice to be able to run one's own email server without having to stress over whether the recipient got your email.
- jahewson 5y ago
  You can make it easy to send email or you can reduce spam but you can’t do both.
ignoramous 5y ago
We are building a similar serverless no-code inbox (with SES and S3) not for outgoing emails, but to only receive them [0]. Our service doesn't require a user to sign-up with any existing identity (like email or phone number), and so, when they make payments, we send the invoices from the payments-processor to an email (corresponding to the customer-id) on our subdomain that goes to SES which promptly plonks it into an S3 bucket set up to life-cycle it out after 60 days, encrypting it with keys associated with customer-id. This encryption serves us well when the customer chooses to rotate / delete their pseudo-anonymous identity (and the keys along with it), as then all data in our systems associated with their id (the archived emails, for example) is essentially tombstoned. The reason we prefer serverless over other solutions is not because of its scale or low-cost, but high-availability. [0] https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-consider-use-case.html https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiv...
rahimnathwani 5y ago
I was hoping this would include a lambda set up to respond to incoming requests on ports 25/465/587, but all the SMTP parts (both server/inbound and client/outbound) are being handled by Amazon SES, not by the serverless functions.
- OJFord 5y ago
  It's not possible. I wanted to do that with mine (similar thing to OP) - not actually for SMTP but for IMAP or even POP3, but you can't with Lambda - you can only reach it (from outside AWS) with HTTP(S). (I use a Lambda to forward to another provider currently, and use their client. I plan to write my own to replace that; ultimately I'd just like to run a JMAP server on a Lambda, that you could do.)
hk1337 5y ago
It's a nice POC but I wouldn't actually use it.
awinder 5y ago
This is awesome & I totally intend to play with it :D Do you have any plans / any ideas on feasibility to plop an imap server on top of the s3 store? I have really wanted a serverless aws solution and this moves the needle on work I did not wanna do so I’m reinterested again. Something like an s3 object change -> dynamodb index might work well for latency concerns and keep the serverless spirit alive. I just have such a starting problem on “building/learning imap intricacies when it may never work”.
- fooblat 5y ago
  At that point why not use Amazon WorkMail[0]? Then you don't have to maintain anything at all. 0. https://aws.amazon.com/workmail/ https://aws.amazon.com/workmail/
  - awinder 5y ago
    Uh because I didn’t know workmail came out, thank you kind human.
  - accelbred 5y ago
    Workmail doesnt have wildcard addresses and you pay per address which is why I was looking at something like this.
sneak 5y ago
Note that using the + address "trick" in email addresses is silly: everyone knows it and can strip it out with a simple regex. It doesn't accomplish the main things that it exists to do, which are to a) make it non-obvious what your username for a site is and b) make it obvious where a spammer got your email.
- caymanjim 5y ago
  The main thing it exists to do is to route email to folders or special handlers. It has nothing to do with spam or obfuscation, and predates the concept of spam.
kureikain 5y ago
This is great but it's a bit misleading. It only store email on S3, that mean you cannot read email with your email client. Seem thing with outgoing, you upload email to s3 in a certain format to be send out, not smtp protocol This is more like an API for sending and receiving. You cannot say generate SMTP credential and drop in your mail client. Also, one of the limitation with AWS SES is you cannot sent FROM or RETURN-PATH to anything, even in production mode, you can only set frm or return-path to verified domain. I have an internal AWS SES/Lambda stacks that I used for email forwarding(only receiving) which use SES to accept email, trigger a lambda and use SES to send a copy to a personal inbox. Plan to cleanup it a bit and release
- hnlmorg 5y ago
  In fairness, you could write support for those in lambda too if you wanted. As a proof of concept though, this is interesting.
- OJFord 5y ago
  I do something similar to OP [0] - and currently forward to another provider (single address, but that's only the 'envelope' To; so it doesn't matter, I preserve the original whatever.to.address). That's mostly an artefact of transitioning from that provider to my own on AWS, but I stopped working on it for some time and never continued. My plan was/still is to write a client that can read 'Just a Bunch of .Eml' format, and then I can point that at S3 (or via S3FS or whatever). Everything that exists for mailbox, maildir, and what-have-you assumes you need it to act as an MTA (I think that's right) and do the actual receiving too; and is difficult/impossible to stop that and say 'no, here's a directory full of email, just be a reader'. Longer-term plan/idle thought is to have a JMAP server on Lambda. (IMAP isn't possible, because it has to look like/be HTTP to reach the Lambda.) [0] - https://github.com/OJFord/amail https://github.com/OJFord/amail
  - kureikain 5y ago
    Yes, I used something like that as well. But, are you sure you can set "From" header to anything(even for email/domain that aren't verified with SES). So that on receiver side(gmail for example) it will show the original sender. In my account(in production mode, not being sandbox), I couldn't. When I attempted to set "From" header(or "Return-Path") I got these error 554 Message rejected: Email address is not verified. The following identities failed the check in region US-WEST-2: can-we-overwrite-this@my-domain.com, Test random <raindom@gmail.com>... On their example doc, using SMTP credential, they also mention this: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/examples-send-using-smtp.html https://docs.aws.amazon.com/ses/latest/DeveloperGuide/exampl...
someguy101010 5y ago
I use this myself for forwarding emails and signing up for random sites. Pretty easy to set up and 0 overhead to maintain. One of these days I'll set up a ui around it to send emails. Having both set up and maintained an email server "the old fashioned way" and using this, the over head on this is way less than postfix et al. This only goes for receiving email though, sending email with this is a biiit clunky, and doesn't support attachments (yet). If you have an aws account and a spare domain give this project a shot! It has a surprising amount of utility.
- kadoban 5y ago
  If you only want forwarding, you could look at https://www.nearlyfreespeech.net/services/email https://www.nearlyfreespeech.net/services/email . You'd have to do a bit of math to figure out the total price (depending on if you want/need DNS there to make it work), but it is _really_ cheap and easy. (not affiliated in any way, just a customer)
gruez 5y ago
>Pricing >$1 per active CodePipeline (must run at least once a month to be considered active) I'm not too versed in how aws services work, but how many active codepipelines are you expected to have running every month? A few paragraphs up it says that there are 3x CodePipelines in total, but they might not be all active every month. If all of them are active, then that would come out to $3/month, which seems a bit pricey considering you can get an actual hosted email plan for $5[1]. If you're looking to save money you can go even lower[2][3] [1] https://www.fastmail.com/pricing/ https://www.fastmail.com/pricing/ [2] https://purelymail.com/pricing https://purelymail.com/pricing [3] https://tutanota.com/pricing/ https://tutanota.com/pricing/
adontz 5y ago
From one side, great work! From another side, this looks much more complex, that getting an email service with catch-all address and use rules/filters to do the same. I'd programmatically manage rules/filters and folders, rather than this entire system. All email providers I have used on scale (Google, Microsoft, Rackspace) support this. I'd still have a usual email service with SMTP and IMAP, rather than incompatible custom JSON APIs and unknown failure modes. What is really interesting is cost on scale. Because if it is cheaper to send/receive 100M emails though this, than it's worth trying. However this is exactly the part not covered at all. And if you are sending not a lot of emails, maybe pay $50 to major email provider and spend your time on something more important?
GuB-42 5y ago
A serverless server? It is a great illustration of how the term "serverless" has shifted from literally no server (ex: sqlite database) to "somebody else's computer".
- thehappypm 5y ago
  "Serverless" is a ridiculous stupid term. You'd think it meant something like Photoshop, some system that doesn't rely a server, something that lives locally. Nope. It's just a server that someone else manages 90% of.
  - specialp 5y ago
    No we call those a more succinct term: local processes. We call "serverless" for processes running not locally that do not require standing up a server that you care about for, are billed by usage only ($0 for none, infinity for infinite), and you are not concerned where or how it is running. Are there computers listening for requests somewhere that someone is managing that is doing this? Yes. But from my frame of reference it is a service, not a server. I know a lot of people on here get hung up on it, but it is the term and it is not going to go away.
- bambam24 5y ago
  Google serverless to start learning about it
- ionwake 5y ago
  I literally had one of the heads of a crucial flight system project loudly exclaim to me in front of the senior managers. "It's serverless - there is no server!" To which I honestly, just kept my mouth shut and stood aside.
  - dec0dedab0de 5y ago
    To which I honestly, just kept my mouth shut and stood aside. I can never keep my mouth shut in these situations. Sometimes it leads to promotions, but I suspect it puts a ceiling on how high I can climb the corporate ladder.
bambam24 5y ago
By default, you can't send emails to unverified addresses. If you'd like to be able to send (as opposed to just receiving), you'll need to reach out to AWS to remove this limitation from your account.
kennu 5y ago
I'm not sure what the point is, since this solution doesn't give you an IMAP server to connect to with your email client. Also the solution seems a bit complex with all those CodePipelines and CodeBuilds. I think all you really need is one or two Lambda functions if you're just going to store stuff in S3.
- karlkatzke 5y ago
  We use exactly this solution internally at $work to handle our customer support, inbound sales, and other 'generic' group email that gets read by software and eventually gets to humans. It all goes through a separate inbound path and domain vs. our corporate email domain. Our outbound email from all of our apps goes through sendmail locally on the app servers and then out via SES.
- garettmd 5y ago
  All you'd really need is a file explorer that works with S3. Create a new file in the TMP folder, and it gets sent out. View received emails in the appropriate folders. It would also be trivial to setup an app or web form for creating the JSON file and uploading to S3.
JackGardner 5y ago
This is really interesting! Currently trying to solve a similar problem thing myself, so having a look through to learn from your solution. I'd recommend having a look at AWS CDK to manage the AWS resource creation, is much nicer than raw CloudFormation: https://github.com/aws/aws-cdk/ https://github.com/aws/aws-cdk/ For my solution to this I've also then got a frontend ontop of my backend to setup view mailboxes, email list per mailbox, email viewer (read/unread status), and email sending. I'm doing the same with S3/SES/Lambda + DynamoDB for email sending/recieving/processing + email metadata in DynamoDB. Mine is currently WIP and needs some real auth (using a URL API key, not the most secure...) + email replying + tidying up. Will definitely open source my solution and post to HN when ready.
- jedahan 5y ago
  I'd love to see a fork/branch that does the same as the linked repo, but using the cdk for comparison.
  - JackGardner 5y ago
    Yeah potentially I could extract the relevant CDK parts of my project out and combine with the code here to produce what you're asking for. I'll try and set aside some time and have a look. Thanks for the suggestion!
throwaway894345 5y ago
I was trying to set up Amazon Cognito (an authentication service) recently which basically needs a verified email address to send users emails for user verification and password reset, and I was shocked that Amazon has no basic email hosting service. Of course it’s easy enough to work around for my use case, but I was surprised nonetheless.
- stevehawk 5y ago
  What's extra annoying is that they need a verified email address and can't use a verified domain via SES.
- mvanbaak 5y ago
  They have workmail.
  - throwaway894345 5y ago
    I saw that, but it was like $5/user/month and geared toward employee email rather than automation (my use case is my hobby projects for which my goal is <$5/month in total).
andrew_ 5y ago
I'd love to see the cost difference between this solution and running an EC2 instance with a traditional lite mail server (like perhaps courier).
- wefarrell 5y ago
  Email from EC2 instances will be blocked because of the high amount of spam that comes from those IPs.
  - andrew_ 5y ago
    Interesting, thanks for sharing. Not sure why folks downvoted the comment, it being a plea for more information.
  - joana035 5y ago
    Yeah, you have to request amazon to remove the throttling from your network. It took few year for aws to document that they have this limitation on their network. https://aws.amazon.com/premiumsupport/knowledge-center/ec2-port-25-throttle/ https://aws.amazon.com/premiumsupport/knowledge-center/ec2-p...
unilynx 5y ago
There's another much more important limits on AWS SES that the article doesn't mention if you try to do this for normal email, the 10MB message size limit, which translates to about a 6MB maximum attachment size. It works fine for transactional mail and most newsletters, but quite a few users expect to be email larger files than that.
- philliphaydon 5y ago
  They shouldn’t. Most ISP email providers don’t support more than 10mb.
  - edoceo 5y ago
    gmail is 25MB, yandex is 30MB.
    - philliphaydon 5y ago
      ?!? I said ISP emails, which a lot of people still use. Outlook/Hotmail are 10mb though. And outlook client is 20mb received.
      - edoceo 5y ago
        I'm only trying to help. Like, every PM has used gmail regard it as "normal" - so if you're different, you're abnormal. also, if your PMs listen to reason here's a site with a table https://www.outlook-apps.com/maximum-email-size/ https://www.outlook-apps.com/maximum-email-size/ which show the 10MB size at the low-end but still used by a huge carrier
        philliphaydon 5y ago
        Sorry, I mistook your comment as asserting that because gmail supports 25mb then everything does. My bad.
    - jeffbee 5y ago
philliphaydon 5y ago
I setup a super super basic version of this which basically allows me to accept emails from one of my domains and forwards it to my real email. (Cos I use AWS DNS and don’t have the ability to forward like I do in name cheap)
fxleach 5y ago
It's just a cloud server. Let's call it what it is. "Serverless email server"? Come on...
mark242 5y ago
When the documentation says "this may cost you money" what they really mean is "this will cost you a lot of money the moment your email address gets onto any spam lists". The number of lambda invocations will go through the roof, and even if you're able to catch 90% of the spam before dumping messages into S3, the cost of putting a message into a bucket will make your bill skyrocket periodically as spammers temporarily find their way past SES's built-in filters, which happens every so often.
- maerF0x0 5y ago
  depends how you define a lot of money. How much would this cost to receive 30k messages per month?
  - mark242 5y ago
    Assuming a default message size of ~10kbytes the SES pricing is about $6 for the connections plus the chunks. Your lambda pricing is negligible and the S3 pricing gets you another dollar or so. But you're operating at the wrong scale-- 30k messages to a domain can happen in an hour by a spammer doing a directory attack. The point is that this solution is extremely open to abuse that happens all day every day over SMTP. Without a fixed-cost firewall to reject messages and only allow clean messages through to this solution, you are going to have escalating costs.
    - megous 5y ago
      That's a lot of money for a LKML subscription. :)
Exuma 5y ago
All the 'client' images on your site are 404.
praveen9920 5y ago
Great initiative but I see one major concern: Spam filtering. Most of the big mail providers provide spam mail filtering out of the box without which your inboxes are bombarded with spam. This can be probably solved by spam checking by adding additional lambda, now you need more compute per mail, translating to more cost per mail, which makes it easy for malicious parties to send mail bombs and bankrupt you.
irae 5y ago
This is a brilliant architecture. Everyone here talking about no email clients, IMAP and AWS restrictions is missing the point. Although you could use this as a personal account, or as a company. This fully automated system is a perfect ground layer to a number of applications with a very cleaver API. Say you are a small startup, and you will need some email functionality? Give your web server access to writing to outbox and you can send email. Use the sender with + rules and you get replies with customerId, threadId etc. Need to show email interactions in your service UI? Subscribe to the notification channels. Need customer support platform, it is almost built-in. If you don't see value in this solution, you probably never tried to setup infrastructure to receive email. In fact, at $work, I developed a very similar architecture and it is running flawlessly for 15 months or so. We just don't do all this git+lambda craziness, we just had already setup EC2 that we added some routes.
- StreamBright 5y ago
  This is how we handle notification emails and some incoming emails as well. There are few gotchas only (using separate domains is one of those)
- danielvinson 5y ago
  As somebody who has been an early engineer at many startups, I would never use this in almost any case because a managed solution is just better. If I can pay somebody else to do something hard with zero maintenance so I can focus on shipping a good product, I’m going to do that every time.
  - hipjiveguy 5y ago
    200% - outsource managing until you're making so much money that you can afford to bring it home, and pay the difference for the scale
  - pbreit 5y ago
    The main reason managed is better is they spend a lot of energy on deliverability.
  - 5y ago
sequoia 5y ago
Totally cool project! I can't quite figure out where in the repo the code is, or maybe this repo just contains the CF configs? If so a link to the lambdas etc. would be useful.
r2d2klapa 5y ago
This is great. I tried a similar approach but couldn't use it as a transactional email server because of this (taken from OP's readme): > By default, you can't send emails to unverified addresses. If you'd like to be able to send (as opposed to just receiving), you'll need to reach out to AWS to remove this limitation from your account. I reached out AWS to remove this limitation but they refused my request 3 times claiming that my use case would impact the deliverability of their service.
- ttul 5y ago
  AWS is not wrong. The instant that you stand up a service that allows anyone to send email, spammers will take advantage of it and spam the world. This ruins your reputation and you get blocked everywhere.
the_arun 5y ago
Very creative solution! Building an enterprise product using this will be fun!
superkuh 5y ago
I'm confused, does this use different IP addresses constantly? How is this mailserver able to send email without an IP based reputation? Is it just that Amazon is so giant other mega-corp email providers are wary to blacklist their IP range? If so this is a wonderful way to send spam without ever worrying about IP reputation. I imagine it will be abused and then blacklisted quickly.
- onefuncman 5y ago
  The Lambda doesn't send or receive email directly, it uses SES. SES handles this in a delegated way via DKIM signatures (Domain Keys Identified Mail).
  - superkuh 5y ago
    The domain signing with DKIM still doesn't get you a valid IP reputation. It may help to gradually build one if it's the same IP over and over, but it's still a brand new IP suddenly sending mail from $domain. If it weren't an amazon IP it'd certainly get refused or at least marked as spam and hidden after acceptance. Just another example of one set of rules for human people and another for corporate people. Either pledge yourself to a corporate feudal lord (amazon here) or suffer.
init 5y ago
Cool project! I designed a similar system many years ago with SES and the AppEngine data store instead of S3.
maerF0x0 5y ago
@st_goliath does David Gatti know you posted? I would love to hear from the creator :)
oblib 5y ago
I dunno... I run my own email server and it feels to me like the giant gap in the ease of getting your own email server (or services) up and running that you can connect to a web app still hasn't closed much here, especially for me because I don't use AWS and I don't want to spend the time to learn. My experience with 3rd party email services is they're also a pita to use and that can become huge when they merge or get bought up by a competing service or die, or raise their prices. With the web browser we can configure very simple emails that they can send with a link to click via the user's email client but as a web app developer what I really need is to be able to send a html email that way. Maybe web browser and email app makers could add a way to do that by letting apps request permissions and users approve it for a specific web app. That would really make life easier for me and my apps users.
lisper 5y ago
This is a very misleading title. It is not serverless. It uses SES which stands for Simple Email Service, which is provided by (of course) servers. It's just that they are servers that Amazon maintains for you. It is no more "serverless" than any other hosted email provider. This is not to say that this is without value. Having your emails end up in an S3 bucket is tremendously handy. But it's not serverless.
- Diederich 5y ago
  The word 'serverless' is certainly a bit misleading. By your definition, is there any possible architecture that is truly 'serverless'?
  - lisper 5y ago
    That depends on what you mean. If you mean, "Is it possible to have SMTP without a server?" the answer is no, by definition. SMTP is a client-server protocol. The same is true of IMAP and POP. If you mean, "Is it possible to have something that has an email-like UI/UX without a server" the answer is yes. But it won't interoperate with what we call email today, because that is fundamentally based on SMTP. In fact, the original email was serverless because it only worked on a single machine. That protocol is actually still in use today. Cron jobs error logging message of last resort is to send email to root@localhost, which is done without any networking at all.
verst 5y ago
The actual email sending / receiving part is handled by Amazon SES here. Similarly you could use Sendgrid in that place. This project is a wrapper around those. It also doesn't aim to provide an email inbox - this is for programmatic handling of email. Now the interesting part to me is using the email address local part to create an email address hierarchy upon receipt. This project stores incoming emails to S3 and the '+' in the email address indicates subfolders from left to right. That's a very creative way to organize emails for processing.
hrez 5y ago
This is as "serverless" as gmail. SES is literally Simple Email Service. Now do that without SES.
- ctippett 5y ago
  | Now do that without SES. Why, so they can prove they know how to host an email server? That’s not the point of the OP.
  - hrez 5y ago
    So they can say it's their service. Using somebody else's email service isn't their "serviceless". Of course there is a question of where you draw the line. But IMHO, using somebody's email service is that line here. Otherwise it's not far from using gmail or aws WorkMail for that matter.
lhoff 5y ago
Something link that could be useful as disaster recovery action. I case of problems with the Mail infrastructure redirect the DNS to something like that and incoming e-mails won't get lost in the meantime. The standby cost for something like that is much lower then a independent secondary e-mail system.
- megous 5y ago
  Why redirect anything? You can have backup MX records. This is already a solved problem. And backup system can be quite lightweight, if it only needs to store emails for the time primary is down and then push them to primary once it's up again.
greatgib 5y ago
I'm a little bit disappointed by the solution of this article. They start with: "This stack was created out of frustration due to the fact that to this day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages." Ok, so the point is that it is too complicated/frustrating to deploy an email server. But look at their solution in "what will deploy"... in the graph there are over 40 entities! In the following they list 13 different aws components needed. And all of that not even really doing the core feature of email sending/receiving that is provided by SES. Maybe you can deploy it with a single cloudformation recipe, but then it is so so many systems that you will have to watch and check for issues, quota, policies, usage billing and configuration. And you are not even in control, you have to go lick aws ass for easing the restrictions on the number of emails sent and to who. Plus there are probably a number of drawbacks and hidden limits that are not exposed to you by aws and that you will discover the hard way. For exemple, looking at the description I'm wondering, how do you do coherent backups at a random time? Knowing the quality of service of aws api, what do you do when there are random crash, network errors and co in the middle of the pipelines/lambdas? In my opinion this is a good exemple of an over-engineered solution.
- dahfizz 5y ago
  There are a class of engineers who have very little Linux experience, but use AWS extensively. For them, "Setting up a server" is always going to be more complex and frustrating than using the platform that they are familiar with, even when the AWS setup they come up with to replace a single server is way more complex to an "outsider". Personally, I don't think there is an excuse to not learn Linux. It's usually the right tool for the job, and you can use tooling like ansible and AWS instances to get your stack cloud based, stateless, automated, yada yada buzzword . But I can't really blame people for using the tools they are comfortable with.
  - antihero 5y ago
    Frankly I don't understand why they didn't use this using CDK.
- 5y ago
lyjackal 5y ago
Previous discussion in January 2020 https://github.com/0x4447/0x4447_product_s3_email https://github.com/0x4447/0x4447_product_s3_email
rdsedmundo 5y ago
I built something on those lines, although way simpler in terms of functionalities and resources used. All I wanted was to receive/send emails from a custom address (@acme.com) using Gmail, but I didn't want to pay Google Suite for that. So I used SES+S3+Lambda to receive emails and forward then to a Gmail alias (email+acme@gmail.com) so I can attach a filter on them using a label. And for sending emails using the address, I simply authenticated it on Gmail using the "Send mail as" option using AWS SES SMTP credentials.
tonfreed 5y ago
I wrote a catch-all lambda for SES too forward anything I sent it. That was 5 hours of my life I'm never getting back

406 points by st_goliath 5y ago

> This stack was created out of frustration due to the fact that to this day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages. Looking at diagram, I'm not sure Postfix + Cyrus is more complicated than what you've got here. Congrats though, I'm sure this will help a lot of people wanting to manage mail themselves. Just not sure how you're going to get mail accepted by others from constantly changing (and possibly blacklisted) Amazon IPs without using trusted smart hosts.

goodpoint 5y ago
You spotted the issue: maintaining a mailer requires constant monitoring of blocklists and contacting the list owners to be unblocked. It required a fixed ipaddr and takes months to gain trust. Using random AWS IP addresses is a lost cause.
xenophonf 5y ago
> Just not sure how you're going to get mail accepted by others from constantly changing (and possibly blacklisted) Amazon IPs without using trusted smart hosts. Are you thinking about EC2 public IP addresses here? That service uses a different IP range than SES, which you can verify by comparing https://ip-ranges.amazonaws.com/ip-ranges.json https://ip-ranges.amazonaws.com/ip-ranges.json to `dig amazonses.com TXT`. In my experience, we haven't had any issues with SES emails getting blocked, but you must set up DMARC properly and respond to bounces quickly.

eljimmy 5y ago

Am I reading this correctly, you have to upload a JSON file to an S3 bucket to send an email? That’s not really email then, is it?

fnord77 5y ago
it wouldn't be hard to have a JS ui that presents an email form, but converts that to JSON and posts it to S3.
- deleted 5y ago
  [deleted]
- bdcravens 5y ago
  Sure, but pretty much every other email option out there has protocol support. Without that, I think it's a stretch to call this an email server. (Though there are a few that are only accessible via their client, such as Hey)
ydant 5y ago
SES supports using SMTP to send emails - so I don't know why this particular setup would prohibit that. https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp.html https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-e...
- eljimmy 5y ago
  > This flow was designed to take advantage of the S3 trigger system and break each action into a small Lambda. The email flows from S3 => Lambda => SES so the underlying design seems to prohibit sending of emails directly from SES. It sounds like SES could still be used directly, but you then need to add in authorization and user management if you're accessing it externally. It's a cool project and I can envision it being used for internal tooling for automated email notifications or something of that nature. Just doesn't seem practical for users.
  - weego 5y ago
    AFAIK Amazon provides no guarantees that s3 file triggers will actually run, though they 'probably always will'

habibur 5y ago

Reading it I get that AWS now by default restricts email by - 200 mails per day max. 1 per second. - Can only send mail to verified addresses. Not sure how they verify though. - Can only send mail from the domain you own and verified. All these are good steps. Guess that will gradually take AWS IPs out of all those mail blacklists. And while we are on the topic, I find IP based mail blacklist services such as BRBL obsolete and possibly harmful. Now that we have domain verification services like DKIM SPF etc. Blacklist the domain. Not ip addresses as most institutions don't own their IPs like in the past. Most of all are hosted, and that IP might get assigned to someone else a few days later.

ydant 5y ago
Verifying emails just involves SES sending an opt-in / verification link to the recipient email address in question.
deleted 5y ago
[deleted]
cutthegrass2 5y ago
SES is pretty good at monitoring your email usage, such that it maintains a "reputation" score calculated by monitoring the hard bounce rate generated by your account. The premise being that high bounce rates (AWS define this as >5%) strongly indicate you're not managing your email recipient list properly i.e. sending untargeted SPAM.
manishsharan 5y ago
>> I find IP based mail server blacklist services such as BRBL as obsolete and possibly harmful. Now that we have domain verification services like DKIM SPF etc. Blacklist the domain. Not the ip Agree with you 100%. I want to run a mail server on my AWS server and not use Yahoo so that Verizon doesnt snoop on my mail and insert creepy ads that seem like emails. But this is impossible as my mail would get dropped into a blackhole.
acdha 5y ago
> Guess that will gradually take AWS IPs out of all those mail blacklists. Are you thinking of EC2? SES has never had a problem with this in my experience. Regarding verification, you can verify any email address where you can receive a link long enough to click on it. Domain validation is only needed if you want to send mail from arbitrary addresses without that step.

joana035 5y ago

One micro instance with Postfix is more than enough and can beat this setup not only in performance but in simplicity. What a year to be alive ...

ungrateful-dead 5y ago
In cost? Running postfix.org mail sever on a VM must have a nonzero cost, with the additional need to maintain the underlying OS.
- caymanjim 5y ago
  EC2 micro instances are free.
  - beagle3 5y ago
    but are often blacklisted as SMTP sources, unfortunately.
    - joana035 5y ago
      Not really, only if it the IP was used to delivery spam. But IP reputation can be established with a bit of warm up and human interaction.

fareesh 5y ago

I feel like the current methods for spam filtration are getting so many false positives that it makes email unnecessarily difficult for users with good intentions. I also don't know how much of this is unethical behaviour by the big email providers, forcing you to sign up for their cloud email solution instead of doing your own. It sure would be nice to be able to run one's own email server without having to stress over whether the recipient got your email.

jahewson 5y ago
You can make it easy to send email or you can reduce spam but you can’t do both.

ignoramous 5y ago

We are building a similar serverless no-code inbox (with SES and S3) not for outgoing emails, but to only receive them [0]. Our service doesn't require a user to sign-up with any existing identity (like email or phone number), and so, when they make payments, we send the invoices from the payments-processor to an email (corresponding to the customer-id) on our subdomain that goes to SES which promptly plonks it into an S3 bucket set up to life-cycle it out after 60 days, encrypting it with keys associated with customer-id. This encryption serves us well when the customer chooses to rotate / delete their pseudo-anonymous identity (and the keys along with it), as then all data in our systems associated with their id (the archived emails, for example) is essentially tombstoned. The reason we prefer serverless over other solutions is not because of its scale or low-cost, but high-availability. [0] https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-consider-use-case.html https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiv...

rahimnathwani 5y ago

I was hoping this would include a lambda set up to respond to incoming requests on ports 25/465/587, but all the SMTP parts (both server/inbound and client/outbound) are being handled by Amazon SES, not by the serverless functions.

OJFord 5y ago
It's not possible. I wanted to do that with mine (similar thing to OP) - not actually for SMTP but for IMAP or even POP3, but you can't with Lambda - you can only reach it (from outside AWS) with HTTP(S). (I use a Lambda to forward to another provider currently, and use their client. I plan to write my own to replace that; ultimately I'd just like to run a JMAP server on a Lambda, that you could do.)

hk1337 5y ago

It's a nice POC but I wouldn't actually use it.

awinder 5y ago

This is awesome & I totally intend to play with it :D Do you have any plans / any ideas on feasibility to plop an imap server on top of the s3 store? I have really wanted a serverless aws solution and this moves the needle on work I did not wanna do so I’m reinterested again. Something like an s3 object change -> dynamodb index might work well for latency concerns and keep the serverless spirit alive. I just have such a starting problem on “building/learning imap intricacies when it may never work”.

fooblat 5y ago
At that point why not use Amazon WorkMail[0]? Then you don't have to maintain anything at all. 0. https://aws.amazon.com/workmail/ https://aws.amazon.com/workmail/
- awinder 5y ago
  Uh because I didn’t know workmail came out, thank you kind human.
- accelbred 5y ago
  Workmail doesnt have wildcard addresses and you pay per address which is why I was looking at something like this.

sneak 5y ago

Note that using the + address "trick" in email addresses is silly: everyone knows it and can strip it out with a simple regex. It doesn't accomplish the main things that it exists to do, which are to a) make it non-obvious what your username for a site is and b) make it obvious where a spammer got your email.

caymanjim 5y ago
The main thing it exists to do is to route email to folders or special handlers. It has nothing to do with spam or obfuscation, and predates the concept of spam.

kureikain 5y ago

This is great but it's a bit misleading. It only store email on S3, that mean you cannot read email with your email client. Seem thing with outgoing, you upload email to s3 in a certain format to be send out, not smtp protocol This is more like an API for sending and receiving. You cannot say generate SMTP credential and drop in your mail client. Also, one of the limitation with AWS SES is you cannot sent FROM or RETURN-PATH to anything, even in production mode, you can only set frm or return-path to verified domain. I have an internal AWS SES/Lambda stacks that I used for email forwarding(only receiving) which use SES to accept email, trigger a lambda and use SES to send a copy to a personal inbox. Plan to cleanup it a bit and release

hnlmorg 5y ago
In fairness, you could write support for those in lambda too if you wanted. As a proof of concept though, this is interesting.
OJFord 5y ago
I do something similar to OP [0] - and currently forward to another provider (single address, but that's only the 'envelope' To; so it doesn't matter, I preserve the original whatever.to.address). That's mostly an artefact of transitioning from that provider to my own on AWS, but I stopped working on it for some time and never continued. My plan was/still is to write a client that can read 'Just a Bunch of .Eml' format, and then I can point that at S3 (or via S3FS or whatever). Everything that exists for mailbox, maildir, and what-have-you assumes you need it to act as an MTA (I think that's right) and do the actual receiving too; and is difficult/impossible to stop that and say 'no, here's a directory full of email, just be a reader'. Longer-term plan/idle thought is to have a JMAP server on Lambda. (IMAP isn't possible, because it has to look like/be HTTP to reach the Lambda.) [0] - https://github.com/OJFord/amail https://github.com/OJFord/amail
- kureikain 5y ago
  Yes, I used something like that as well. But, are you sure you can set "From" header to anything(even for email/domain that aren't verified with SES). So that on receiver side(gmail for example) it will show the original sender. In my account(in production mode, not being sandbox), I couldn't. When I attempted to set "From" header(or "Return-Path") I got these error 554 Message rejected: Email address is not verified. The following identities failed the check in region US-WEST-2: can-we-overwrite-this@my-domain.com, Test random <raindom@gmail.com>... On their example doc, using SMTP credential, they also mention this: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/examples-send-using-smtp.html https://docs.aws.amazon.com/ses/latest/DeveloperGuide/exampl...

someguy101010 5y ago

I use this myself for forwarding emails and signing up for random sites. Pretty easy to set up and 0 overhead to maintain. One of these days I'll set up a ui around it to send emails. Having both set up and maintained an email server "the old fashioned way" and using this, the over head on this is way less than postfix et al. This only goes for receiving email though, sending email with this is a biiit clunky, and doesn't support attachments (yet). If you have an aws account and a spare domain give this project a shot! It has a surprising amount of utility.

kadoban 5y ago
If you only want forwarding, you could look at https://www.nearlyfreespeech.net/services/email https://www.nearlyfreespeech.net/services/email . You'd have to do a bit of math to figure out the total price (depending on if you want/need DNS there to make it work), but it is _really_ cheap and easy. (not affiliated in any way, just a customer)

gruez 5y ago

>Pricing >$1 per active CodePipeline (must run at least once a month to be considered active) I'm not too versed in how aws services work, but how many active codepipelines are you expected to have running every month? A few paragraphs up it says that there are 3x CodePipelines in total, but they might not be all active every month. If all of them are active, then that would come out to $3/month, which seems a bit pricey considering you can get an actual hosted email plan for $5[1]. If you're looking to save money you can go even lower[2][3] [1] https://www.fastmail.com/pricing/ https://www.fastmail.com/pricing/ [2] https://purelymail.com/pricing https://purelymail.com/pricing [3] https://tutanota.com/pricing/ https://tutanota.com/pricing/

adontz 5y ago

From one side, great work! From another side, this looks much more complex, that getting an email service with catch-all address and use rules/filters to do the same. I'd programmatically manage rules/filters and folders, rather than this entire system. All email providers I have used on scale (Google, Microsoft, Rackspace) support this. I'd still have a usual email service with SMTP and IMAP, rather than incompatible custom JSON APIs and unknown failure modes. What is really interesting is cost on scale. Because if it is cheaper to send/receive 100M emails though this, than it's worth trying. However this is exactly the part not covered at all. And if you are sending not a lot of emails, maybe pay $50 to major email provider and spend your time on something more important?

GuB-42 5y ago

A serverless server? It is a great illustration of how the term "serverless" has shifted from literally no server (ex: sqlite database) to "somebody else's computer".

thehappypm 5y ago
"Serverless" is a ridiculous stupid term. You'd think it meant something like Photoshop, some system that doesn't rely a server, something that lives locally. Nope. It's just a server that someone else manages 90% of.
- specialp 5y ago
  No we call those a more succinct term: local processes. We call "serverless" for processes running not locally that do not require standing up a server that you care about for, are billed by usage only ($0 for none, infinity for infinite), and you are not concerned where or how it is running. Are there computers listening for requests somewhere that someone is managing that is doing this? Yes. But from my frame of reference it is a service, not a server. I know a lot of people on here get hung up on it, but it is the term and it is not going to go away.
bambam24 5y ago
Google serverless to start learning about it
ionwake 5y ago
I literally had one of the heads of a crucial flight system project loudly exclaim to me in front of the senior managers. "It's serverless - there is no server!" To which I honestly, just kept my mouth shut and stood aside.
- dec0dedab0de 5y ago
  To which I honestly, just kept my mouth shut and stood aside. I can never keep my mouth shut in these situations. Sometimes it leads to promotions, but I suspect it puts a ceiling on how high I can climb the corporate ladder.

bambam24 5y ago

By default, you can't send emails to unverified addresses. If you'd like to be able to send (as opposed to just receiving), you'll need to reach out to AWS to remove this limitation from your account.

kennu 5y ago

I'm not sure what the point is, since this solution doesn't give you an IMAP server to connect to with your email client. Also the solution seems a bit complex with all those CodePipelines and CodeBuilds. I think all you really need is one or two Lambda functions if you're just going to store stuff in S3.

karlkatzke 5y ago
We use exactly this solution internally at $work to handle our customer support, inbound sales, and other 'generic' group email that gets read by software and eventually gets to humans. It all goes through a separate inbound path and domain vs. our corporate email domain. Our outbound email from all of our apps goes through sendmail locally on the app servers and then out via SES.
garettmd 5y ago
All you'd really need is a file explorer that works with S3. Create a new file in the TMP folder, and it gets sent out. View received emails in the appropriate folders. It would also be trivial to setup an app or web form for creating the JSON file and uploading to S3.

JackGardner 5y ago

This is really interesting! Currently trying to solve a similar problem thing myself, so having a look through to learn from your solution. I'd recommend having a look at AWS CDK to manage the AWS resource creation, is much nicer than raw CloudFormation: https://github.com/aws/aws-cdk/ https://github.com/aws/aws-cdk/ For my solution to this I've also then got a frontend ontop of my backend to setup view mailboxes, email list per mailbox, email viewer (read/unread status), and email sending. I'm doing the same with S3/SES/Lambda + DynamoDB for email sending/recieving/processing + email metadata in DynamoDB. Mine is currently WIP and needs some real auth (using a URL API key, not the most secure...) + email replying + tidying up. Will definitely open source my solution and post to HN when ready.

jedahan 5y ago
I'd love to see a fork/branch that does the same as the linked repo, but using the cdk for comparison.
- JackGardner 5y ago
  Yeah potentially I could extract the relevant CDK parts of my project out and combine with the code here to produce what you're asking for. I'll try and set aside some time and have a look. Thanks for the suggestion!

throwaway894345 5y ago

I was trying to set up Amazon Cognito (an authentication service) recently which basically needs a verified email address to send users emails for user verification and password reset, and I was shocked that Amazon has no basic email hosting service. Of course it’s easy enough to work around for my use case, but I was surprised nonetheless.

stevehawk 5y ago
What's extra annoying is that they need a verified email address and can't use a verified domain via SES.
mvanbaak 5y ago
They have workmail.
- throwaway894345 5y ago
  I saw that, but it was like $5/user/month and geared toward employee email rather than automation (my use case is my hobby projects for which my goal is <$5/month in total).

andrew_ 5y ago

I'd love to see the cost difference between this solution and running an EC2 instance with a traditional lite mail server (like perhaps courier).

wefarrell 5y ago
Email from EC2 instances will be blocked because of the high amount of spam that comes from those IPs.
- andrew_ 5y ago
  Interesting, thanks for sharing. Not sure why folks downvoted the comment, it being a plea for more information.
- joana035 5y ago
  Yeah, you have to request amazon to remove the throttling from your network. It took few year for aws to document that they have this limitation on their network. https://aws.amazon.com/premiumsupport/knowledge-center/ec2-port-25-throttle/ https://aws.amazon.com/premiumsupport/knowledge-center/ec2-p...

unilynx 5y ago

There's another much more important limits on AWS SES that the article doesn't mention if you try to do this for normal email, the 10MB message size limit, which translates to about a 6MB maximum attachment size. It works fine for transactional mail and most newsletters, but quite a few users expect to be email larger files than that.

philliphaydon 5y ago
They shouldn’t. Most ISP email providers don’t support more than 10mb.
- edoceo 5y ago
  gmail is 25MB, yandex is 30MB.
  - philliphaydon 5y ago
    ?!? I said ISP emails, which a lot of people still use. Outlook/Hotmail are 10mb though. And outlook client is 20mb received.
    - edoceo 5y ago
      I'm only trying to help. Like, every PM has used gmail regard it as "normal" - so if you're different, you're abnormal. also, if your PMs listen to reason here's a site with a table https://www.outlook-apps.com/maximum-email-size/ https://www.outlook-apps.com/maximum-email-size/ which show the 10MB size at the low-end but still used by a huge carrier
      - philliphaydon 5y ago
        Sorry, I mistook your comment as asserting that because gmail supports 25mb then everything does. My bad.
  - jeffbee 5y ago

philliphaydon 5y ago

I setup a super super basic version of this which basically allows me to accept emails from one of my domains and forwards it to my real email. (Cos I use AWS DNS and don’t have the ability to forward like I do in name cheap)

fxleach 5y ago

It's just a cloud server. Let's call it what it is. "Serverless email server"? Come on...

mark242 5y ago

When the documentation says "this may cost you money" what they really mean is "this will cost you a lot of money the moment your email address gets onto any spam lists". The number of lambda invocations will go through the roof, and even if you're able to catch 90% of the spam before dumping messages into S3, the cost of putting a message into a bucket will make your bill skyrocket periodically as spammers temporarily find their way past SES's built-in filters, which happens every so often.

maerF0x0 5y ago
depends how you define a lot of money. How much would this cost to receive 30k messages per month?
- mark242 5y ago
  Assuming a default message size of ~10kbytes the SES pricing is about $6 for the connections plus the chunks. Your lambda pricing is negligible and the S3 pricing gets you another dollar or so. But you're operating at the wrong scale-- 30k messages to a domain can happen in an hour by a spammer doing a directory attack. The point is that this solution is extremely open to abuse that happens all day every day over SMTP. Without a fixed-cost firewall to reject messages and only allow clean messages through to this solution, you are going to have escalating costs.
  - megous 5y ago
    That's a lot of money for a LKML subscription. :)

Exuma 5y ago

All the 'client' images on your site are 404.

praveen9920 5y ago

Great initiative but I see one major concern: Spam filtering. Most of the big mail providers provide spam mail filtering out of the box without which your inboxes are bombarded with spam. This can be probably solved by spam checking by adding additional lambda, now you need more compute per mail, translating to more cost per mail, which makes it easy for malicious parties to send mail bombs and bankrupt you.

irae 5y ago

This is a brilliant architecture. Everyone here talking about no email clients, IMAP and AWS restrictions is missing the point. Although you could use this as a personal account, or as a company. This fully automated system is a perfect ground layer to a number of applications with a very cleaver API. Say you are a small startup, and you will need some email functionality? Give your web server access to writing to outbox and you can send email. Use the sender with + rules and you get replies with customerId, threadId etc. Need to show email interactions in your service UI? Subscribe to the notification channels. Need customer support platform, it is almost built-in. If you don't see value in this solution, you probably never tried to setup infrastructure to receive email. In fact, at $work, I developed a very similar architecture and it is running flawlessly for 15 months or so. We just don't do all this git+lambda craziness, we just had already setup EC2 that we added some routes.

StreamBright 5y ago
This is how we handle notification emails and some incoming emails as well. There are few gotchas only (using separate domains is one of those)
danielvinson 5y ago
As somebody who has been an early engineer at many startups, I would never use this in almost any case because a managed solution is just better. If I can pay somebody else to do something hard with zero maintenance so I can focus on shipping a good product, I’m going to do that every time.
- hipjiveguy 5y ago
  200% - outsource managing until you're making so much money that you can afford to bring it home, and pay the difference for the scale
- pbreit 5y ago
  The main reason managed is better is they spend a lot of energy on deliverability.
- 5y ago

sequoia 5y ago

Totally cool project! I can't quite figure out where in the repo the code is, or maybe this repo just contains the CF configs? If so a link to the lambdas etc. would be useful.

r2d2klapa 5y ago

This is great. I tried a similar approach but couldn't use it as a transactional email server because of this (taken from OP's readme): > By default, you can't send emails to unverified addresses. If you'd like to be able to send (as opposed to just receiving), you'll need to reach out to AWS to remove this limitation from your account. I reached out AWS to remove this limitation but they refused my request 3 times claiming that my use case would impact the deliverability of their service.

ttul 5y ago
AWS is not wrong. The instant that you stand up a service that allows anyone to send email, spammers will take advantage of it and spam the world. This ruins your reputation and you get blocked everywhere.

the_arun 5y ago

Very creative solution! Building an enterprise product using this will be fun!

superkuh 5y ago

I'm confused, does this use different IP addresses constantly? How is this mailserver able to send email without an IP based reputation? Is it just that Amazon is so giant other mega-corp email providers are wary to blacklist their IP range? If so this is a wonderful way to send spam without ever worrying about IP reputation. I imagine it will be abused and then blacklisted quickly.

onefuncman 5y ago
The Lambda doesn't send or receive email directly, it uses SES. SES handles this in a delegated way via DKIM signatures (Domain Keys Identified Mail).
- superkuh 5y ago
  The domain signing with DKIM still doesn't get you a valid IP reputation. It may help to gradually build one if it's the same IP over and over, but it's still a brand new IP suddenly sending mail from $domain. If it weren't an amazon IP it'd certainly get refused or at least marked as spam and hidden after acceptance. Just another example of one set of rules for human people and another for corporate people. Either pledge yourself to a corporate feudal lord (amazon here) or suffer.

init 5y ago

Cool project! I designed a similar system many years ago with SES and the AppEngine data store instead of S3.

maerF0x0 5y ago

@st_goliath does David Gatti know you posted? I would love to hear from the creator :)

oblib 5y ago

I dunno... I run my own email server and it feels to me like the giant gap in the ease of getting your own email server (or services) up and running that you can connect to a web app still hasn't closed much here, especially for me because I don't use AWS and I don't want to spend the time to learn. My experience with 3rd party email services is they're also a pita to use and that can become huge when they merge or get bought up by a competing service or die, or raise their prices. With the web browser we can configure very simple emails that they can send with a link to click via the user's email client but as a web app developer what I really need is to be able to send a html email that way. Maybe web browser and email app makers could add a way to do that by letting apps request permissions and users approve it for a specific web app. That would really make life easier for me and my apps users.

lisper 5y ago

This is a very misleading title. It is not serverless. It uses SES which stands for Simple Email Service, which is provided by (of course) servers. It's just that they are servers that Amazon maintains for you. It is no more "serverless" than any other hosted email provider. This is not to say that this is without value. Having your emails end up in an S3 bucket is tremendously handy. But it's not serverless.

Diederich 5y ago
The word 'serverless' is certainly a bit misleading. By your definition, is there any possible architecture that is truly 'serverless'?
- lisper 5y ago
  That depends on what you mean. If you mean, "Is it possible to have SMTP without a server?" the answer is no, by definition. SMTP is a client-server protocol. The same is true of IMAP and POP. If you mean, "Is it possible to have something that has an email-like UI/UX without a server" the answer is yes. But it won't interoperate with what we call email today, because that is fundamentally based on SMTP. In fact, the original email was serverless because it only worked on a single machine. That protocol is actually still in use today. Cron jobs error logging message of last resort is to send email to root@localhost, which is done without any networking at all.

verst 5y ago

The actual email sending / receiving part is handled by Amazon SES here. Similarly you could use Sendgrid in that place. This project is a wrapper around those. It also doesn't aim to provide an email inbox - this is for programmatic handling of email. Now the interesting part to me is using the email address local part to create an email address hierarchy upon receipt. This project stores incoming emails to S3 and the '+' in the email address indicates subfolders from left to right. That's a very creative way to organize emails for processing.

hrez 5y ago

This is as "serverless" as gmail. SES is literally Simple Email Service. Now do that without SES.

ctippett 5y ago
| Now do that without SES. Why, so they can prove they know how to host an email server? That’s not the point of the OP.
- hrez 5y ago
  So they can say it's their service. Using somebody else's email service isn't their "serviceless". Of course there is a question of where you draw the line. But IMHO, using somebody's email service is that line here. Otherwise it's not far from using gmail or aws WorkMail for that matter.

lhoff 5y ago

Something link that could be useful as disaster recovery action. I case of problems with the Mail infrastructure redirect the DNS to something like that and incoming e-mails won't get lost in the meantime. The standby cost for something like that is much lower then a independent secondary e-mail system.

megous 5y ago
Why redirect anything? You can have backup MX records. This is already a solved problem. And backup system can be quite lightweight, if it only needs to store emails for the time primary is down and then push them to primary once it's up again.

greatgib 5y ago

I'm a little bit disappointed by the solution of this article. They start with: "This stack was created out of frustration due to the fact that to this day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages." Ok, so the point is that it is too complicated/frustrating to deploy an email server. But look at their solution in "what will deploy"... in the graph there are over 40 entities! In the following they list 13 different aws components needed. And all of that not even really doing the core feature of email sending/receiving that is provided by SES. Maybe you can deploy it with a single cloudformation recipe, but then it is so so many systems that you will have to watch and check for issues, quota, policies, usage billing and configuration. And you are not even in control, you have to go lick aws ass for easing the restrictions on the number of emails sent and to who. Plus there are probably a number of drawbacks and hidden limits that are not exposed to you by aws and that you will discover the hard way. For exemple, looking at the description I'm wondering, how do you do coherent backups at a random time? Knowing the quality of service of aws api, what do you do when there are random crash, network errors and co in the middle of the pipelines/lambdas? In my opinion this is a good exemple of an over-engineered solution.

dahfizz 5y ago
There are a class of engineers who have very little Linux experience, but use AWS extensively. For them, "Setting up a server" is always going to be more complex and frustrating than using the platform that they are familiar with, even when the AWS setup they come up with to replace a single server is way more complex to an "outsider". Personally, I don't think there is an excuse to not learn Linux. It's usually the right tool for the job, and you can use tooling like ansible and AWS instances to get your stack cloud based, stateless, automated, yada yada buzzword . But I can't really blame people for using the tools they are comfortable with.
- antihero 5y ago
  Frankly I don't understand why they didn't use this using CDK.
5y ago

lyjackal 5y ago

Previous discussion in January 2020 https://github.com/0x4447/0x4447_product_s3_email https://github.com/0x4447/0x4447_product_s3_email

rdsedmundo 5y ago

I built something on those lines, although way simpler in terms of functionalities and resources used. All I wanted was to receive/send emails from a custom address (@acme.com) using Gmail, but I didn't want to pay Google Suite for that. So I used SES+S3+Lambda to receive emails and forward then to a Gmail alias (email+acme@gmail.com) so I can attach a filter on them using a label. And for sending emails using the address, I simply authenticated it on Gmail using the "Send mail as" option using AWS SES SMTP credentials.

tonfreed 5y ago

I wrote a catch-all lambda for SES too forward anything I sent it. That was 5 hours of my life I'm never getting back