11 ms·
Show HN: WildDuck – Self-hosted modern email server
- nicolaslem 7y agoBack when I was hosting my emails myself, my main complaint was that it required plugging many pieces together, in a traditional Unix fashion. This was before configuration management was really a thing, so it resulted in a Frankenstein setup that worked but was afraid to touch. If there was a more modern all-in-one-binary solution handling MTA/DKIM/IMAP/webmail with sane defaults I would maybe go back to self hosting.
- efesak 7y agoSee poste.io, it solves exactly this problem. (I've created poste because I administer multiple smaller mailserevers and thus scratching my own itch)
- eps 7y agoThis is really impressive. Perhaps do a Show HN for it?
- lifty 7y agomailu.io is an open source alternative to poste.io and bundles a classing email stack in an easy to use package. But it’s still quite complex under the hood. There is also an upcoming project called maddy( https://github.com/emersion/maddy https://github.com/emersion/maddy) that tries to package everything in a single binary, written in Go. I’m excited about about this and hope that the good folks that are developing it will take it to a usable state.
- jseliger 7y agoThese guys are selling a box, too: https://thehelm.com https://thehelm.com
- reaperducer 7y agoWish I could learn more about their box. But when I go to the web site I get a giant "Get updates and news from Helm" pop-up that blocks all other content except a sliver of an animation on one side of the screen. How do I know if I want news and updates from Helm if I can't even find out what Helm does?
- gscott 7y agoI use https://mailinabox.email/ https://mailinabox.email/ it does pretty much everything for you very easy just host on linode.com.
- h1d 7y ago> all-in-one-binary solution handling MTA/DKIM/IMAP/webmail Sounds like a very wrong idea there. One mistake in part of it will crash or get breached in the entire stack. You know why postfix runs many small binaries.
- js4ever 7y agoWow I was looking for something like that since ages. Best part, it's open source an made with Node.js/Redis, a stack I deeply love :)
- amolo 7y agoCool project. Been looking for something like this for ages. But still cant figure out from the docs how to send email.
- toomuchtodo 7y agoIs JMAP support planned?
- andris9 7y agoNo, at least not any time soon. JMAP is a beast that should not have been invented. It tries to convert TCP/IMAP (already a bad protocol by todays standards) into HTTP/JSON, just using a different transport while keeping the spirit. Instead WildDuck has its own simple REST API to build email clients against (https://api.wildduck.email/ https://api.wildduck.email/). It is not standard by any means and there are no wide client support but it is really easy to integrate with other projects as it is basically just a wrapper against database access, just like you would access blog posts or whatever via an API.
- papaf 7y agoJust to add to this point, I wanted to implement JMAP after only half completing an IMAP server implementation. Unfortunately, its just as bad only "this time with HTTP and Json". The batching advantages of JMAP could be generically done by using HTTP 2 and/or GraphQL.
- chrismorgan 7y agoI shall reply to your comment and your parent comment in one go here: I work on FastMail’s web interface, which is now based on JMAP. Here are my opinions on the matter. What papaf says is flatly not true. What andris9 says is generally unreasonable. At its core, the API part of JMAP is an RPC framework, with defined semantics for object synchronisation and querying. The most important part of it is those semantics, and they are necessarily more complex than what web developers are generally used to. Simple traditional-REST APIs or the likes of GraphQL lack those object synchronisation semantics altogether, and are wildly inferior for practical email clients. There is just no comparison at all. There is more to JMAP than just the API calls, though. What I think is the most obviously important part is that it also defines a push mechanism, including the ability for email clients to be notified of changes; this is tied into the object synchronisation semantics with the ability to ask questions like “what changed between these two states?”—so that your email client (be it on the web or not) can be told “something changed in the emails” and efficiently fetch the delta and apply it to the user interface and any persistent storage you may use. This is something that the shown WildDuck API does not appear to support at all, and is in fact something entirely unsupportable on such a style of API. These things are hard to do properly, in a way that will give any semblance of performance or efficiency. (It looks like WildDuck’s web interface does have some way arranged so that changes get pushed to the interface, but that doesn’t appear to be part of its public API, and I can’t imagine that it’s actually a good, flexible and efficient solution. I haven’t looked into how it is implemented at all, however. I’m open to discussion of what I’m confident will be its shortcomings, if you don’t like the aspersions I am casting.) Now to return to papaf’s remark: JMAP’s method batching is not achievable with HTTP/2 or GraphQL; the key difference is backreferences, whereby you can establish data dependencies between method calls, so that you pass part of one method call as an argument to subsequent method calls. To pick a common example of something that the FastMail web interface does, this allows you to express in one HTTP request the following: “find the first ten emails in Inbox, collapsing threads (so that you get one message from each thread only); then, given those, get the threads that they are in; then, given those threads, get the basic details of the emails that are contained in those threads.” In the absence of backreferences, you would need probably three round trips, or a special case in the API that limits flexibility. I will add that JMAP can be usefully combined with HTTP/2 so that you can issue multiple batches of requests simultaneously and use the results as they return. Please read through all the content on the front page of https://jmap.io/ https://jmap.io/. It provides good justification for why quite a few things are how they are. Remember this most important fact: JMAP is an object synchronisation protocol.
- indigodaddy 7y ago"it does not touch the filesystem" How does this work exactly? Mongo would have to touch a FS at the end of the day, no?
- andris9 7y agoSure, MongoDB does manage files but for the email app this is behind an abstraction and possibly in a different machine. I was more referring to maildir/mbox where the email server has to write and read files while juggling with user/group permissions, invoking shell and such. WildDuck just writes and reads stuff from a database like a regular web server software.
- indigodaddy 7y agoGot it. I suspected that's where you were going with that but just wanted to clarify. Thanks.
- mamon 7y agoDoes it come with official Docker image? :)
- srameshc 7y agoThanks, this is so timely. I was looking for something like this for weeks. All I want is IMAP server which can be managed via IMAP client api.
- Felz 7y agoThis is cool. The world could definitely use more email server implementations, even if JS isn't my cup of tea. Have you considered supplementing Mongo with block storage like S3, though? I've found it to be a pretty much perfect match for storing immutable mail messages.
- andris9 7y agoFor messages no, for attachments yes. WildDuck parses stored messages and separates attachments from rest of message data. Deduplicated attachments are then stored to MongoDB as GridFS files but this could be just as well S3 in the future. Message data needs to be in Mongo because of indexing and speed, attachments do not have such requirements and could be stored anywhere.
- dead_mall 7y agoI feel like self hosted emails is gonna be the next big thing, kinda like the pi-hole trend. This is awesome
- acidburnNSA 7y agoBeen running a postfix/dovecot/spamassassin stack for years. Absolutely love it. Want to wrote up a joke server side filter that bounces emails that are too snarky by sentiment analysis? That's a simple dovecot-pigeonhole-sieve filter. Lots of fun for all.
- diegoperini 7y agoI just googled pi-hole and now I'm converted. Thank you!
- icedchai 7y agoWhat's old is new again... Back in the 90's, it was very common to run your own email server. I've been hosting my own since 1996.
- mgamache 7y agoHow do you handle spam filtering for self hosted servers? I know there are black lists for SMTP servers, but I mean the content based filters that are updated based on recent spam signatures?
- geocrasher 7y agoI run a few servers that use Virtualmin + Postfix. I'm militant about outbound mail, and I make sure my accounts don't get compromised. If my mail queue reaches a certain size, I get notified and I go mop up the mess. Most recently it was an unprotected form that wasn't linked to, but existed. As for inbound spam, I can use RBL's but the most effective thing has been to block email servers that have rDNS that is either mismatched or nonexistent. I also host at a provider that isn't known for having their IP's on the "forever banned" lists.
- indigo945 7y ago> As for inbound spam, I can use RBL's but the most effective thing has been to block email servers that have rDNS that is either mismatched or nonexistent. This is also a recipe for never receiving mail from the SMB you do business with. (The same is true for enforcing SPF hard fails, unfortunately.) If you really want to be smart about this, use an SMTP soft error (greylisting) or an SMTP greeting pause. This effectively singles out spam bots, while leaving your regular mail traffic unaffected.
- mark242 7y agoSpeaking as someone who runs a commercial antispam SAAS, content filtering is increasingly irrelevant, and isn't nearly as effective as reputation filtering or bayesian classification.
- 55555 7y agoI would love to set this up but I've seen too many posts on the front page of HN about how gmail is blocking their incoming mail and they are powerless to do anything about it.
- StudentStuff 7y agoIf your hosting email on a residential ISP, your gonna have a rough time. Comparatively, hosting on any of the notable VPS providers (DO, Linode, OVH, etc) has had fine deliverability in my experience, though emails may occasionally get flagged as spam. Beyond that, the only insidious behavior I've seen has been Outlook.com hosted addresses accepting email, but then not delivering the email to the Inbox or Spam. Mysteriously, this issue disappears whenever I start going up the IT chain on the reciever's side. If the MTA is going to drop email, it should reject it rather than accept it (as otherwise the Outlook MTAs are falsely accepting mail that they will not deliver).
- nothrabannosir 7y ago> hosting on any of the notable VPS providers (DO, Linode, OVH, etc) has had fine deliverability in my experience, though emails may occasionally get flagged as spam. I can share a polar opposite experience: Yahoo and Outlook would categorically reject anything DO at the SMTP level with a 502. Luckily enough, at least they were explicit about it. But no leeway, not even accepting the envelope body. So YMMV, but it will almost certainly be poor :)
- StudentStuff 7y agoI don't run my mailserver on DO (I know a few people who do tho), but I've had few issues operating one on a no name host and another on OVH. If that was a persistent issue, it might be advisable to move to a host that isn't blacklisted.
- h1d 7y agoYou would use SMTP delivery service instead of moving around wishing for a lucky host / IP.
- niftylettuce 7y agoSee also https://forwardemail.net https://forwardemail.net. Thank you Andris for making this possible.
- konart 7y ago>андрис@уайлддак.орг Russian inside me chuckled.
- markvdb 7y ago> While WildDuck is more secure by definition than most alternatives > ... > You can run WildDuck on any system that supports Node.js, Node.js and more secure by definition? That sounds _very_ strange to me...
- pictur 7y agoDo large companies provide data security with very well-known languages?
- stingraycharles 7y agoI guess they mean “Secure by Design” but it doesn’t inspire confidence indeed..
- vince14 7y agoMaybe because "Microsoft: 70 percent of all security bugs are memory safety issues"
- Tankenstein 7y agoI get that this is a meme, but v8 is one of the most battle-hardened runtimes out there, plus you don't manage your own memory or concurrency primitives, which are the sources of most security issues.
- masukomi 7y agov8 may be badass. but the quality of many of the libraries in the node ecosystem is .... questionable at best. Lots of newbs with no clues about security throwing out packages that people end up with in their dependency trees and never stop to ponder the security / quality of.
- andris9 7y agoFor secure by definition I meant: * written in memory safe language * no special permissions needed (besides binding to privileged ports at start but there are workarounds) so no need to be able to chown or spawn workers under different user id's * no file access for the running daemon, once the daemon starts then it does not read nor write to the local file system * no spawning shell commands each of these things is a target for a different attack vector
- scblzn 7y agoMade in Estonia as I see. Congrats ! Very cool project, I really like the GUI part.
- SillyPosition 7y agoNice! I recently migrated my home server from Ubuntu to freebsd and was lazy to migrate my postfix/dovecot/sieve/all-the-other-parts and eventually kept it running in a VM instead.. This gives me the idea to try and migrate to Wildduck as it seems easier to deploy. Two questions that I have: 1. Is it possible to plug it into a mysql db instead of mongo?Self hosted apps are heavily relying on SQL db, so it feels more fluent to have a sql support 2. Is there a way to migrate existing postfix/dovecot mail server to this if I end up liking it?
- andris9 7y ago1. no. WildDuck is built to use MongoDB sharding to be able to easily scale both vertically and horizontally without single point of failures - each component of WildDuck (DB, IMAP, POP3, MX, SMTP) can be replicated. SQL does not fit too well into this model. If you want to use SQL then you could look into DBMail 2. migrating is possible only syncing via IMAP. There is a maildir importer but it is not open sourced (yet)
- F_r_k 7y agoWhat's the difference with mailcow (a fully dockerized email solution) ?
- andris9 7y agoMailcow packages Dovecot, Postfix etc. in an easy to use solution. WildDuck has its own and quite different implementations for most of these things.
- Tepix 7y agoAlways interesting to see new MTAs! I'm not convinced that storing mails (including attachments) in MongoDB is a good solution. What's wrong with the filesystem? It's designed for that purpose. You can (and should) of course store metadata in a DB to make searching easier, but when it comes to the actual emails I think the filesystem is the best place. One less thing to break. Easier to backup, mangle etc.
- mboehm 7y agoWell, scaling a filesystem does not really sound easy to me. Scaling a dbms is rather common, I guess. I rather question, why do they store their binaries in mongodb? imho, s3 would be the perfect fit for that requirement.
- nijave 7y agoIt's significantly easier to make an external data store resilient than to try to do that with a local filesystem. Backups, clustering, HA are all easier when there's a strong separation. In addition things like querying and lookups will be faster if you're utilizing a database
- Tepix 7y agoA filesystem doesn't have to be local. There are plenty of filesystems to choose from with different properties.
- andris9 7y agoUsing maildir (the current standard for mail storage) with any kind of remote file system is not the best environment to have. Maildir stores every email into a separate file and then uses rename() for every flag change. Marking 1000 messages as "Seen/Unseen" or adding a flag or moving to Trash means 1000 rename() calls in the file system. Multiply that with thousands of users also logged in to the system and doing their own things - you get many millions of inodes and rename()'s flying around all over. Good luck keeping a consistent copy or not crashing the mail server software as it waits for some queued rename() call over network FS to finish (and expects it to take nanoseconds as this is how long it takes in a local FS) while blocking everything else.
- Ente 7y agoI would love to try this. Is there a way to configure catch-all addresses for domains?