11 ms·

The Hostile Email Landscape

• SHIT_TALKER 11y ago
  I see lots of threads shitting on the guy for doing it wrong vis-a-vis his configuration whilst ignoring his actual problem: An IP address without a reputation score. I've had the same problem and reached the same conclusion. The address can't just be clean, as in not on a blacklist, but has to essentially already be whitelisted via a "known good" reputation score or mail automatically gets blackholed. How do I get my VPS provider of choice to give me an IP address with a good reputation score?
• grey-area 11y ago
  Perhaps the problem here is that there is no verified identity for email servers?
  • jimktrains2 11y ago
    What's that even mean?
    • gruez 11y ago
      something like EV certificates for smtp servers?
      • ryan-c 11y ago
        Certificates for SMTP servers are meaningless without DNSSEC because crossdomain email servers are a thing.
        • geofft 11y ago
          Crossdomain web hosts are also a thing, and HTTPS works fine. (Sometimes with particularly hilarious definitions of "fine", like CloudFlare's former practice of putting dozens of customers' websites in the same certificate, via subject alternative names.) If you're worried about the fact that your mail host and web host can now impersonate each other, we can just define a new X.509 extension for "I can only be used for email". You might even be able to get away adding a new option to the existing Extended Key Usage field, but it's possible that enough clients don't enforce it rigorously enough. Ideally, you'd also want an SMTP equivalent of strict transport security.
          • jimktrains2 11y ago
            SMTP can use TLS, though, right? It doesn't _have_ to use STARTTLS? You _could_ use SNI. My concern is that it doesn't get you anywhere. phishing sites can and do get TLS/SSL certificates. The process isn't particularly difficult or labour intensive if you own the domain. As far as spam goes, so what? This only proves I'm talking to the server I intended to, not that it's a reputable and upstanding member of the server society.
            • thetmkay 11y ago
              What about EV itself? Currently there's no EV equivalent for individuals, but it would be a step forward.
              • jimktrains2 11y ago
                If I can't reasonably get it, then how is it a step forward?
                • thetmkay 11y ago
                  It would at least help companies/freelancers to host their own mailserver. The same infrastructure may lead to EV for individuals (or equivalent) once we can figure that out. Waiting for everything to be equally ready leads to inaction.
          • ryan-c 11y ago
            If someone spoofs a CNAME to evil.com in response to DNS query for google.com, your browser will not accept a certificate from evil.com as valid for google.com, whereas if you send mail to gmail.com and someone spoofs an MX record for mx.evil.com, a vaild mx.evil.com certificate will be accepted.
            • geofft 11y ago
              Right, so "don't do that." There isn't a clear standard for what subject to accept, out of the two options. The obvious correct one is to check for google.com, even if you're talking to a server named mx.google.com -- just as you don't trust CNAMEs and you check the original name, don't trust MXes either. It's just that most people don't have those certs configured in their mail servers, so doing that check today would lead to widespread failures and it's a bad default. Postfix, for instance, lets you configure what it checks via the "smtp_tls_verify_cert_match" option. If you set it to "nexthop", it'll require google.com. The default, "hostname", will check against mx.evil.com. http://www.postfix.org/postconf.5.html#smtp_tls_verify_cert_match http://www.postfix.org/postconf.5.html#smtp_tls_verify_cert_...
      • jimktrains2 11y ago
        That proves I own the domain to some extent. Don't DKIM records do the same? For the purposes of anti-spam, a cert doesn't do anything except prove that you showed some CA that you own the domain (which can be done and is done in other ways currently).
        • grey-area 11y ago
          At present SMTP is so broken that arbitrary servers are allowed to send mail for any domain, with fake headers, there is no proper verification of sending server or sending identity, the mail is accepted (along with legitimate mail), and put in spam folders (sometimes, as is real mail sometimes). SPF/DKIM are an attempt to solve that of course, but they are not enforced rigorously or universally, and tactics like IP reputation and email content sniffing still seem to be widely used too (the problem reported in the article). Compared to the web, email seems pretty behind on verifying server identity.
          • jimktrains2 11y ago
            Yes, that's my point. It doesn't really solve any issues that SPF/DKIM don't already solve -- pointing to certs isn't a new solution, it's a rehash of the same solution. And yes, the concept of relays should disappear along with much of the other cruft SMTP brings. (And while we're at it, can we fix/replace IMAP, or at least make the spec say that message ids are eternal and can't be invalidated?)
            • grey-area 11y ago
              IMO Message ids should be defined as a sha512 of headers and content to ensure they are unique. A new protocol is required.
              • jimktrains2 11y ago
                Currently headers get modified in-flight, so a content-based ID wouldn't work. As for a new protocol, there are some out there, but all fail in some way or another. I figured I'd throw my ideas out there and see how much they fail https://github.com/jimktrains/email_ng https://github.com/jimktrains/email_ng
    • grey-area 11y ago
      Certs or some other method to identify servers, so that gmail etc don't attempt to use IP for server identity, which is obviously flawed. The SMTP protocol is pretty flawed itself though, so probably that needs to change to see any real progress here.
  • mike_hearn 11y ago
    There is - this is effectively what DKIM does. Reputation systems calculate reputations over email server identities, not email sender identities (subtle distinction).
    • grey-area 11y ago
      Yes true, there is SPF and DKIM which are used by the big providers, though I'm not sure how strongly they trust those, but they shouldn't be using IP as a signal to determine whether the server is legitimate, or it leads to the catch 22 this article talks about - unknown IP = SPAM.
• jimktrains2 11y ago
  Email has been the my last hold out from switch away from gapps completely. I don't want to have to deal with any of this, especially as I do business communication with clients via it. Email wasn't suppose to be like this, and there has to be a better way to enable non-giants to successfully deliver email.
  • seiji 11y ago
    Joke answer: the blockchain! Real answer: It comes down to trust (duh). But, how do we manage trust online? How do we manage trust in real life? Real life trust is through association of groups. But, groups online are meaningless. A "gmail user" doesn't belong to a community, they belong to The Nation of Google. How do we break down online identities into manageable, trustable communities? How do we bootstrap new communities into the system so we don't end up with a pre-selected list of blessed communities while others languish in obscurity? Open questions, no answers, and if you say Facebook you have to run 20 laps and give me 1000 pushups.
    • jimktrains2 11y ago
      You joke about the blockchain, but HashCash (a similar technique to Bitcoin's Blockchain-difficulty (not the metric itself, but how it's used)) was originally conceived as a means of making email computationally costly to send. The main issue was that spammers would use botnets while normal people would be stuck taking a while sending emails (iirc). Personally, I would like everyone to have their own rsa or ecc key. While not as expensive as HashCash, it does require more computational effort to send an encrypted email. (This precludes web portals from sending mail, but user convince seems to always trump security! :( ). Still not a great solution. Web of Trusts are an interesting idea, but fails for the same reason not everyone has a public encryption key: bad ui, complex idea, and no one cares. I honestly think this is the way forward. If someone isn't a few hops from me, they probably don't have any business contacting me anyway -- if they really want to, a.k.a. read my blog and want to ask me a question, they can use HashCash or something else expensive to make the initial contact and slowly build up trust/reputation as we exchange emails. In the end, everyone is always told they need to check their spam folder for false positives anyway. Maybe we should all just say "screw it" and read all our email. If some combination of the sender and subject doesn't make us think it's legit, then we can delete it. Maybe we keep spam filters, but stop calling it a "Spam Box" and treat it like trash. Maybe we start calling it "Unknown Box" and treat it with a little skepticism instead of a dumpster.
• drumdance 11y ago
  Surprised he didn't mention third party reputation providers such at Return Path.
  • vangale 11y ago
    I'm surprised as well, since these services reinforce his point even more. In other words, good reputation = $$$. Also, I currently use mailgun (and dabbling with mandrill on some new projects) but I'm intrigued by postmarkapp.com take on dedicated IP addresses. Apparently they won't sell dedicated IP addresses because of the time needed to warm them up, so you get a shared IP and their TOS is stricter anti-spam than other ESP's: http://blog.postmarkapp.com/post/14127210172/the-false-promises-of-dedicated-ips http://blog.postmarkapp.com/post/14127210172/the-false-promi...
    • old-gregg 11y ago
      Mmm... not really. Dedicated IPs are hard to beat if you have a steady traffic of high quality and low-latency delivery is a priority (think of PagerDuty). Most legit businesses/customers have that. Most likely they're not offering dedicated IPs because they're tough to get, especially if you are a small company in email business. The reason Mailgun and Mandrill can do this is because they belong to much larger companies with better access to IPv4 stockpiles: Rackspace and Mailchimp. Source: I worked at Mailgun.
• cm2187 11y ago
  The problem is not so much the attitude of the big guys. It is that smtp is fundamentally broken. we need a better mail protocol that ensures: 1. Traffic always encrypted and content always signed 2. Guarantee that the sender is who it claims he is 3. Decorrelating the email from the domain, a lot of users are prisoners of their current provider just because the address they gave everyone ends with the provider's domain name, very much like it is very hard to switch bank accounts 4. Ability to provide disposable adresses which can be deleted when spammed 3 and 4 would require a sort of token system
  • wesleytodd 11y ago
    Maybe something like this? https://github.com/ssbc https://github.com/ssbc https://github.com/ssbc/secure-scuttlebutt https://github.com/ssbc/secure-scuttlebutt https://github.com/ssbc/scuttlebot https://github.com/ssbc/scuttlebot
    • vezzy-fnord 11y ago
      Or djb's IM2000 (later fleshed out by JdeBP): http://homepage.ntlworld.com/jonathan.deboynepollard/Proposals/IM2000/ http://homepage.ntlworld.com/jonathan.deboynepollard/Proposa... That might have since fallen out of favor, I'm unsure.
      • JdeBP 11y ago
        I wrote about the further balkanization of SMTP-based Internet electronic mail, lamented by the headline article here, some ten years ago in http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/smtp-anti-ubm-dont-work.html http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sm... . I am surprised and encouraged to see IM2000 mentioned in this discussion. As to whether such ideas have fallen out of favour, I offer this observation: You are using a pull-style electronic communications system right now. Your WWW browser is the originator and receipient UA, and Hacker News is the message store. A copy of a message is only made and sent across the network when you request it with your User Agent. A lot of Internet-mediated communications are not SMTP-based any more. The shift to pull-style systems has, in some ways, actually already happened. As such, many of the things that people ask about IM2000 are questions that they can answer from their own direct experiences, with a little reflection. How do you deal with senders who alter their messages? Well: how do you the receipient deal with the fact that right here and now I can hit an edit button and modify this comment after people have pulled it for reading it, with no edit history and nothing but the honour system for dealing with letting people know? The fleshing out of IM2000 set out the idea of downloading to folders in the recipient MUA the messages that one wanted to take off the senders' mail stores, by the way.
• karlshea 11y ago
  Having run my own email server about a decade ago I can kind of understand why they are just rejecting things. Dealing with spam for even just a handful of domains was a nightmare.
• ofir_geller 11y ago
  Did you try to acquire reputation by having real people with accounts in "known" services send emails to your server?
  • josteink 11y ago
    If this sort of shit was required to get a basic www-server up and running, do you think anyone would have cared about the internet at all? The internet was supposed to be for everyone and anyone. Current internet email is terribly flawed. There's just no way around it and the current status cannot be defended.
• mike_hearn 11y ago
  This guys story is a sad one but I'm sure we're missing some important information here. New mail servers get set up all the time, it's not impossible for such servers to be accepted by the big players. If none of these major services ever learned that his server was OK, it's likely that users weren't unmarking the mail he sent as spam. And that leads to the question of why not.
  • angelbob 11y ago
    In general, everybody uses a service like MailChimp or SendGrid to do this. That allows them to send email, but keeps them from building up a good reputation for no spam on their IP addresses -- that reputation is built by the email provider, not by you as the customer. For awhile I wondered why so many SaaS apps bothered to pay external (not cheap!) mail services when it's so easy to set up an email server. This is why. My "easy to set up an email server" info was years out of date.
    • gingerlime 11y ago
      sadly, it's even hard to get some email across when using those ESPs (Email Service Providers). Our domain has all the correct SPF records, DKIM etc and we're using a reputable email provider (Mandrill). All is sparking clean. Validated etc. Still we see some activation emails to google get delivered, but severely delayed on the Google side (after we see an OK from google for the email being delivered). Our users can't reset their password or activate their account until a certain time passes... Emails from my mum (who I've emailed on the same account for nearly a decade) suddenly get flagged as spam by gmail. Go figure.
• andrew-lucker 11y ago
  It's only a matter of time before we start seeing peering issues between major email providers. This is not a system for sharing, it is competition.
• sjackso 11y ago
  I've run into similar issues with a similar setup. It's frustrating. You can convince gmail user A to whitelist your messages, and so they'll get through to user A, but gmail user B probably still won't see messages from you unless you tell him to dig them out of the spam trap. And your messages to A might still be classified as spam if they have attachments or hyperlinks in them. (Even if you've been corresponding with A for several years!) Once upon a time, to send email, you needed to use SMTP. Now, you must use SMTP, from a IP block that isn't categorized as residential, and which has never before had any association with outgoing spam, and you also must implement several ad-hoc identification protocols like SPF and reverse DNS. You should also use a domain name that you've owned for some time and which is not expiring soon. Every system to which you want to send mail will give different weights to all these signals. If they don't like you, their behavior is to report successful delivery and then silently hide messages from their intended recipients. Spam is no fun, but the present situation is pretty weak too.
  • unclebucknasty 11y ago
    The email deliverability issues we have had as a legitimate business are insane. Moving to an ESP years ago has helped, but it's far from perfect. I've wondered how a small business without the tech resources could manage this. For instance, several months back some of our account holders suddenly stopped receiving important account info as well as newsletters from us. Tracked it down to a third party filtering service used by various email providers. Contacted them and they indicated that we were sending to honeypot addresses and/or doing other things that landed us on their list. We refuted all of their assertions (all of our recipients are double out-in, etc.) But , they insisted, as if their processes were infallible and refused to remove us. Here they were, a third party with whom we had no agreement, yet they were interfering with our ability to do business (and profiting from it). We also had paid advertising in the newsletters per agreements with advertisers. Consulted counsel and the next step was to send a cease and desist, followed by an injunction on the grounds of tortious interference. Then, just as suddenly, our emails started going through again. So, all of this, just to send email. Yet, I still have to deal with a ton of spam personally when wading through my quarantine folder for routine false positives. Current processes are harder on the good guys than the spammers.
• grinich 11y ago
  I've looked into this a bunch as we've been building Nylas. Our backend is essentially a cloud mail user agent (MUA), but can have similar issues to a MTA or mailbox provider. It turns out that creating a successful product in the email space requires you to build relationships and partnerships with the existing vendors/providers. It takes a lot of work, and is part of what you pay for when using Mailgun/Mandrill/etc. These "greylists" and systems that drop mail from unknown IPs have been pretty carefully designed and tuned to combat the insane amount of spam out there. They work remarkably well, and for most of today's email users, spam is no longer an issue. (The current state of email abuse innovation is promotions/marketing/etc. which is a more subtle challenge.) At the end of this article, the author writes, "This isn't how the internet is supposed to work." Ironically this system of obscure reputation-based email is a direct result of how the email system was actually designed to work, with a total lack of permissions or feedback loop. Many SMTP servers used to not even require a password. Stuff like DKIM/SPF and DMARC is a step in the right direction. But the RFCs upon which our email system is based were written decades ago, and in many cases have fundamental flaws, like SMTP leaking metadata no matter what. I could go on and on about issues with email, and why I care about getting them fixed, but let's just say it was designed in a different era of Internet with different constraints and opportunities. So how do you build a new email service and not get blocked? Well, you spend a few weeks or months emailing, calling, Skyping, and meeting with folks in the current space. You work your way up through marketing support and random protocol discussion lists until you are talking with the folks who can influence which IPs are blocked/unblocked. Then you convince them you are (1) building a legit venture, (2) are worthy of their trust, and (3) don't directly compete with them. Then you'll get a small number of clean IPs and you must not screw up! There are a few hacks, like sometimes a single partner/vendor will sell you a block of clean IPs and help manage the spam reputation. But usually it's just a lot of sweat and annoying phone calls. It takes way, way longer than setting up SPF/DKIM. The challenge is more relationships than technical. Once you have a new email sending provider, the burden shifts to you for doing abuse/spam prevention. And you find yourself implementing many of the strategies and systems you cursed when getting started. But that's the circle of rfc2822 life I guess. Oh, and never use EC2 IPs for sending mail. Most of them have been burned by spammers.
• larrys 11y ago
  Edit: They said they checked the spam lists but the ones that I've listed below may be helpful to others. OP might have had an IP previously used by spammers or in a spam block. (See edit). Important to check the IP before using it if it's been given to you by, say, the VPS Provider or the upstream. To see what, if any, reputation it has. Here are two tools to use: http://multirbl.valli.org/ http://multirbl.valli.org/ http://mxtoolbox.com/SuperTool.aspx http://mxtoolbox.com/SuperTool.aspx And there are a bunch of similar tools.
• deleted 11y ago
  [deleted]
• jfaucett 11y ago
  You can talk about HTTPS and forcing encryption all over the web however much you want but as long as you're stuck with SMTP and the few large coorps as the only viable mail solutions you can forget individual citizen data privacy... at least thats my 2cents.
• waynecochran 11y ago
  Create an email network where is would cost a penny to send email. It would be payed into bitcoin wallet of folks maintaining infrastructure. Every email would be digitally signed and encrypted. Certificate with keys would connected to email address (and bitcoin wallet). Spam would die. Go build it please.
  • HarryHirsch 11y ago
    Hashcash?
    • pmlnr 11y ago
      Botnets won't care :/
  • adamrt 11y ago
    I think that is an interesting idea, but my first reaction is that it wouldn't end spam. In the US there is a much larger cost associated with sending physical spam in the US (stamps). But there seems to be unlimited physical spam still. Maybe it would just become more targeted?
    • waynecochran 11y ago
      Probably not end spam completely, but reduce it to a point it wouldn't be noticable. In any case, I would love to only send/receive encrypted/signed email. I don't think the problems are technical.
  • deleted 11y ago
    [deleted]
  • teddyh 11y ago
    Please read http://craphound.com/spamsolutions.txt http://craphound.com/spamsolutions.txt and check all boxes that apply.
    • waynecochran 11y ago
• codecamper 11y ago
  Here is an almost related story: I am in italy and was waiting for an email from the local Apple store to tell me my macbook's repair was complete. After 6 days waiting, I called the store. They said the computer had been ready for a few days. I checked the spam folder. Gmail had plopped the apple email into the spam folder. Gmail's reason was that the email was in a foreign language, Italian. Didn't matter that the previous 2 weeks all my google.com searches were done from Italy.
  • dredmorbius 11y ago
    Google's killed access to various accounts of mine at different times for: 1. Accessing an account one time over Tor. Subsequent reconnect attempts from the same device I'd previously used, same IP, etc., failed. Took a couple of weeks to get back. 2. Accessing accounts from different IPs while travelling. Ultimately ending up at the same IP as one of the email accounts I'd previously corresponded heavily with. 3. Accessing accounts from new device(s). Old (Android) devices could access email but not Web-based account recovery tools. "Who are you" is the most expensive question in technology. No matter how you get it wrong, you're fucked. https://www.reddit.com/r/dredmorbius/comments/3mo7l6/that_google_identity_thing_again_who_are_you_is/ https://www.reddit.com/r/dredmorbius/comments/3mo7l6/that_go...
    • georgenishimura 11y ago
      I was once disappointed not to hear back from a Google recruiter after an interview. Turns out it was in my Gmail spam folder. Along with any other @google.com address.
• Animats 11y ago
  My experience has been that sending from my server works fine if it's in DNS and RDNS. Sending in that server's name works fine if the SPF records are present. But I don't bulk send; everything I send is something I typed, other than some messages the server sends to me periodically.
• jasode 11y ago
  >This isn't how the internet is supposed to work. The email architecture was started back when it was a smaller network of researchers at universities, governments, etc. Everybody basically trusted each other. Once the "internet" is available to the general public and commercial interests, it becomes vulnerable to the "bad actors" problem (e.g. spam abuse). That's why we have the inevitable situation today of a few entities (e.g. gmail, hotmail) being "trusted", and random residential SMTP servers run by homeowners being "untrusted". I haven't seen a realistic de-centralized trust proposal for email. Even if a proposal is theoretically sound, what incentive is there for other big players to adopt it?
  • tracker1 11y ago
    Combining a notify/pull system with a requirement for a valid domain certificate from a pre-approved list of CA's, similar to those already in the browsers, would go a step farther... increasing the costs for operating a a badly acting domain only to be blacklisted relatively quickly. Unfortunately, that would be less than decentralized, but it may turn out to be the best option in combating spam. I've just opted to pay for sendgrid for my small hobby BBS server's outbound email, because it's easier than setting up an appropriate outbound system myself, and as TFA points out, even then odds are you'll be bitbucketted before you even start.
  • chmike 11y ago
    How do you solve the problem of "cold call" with a trusting system ? How does one build up trust with a new account ? Spammers could create million accounts at once and fake trust build up between them. That's why I'm not convinced that trust will solve the abuse of messaging. A messaging application must allow "cold calls". Relying on trust built up by others is not a reliable system because it can be abused.
    • jasode 11y ago
      >How do you solve the problem of "cold call" with a trusting system ? [...] That's why I'm not convinced that trust will solve the abuse of messaging. You can't remove "trust" or "reputation" from a viable messaging system. The concept of trust always exists whether informally or formally. When the internet was a small private network between universities and government, how did a new unknown scientist "cold-call" an email? It was not an issue. The scientist just sent the cold email. There was already implicit trust within the system to allow that scenario. Everybody trusted that researcher@someuniversity.edu didn't send a million emails about Nigerian money transfers to researcher@army.mil. Back then, email users didn't need "spam filters". The trust was informally created because the institutional "system" outside of the email system vetted email users. (The "system" being the hiring procedures of SomeUniversity, DepartmentOfDefense, etc.) All those gates for reputation are gone when we expose that simplistic email architecture to the general public. Trust doesn't go away. You now must recreate trust at a massive scale. >Spammers could create million accounts at once and fake trust build up between them. That's not the type of "trust" people are talking about in this thread. That type of incestuous "trust" between bad actors won't work because it doesn't have an initial authority (good actor) to "bless" them which starts an acceptable "chain of trust". The concept is similar to Certificate Authorities. We trust Chrome because we trust Google Inc; and in turn, Google Inc trusts Verisign; and in turn, Verisign EV-certificate processing trusts government offices that register businesses.
• calpaterson 11y ago
  I have self-hosted my mailserver for a long time and started to get problems a couple of years ago. The main issue is corporate networks running McAfee's "MxLogic" product that claim to bounce my mail and tell me so, but then go on to deliver it almost all the time. The difficulty in getting feedback is extremely frustrating, particularly compared with getting feedback from, eg, google's webcrawlers.
• kakakakaren 11y ago
  "half-defeating the purpose of having privacy by running my own server." Is there even privacy in e-mail? Say I have a server that talks TLS. There's still no guarantee the other end wants to, and even if it does it's just going to store my mail on disk unencrypted anyway. Even if it's encrypted there, it has to be decrypted and read by a process to make a webpage out of it, or, to allow a client to download it, where it will probably rest unencrypted on their system. You're not getting privacy. If you want privacy, encrypt it (pgp), or, use another protocol that is encrypted end-to-end and stays encrypted even after the recipient reads it.
• deleted 11y ago
  [deleted]
• dredmorbius 11y ago
  Many of the issues we're running into with online systems, particularly those relating to quality and reputation (spam, collaborative filtering or content rating as on HN or Reddit, etc.) have strong analogs in real-world social spaces. And there were real-world mechanisms for dealing with these. For a new businessman or professional setting out in the world, pre-Internet, "establishing your name" was a requirement. These days the concept's often referred to as "creating a personal brand", but the reality was pretty straightforward: how does an unknown quantity become a known quantity? A common method was the professional or social introduction. This is still practiced, where a third-party _matchmaker_ will introduce two parties. The matchmaker usually knows both, and can vouch for the newcomer and smooth the path for introductions with the established party. Essentially, the matchmaker stakes _their_ reputation by speaking for another. Lawrence Lessig describes a similar concept in his book Code and Other Laws of Cypberspace, in a passage describing a physical messaging system, the Yale Wall. This was a board onto which messages could be posted, with the proviso that that they be signed. Unsigned messages would be posted from time to time, effectively presenting an anonymous viewpoint. Removal wasn't instantaneous or automatic, but rather, at some point prior to garbage collection, another individual could review the piece, and, if they felt it warranted posting, sign for it. They weren't registering themselves as author, but as vouching for the merits of the viewpoint -- not necessarily agreement. A messaging system in which a new peer might be able to indicate that, hey, peers X, Y, and Z, with established reputations can vouch for me, and for which those peers could confirm their endorsement, might address the "how to build reputation" issues of new mailservers. I've also found that even large mail systems will frequently have some procedure for getting at least provisionally vetted, effectively a workfactor cost to coming online. Though the process absolutely could be improved.
• Sir_Cmpwn 11y ago
  Mail spammers are truly vile people. They have ruined it for everyone.
• staunch 11y ago
  This largely true and hugely disappointing. Our startup (https://portal.cloud https://portal.cloud) is making it possible for non-hackers to self-host their own email servers. It works very well for the most part, but we have had to explain to a number of them why their email sometimes gets bounced by the big proprietary cloud services. All of our users have their own domain names, IP addresses, SPF records, and correctly configured (and up to date) Postfix SMTP servers. There is absolutely no excuse for not always delivering their email, and yet this is what the big companies do.
  • ams6110 11y ago
    There is no obligation to accept email, from anybody.
    • staunch 11y ago
      There's no good reason a company should prevent their users from receiving email from their self-hosted friends.
  • phit_ 11y ago
    fyi: there's a little mixup on your main page, the $5/mo plan says 512gb ram instead of mb
    • staunch 11y ago
      Thank you. We caught it after a few hours. I should've been reading HN!
• jwr 11y ago
  This is indeed worrying. I've been running my own E-mail servers for the last 20 years or so and even though my problems weren't as severe, I did run into a few cases where the "big ones" were at least delaying my E-mail. But this kind of problem invariably arises when we go from a fragmented Internet with lots of small hosts/providers to an Internet of several walled gardens, run by the big guys. The real answer is to offer a reasonable self-hosting competitor to GMail.
• jmount 11y ago
  It is simple: the more fear uncertainty and doubt the "big email providers" can cast on no using one of the big email providers the more they chase everyone into their business (when they can read it). You can go on and on how it is "technically hard to fix email" but that is a second order effect to not even trying.
  • jmount 11y ago
    "no using" -> "not using". sorry.
• zer0defex 11y ago
  Just goes to show reputation isn't the end all, be all, solution to everything. It's so often championed by the Linux kernel team as a reason why the distributed model works, and it's evident that in that case it certainly does. Here though, much outcry about how you can't setup a box and instantly be as respected as established boxes that have earned rep over time. This post just comes off way too butt hurt for my taste.
• tete 11y ago
  I think it's exaggerated. I am happily running a private mail server on a tiny vserver. My emails get through to whomever I might mail and I do so a lot. I once heard that my email got flagged as spam. Of course, if you don't set up your mail server correctly it might be that it's flagged as spam, but it's not really harder than setting up various other things correctly. It sounds like it was set up correctly by the author. Most people use SpamAssassin, also big companies do. So it should be good. Maybe the network itself wasn't considered to be good by the mentioned big companies.
• teddyh 11y ago
  I sometimes see similar tales of woe, and I can only say that this does not match my experience. I’ve done this many times, you set up the mail server, configure DNS correctly (including reverse lookup), and that’s it. Never had problems being blacklisted or mail getting classified as spam. I suspect that people having trouble are sending a lot of mail, like “newletters”, etc. But I can’t prove this hypothesis.
  • rtehfm 11y ago
    Mail reputations are very real and pose an issue with mail servers. I had a gaming server for years and had many issues with Gmail, Microsoft and Yahoo, to name a few, filtering or blocking emails. Just last week, I setup a mail server using mail in a box on a new server I spun up on Digital Ocean using an IP that was on no blacklists and still had issues with sending emails to various Gmail subscribers. Even when I used to work at HostGator and handled many of their abuse issues, they had many issues with being blacklisted just because RBLs didn't recognize new HostGator IPs or the rate of email being sent from their new gateways. So, at least with my personal and professional experience, I can attest to the issues with using self-hosted mail servers.
    • teddyh 11y ago
      One more point of data from my experience: I have never set up mail servers at hosting providers in remote datacenters, only local servers in my own server rooms which I could physically touch. I guess it’s quite possible that Gmail, etc. have figured out by now where all the IP blocks of hosting providers are, and are very suspicious of them.
    • zhte415 11y ago
      New IPs are often old IPs that have been used for other purposes before, and often blacklisted. Especially IPs on easily spun-up and spun-down hosts like DO. On DO, after every VPS I've removed, and then started up another a week later, it has a different IP. It would be easy for a spammer/phisher/whoever to abuse this quick address re-allocation, but hurts regular users.
  • 11y ago
• phantom_oracle 11y ago
  The problem goes both ways. Spam needs to stop and so too does the convergence of Internet services in general (not just email). Frankly, the solution already exists, and just like how billion-dollar companies start with the tech community, techies need to embrace the idea of decentralization by adopting some trust-based model. Frankly, I do not see why encrypted emails cannot be accepted by users through some peer-sharing agreement where the public-key is online. There's tons of solutions that are easy for tech people and if this community embraces it, it trickles down, like how a lot of other things do in tech.
  • redahs 11y ago
    I don't think we will see a trickle down effect unless there are substantial benefits for the average user to outweigh the cost of switching and learning something new. If there is a single abstract advantages, like 'more security', that might not be good enough. We might as well throw in the kitchen sink and come up with a detailed list of every possible advantage an ideal replacement protocol should offer: - no domain names required to create addresses - 1 terabyte+ attachment file sizes - ability to update and delete messages which recipients have not yet downloaded - mail transfer and storage agent which don't have access to unencrypted message contents - mail agents which can be distributed across multiple personal and home devices - client apps which always and non-optionally use local encryption\decryption - ability to purchase mail serving and storage resources from a competitive commercial market directly from client application. - ability to switch to a different commercial serving\storage resource provider without loosing your previous identity\address.
• mgalka 11y ago
  I suspect many of my messages are getting flagged as spam as well. Is there an easy way of checking whether emails are going through?
• lisa_henderson 11y ago
  This is a true story: I rent some servers from the Rackspace cloud for personal use. I have my own sites on these machines, and my own email servers. Meanwhile, I have a day job, and lately it has been consuming 12 hours a day. We missed a deadline and we have all been working like crazy to catch up. I have fallen behind reading my personal email. Roughly a month ago, my friends who use Gmail stopped getting my email. Or rather, they did not know I was sending them email, because all of my email to them was going to spam. After a few weeks, I finally had a free weekend to catch up on my personal life, so I did some investigations. Turns Rackspace had switched over to IP6 in a way that impacted my email. I did not have a Sender Policy Framework for IP6, only IP4. It's likely that Rackspace sent me an email about this, though I never read it because I was busy. This was easy to fix: I added a SPF for IP6. However, these kinds of issues do make it harder to maintain a personal email server. Its tough for us to keep up with the changes.
• eridius 11y ago
  I feel like this is a solvable problem without making any changes to email whatsoever. The problem is the email recipient hosts are suspicious of the sender (as opposed to the message itself being suspicious). So the solution is to have a standardized way for senders to acquire an instantaneous reputation by tying their real-world identity to it (which lets them be held accountable if they do spam), and perhaps by throwing some money at it too. If there was some company that did identity checks, similar to how EV certificates are given out, then that ties your real-world identity to it (this could in fact be done by literally requiring an EV certificate for the hostname of the sender). This company could also take a decent-sized deposit (so you're staking money on not being a spammer) and hold it in trust for a set amount of time. Once the time has passed, and you've sent enough emails for recipients to draw meaningful conclusions, if you have in fact not spammed, then you get your deposit back (minus a service fee). Then all the big email hosts would pay this company to query it about senders the host doesn't already trust, and similarly they'd report any spam from these hosts back to the service. Heck, this doesn't even have to be a new company. A big host like Google could just start offering this service anyway, as a way to simplify their own handling of unknown senders, although I'd feel more comfortable if this was done by someone else.
  • glogla 11y ago
    You mean extortion. Give us money or we will drop your mail.
  • notatoad 11y ago
    The core problem to this is that reputation is valuable. Google or Twitter or Facebook or any of the other big web presences could offer a method to determine "reputation" of any of their users, and a way to canonically tie your identity to that reputation, but they'd be undermining their own business model. They want you to stay in their network, because the safety of interacting with only reputable users is the big selling point of their walled-garden social networks over open networks like email
    • eridius 11y ago
      What do you mean by "stay in their network", when it comes to sending emails? Google is not in the business of being a paid mass email sender for businesses. What business email they handle is specifically for being the email host for a business's internal email, not sending their newsletter / marketing / account notifications / whatever else.
• tacon 11y ago
  I've managed my own mail server since 1993, and my email address has been the same that entire time. Here are some tips for maintaining sanity: Greylisting still works amazingly well. With a long, long whitelist and greylisting plus DNSBL, I don't even bother running a spam filter, since the little bit of spam and emails from new senders ends up in its own directory as it came from a non-whitelisted sender. Comcast finally started blocking residential mail server ports inbound a few years ago, so I had to migrate to a smarthost environment using a VPS as email server for $15/yr.[1] Last year for a few months, Gmail was dropping everything I sent into the spam folder, even after recipients were marking it not spam. I eventually discovered the "Authentication-Results:" header that Gmail adds to every inbound message. It is under the "Show Original" dropdown menu. That showed that I "hadn't changed anything"(!) on my mail server, but suddenly Gmail was connecting to my mail server over an IPv6 interface, and I had never bothered to put the IPv6 block into the SPF record. Gmail was nice enough to explain exactly what it didn't like about those emails. [1] http://lowendbox.com/blog/top-provider-poll-2014-q3-the-results/ http://lowendbox.com/blog/top-provider-poll-2014-q3-the-resu...
  • niravshah 11y ago
    > Greylisting still works amazingly well. With a long, long whitelist and greylisting plus DNSBL, I don't even bother running a spam filter, since the little bit of spam and emails from new senders ends up in its own directory as it came from a non-whitelisted sender. Any good tips on this section in particular? If I'm running my own mail server, how would I get started making sure this is in order?
    • dijit 11y ago
      old but valid HOWTO blog post. http://blog.philippheckel.com/2010/01/28/how-to-postfix-as-mail-relay-with-greylisting-support/ http://blog.philippheckel.com/2010/01/28/how-to-postfix-as-m...
    • tacon 11y ago
      I have been running exim4 for years, but I'm in the process of moving to postfix, as postfix is considerably easier to set up all the DKIM, etc., machinery that is now required. Inbound email comes through procmail and is mainly read in emacs (mh-e), which is kind of old fashioned. I have a small script that makes a new email address within my domain for each new use. I sign up for a lot of mailing lists and groups, and my /etc/aliases is more than 5300 lines. I can track if domainA's address starts coming from domainB and disable that address, but that doesn't happen very often, which is a pleasant surprise. I also have a small script that puts a new sender on my whitelist of sender email addresses. My whitelist is 12000+ lines right now, collected over many years. Procmail sorts to mailing lists and vendor folders, and finally puts things that are not on the whitelist into a "possible spam" folder. From the five or ten items a day, it is easy to spot legitimate emails and I add those to the whitelist. The majority of spam is blocked by the combination of greylisting and DNSBL lists, as the delay of greylisting (ten minutes for me) is enough for them to make the blackhole list, if they happen to ever attempt delivery again. I was thinking recently that I should be collecting statistics on the use of a lot of those aliases and whitelisted emails, and maybe start garbage collecting my lists. There are various reputation reports and services that can tell you how your mail is doing in the major ISPs, but a lot of those require higher traffic than a personal or small business generates. There is one service, DMARC[1], that is free and can give you some visibility into how email from your domain is being processed. I put the txt record in my DNS, and Google, Facebook, Comcast, Yahoo, Fastmail, and a few others send me reports about email they have processed from my domain. It's not that interesting at the moment because things are working, but it might help to debug issues if your email was being rejected. At least I see a few spammers are trying to use my domain from their servers. [1] https://dmarc.org/ https://dmarc.org/
• gwu78 11y ago
  It may be infeasible to run a new SMTP-based mail service from "residential IP's" that can interact with the existing email empire, dominated by store and forward middlemen who expect to make money from the "free" email service they provide. That empire amounts to a junk email delivery service and later a way to gather information about email users. The later purpose is probably why you want to run a new email service? However it is certainly feasible to run a new SMTP-based email service from residential IP's that does NOT interact with the existing email empire. One with no middlemen. The sender's SMTP server talks directly to the recipient's SMTP server. You decide what port you want to use. There are thousands to choose from. There are multiple ways to do this, but I rarely if ever see this option discussed. I suspect it's because like DNS most users are not comfortable configuring mail servers nor with NAT traversal. If indeed the motivation for running your own mail service is because you do not want your mail stored on third part servers (whether in the sender's mail folders or the recipient's), then the ability to interact with the existing store and forward email providers seems a counterproductive requirement.
• 9248 11y ago
  > This isn't how the internet is supposed to work. As we continue to consolidate on a few big mail services, it's only going to become more difficult to start new servers. And this is exactly the reason I setup my own mail server. I'm only 1 man, but I hope more people will do so with time, thus requiring the "big ones" to work on better algorithms for filtering and not base it on reputation.
• maerF0x0 11y ago
  Nobody else has mentioned it, so i will; This is a great situation for the NSA. Get all 400M accounts in one fell swoop by using a gag ordered warrant on Google (or microsoft) Ez peasy. Much harder than to contact a sysadmin about their 1 account that isn't being fed into the behemoth..
  • patrickaljord 11y ago
    > Nobody else has mentioned it, so i will Do we really need a comment against the NSA in every single thread mentioning emails or hosting in general? Not that I'm a NSA supporter or anything.
• grey-area 11y ago
  Everyone seems to agree that email is broken (and yet incredibly useful and almost universal in reach). So moving on from there, how do we fix it? Who is currently working on fixing it? What would a new protocol look like?
  • bachmeier 11y ago
    Chat services such as Slack, and social media like Twitter, are excellent ways to communicate. It's pretty simple. If you implement a request system so that both parties have to agree to let the other send messages, all of the problems are solved. There's no good reason that we need a system that allows anyone to send an arbitrary number of messages to anyone else.
    • deleted 11y ago
      [deleted]
      • bachmeier 11y ago
        > but there is a compelling reason -- this ability to send junk to anyone has allowed some people to make money consistently for decades How is that a compelling reason? Why would I want to open myself up to spam so that others can make money? There is a compelling reason that we use the current email system. We do it because we have to. Nonetheless, communication over the internet without spam is a solved problem, whether or not we choose to take advantage of the solutions.
        • deleted 11y ago
          [deleted]
  • pmlnr 11y ago
    This is the big question I'd like to hear an answer for as well.
    • JdeBP 11y ago
• bad_user 11y ago
  The irony is that at least half the spam I get comes from @gmail.com addresses.
• deleted 11y ago
  [deleted]
• gull 11y ago
  It's a lost cause. The blacklisting ugliness is one more sign email was badly designed. One more nail in the coffin.
• fensipens 11y ago
  > this server was configured perfectly: (...) SPF, DKIM and DMARC policies in place SPF, DKIM and DMARC are no indicators of spamminess of a source. These systems have a completely different purpose.
  • leni536 11y ago
    They should shift IP reputation to domain reputation though.
• atmosx 11y ago
  I did not realize that I am running my private SMTPd (+ imaps) for nearly 8 years. I never had issues. My score on test-mail is 9/10 because of lack of DKIM. I will implement DKIM, see if I can get 10/10.
• MortenK 11y ago
  OP, I don't know if you are reading the comments here but in case you do: Don't get discouraged so quickly. The reason this is happening is as the blurb from the MS postmaster help page: Your IP doesn't have a reputation yet. The reason these rules are in place aren't about email monopoly, it's about spam. If anybody could setup a SMTP server and start firing off large amounts of mail, spam would be even more endemic than today. You can configure your server perfectly, but that doesn't mean much, since it's your IP that's the problem. If you have legit objectives, it's a pain in the ass for sure. But you are not the only one having this problem, and there's a solution for it. All the big email service providers (ESP's) like Neolane, Exact Target, Mailchimp, Campaign monitor etc share this problem when they onboard a new client, who requires their own IP. Deliverability is a surprisingly deep, technical topic, and all major ESP's have entire teams of specialists working on this. If you want to make such a service as Fastmail, you need to get really into deliverability. It's not a walk in the park, but it's not impossible either. I'm not a specialist in this particular area myself, so I can't give you that much specific advice. I've just worked elbow to elbow with a lot of these guys, so I know what kind of challenges they work with. One thing I know for sure is really important, is the "warming up" of IP's. Basically the IP you are sending from needs to accumulate some reputation over a period of time, typically a month or two. If you send out reasonably small amounts of mail to email addresses that exists and the recipients does not explicitly report you for junk mail, your IP get whitelisted and you will get a much higher delivery rate. There's no quick fix unfortunately, and email reputation is hard to gain and fast to lose. But it certainly can be done. You sound very competent on the server side of things, so to get your fastmail-like service up, I think it's just a matter of a bit more persistence and studying deliverability as a technical subject. Hope this helps.
  • jodyribton 11y ago
    Thanks for the encouragement! I might take another shot at it sometime. I had this server running for about 3 months with just my personal mail, sending probably 1 email per day on average. It's possible slightly higher volume would do a better job of "warming up" the IP. I self-hosted from about 2007-2008 and 2011-2013, and had nowhere near as much trouble with deliverability. It came as a bit of a surprise how much more difficult it is these days.
• zrm 11y ago
  I was thinking about this a while ago and have been meaning to write it up and post it somewhere, so I guess this is as good a time as any. Hashcash (also known as the precursor to Bitcoin) was proposed to solve this problem in 1997: https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash The trouble with it is that it requires computation for each sent message, which is bad for senders with low resource devices or legitimate mailing lists. I want to propose a variant. Instead of creating an expensive hash against the message, create an even more expensive hash against the (sender TLS certificate, receiver domain name) pair. This implies using TLS but it works just as well even if the certificate is self-signed. Then each mail server only has to generate a hash once per recipient domain, ever. Every message that mail server sends to that domain is tagged with that hash. A legitimate mail server will have already computed hashes for all the domains its users regularly correspond with and rarely if ever need to do any more expensive computations. If spammers do the same thing then the receiving server can mark all messages sent with that hash as spam. So there is a highly disproportionate cost to spammers (even if they have more computing power) because to avoid that they have to continuously generate expensive new hashes. Which can be made arbitrarily expensive because legitimate servers only need to do it once. And a new hash is much less valuable to a spammer than a domain name or IP address is today because each hash can only be used against one recipient domain. The required amount of computation can be set by the receiving server so domains with more users can require more computation. If a legitimate mail server is compromised by a spammer then it will have to generate all new hashes (because the spammer will presumably immediately ruin the reputation of the compromised ones), but the reputation of the legitimate sender's email domains is unharmed because the reputation is tied to the hash computation, not the sender's domain name(s). And adding support to mail servers would require no configuration whatsoever. You install server version N+1 and it starts tagging outgoing messages with hashes that receiving servers can verify. So here's the traditional form: http://craphound.com/spamsolutions.txt http://craphound.com/spamsolutions.txt How'd I do?
  • tromp 11y ago
    Sounds like a plan! With a memory-bound proof-of-work system like my Cuckoo Cycle, computing the hash could require the use of more than 4GB of memory for over 5 minutes on a 20-thread server, thus preventing the use of botnets for avoiding the expense.
• z3t4 11y ago
  I guess big providers has to deal with "newbies" all the time, that don't know how to configure their server and run open relays. And don't know how to add extra headers etc. That said, silently dropping messages without a notification is probably illegal! And pretty serious! So if you know what you are doing (and you are not a spammer) you should send a cease and desist! Just make sure you use double opt-in and that providing an e-mail address in sign-ups/etc is optional! Some will however put your mail in a spam-folder and it's not much you can do about it, just hope your readers complain to their provider. So basically: setup your smtp relay correctly! Make sure you are not on any black-lists. Add extra headers like dkmi / precedence, add spf (don't froget ipv6) and ptr. Add your relays to white-lists. Publicly publish a privacy and e-mail policy (important! with opt in and optional clause); Link to them and fill out some "email provider" forms at gmail/microsoft. Send out a bunch of test mails. This will take a whole day, but if you do this, you will have no problems unless your IP or domain is perma-banned.
  • cortesoft 11y ago
    There is absolutely no law that says you have to accept an email, nor that you have to send a notification if you don't. A cease and desist?! Are you kidding?
• k2enemy 11y ago
  The anti-competitive consequences of this are really interesting. Does anyone know if there are historical statistics on email provider market shares? It would be interesting to see how things have changed over time.
• motoboi 11y ago
  McAfee Mail Gateway also uses a reputation, called GTI. Appliance hashes the message, looks up it on this online service and get back a score. There is a KB explaining that new senders get a high score.
• hannob 11y ago
  I have a bit of experience with running email servers. I can't really say that I had similar encounters. In my experience if you get blocked by big mail providers it's almost always due to some reason. What's tricky is that it may be hard to tell what exactly is wrong, because they won't necessarily tell you (or not in an easy way). Some advice what I'd do to try to find out what's going on: 1. Take a sent example mail that is like the blocked one (but obviously one that reached its target destination) with all headers and run it through spamassassin. Don't just look if it hit the spam score (then you did something terribly wrong), look at each individual rule that spamassassin hit. They might give you a clue. A proper mail usually shouldn't hit any or very few positive spamassassin rules. 2. Check your IP at a service like valli where you can query multiple DNS black lists. If it is on any blacklist try to find out how you can be delisted. There are some rogue blacklists that make it impossible to be delisted at all, you may ignore them (google for them, their behavior is well documented), but these shouldn't be more than 1 or 2. As already said by other commenters, don't forget IPv6. 3. Read whatever error message you can get your hands on. If you're blocked on the SMTP level read the error message. If your message got sorted into a spam folder look at all the headers. If the provider blocking you has some online docs about their spam filtering read that. If they have some sort of service for mail ISPs where you can sign up to get warnings sign up there. Of course also the obvious stuff. If you do anything that is mass mailing you are in extra danger. Make sure that you allow people to unsubscribe easily, don't ignore manual attempts by them to unsubscribe ("I want to get off this mailing list") and delete invalid mail addresses.
• TazeTSchnitzel 11y ago
  I use a private email provider (privateemail.com, via Namecheap), which doesn't seem to stop others receiving my emails, but sites sometimes have trouble emailing me, oddly enough. Though that might just be because I have a 6-character domain (ajf.me).
• jwatte 11y ago
  Charge the sender one cent per email through a combination of legislation and technology. The problem will instantly go away. The transfer of mailing lists to online forums will be a very small price to pay.
• skrebbel 11y ago
  It bothers me that this entire thread is about technology. This is anticompetitive behavior by a small group of oligopolists, plain and simple. The big providers need to build an accessible system for entering the email market or face tremendous fines from worldwide governments. There are plenty ways imaginable to do this, especially if you allow the process to involve real people and paperwork. It shouldn't matter whether this is simple negligence or conscious work to keep out competition. The end result is that it's made nigh on impossible by a few market leaders to enter said market.
• weihaw 11y ago
  Gmail has Postmaster tools that let domains with moderately large mail flow be able to monitor their spam reputation scores, email authentication e.g DKIM / SPF, DMARC rejection rates, etc. This tool can help address a number issues folks here have mentioned wrt Gmail. In particular please look at the "Delivery Errors Dashboard" description in the help center article. help center post: https://goo.gl/7QHoqc https://goo.gl/7QHoqc Postmaster tool: https://gmail.com/postmaster/ https://gmail.com/postmaster/ (disclosure I work in Gmail)
  • kintamanimatt 11y ago
    What about for the small time, personal servers? How do we get our mail delivered without having to use a third party service like Mandrill? I'm not blaming you personally, but the way Gmail handles email originating from personal servers is reprehensible.
  • antichrist 11y ago
    We are using Gmail Postmaster tools (I work at a small ESP), and while this tool is certainly useful for diagnosing rejections, delivery to Gmail inbox / promotions is still a black box. In our case, a number of IP addresses we are using started showing up as "bad" in Postmaster tools, despite having no increase in complaint rates (actually, in Spam rate dashboard it shows 0.0% for the past 90 days). We have implemented Gmail feedback loop headers into outgoing emails to diagnose the issue, but the Feedback Loop dashboard also does not show any data. All of these IPs have a Senderscore reputation in the high 90s and deliver perfectly fine to all ISPs, except Gmail. Contacting Gmail team through their form also yields no response. My takeaway from this situation is that once Gmail starts not liking your emails for some obscure reason, you have practically no way to fix that, apart from getting a new IP / sending domain and starting to build sending reputation from scratch.
  • gdr 11y ago
    It's not a solution for low-traffic personal email servers. I've tested 3 domains in Postmaster Tools and for all 3 there's no data (I've added them a month ago or so - there should be data already) No data to display at this time. Please come back later. Postmaster Tools requires that your domain satisfies certain conditions before data is visible for this chart. Refer to the help page for more details.
• chmike 11y ago
  SMTP Mail is broken. The best justification is that error messages can't even be sent back to avoid abuse by spammers. People are constrained to use services like gmail to avoid troubles. This is a serious privacy threat.
• pilif 11y ago
  OP might still have some configuration issue (check the headers of the mail that has been placed in the spam folder. Gmail usually tells you what the problem is). Last August, I have added IPv6 to our mail server, so its address for sure didn't have any prior reputation. Once the PTR and SPF records were correct, sending mail over v6 to gmail was no problem at all. I have yet to see any delivery issue over v6 that's caused by a lack of reputation
• technion 11y ago
  What has completely decimated mail management in my experience had been the rise of cryptolocker related emails. You could always deal with traditional viruses by blocking certain types of attachments. A false negative on spam just annoyed someone with some Viagra sale they probably didn't want. Nowadays, you fail to block an email and suddenly a user loses his entire department's file share for the day.
• leni536 11y ago
  So, why they look at the IP address at all when there is DKIM in place?
• al2o3cr 11y ago
  Drinking game for this thread. Start at the top of the comments: * every time someone implies spam filtering is a massive conspiracy against "the little guy", drink * every time someone suggests Bitcoin be used for something, drink * every time a solution utterly fails to account for compromised end-user machines sending spam, drink * every time a comment can be summed up as "assuming a PKI exists, this problem is trivially solved", CHUG THE REST OF THE BOTTLE
  • dcposch 11y ago
    LOL That reminds me of an old classic: http://craphound.com/spamsolutions.txt http://craphound.com/spamsolutions.txt There are a lot of obvious problems with email. * It's hard to run your own mail server, as described in the original post * Your address is tied to your provider * The standards involved in delivering mail (SMTP, POP, IMAP, etc) are complex, layered with hacks (SPF, STARTTLS, etc), hard to implement correctly, hard to secure. * The standards that govern the actual body of email (multipart MIME, a 90s-vintage subset of "HTML" and "CSS") are even worse But despite all those flaws, email is built on ubiquitous open standards running on a federated infrastructure. It's the best thing we have. Proprietary standards and centralized designs like AIM, ICQ, MSN, FB Messenger, Slack, and Twitter DMs come and go. Email is forever, and is more useful than all of those combined. The flipside is that it's very hard to "fix".
• jorangreef 11y ago
  Outlook.com is possibly the worst offender at present: 1. They set SPF policies of "-all" on customer domains. These customers have no idea about SPF and often use 3rd party software to send account statements that never get delivered. This breaks email for their customers and for everyone else. They should set "~all" by default as Google Apps does. 2. They claim not to, but they definitely block entire IP ranges rather than a single IP address. A noisy neighbour in the same datacenter as you can get you blacklisted. Use http://www.senderbase.org/lookup/ip http://www.senderbase.org/lookup/ip to monitor your neighbours and report them to your hosting company. If you spot a bad neighbour, you will no doubt see your own IP blacklisted at Outlook.com's https://postmaster.live.com/snds/ipStatus.aspx https://postmaster.live.com/snds/ipStatus.aspx within a few hours. 3. They are well known for accepting mail and then silently dropping it, without bouncing sender or receiver and without putting it into a spam folder, even when the sender is a longtime sender to that receiver and when the sender is sending personal anticipated email. They should either reject at the SMTP transaction (sending a bounce later is bad practice and leads to backscatter) or else deliver to a spam folder. Silently dropping email breaks email for everyone. The sad thing is that many businesses are switching to Office 365 and Outlook.com.
• chei0aiV 11y ago
  Another post lamenting this situation: https://sfconservancy.org/blog/2015/sep/15/email/ https://sfconservancy.org/blog/2015/sep/15/email/
• andrewrothman 11y ago
  Email is such a broken system. It's unacceptably hard to get a message from one machine to another and have it not marked as spam. Furthermore I can't stand when I have to setup all of my accounts across the multitude of devices that I own. Way too many settings to configure regarding ports, domain names, encryption techniques, folder mappings, etc. Don't get me wrong, I'm super impressed that email has managed to continue to be an essential tool in many people's day to day workflow despite being old enough to be found in a museum. But it's time for a change, and for us to put all we've learned about encryption, data serialization, and user workflow out on the table to design a better future for the universal method of digital asynchronous threaded communication. And if we've learned anything from Google Wave and friends, these changes have to make sense, and not stray too far from what we're already familiar with and know that works. Anyone want to weigh in?