14 ms·
I fought the battle to keep my SMTP server IP off blacklists, and lost. You can do everything possible, have a perfectly clean IP, have a good amount of outbou
by mjrpes 5y ago
I fought the battle to keep my SMTP server IP off blacklists, and lost.
You can do everything possible, have a perfectly clean IP, have a good amount of outbound email traffic, only send transactional email, etc. Still, there will be edge cases where email does not go through. AT&T email servers would constantly blacklist me and not respond to requests to remove me, gmail/yahoo/outlook would silently put emails in the spam folder, and companies using email firewall products would blacklist me, with an IT Dept too inept to fix it.
The solution was to pay a small fee and proxy all outbound email through a transactional SMTP sender, like Postmark or Mailgun. It's easy to do, with one line of code in Postfix. You can be selective, and only proxy emails sent to certain troublesome domains. If you try an email provider and it's not working out, it's one line of code to change to another provider.
This allows me to still manage nearly all aspects of hosting my email server and control my email data, while not dealing with deliverability issues. I use Postmark and I have not dealt with a deliverability issue in two years.
- Silhouette 5y agoWhile I am happy for you that you have found a solution, the solution you found is symptomatic of a very dangerous situation: it is increasingly impossible for individuals or SMEs to use essential online facilities like sending messages or transferring money reliably unless they use a broker service as an intermediary. We are allowing small numbers of tech firms to take control of vital functionality that should be using open, standardised protocols in a decentralised way. This leaves everyone who isn't big enough to run their own implementation that others can't afford to ignore beholden to those brokers and subject to arbitrary charges and/or denial of service.
- YPCrumble 5y agoI agree completely with this, but the one problem is that you haven’t addressed how we control spam without these “trusted” intermediaries. “Trusted” here meaning that they aren’t spammers.
- netr0ute 5y agoThe solution is to eliminate spam filters because that is the only good choice.
- uoaei 5y agoNo one is free until we are all free to send spam.
- Silhouette 5y agoSpam has largely been a solved problem for decades IME. You don't need some big-data-crunching mega-mail-host to block it successfully. For my personal mail, I use a small provider that isn't configured to block anything automatically and the built-in tools in my mail software. For my businesses, we have a pretty standard SpamAssassin-style setup. Either way, I see hardly any spam in my inbox despite receiving mail to multiple published contact addresses for those businesses, and I also can't remember the last time a false positive resulted in missing a legitimate mail. Meanwhile I've seen people miss events because $BIG_MAIL_PROVIDER decided the invitation was spam, I've seen recruitment go wrong because a CV from an excellent candidate was blocked on its way to the designated email address for applications, and countless other examples where bad spam blocking was throwing baby out with bathwater.
- Karrot_Kream 5y agoSure for you, but all of these other mailservers still obviously find value in applying spam filters or they wouldn't keep filtering. The problem is that it's much easier to send email than it is to receive it. This puts the onus of spam filtering on the recipient. I'm sad that HashCash or some other PoW scheme was never adopted as a way to force rate limiting of mailers.
- Silhouette 5y agoSure for you, but all of these other mailservers still obviously find value in applying spam filters or they wouldn't keep filtering. But since they have no accountability, their incentive is to eliminate anything potentially hostile regardless of the collateral damage from false positives. What are their users going to do? Take their business to another big provider that will probably be doing the same, and in the process lose access to a bunch of online services they like because their accounts on those services are tied to the old address? Move to a small provider, which is being forced out of the network by the giants so can't deliver mail reliably? We are rapidly heading for a world where email is no longer a reliable communications medium for genuine messages between friends, family, work colleagues, etc. The giants are killing email through death by 1,000 cuts instead of a shotgun to the face but the end result is the same.
- SturgeonsLaw 5y agoThis is the real value of cryptocurrencies. Yes, I know HN doesn't like them, yes there's a bunch of get-rich-quick bros and scammers out there, please try and separate the grift from the tech and consider how vital it is that people are able to control their finances without a third party having the ultimate say as to whether a transaction takes place or not.
- walrus01 5y agocryptocurrencies and not screwing up global implementation of SMTP are completely different things. cryptocurrencies are not in any way going to solve the problem of people centralizing their MX all onto office365 and gsuite.
- dtech 5y agoThe only currently realistic way to acquire cryptocurrency or for non-tech people to use it is through a 3rd party broker. It's about as difficult as running your own SMTP server I'd say. edit see this current front-page submission about how Bitcoin fails to provide this despite being centralized in exactly the same way as the decentralized SMTP: https://news.ycombinator.com/item?id=30224637 https://news.ycombinator.com/item?id=30224637
- dane-pgp 5y ago> for non-tech people to use it Then it should be only the SMTP server operators who have to handle the crypocurrency side of things, so email users never have to worry about it. As you say, acquiring and using the cryptocurrency would be about as technically difficult as what the SMTP server operators are already doing, and there are a variety of 3rd party brokers they can choose if they want to simplify things and not run a node themselves. The system I'm imagining is one where each newly registered domain has to put up a cryptocurrency bond if the registrant wants to send email from it. Existing domains would be grandfathered in (having already built up their reputation) and new domains would have their bonds burned if some N-of-M stakeholders agreed that they were sending (DKIM-signed) spam. Choosing those stakeholders would be controversial, but hopefully no less controversial than the system we have today where Google can use the threat of a Gmail blacklist to make every SMTP server in the world follow its wishes. Ideally some of the stakeholders would be non-profits like the ISRG and Mozilla Foundation.
- kbenson 5y agoThis is just the internet moving to match the real world. In the real world, reputation matters and some people don't want to talk to you unless someone can vouch for you. For areas where the general public needs to interact, third party intermediary services spring up to fill this need. This is why for any store over the size of a mom and pop operation in a neighborhood, you can't just tell the owner who you know by name to put it on your account, and instead for credit purchases you use a credit card company which acts as an intermediary and smooths problems over on both sides, and refuses to work with stores and people that are untrustworthy. This is why there are mailing list (mass email) services and why mail servers allow them. They keep their customers working within the accepted bounds (they ensure removal works and fire clients that abuse), and this allows mass email for accepted reasons while still being able to come down hard in random exploited servers/accounts. This is why big email services are very selective about what servers they talk to. I work at an ISP where our main outbound mail servers are on IPs that we try not to change because they've got decades of reputation attached. Even so, we recently brought up two new servers for email forwards, and shifted a small percentage of our mail queue traffic to them and ramped it up over a couple weeks, and that seemed to work "warming them up" to the likes of Gmail and yahoo, etc. It used to be there were lists of mail operators you could be part of and you could use reputation within that to get them to be lenient with you when you started. These days it's all so centralized in a few very large players that they really likely just talk to each other.
- lytefm 5y ago> and that seemed to work "warming them up" to the likes of Gmail and yahoo, etc Yes, warming up a "fresh" IP definely works. If you suddenly send thousands of mails from a new server - sure, you'll be labeled as Spam. If you slowly increase the volume over time, have a good domain and recipients that interact with the email, things should be fine. GMail has quite helpful guidelines [1]. 1: https://support.google.com/mail/answer/81126?hl=en https://support.google.com/mail/answer/81126?hl=en
- deleted 5y ago[deleted]
- 5y ago
- jart 5y agoThe open internet is dead. Spammers, fraudsters, and abusers have killed it. I can see how you might hate me, but I only speak the truth. Once upon a time the Internet consisted of a bunch of anonymous university students and u.s. government, Bell, IBM, etc. employees doing unfettered research. Today the Internet consists of anonymous grifters trying to squeeze a buck out of you any way they can.
- RunSet 5y agoI hope to see bitmessage supplant email for many use cases. It is decentralized and has built-in mechanisms to fight spam. Also all messages are end-to-end encrypted. Despite the name it has nothing to do with bitcoin or cryptocurrency. https://wiki.bitmessage.org/ https://wiki.bitmessage.org/
- Silhouette 5y agoThe open internet is dead. Spammers, fraudsters, and abusers have killed it. I don't accept that. Nothing about defending against those threats requires the kind of guilt-by-association spam blocking we have been discussing. There will always be bad people in the world but that doesn't mean we allow some random big business to tell us where we may go, what we may do or who we may talk to because it is in that business's interests and (coincidentally or otherwise) it might make us safer.
- deleted 5y ago[deleted]
- tinus_hn 5y agoThe flip side is that before, when the rules were lax and everybody could send email, everyone would get at least 10 spam mails every day, more if your address was online somewhere. Where now most people get one a week.
- js2 5y agoHow do you know whether your mail is going through? I can understand for messages that bounce, but how about mails that are silently dropped or end up in spam folders?
- noduerme 5y agoIn our case it's quickly obvious since our server sends verification codes when people sign up for online accounts or change their passwords. We'll see a pattern of customer complaints from certain mail services within a day of being on any major blacklist.
- mjrpes 5y agoI can never know for sure if email is avoiding the spam folder, but everything tells me if it does happen, it's quite rare now. First, I will send emails to test accounts I have set up with major email providers (gmail, yahoo, outlook, etc). I used to do this often, but they have all been going to the inbox on my last few tests, so I haven't done this recently. Second, is feedback I get from recipients. When I send an email and ask for a reply, I'll get a reply. I also used to hear a few times a month, "sorry for the delay. I found your email in my spam folder!". This has gone away.
- johnklos 5y agoHonestly, we usually find out after communication has failed and some other form of communication is used. It's usually worthwhile to remind businesses that when they use "free" services like privacy-paid outlook.com and Gmail, they'll get what they pay for, and if their communications really matter, they should find proper email providers.
- noduerme 5y agoI manage an outbound mail server for a mid-sized company. I happen to also use it for my own personal mail. We have had on and off deliverability issues for years (AT&T and Comcast being the worst). As head of IT it fell to me to post whitelisting requests and try to get mail delivering again. I decided after awhile that this really isn't my job, and made a suggestion to the CEO which he took to heart: There is another solution besides changing IPs, using a paid sender, or filling out whitelist requests into the void: Get your legal department involved. We have repeatedly been taken off various public and private blacklists by having lawyers do their job. Once we went this path, it was like magic. Same day responses from those companies, and we haven't been on any blacklists for a couple of years.
- walrus01 5y agoNot everyone can spend $500 on lawyer billable hours per SMTP destination multiplied by N number of destinations. I also think that the likelihood of success in sending legal threats to somebody that demand they accept your SMTP traffic will not stand up in court, if you ever escalated it that far. As somebody who runs postfix MX on the receiving side of things, I can guarantee you that the day I receive a legal threat from some unknown third party with which I don't have a pre-existing business/contract relationship, demanding that I accept their email, is the day that I blacklist their entire organization and tell them "okay, I'll await service of your statement of claim". You actually think that the best answer to a network engineering problem is to make legal threats at third party ISPs? Companies with which you don't have a signed service order contract and/or master services agreement? You say you're a mid sized company. I think you're running a huge legal risk of angering a Comcast or AT&T size entity that has much deeper pockets and legal resources than you. The day that one of those giants calls you out on your bluff is going to be very expensive. On an ISP-to-ISP relationship level, this is not how you solve SMTP flow traffic problems. I can tell you that if I went to a NANOG conference representing my AS and proudly told other people "oh yeah, we've started sending threats from our lawyers to $OTHERISP1 and $OTHERISP2 because they won't take our mail traffic", that I would quickly be treated as a pariah.
- mhh__ 5y ago
- TedDoesntTalk 5y agoI did not even know services like Postmark and Mailgun solved this problem. Thank you!
- mjrpes 5y agoTo be clear, Postfix is the Open Source Linux software for sending email. In a perfect world, Postfix would be all you need for sending. Providers like Postmark and Mailgun let you forward and send email through them. Essentially, you are paying these companies to deal with IP address reputation management and ensure high deliverability.
- suzzer99 5y agoMy Dad still sends all emails to my main address and my gmail address, due to the occasional spat between netzero and godaddy that would block emails to my main address.
- AviationAtom 5y agoI was flabbergasted that relaying their mail through a service that does high volume, and enforces abuse policies, wasn't one of his listed options. As the Internet has become, due to rampant spamming, it's become best to just outsource mail relaying.
- deleted 5y ago[deleted]
- makecheck 5y agoThe funny thing about that work-around is, what stops spammers from doing it either?
- dqv 5y agoA spammer is going to get blocked and then have to send another letter ad infinitum. It's not financially feasible to do that. The respondent should also have evidence available to prove that the spammer is acting in bad faith (if they don't, then maybe they should get some before initiating a block).
- dataflow 5y ago> and control my email data Are you though? Aren't you giving your outbound emails to Postmark or Mailgun in that case?
- mjrpes 5y agoYes, Postmark keeps a 45 day history of emails for user reporting. After that it's deleted: https://postmarkapp.com/eu-privacy#security-and-privacy https://postmarkapp.com/eu-privacy#security-and-privacy It's a decision I am fine with. Others may not be.
- JohnTHaller 5y agoI've had a few issues with Mailgun getting blocked over the years. Especially with Hotmail/Outlook.
- nottorp 5y ago> through a transactional SMTP sender, like Postmark or Mailgun. ... cheapest is $10/month. Not paying that for my meager email traffic. Are there other options for personal domains?
- watchdogtimer 5y agoI've been using the free tier at MailJet to relay mail for my server for several years and have had few problems. Several others offer free tiers too (SendinBlue, etc.)
- nottorp 5y agoBoth of those seem to insert their logo in the emails for the free tier. I'd rather pay a little and get unmodified emails. Guess they all cater to marketers not personal use.
- maccolgan 5y agoAWS SES