14 ms·

Sendgrid under siege from hacked accounts

199 points by elsewhen 6y ago

jbverschoor 6y ago
Well, I've tried contacting sendgrid about phishing emails that look like they're from @sendgrid.com , using their own server. The SPF records match, so it isn't marked as junk mail. It's similar to the github pages exploit from years ago, which is why we now have github.com and gitub.io The purpose of these mails is of course to get credentials and spam some more. At first I checked for a bounty, but as this is a "configuration error" (SPF), there is no bounty. Then I sent it by mail, but yeah.. No reply.
- nodesocket 6y ago
  Can you elaborate how a 3rd party is able to send e-mail from @sendgrid.com?
  - jbverschoor 6y ago
    Simple. Their ip addresses of their client/guest-servers are whitelisted in the SPF records. So anyone can impersonate them. Not 100% sure if that was the case now, as I can't find the email anymore, but I've been getting them from both sendgrid and mailgun. Mailgun's SPF (was at least): v=spf1 include:spf1.mailgun.org include:spf2.mailgun.org include:_spf.google.com include:aspmx.pardot.com include:mail.zendesk.com include:spf.mailjet.com ~all spf1.mailgun.org: v=spf1 ip4:104.130.122.0/23 ip4:146.20.112.0/26 ip4:141.193.32.0/23 ip4:161.38.192.0/20 ~all v=spf1 ip4:209.61.151.0/24 ip4:166.78.68.0/22 ip4:198.61.254.0/23 ip4:192.237.158.0/23 ip4:23.253.182.0/23 ip4:104.130.96.0/28 ip4:146.20.113.0/24 ip4:146.20.191.0/24 ip4:159.135.224.0/20 ip4:69.72.32.0/20 ~all In the mailgun case I received mails from for example 198.61.254.24 with from: invoice@mailgun.com This is actually part of a bigger issue, where lots of people use the same email service to send out emails. In theory that should work with gmail or office365, but iirc gmail forces your from sometimes, and google+ms probably thought about this, right? ;)
    - monadic2 6y ago
      I'm surprised they give clients access to their own DKIM.
      - perennate 6y ago
        It doesn't pass DKIM. But the fact that it's sent through Sendgrid's platform gives the e-mail credibility. Because no one expects Sendgrid to allow random customers to set e-mail (not necessarily envelope) from address to @sendgrid.com.
        monadic2 6y ago
        Based on my experience with sendgrid it's par for the course.
  - XCSme 6y ago
    Someone sent spam emails using my domain name, so I looked a bit into it and made a summary of what I learned: https://www.usertrack.net/blog/stop-others-use-your-domain-emails https://www.usertrack.net/blog/stop-others-use-your-domain-e...
- perennate 6y ago
  Same -- I got several e-mails with subject "The following services failed to auto-renew and are about to expire" that I almost believed were real because they came from an actual sendgrid server (wrqvpkzw.outbound-mail.sendgrid.net [149.72.49.233]). If SendGrid can't develop simple filters to deter phishing e-mails sent through their platform targeting their own service, then obviously they're going to end up with a lot of users being hacked. I mean seriously, why would an e-mail provider like this allow e-mails to be sent from "renewaldepartment@sendgrid.com" and "support@Sendgrid.com"??
moomin 6y ago
I still remember when SendGrid got DDoSed after an incident at PyCon* and they caved. This told me two things: 1) they weren’t prepared to stand up for their own employees when the chips were down 2) their engineering wasn’t good enough to cope with a bunch of amateurs attacking them. * Let’s not derail this into a discussion of whether or not they should have done it in the first place. They didn’t, they only acted after they couldn’t cope with the DDoS.
- Drdrdrq 6y ago
  For those of us who don't know / remember the incident: https://venturebeat.com/2013/03/21/sendgrid-under-ddos-attack-after-its-developer-evangelist-complains-about-sexual-jokes-at-pycon/ https://venturebeat.com/2013/03/21/sendgrid-under-ddos-attac...
- deleted 6y ago
  [deleted]
bytecycle 6y ago
My university straight up blocks any mail coming from a SendGrid IP because of the massive amount of phishing. What would be a huge UX and security improvement is showing which team members are using 2FA. Just like GitLab shows a little blue label next to the username.
pmlnr 6y ago
That's actually good. Sendgrid & the rest are excluded from the paranoid gmail, m$, etc spam filters, while small senders suffer from being sent to spam immediately. There should be no exceptions.
- stefan_ 6y ago
  Yes, time for GMail et al to block them or admit it's just more anti-competitive nonsense.
- gruez 6y ago
  But is sendgrid "too big to fail"?
  - calmworm 6y ago
    Not even close.
  - dimitrios1 6y ago
    Too big to fail also was meant to communicate the vast amount of effort, time, and money that would be involved in rebuilding the financial infrastructure if indeed it had collapse, not to mention the immense short term pain and suffering that would occur in the interim. A SendGrid sized void could be filled in a matter of weeks to months by teams of highly motivated engineers looking to fill a market gap. The cost of starting up there is virtually nonzero (compared to say, starting a bank), and the regulatory burden is virtually nonzero (Email regulations are arguably some of the most clear and concise). All that to say, drawing a comparison with the "too big to fail" bailout of major US lenders is, frankly, silly.
    - Sebguer 6y ago
      I think you meant 'zero' not nonzero?
      - dimitrios1 6y ago
        Yes thank you
walrus01 6y ago
Sendgrid and its cohorts have always been a source of spam. The spamassassin setup on my postfix SMTP gateway ranks all of the bulk email newsletter sending services with +2.5 points or higher. This is like a sewage treatment plant complaining someone has come at 3am, cut the lock off their gate and dumped a tanker truck full of additional sewage.
- techsupporter 6y ago
  > Sendgrid and its cohorts have always been a source of spam. Sendgrid is the source of a lot of e-mail spam and their parent company Twilio is the source of a lot of SMS spam. It's really a synergy match made in MBA heaven.
aspectmin 6y ago
Sigh. Same happened to me. A few months ago. Sendgrid account hacked, someone set up a campaign and sent a huge amount of spam mail. Big bill on my credit card. Contacted Sendgrid, they blamed it on me. I cancelled my account, never going back. Screw them
- jeffbee 6y ago
  Was it not your fault?
  - PakG1 6y ago
    Is that sarcasm?
    - bawolff 6y ago
      Users are at least partially responsible if they use a non unique password and then get hacked.
      - tialaramex 6y ago
        However, passwords are a shared secret, so you actually don't necessarily know who was responsible for the secret leaking, you or the other party. Most proposed replacements or supplements are no better except (you can guess I'm about to say) WebAuthn. WebAuthn is built out of public key signatures, so a Relying Party has no idea how to present valid credentials even for their own system. Obviously you could just add a back door if you want one, but the WebAuthn credentials you're storing aren't like a password, or a password hash, or a TOTP seed. They're just an opaque ID and a public key.
        bawolff 6y ago
        Thats very true. But i think if sendgrid leaked their password db the level of spam would be mich higher. Of course its always possible the password db leaked but it was using argon2 or something so only really weak passwords could be decoded. I agree webauthn is great, and its the only option (other than TLS client certs which is a usability disaster, and maybe push notice 2fa depending on how its implemented) that protect against server side breach. However the vast majority of issues are password reuse, and to a much lesser extent dictionary attacks. TOTP effectively stops those. Then again so would choosing the users passwords for them.
crizzlenizzle 6y ago
This has been ongoing for months already. Nobody at Sendgrid cared. We had several issues with them leading to penalize mails from their IP space. I’m actively discouraging my customers from using their service.
jlgaddis 6y ago
> ... over the past few months there has been a marked increase in malicious, phishous and outright spammy email being blasted out via Sendgrid’s servers. Fortunately, I hadn't noticed the recent increases. After it became apparent that they didn't bother looking ibto -- or even reading -- abuse reports, we blacklisted basically their entire IP ranges a long time ago and then added exceptions when they were requested. I certainly did notice the marked decrease in spam then, though.
urda 6y ago
E-mail is a hard problem to solve, and I'm honestly really glad I do not have to manage it day-to-day myself.
- pmlnr 6y ago
  Enough with this "hard to solve" nonsense; email has been around since the 70s. Google, Yahoo!, and Microsoft are f*king it up, that's all.
statquontrarian 6y ago
I've been a paying customer for years on their shared server plan ($15/mo) and I've had terrible customer service and lots of emails marked as spam (e.g. to my Mom) due to a bad reputation of the shared box. I haven't migrated out of laziness and inertia. This article and the fact that other comments mention that SendGrid is seen as spammy is the final straw and I'm going to migrate. What paid email senders do people recommend that aren't excessively marked as spammy by gmail, etc.?
- throwaway200829 6y ago
  Throwaway since I’m a former Twilion. Even though people love Twilio for their developer friendliness and nice APIs, what no one wants to talk about publicly is their disproportionate amount of revenue that comes from spammy behavior. Either as a company or it’s customers. I was in sales. I saw the revenue numbers and sold these deals. It’s staggering. Yet wall street looooooves Twilio because of their revenue growth. Note: Twilio owns Sendgrid.
  - lowdose 6y ago
    What I would like to know from a Twilion: Is Twilio used by commercial parties in the past to participate in voting in the European Song Festival?
    - tialaramex 6y ago
      Did you mean the Eurovision Song Contest, or something else?
- Shared404 6y ago
  You may have luck with protonmail's paid tiers. I use the free tier currently and have been pretty happy.
  - statquontrarian 6y ago
    Thanks but I host my own email so I'm only looking for an outbound SMTP server. I went to ProtonMail's business page and the pricing plan only shows full email hosting which I don't need.
afrcnc 6y ago
how I hate this kind of reporting 3 people complain on reddit and it's now "[COMPANY X] under siege"
- jlgaddis 6y ago
  Between the comments on the article itself and this HN thread, there's already more than 10 times that. It's not being exaggerated quite the way that you "hate". To further prove how bad SendGrid is, Invaluement recently spent a ton of time and energy to create an entirely new "Service Provider DNSBL" [0] -- just for SendGrid! In fact, they just announced it about a week ago [1]. [0]: https://www.invaluement.com/serviceproviderdnsbl/ https://www.invaluement.com/serviceproviderdnsbl/ [1]: https://www.mail-archive.com/mailop@mailop.org/msg11403.html https://www.mail-archive.com/mailop@mailop.org/msg11403.html
- krebsonsecurity 6y ago
  Spend a few minutes looking at the spam list threads linked in the article. This is not just a few people complaining. E.g.: https://www.mail-archive.com/search?l=mailop%40mailop.org&q=sendgrid https://www.mail-archive.com/search?l=mailop%40mailop.org&q=... There was a ton of material I did not include in the story, including a story from a company that had a client have 40 million phishing emails sent through their Sendgrid account, which it turns out was set up by an employee long ago who was no longer with the company and had not turned on 2FA (and probably was re-using passwords). I started reporting this story almost a month ago after receiving more than 3 emails from different IT experts who were really frustrated with the amount of malware and phishing coming from Sendgrid accounts. They were frustrated because they couldn't block Sendgrid outright because too many companies they were expecting regular emails from used the platform. The day before I published the story, I head from someone else who was getting phishing attacks spoofing Aruba Networks.
- tobltobs 6y ago
  You also could have a look on twitter how many people with hacked accounts try to reach sendgrid support via this channel.
jlgaddis 6y ago
The CSO said: > "... 2FA for customer accounts is the right thing to do, and we’re working towards that end ... This is part of the reason we acquired Authy ..." Cool, it sounds like they're making a effort to implement -- and require -- 2FA... oh wait, FIVE YEARS have passed since then!
- nodesocket 6y ago
  Twilio supports and soon will require TFA. I personally don't use SendGrid (use Mailgun), but I have to believe it should be coming soon (if not already supported).
- maerF0x0 6y ago
  see my comment here: https://news.ycombinator.com/item?id=24319007 https://news.ycombinator.com/item?id=24319007
jeffbee 6y ago
If you run a service that can be abused like this, you're going to need objective account abuse metrics to warn you before these things happen. Specifically you're going to need a Russian speaker to prowl around on Russian forums buying your own hacked accounts, so you can figure out the market price. If accounts on your system are selling for ten cents, might be time to reflect on account security.
- neolog 6y ago
  Do big companies actually do this (check market price by buying their own accounts on forums)?
  - timClicks 6y ago
    Absolutely. Edit: Although it's rarely direct. Normally big corp will engage a security consultancy and discuss type of reconnaissance as one part of a wider strategy.
PakG1 6y ago
I run IT for my organization and I am seeing a lot of phishing emails from Sendgrid. I haven't counted or done data analysis for proportion, but it's enough volume for me to really notice it.
jlgaddis 6y ago
This list is probably incomplete / out-of-date but ... 149.72.0.0/16 167.89.0.0/17 168.245.0.0/17 192.254.112.0/20 198.21.0.0/21 198.37.144.0/20
- jeffbee 6y ago
  Isn't their SPF record authoritative? ip4:167.89.0.0/17 ip4:208.117.48.0/20 ip4:50.31.32.0/19 ip4:198.37.144.0/20 ip4:198.21.0.0/21 ip4:192.254.112.0/20 ip4:168.245.0.0/17 ip4:149.72.0.0/16
  - jlgaddis 6y ago
    Usually, yeah, but not necessarily (e.g., for those with dedicated IP addresses). Even then, it usually will be since most customers will just add the (customer-specific) "include:nnn.wl.sendgrid.net", or even just "include:sendgrid.net". But, for whatever reason, I've come across several that add an "a:" or "ip4:" for their dedicated IPs instead (I assume there's a good reason for that, but I don't know what it is).
    - megous 6y ago
      Probably so that they can also send some mail from their own servers, in addition to sendgrid's.
Farfromthehood 6y ago
Sendgrid user for many, many years. I believe we disabled 2FA because it broke /prevented our client's ability to send via SMTP. No idea if they've fixed the issue (and honestly don't care). Make security easy to adopt of you want us to implement it. Also, organization-wide, we don't use 2FA that requires a mobile phone number/SMS (because we don't trust vendors with our phone number). *IIRC, Sendgrid initially only offered 2FA via SMS, but now you can also use an authenticator app.
- dylz 6y ago
  Phone 2FA is broken by default. Yes, it broke SMTP and it also broke all API access until they kinda-fixed API keys. Pretty much anything you used basic auth for at a touch point.
maerF0x0 6y ago
> Neil Schwartzman, executive director of the anti-spam group CAUCE, said Sendgrid’s 2FA plans are long overdue, noting that the company bought Authy back in 2015. It's worth noting that this mixes up history. Authy was acquired by Twilio in 2015. Sendgrid was acquired by Twilio in 2019, there's no reason to assume in the 4 years interim that SG should have used Authy.
- dylz 6y ago
  I think the point was that they own a literal 2FA company, but don't have 2FA
  - tialaramex 6y ago
    They do have 2FA they just don't require 2FA.
    - jaxn 6y ago
      But they only have SMS 2fa.
- psanford 6y ago
  For a company that went out and acquired a 2FA subsidiary, Twilio really doesn't seem to have their 2FA story together. For example you'd think a company with an division dedicated to authentication would support what the industry considers the best 2FA solution: WebAuthn. Sadly that is not an option for your Twilio account (other Auth providers like Duo support WebAuthn). Twilio has also been emailing me the past few months telling me I must turn on 2FA on my account. This is weird since I have TOTP enabled. I feel like Twilio must be internally conflicted about this. As a company dedicated to phone services they really want to push SMS solutions even though SMS based 2FA is vastly inferior to WebAuthn.
  - kelnos 6y ago
    > Twilio has also been emailing me the past few months telling me I must turn on 2FA on my account. This is weird since I have TOTP enabled. There's been a bit of confusion about this email: you can turn on 2FA separately for your Twilio account (identified by a 32-char hex string prefixed with "AC") and your user account (your email address). The email is talking about your user account. Even if you have 2FA turned on for your Twilio account, the email is asking you to turn it on for your user account. For some context for those who may not be familiar: the Twilio account is essentially a billing/project unit that is a container for Twilio resources you've purchased or configured. Multiple user accounts can have access to a single Twilio account, and a single user account can have access to multiple Twilio accounts. Enabling 2FA on the Twilio account means all users who want to sign into that account (regardless of their user account setting) will have to use 2FA. Similarly, enabling 2FA on your user account will require 2FA whenever you sign in, regardless of the settings on any accounts you may have access to. (Source: I work there, got confused about this email myself, and managed to clarify internally what was going on.)
adrianh 6y ago
What service do people recommend in place of SendGrid? Personally I need a service to send transactional emails from my company website with a low chance that they’ll be marked as spam. Tracking is not necessary (I disabled that in SendGrid).
- xenospn 6y ago
  Amazon SES can work for that.
  - nucleardog 6y ago
    And depending how much you need to send, if you’re already in AWS the first 62k messages every month are free. The only annoying thing is their sandbox and usage limits. You’re supposed to accept a lower limit, send high quality mail, and then request a higher limit once they’ve seen that you’re not generating a bunch of abuse reports. Slowly on boarding senders is good for delivery but makes it difficult to just do a simple cut-over if you’re currently sending more than whatever quota they’ll give you right now. You need to continue sending through your old service and slowly move your usage over.
- moltar 6y ago
  Postmark has been great for us for ages. They are strictly transactional and will flag accounts quickly that don’t follow this rule. This detracts spammers.
  - tobltobs 6y ago
    Sadly they do now also allow bulk emails and therefore will soon be as shitty as all other emails sender services.
social_quotient 6y ago
Maybe Twilio will use that run up (bubble) in stock price to allocate to some tech debt like this. 169% YTD 33.8% last 3 months I’ve always looked at SendGrid as a premium provider for transactional mail. Why recommendations for something better but similarly priced?
- a2tech 6y ago
  We saw our deliverability go way up with SES. It’s pricey for what it is but we’ve had really good luck with it.
- cortesoft 6y ago
  Unless a company is selling new stock, an increase in stock price doesn't give them any extra money to spend. It just means their shareholders are worth more.
darkport 6y ago
I developed a tool called shhgit.com that watches code commits across GitHub, Gitlab and Bitbucket in real time. In a ~24 hour period you will find 100s of valid SendGrid credentials/API keys so this does not surprise me. They really need to enforce 2FA and IP whitelisting for API key use.
edoceo 6y ago
Oof. So,I tried sending mails from our own domain and our mail server - but Google tagged us as spam. We migrated into Google hosting and it was fixed, now back to Spam. So we move to a provider like sendgrid, but then they have a hack. Damnit! When we hosted our own, we had DKIM, SPF, DMARC and such, clean IP and clean messages, we only sent account create and transaction notices - low volume, relevant messages to folks who opt-in. We had our little box setup right and we're polite actors but got tagged which killed our user experience (password resets went to Spam and folk called support) But it feels the "big-web" forced us into vulnerable providers for the spam game. We need more self hosted email and less centralized control.
- smnplk 6y ago
  `We need more self hosted email and less centralized control` We need to start over and build a truly open internet.
  - Reelin 6y ago
    > We need to start over and build a truly open internet. I want to agree, but email is actually an example of an open and federated protocol achieving success. It's pretty much everything that could have been hoped for as far as openness and interoperability go at the protocol level. Unfortunately, balkanization still occurs at the social (ie network) level.
  - dredmorbius 6y ago
    The open email Internet was buried in an avalanche of spam. Open alone is insufficient.
    - paulryanrogers 6y ago
      If email were architected to put the cost of storage on senders then I think the spam problem would have been more manageable. Instead it's so closely modeled on physical mail that recipients get overwhelmed quickly.
tialaramex 6y ago
I am not a Sendgrid customer, however examining their documentation what I think the "attacks" look like is this: 14 year old Jimmy guesses that Foo Corp has the Sendgrid username "FooCorp" and the password "FooCorp0" obeying a company requirement to use "At least 8 characters" and "At least one digit". Very secure. Much password. Jimmy connects to Sendgrid's old API and literally just provides username "FooCorp" password "FooCorp0" to send as much email as Foo Corp have paid to be able to send. No other steps needed. Insisting customers get 2FA would in practice mean: 1. Get all customers to sign up to Authy's phone-based 2FA and use it for Sendgrid. Accept that this will have a revenue impact, you're essentially firing customers who either were too lazy to sign up for this or regard this type of security as a joke. 2. Deprecate older APIs that allow username + password auth 3. Thus force users into APIs that rely on a bearer token they can only get using 2FA. And yet after all this work if Jimmy instead gets a copy of your bearer token (e.g. because some idiot commits it to github) then he can once again send emails as Foo Corp until somebody disables the stolen bearer token (and then generates a new one and updates all their running code). It's a poor show that in 2020 this is their goal, it's not even the thing they have and need to improve upon, it's just a future goal.
- cnst 6y ago
  The whole idea behind Sendgrid is a joke. It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be. As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless.) That's how horrible Sendgrid is, and how easy it is to make other people switch. P.S. I often send mail through shell scripts. Try selling me Sendgrid when my laptop already comes with Postfix/Sendmail, with spool support. Does Sendgrid even work when you have no internet connection? (Rhetorical question: no, Sendgrid does not when internet is not available; but Postfix/Sendmail works just fine even without any internet, and will spool all your mail until you get connected to an internet with port 25 not blocked.) Sendgrid is a downgrade however you look at it.
cnst 6y ago
The whole idea behind Sendgrid is a joke. It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day, if not less. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be. As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless, and actually improved reliability.) That's how horrible Sendgrid is, and how easy it is to make other people switch. P.S. I often send mail through shell scripts. Try selling me Sendgrid when my laptop already comes with Postfix/Sendmail, with spool support. Does Sendgrid even work when you have no internet connection? (That was a rhetorical question; the answer is, No, Sendgrid does not work when internet is not available; but Postfix/Sendmail works just fine even without any internet at all, and Postfix/Sendmail will spool all your mail until you get connected to an internet with port 25 not blocked.) Sendgrid is a downgrade however you look at it. When they move to MFA, it'll probably be easier to just switch to one of their plentiful competitors than deal with any new restrictions; that's why Sendgrid will probably be very careful around any new requirements; because their service is 100% disposable.
- cortesoft 6y ago
  You seem to not understand the sendgrid service at all. They aren't selling SMTP service, they are selling deliverability (along with tracking and metrics) You seem to not understand the actual problem at all if you think you will be able to successfully deliver email sent directly from a local sendmail client on any random IP that lets you connect over port 25. No major email provider will accept your email coming from a residential IP.
  - cnst 6y ago
    If that's what they're selling, they're not very good at it. Sendgrid doesn't support all applicable RFCs, and they don't support spool, so, they have exactly 0% deliverability if the recipient host is down for any reason, or uses greylisting. I've never had any issues with mail from my WiFi IP accepted. You'd normally send it just to your own server from your local laptop w/o any extra configuration, so, it's up to you how you configure your own server. I just tried sending to Gmail directly from Postfix on my MacBook, without any extra configuration, and it's been accepted by Gmail as well. Zero configuration on the laptop. Postfix and the mail CLI clients come preinstalled, and the mail just works.
sarreph 6y ago
At work we had a Sendgrid account that got compromised, after which all of our comms were (predictably) being marked as spam. After back and forth with support, we were told to continue sending as normal and that "eventually" our genuine emails would stop getting sent to spam. That lacklustre support response was enough to put me off the service. And then, funnily enough, a few days ago I received a reply from Sendgrid support about another account I had opened for a side project. This reply came in ~2 months late, where I was asking if I could be moved to a different IP range as all my emails were getting blocked by Outlook services due to the shared IP I was assigned being (seemingly) misused by spammers. Again, the response was "just keep trying as normal" and eventually my emails would work... Apparently. That's obviously not a viable business communications strategy. The funniest part about both these support responses was that despite one of them taking several weeks to get a response, I'm being pinged every few days asking me if my issue was solved. It's all well and good for me to wait months when I need a response as to why all my emails are getting sent to spam, but as soon as the ball is in their court, they don't mind pestering me for a response. I don't have enough bandwidth for a dedicated IP, so I signed up for a service that requires a credit card even for its free plan, which I believe is a decent enough barrier against spamming.
- old-gregg 6y ago
  > That lacklustre support response Frankly, this makes my blood boil. Full disclosure: I used to work at Sendgrid's competitor (Mailgun) and I'm quite familiar with what's happening under the hood. You allowed spam to be sent from your account. You poisoned the IP space (not just one address you were sending from, but the IP block, affecting others) with spammy reputation. These blocks are expensive and hard to acquire (due to IPv4 shortage) and at this point you have caused more damage to Sendgrid and their customers, than your account value can probably compensate for. What kind of "support" do you expect at this point? You rented a hotel room, gave the key to someone who erected a meth lab there, and you're unhappy with the hotel's reaction? Sending email on behalf of other people is hard work. The margins are not great, the ratio of spam-to-ham for new accounts is insane, and they're constantly under pressure between their customers who pay very little but want to spam the entire universe (I am sorry but most of you do!) and ESPs not wanting that traffic.
te_chris 6y ago
As a sendgrid customer, this explains why, even after upgrading to dedicated ips and setting all our auth/security stiff correctly we still have emails being blocked inexplicably. Top off the rubbish customer service - they didn’t respond to me about getting blocked by outlook on our private IP until I tweeted them... - and next week looks like my job is to move to postmark.
redm 6y ago
Our customers are also getting our emails flagged as malware. Great way to onboard a service and verify your email, with a Google Malware alert. :/ Its Google's world and Twilio is living in it. :|
_nickwhite 6y ago
Sendgrid have a HUGE problem with spammers using their service. I've probably reported 20 SPAM campaigns using their official channels, but I think they route such e-mails to /dev/null. Never gotten a response- just more SPAM through their service. And I can't block their IPs, as a lot of legitimate sites utilize their services.
PaulBGD_ 6y ago
Related, 2 months ago my sendgrid account gets hacked and a stolen (I presume) credit card gets added. They buy a plan, then proceed to send spam/phishing. I of course contact their support within minutes of receiving an email that $750 was spent on my account. They've now spent $1500 through my account on these spam emails, yet sendgrid finally dealt with the issue yesterday by deleting my account. https://mobile.twitter.com/PaulBGD/status/1295918800337084416 https://mobile.twitter.com/PaulBGD/status/129591880033708441...
electro_blah 6y ago
looks like they cracked them? no rate limits?
tobltobs 6y ago
This article sounds like sendgrid is the victim. But in reality this is going on since months. My account was hacked three months ago and abused by spammers. There was no way to stop it in the UI and the support didn't react for days. In the end I had to pay ~1k$. I guess Twilio will close down sendgrid soon anyway and this is a way to monetize the last few months. Stay away from Sendgrid and Twilio, they are incompetent and/or aholes.
rootsudo 6y ago
Yep, this happened to our ORG a while back, and Sendgrid took no responsibility. Wish I could go into details, but... they're finally implementing 2FA and revamping how they deal with identities.
nickjoker 6y ago
Why are people using email? Doesn't USPS/Fedex work?

199 points by elsewhen 6y ago

Well, I've tried contacting sendgrid about phishing emails that look like they're from @sendgrid.com , using their own server. The SPF records match, so it isn't marked as junk mail. It's similar to the github pages exploit from years ago, which is why we now have github.com and gitub.io The purpose of these mails is of course to get credentials and spam some more. At first I checked for a bounty, but as this is a "configuration error" (SPF), there is no bounty. Then I sent it by mail, but yeah.. No reply.

nodesocket 6y ago
Can you elaborate how a 3rd party is able to send e-mail from @sendgrid.com?
- jbverschoor 6y ago
  Simple. Their ip addresses of their client/guest-servers are whitelisted in the SPF records. So anyone can impersonate them. Not 100% sure if that was the case now, as I can't find the email anymore, but I've been getting them from both sendgrid and mailgun. Mailgun's SPF (was at least): v=spf1 include:spf1.mailgun.org include:spf2.mailgun.org include:_spf.google.com include:aspmx.pardot.com include:mail.zendesk.com include:spf.mailjet.com ~all spf1.mailgun.org: v=spf1 ip4:104.130.122.0/23 ip4:146.20.112.0/26 ip4:141.193.32.0/23 ip4:161.38.192.0/20 ~all v=spf1 ip4:209.61.151.0/24 ip4:166.78.68.0/22 ip4:198.61.254.0/23 ip4:192.237.158.0/23 ip4:23.253.182.0/23 ip4:104.130.96.0/28 ip4:146.20.113.0/24 ip4:146.20.191.0/24 ip4:159.135.224.0/20 ip4:69.72.32.0/20 ~all In the mailgun case I received mails from for example 198.61.254.24 with from: invoice@mailgun.com This is actually part of a bigger issue, where lots of people use the same email service to send out emails. In theory that should work with gmail or office365, but iirc gmail forces your from sometimes, and google+ms probably thought about this, right? ;)
  - monadic2 6y ago
    I'm surprised they give clients access to their own DKIM.
    - perennate 6y ago
      It doesn't pass DKIM. But the fact that it's sent through Sendgrid's platform gives the e-mail credibility. Because no one expects Sendgrid to allow random customers to set e-mail (not necessarily envelope) from address to @sendgrid.com.
      - monadic2 6y ago
        Based on my experience with sendgrid it's par for the course.
- XCSme 6y ago
  Someone sent spam emails using my domain name, so I looked a bit into it and made a summary of what I learned: https://www.usertrack.net/blog/stop-others-use-your-domain-emails https://www.usertrack.net/blog/stop-others-use-your-domain-e...
perennate 6y ago
Same -- I got several e-mails with subject "The following services failed to auto-renew and are about to expire" that I almost believed were real because they came from an actual sendgrid server (wrqvpkzw.outbound-mail.sendgrid.net [149.72.49.233]). If SendGrid can't develop simple filters to deter phishing e-mails sent through their platform targeting their own service, then obviously they're going to end up with a lot of users being hacked. I mean seriously, why would an e-mail provider like this allow e-mails to be sent from "renewaldepartment@sendgrid.com" and "support@Sendgrid.com"??

moomin 6y ago

I still remember when SendGrid got DDoSed after an incident at PyCon* and they caved. This told me two things: 1) they weren’t prepared to stand up for their own employees when the chips were down 2) their engineering wasn’t good enough to cope with a bunch of amateurs attacking them. * Let’s not derail this into a discussion of whether or not they should have done it in the first place. They didn’t, they only acted after they couldn’t cope with the DDoS.

Drdrdrq 6y ago
For those of us who don't know / remember the incident: https://venturebeat.com/2013/03/21/sendgrid-under-ddos-attack-after-its-developer-evangelist-complains-about-sexual-jokes-at-pycon/ https://venturebeat.com/2013/03/21/sendgrid-under-ddos-attac...
deleted 6y ago
[deleted]

bytecycle 6y ago

My university straight up blocks any mail coming from a SendGrid IP because of the massive amount of phishing. What would be a huge UX and security improvement is showing which team members are using 2FA. Just like GitLab shows a little blue label next to the username.

pmlnr 6y ago

That's actually good. Sendgrid & the rest are excluded from the paranoid gmail, m$, etc spam filters, while small senders suffer from being sent to spam immediately. There should be no exceptions.

stefan_ 6y ago
Yes, time for GMail et al to block them or admit it's just more anti-competitive nonsense.
gruez 6y ago
But is sendgrid "too big to fail"?
- calmworm 6y ago
  Not even close.
- dimitrios1 6y ago
  Too big to fail also was meant to communicate the vast amount of effort, time, and money that would be involved in rebuilding the financial infrastructure if indeed it had collapse, not to mention the immense short term pain and suffering that would occur in the interim. A SendGrid sized void could be filled in a matter of weeks to months by teams of highly motivated engineers looking to fill a market gap. The cost of starting up there is virtually nonzero (compared to say, starting a bank), and the regulatory burden is virtually nonzero (Email regulations are arguably some of the most clear and concise). All that to say, drawing a comparison with the "too big to fail" bailout of major US lenders is, frankly, silly.
  - Sebguer 6y ago
    I think you meant 'zero' not nonzero?
    - dimitrios1 6y ago
      Yes thank you

walrus01 6y ago

Sendgrid and its cohorts have always been a source of spam. The spamassassin setup on my postfix SMTP gateway ranks all of the bulk email newsletter sending services with +2.5 points or higher. This is like a sewage treatment plant complaining someone has come at 3am, cut the lock off their gate and dumped a tanker truck full of additional sewage.

techsupporter 6y ago
> Sendgrid and its cohorts have always been a source of spam. Sendgrid is the source of a lot of e-mail spam and their parent company Twilio is the source of a lot of SMS spam. It's really a synergy match made in MBA heaven.

aspectmin 6y ago

Sigh. Same happened to me. A few months ago. Sendgrid account hacked, someone set up a campaign and sent a huge amount of spam mail. Big bill on my credit card. Contacted Sendgrid, they blamed it on me. I cancelled my account, never going back. Screw them

jeffbee 6y ago
Was it not your fault?
- PakG1 6y ago
  Is that sarcasm?
  - bawolff 6y ago
    Users are at least partially responsible if they use a non unique password and then get hacked.
    - tialaramex 6y ago
      However, passwords are a shared secret, so you actually don't necessarily know who was responsible for the secret leaking, you or the other party. Most proposed replacements or supplements are no better except (you can guess I'm about to say) WebAuthn. WebAuthn is built out of public key signatures, so a Relying Party has no idea how to present valid credentials even for their own system. Obviously you could just add a back door if you want one, but the WebAuthn credentials you're storing aren't like a password, or a password hash, or a TOTP seed. They're just an opaque ID and a public key.
      - bawolff 6y ago
        Thats very true. But i think if sendgrid leaked their password db the level of spam would be mich higher. Of course its always possible the password db leaked but it was using argon2 or something so only really weak passwords could be decoded. I agree webauthn is great, and its the only option (other than TLS client certs which is a usability disaster, and maybe push notice 2fa depending on how its implemented) that protect against server side breach. However the vast majority of issues are password reuse, and to a much lesser extent dictionary attacks. TOTP effectively stops those. Then again so would choosing the users passwords for them.

crizzlenizzle 6y ago

This has been ongoing for months already. Nobody at Sendgrid cared. We had several issues with them leading to penalize mails from their IP space. I’m actively discouraging my customers from using their service.

jlgaddis 6y ago

> ... over the past few months there has been a marked increase in malicious, phishous and outright spammy email being blasted out via Sendgrid’s servers. Fortunately, I hadn't noticed the recent increases. After it became apparent that they didn't bother looking ibto -- or even reading -- abuse reports, we blacklisted basically their entire IP ranges a long time ago and then added exceptions when they were requested. I certainly did notice the marked decrease in spam then, though.

urda 6y ago

E-mail is a hard problem to solve, and I'm honestly really glad I do not have to manage it day-to-day myself.

pmlnr 6y ago
Enough with this "hard to solve" nonsense; email has been around since the 70s. Google, Yahoo!, and Microsoft are f*king it up, that's all.

statquontrarian 6y ago

I've been a paying customer for years on their shared server plan ($15/mo) and I've had terrible customer service and lots of emails marked as spam (e.g. to my Mom) due to a bad reputation of the shared box. I haven't migrated out of laziness and inertia. This article and the fact that other comments mention that SendGrid is seen as spammy is the final straw and I'm going to migrate. What paid email senders do people recommend that aren't excessively marked as spammy by gmail, etc.?

throwaway200829 6y ago
Throwaway since I’m a former Twilion. Even though people love Twilio for their developer friendliness and nice APIs, what no one wants to talk about publicly is their disproportionate amount of revenue that comes from spammy behavior. Either as a company or it’s customers. I was in sales. I saw the revenue numbers and sold these deals. It’s staggering. Yet wall street looooooves Twilio because of their revenue growth. Note: Twilio owns Sendgrid.
- lowdose 6y ago
  What I would like to know from a Twilion: Is Twilio used by commercial parties in the past to participate in voting in the European Song Festival?
  - tialaramex 6y ago
    Did you mean the Eurovision Song Contest, or something else?
Shared404 6y ago
You may have luck with protonmail's paid tiers. I use the free tier currently and have been pretty happy.
- statquontrarian 6y ago
  Thanks but I host my own email so I'm only looking for an outbound SMTP server. I went to ProtonMail's business page and the pricing plan only shows full email hosting which I don't need.

afrcnc 6y ago

how I hate this kind of reporting 3 people complain on reddit and it's now "[COMPANY X] under siege"

jlgaddis 6y ago
Between the comments on the article itself and this HN thread, there's already more than 10 times that. It's not being exaggerated quite the way that you "hate". To further prove how bad SendGrid is, Invaluement recently spent a ton of time and energy to create an entirely new "Service Provider DNSBL" [0] -- just for SendGrid! In fact, they just announced it about a week ago [1]. [0]: https://www.invaluement.com/serviceproviderdnsbl/ https://www.invaluement.com/serviceproviderdnsbl/ [1]: https://www.mail-archive.com/mailop@mailop.org/msg11403.html https://www.mail-archive.com/mailop@mailop.org/msg11403.html
krebsonsecurity 6y ago
Spend a few minutes looking at the spam list threads linked in the article. This is not just a few people complaining. E.g.: https://www.mail-archive.com/search?l=mailop%40mailop.org&q=sendgrid https://www.mail-archive.com/search?l=mailop%40mailop.org&q=... There was a ton of material I did not include in the story, including a story from a company that had a client have 40 million phishing emails sent through their Sendgrid account, which it turns out was set up by an employee long ago who was no longer with the company and had not turned on 2FA (and probably was re-using passwords). I started reporting this story almost a month ago after receiving more than 3 emails from different IT experts who were really frustrated with the amount of malware and phishing coming from Sendgrid accounts. They were frustrated because they couldn't block Sendgrid outright because too many companies they were expecting regular emails from used the platform. The day before I published the story, I head from someone else who was getting phishing attacks spoofing Aruba Networks.
tobltobs 6y ago
You also could have a look on twitter how many people with hacked accounts try to reach sendgrid support via this channel.

jlgaddis 6y ago

The CSO said: > "... 2FA for customer accounts is the right thing to do, and we’re working towards that end ... This is part of the reason we acquired Authy ..." Cool, it sounds like they're making a effort to implement -- and require -- 2FA... oh wait, FIVE YEARS have passed since then!

nodesocket 6y ago
Twilio supports and soon will require TFA. I personally don't use SendGrid (use Mailgun), but I have to believe it should be coming soon (if not already supported).
maerF0x0 6y ago
see my comment here: https://news.ycombinator.com/item?id=24319007 https://news.ycombinator.com/item?id=24319007

jeffbee 6y ago

If you run a service that can be abused like this, you're going to need objective account abuse metrics to warn you before these things happen. Specifically you're going to need a Russian speaker to prowl around on Russian forums buying your own hacked accounts, so you can figure out the market price. If accounts on your system are selling for ten cents, might be time to reflect on account security.

neolog 6y ago
Do big companies actually do this (check market price by buying their own accounts on forums)?
- timClicks 6y ago
  Absolutely. Edit: Although it's rarely direct. Normally big corp will engage a security consultancy and discuss type of reconnaissance as one part of a wider strategy.

PakG1 6y ago

I run IT for my organization and I am seeing a lot of phishing emails from Sendgrid. I haven't counted or done data analysis for proportion, but it's enough volume for me to really notice it.

jlgaddis 6y ago

This list is probably incomplete / out-of-date but ... 149.72.0.0/16 167.89.0.0/17 168.245.0.0/17 192.254.112.0/20 198.21.0.0/21 198.37.144.0/20

jeffbee 6y ago
Isn't their SPF record authoritative? ip4:167.89.0.0/17 ip4:208.117.48.0/20 ip4:50.31.32.0/19 ip4:198.37.144.0/20 ip4:198.21.0.0/21 ip4:192.254.112.0/20 ip4:168.245.0.0/17 ip4:149.72.0.0/16
- jlgaddis 6y ago
  Usually, yeah, but not necessarily (e.g., for those with dedicated IP addresses). Even then, it usually will be since most customers will just add the (customer-specific) "include:nnn.wl.sendgrid.net", or even just "include:sendgrid.net". But, for whatever reason, I've come across several that add an "a:" or "ip4:" for their dedicated IPs instead (I assume there's a good reason for that, but I don't know what it is).
  - megous 6y ago
    Probably so that they can also send some mail from their own servers, in addition to sendgrid's.

Farfromthehood 6y ago

Sendgrid user for many, many years. I believe we disabled 2FA because it broke /prevented our client's ability to send via SMTP. No idea if they've fixed the issue (and honestly don't care). Make security easy to adopt of you want us to implement it. Also, organization-wide, we don't use 2FA that requires a mobile phone number/SMS (because we don't trust vendors with our phone number). *IIRC, Sendgrid initially only offered 2FA via SMS, but now you can also use an authenticator app.

dylz 6y ago
Phone 2FA is broken by default. Yes, it broke SMTP and it also broke all API access until they kinda-fixed API keys. Pretty much anything you used basic auth for at a touch point.

maerF0x0 6y ago

> Neil Schwartzman, executive director of the anti-spam group CAUCE, said Sendgrid’s 2FA plans are long overdue, noting that the company bought Authy back in 2015. It's worth noting that this mixes up history. Authy was acquired by Twilio in 2015. Sendgrid was acquired by Twilio in 2019, there's no reason to assume in the 4 years interim that SG should have used Authy.

dylz 6y ago
I think the point was that they own a literal 2FA company, but don't have 2FA
- tialaramex 6y ago
  They do have 2FA they just don't require 2FA.
  - jaxn 6y ago
    But they only have SMS 2fa.
psanford 6y ago
For a company that went out and acquired a 2FA subsidiary, Twilio really doesn't seem to have their 2FA story together. For example you'd think a company with an division dedicated to authentication would support what the industry considers the best 2FA solution: WebAuthn. Sadly that is not an option for your Twilio account (other Auth providers like Duo support WebAuthn). Twilio has also been emailing me the past few months telling me I must turn on 2FA on my account. This is weird since I have TOTP enabled. I feel like Twilio must be internally conflicted about this. As a company dedicated to phone services they really want to push SMS solutions even though SMS based 2FA is vastly inferior to WebAuthn.
- kelnos 6y ago
  > Twilio has also been emailing me the past few months telling me I must turn on 2FA on my account. This is weird since I have TOTP enabled. There's been a bit of confusion about this email: you can turn on 2FA separately for your Twilio account (identified by a 32-char hex string prefixed with "AC") and your user account (your email address). The email is talking about your user account. Even if you have 2FA turned on for your Twilio account, the email is asking you to turn it on for your user account. For some context for those who may not be familiar: the Twilio account is essentially a billing/project unit that is a container for Twilio resources you've purchased or configured. Multiple user accounts can have access to a single Twilio account, and a single user account can have access to multiple Twilio accounts. Enabling 2FA on the Twilio account means all users who want to sign into that account (regardless of their user account setting) will have to use 2FA. Similarly, enabling 2FA on your user account will require 2FA whenever you sign in, regardless of the settings on any accounts you may have access to. (Source: I work there, got confused about this email myself, and managed to clarify internally what was going on.)

adrianh 6y ago

What service do people recommend in place of SendGrid? Personally I need a service to send transactional emails from my company website with a low chance that they’ll be marked as spam. Tracking is not necessary (I disabled that in SendGrid).

xenospn 6y ago
Amazon SES can work for that.
- nucleardog 6y ago
  And depending how much you need to send, if you’re already in AWS the first 62k messages every month are free. The only annoying thing is their sandbox and usage limits. You’re supposed to accept a lower limit, send high quality mail, and then request a higher limit once they’ve seen that you’re not generating a bunch of abuse reports. Slowly on boarding senders is good for delivery but makes it difficult to just do a simple cut-over if you’re currently sending more than whatever quota they’ll give you right now. You need to continue sending through your old service and slowly move your usage over.
moltar 6y ago
Postmark has been great for us for ages. They are strictly transactional and will flag accounts quickly that don’t follow this rule. This detracts spammers.
- tobltobs 6y ago
  Sadly they do now also allow bulk emails and therefore will soon be as shitty as all other emails sender services.

social_quotient 6y ago

Maybe Twilio will use that run up (bubble) in stock price to allocate to some tech debt like this. 169% YTD 33.8% last 3 months I’ve always looked at SendGrid as a premium provider for transactional mail. Why recommendations for something better but similarly priced?

a2tech 6y ago
We saw our deliverability go way up with SES. It’s pricey for what it is but we’ve had really good luck with it.
cortesoft 6y ago
Unless a company is selling new stock, an increase in stock price doesn't give them any extra money to spend. It just means their shareholders are worth more.

darkport 6y ago

I developed a tool called shhgit.com that watches code commits across GitHub, Gitlab and Bitbucket in real time. In a ~24 hour period you will find 100s of valid SendGrid credentials/API keys so this does not surprise me. They really need to enforce 2FA and IP whitelisting for API key use.

edoceo 6y ago

Oof. So,I tried sending mails from our own domain and our mail server - but Google tagged us as spam. We migrated into Google hosting and it was fixed, now back to Spam. So we move to a provider like sendgrid, but then they have a hack. Damnit! When we hosted our own, we had DKIM, SPF, DMARC and such, clean IP and clean messages, we only sent account create and transaction notices - low volume, relevant messages to folks who opt-in. We had our little box setup right and we're polite actors but got tagged which killed our user experience (password resets went to Spam and folk called support) But it feels the "big-web" forced us into vulnerable providers for the spam game. We need more self hosted email and less centralized control.

smnplk 6y ago
`We need more self hosted email and less centralized control` We need to start over and build a truly open internet.
- Reelin 6y ago
  > We need to start over and build a truly open internet. I want to agree, but email is actually an example of an open and federated protocol achieving success. It's pretty much everything that could have been hoped for as far as openness and interoperability go at the protocol level. Unfortunately, balkanization still occurs at the social (ie network) level.
- dredmorbius 6y ago
  The open email Internet was buried in an avalanche of spam. Open alone is insufficient.
  - paulryanrogers 6y ago
    If email were architected to put the cost of storage on senders then I think the spam problem would have been more manageable. Instead it's so closely modeled on physical mail that recipients get overwhelmed quickly.

tialaramex 6y ago

I am not a Sendgrid customer, however examining their documentation what I think the "attacks" look like is this: 14 year old Jimmy guesses that Foo Corp has the Sendgrid username "FooCorp" and the password "FooCorp0" obeying a company requirement to use "At least 8 characters" and "At least one digit". Very secure. Much password. Jimmy connects to Sendgrid's old API and literally just provides username "FooCorp" password "FooCorp0" to send as much email as Foo Corp have paid to be able to send. No other steps needed. Insisting customers get 2FA would in practice mean: 1. Get all customers to sign up to Authy's phone-based 2FA and use it for Sendgrid. Accept that this will have a revenue impact, you're essentially firing customers who either were too lazy to sign up for this or regard this type of security as a joke. 2. Deprecate older APIs that allow username + password auth 3. Thus force users into APIs that rely on a bearer token they can only get using 2FA. And yet after all this work if Jimmy instead gets a copy of your bearer token (e.g. because some idiot commits it to github) then he can once again send emails as Foo Corp until somebody disables the stolen bearer token (and then generates a new one and updates all their running code). It's a poor show that in 2020 this is their goal, it's not even the thing they have and need to improve upon, it's just a future goal.

cnst 6y ago
The whole idea behind Sendgrid is a joke. It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be. As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless.) That's how horrible Sendgrid is, and how easy it is to make other people switch. P.S. I often send mail through shell scripts. Try selling me Sendgrid when my laptop already comes with Postfix/Sendmail, with spool support. Does Sendgrid even work when you have no internet connection? (Rhetorical question: no, Sendgrid does not when internet is not available; but Postfix/Sendmail works just fine even without any internet, and will spool all your mail until you get connected to an internet with port 25 not blocked.) Sendgrid is a downgrade however you look at it.

cnst 6y ago

The whole idea behind Sendgrid is a joke. It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day, if not less. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be. As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless, and actually improved reliability.) That's how horrible Sendgrid is, and how easy it is to make other people switch. P.S. I often send mail through shell scripts. Try selling me Sendgrid when my laptop already comes with Postfix/Sendmail, with spool support. Does Sendgrid even work when you have no internet connection? (That was a rhetorical question; the answer is, No, Sendgrid does not work when internet is not available; but Postfix/Sendmail works just fine even without any internet at all, and Postfix/Sendmail will spool all your mail until you get connected to an internet with port 25 not blocked.) Sendgrid is a downgrade however you look at it. When they move to MFA, it'll probably be easier to just switch to one of their plentiful competitors than deal with any new restrictions; that's why Sendgrid will probably be very careful around any new requirements; because their service is 100% disposable.

cortesoft 6y ago
You seem to not understand the sendgrid service at all. They aren't selling SMTP service, they are selling deliverability (along with tracking and metrics) You seem to not understand the actual problem at all if you think you will be able to successfully deliver email sent directly from a local sendmail client on any random IP that lets you connect over port 25. No major email provider will accept your email coming from a residential IP.
- cnst 6y ago
  If that's what they're selling, they're not very good at it. Sendgrid doesn't support all applicable RFCs, and they don't support spool, so, they have exactly 0% deliverability if the recipient host is down for any reason, or uses greylisting. I've never had any issues with mail from my WiFi IP accepted. You'd normally send it just to your own server from your local laptop w/o any extra configuration, so, it's up to you how you configure your own server. I just tried sending to Gmail directly from Postfix on my MacBook, without any extra configuration, and it's been accepted by Gmail as well. Zero configuration on the laptop. Postfix and the mail CLI clients come preinstalled, and the mail just works.

sarreph 6y ago

At work we had a Sendgrid account that got compromised, after which all of our comms were (predictably) being marked as spam. After back and forth with support, we were told to continue sending as normal and that "eventually" our genuine emails would stop getting sent to spam. That lacklustre support response was enough to put me off the service. And then, funnily enough, a few days ago I received a reply from Sendgrid support about another account I had opened for a side project. This reply came in ~2 months late, where I was asking if I could be moved to a different IP range as all my emails were getting blocked by Outlook services due to the shared IP I was assigned being (seemingly) misused by spammers. Again, the response was "just keep trying as normal" and eventually my emails would work... Apparently. That's obviously not a viable business communications strategy. The funniest part about both these support responses was that despite one of them taking several weeks to get a response, I'm being pinged every few days asking me if my issue was solved. It's all well and good for me to wait months when I need a response as to why all my emails are getting sent to spam, but as soon as the ball is in their court, they don't mind pestering me for a response. I don't have enough bandwidth for a dedicated IP, so I signed up for a service that requires a credit card even for its free plan, which I believe is a decent enough barrier against spamming.

old-gregg 6y ago
> That lacklustre support response Frankly, this makes my blood boil. Full disclosure: I used to work at Sendgrid's competitor (Mailgun) and I'm quite familiar with what's happening under the hood. You allowed spam to be sent from your account. You poisoned the IP space (not just one address you were sending from, but the IP block, affecting others) with spammy reputation. These blocks are expensive and hard to acquire (due to IPv4 shortage) and at this point you have caused more damage to Sendgrid and their customers, than your account value can probably compensate for. What kind of "support" do you expect at this point? You rented a hotel room, gave the key to someone who erected a meth lab there, and you're unhappy with the hotel's reaction? Sending email on behalf of other people is hard work. The margins are not great, the ratio of spam-to-ham for new accounts is insane, and they're constantly under pressure between their customers who pay very little but want to spam the entire universe (I am sorry but most of you do!) and ESPs not wanting that traffic.

te_chris 6y ago

As a sendgrid customer, this explains why, even after upgrading to dedicated ips and setting all our auth/security stiff correctly we still have emails being blocked inexplicably. Top off the rubbish customer service - they didn’t respond to me about getting blocked by outlook on our private IP until I tweeted them... - and next week looks like my job is to move to postmark.

redm 6y ago

Our customers are also getting our emails flagged as malware. Great way to onboard a service and verify your email, with a Google Malware alert. :/ Its Google's world and Twilio is living in it. :|

_nickwhite 6y ago

Sendgrid have a HUGE problem with spammers using their service. I've probably reported 20 SPAM campaigns using their official channels, but I think they route such e-mails to /dev/null. Never gotten a response- just more SPAM through their service. And I can't block their IPs, as a lot of legitimate sites utilize their services.

PaulBGD_ 6y ago

Related, 2 months ago my sendgrid account gets hacked and a stolen (I presume) credit card gets added. They buy a plan, then proceed to send spam/phishing. I of course contact their support within minutes of receiving an email that $750 was spent on my account. They've now spent $1500 through my account on these spam emails, yet sendgrid finally dealt with the issue yesterday by deleting my account. https://mobile.twitter.com/PaulBGD/status/1295918800337084416 https://mobile.twitter.com/PaulBGD/status/129591880033708441...

electro_blah 6y ago

looks like they cracked them? no rate limits?

tobltobs 6y ago

This article sounds like sendgrid is the victim. But in reality this is going on since months. My account was hacked three months ago and abused by spammers. There was no way to stop it in the UI and the support didn't react for days. In the end I had to pay ~1k$. I guess Twilio will close down sendgrid soon anyway and this is a way to monetize the last few months. Stay away from Sendgrid and Twilio, they are incompetent and/or aholes.

rootsudo 6y ago

Yep, this happened to our ORG a while back, and Sendgrid took no responsibility. Wish I could go into details, but... they're finally implementing 2FA and revamping how they deal with identities.

nickjoker 6y ago

Why are people using email? Doesn't USPS/Fedex work?