5 ms·
Sendgrid is down
- Raynh 14y agolol ddos
- lifeisstillgood 14y agoOK, I'll bite - is this a coincidence - or are they being DDOS'd over the PyCon incident?
- joshrotenberg 14y agoThe dreaded FDDOS: Forking Dongle Denial of Service.
- Hopka 14y agoThey could just use bigger Dongles to connect their servers to the internet.
- rmrfrmrf 14y agoStill unfunny and tasteless. Imagine that.
- kevincrane 14y agoWell I laughed at his joke. Maybe it's time to grow up and understand different people like different jokes?
- joshrotenberg 14y agoIt's totally unfunny and tasteless, I agree. That doesn't make it worth the hullabaloo that it caused. And I think that's the point.
- deleted 14y ago[deleted]
- whaevr 14y agoIt seems that Adria's personal blog is having similar problems.. http://butyoureagirl.com/ http://butyoureagirl.com/
- canttestthis 14y agoYes she mentioned that her website was getting DDOSed and that she was going to put it behind Cloudflare. I'm not sure its a DDOS though, it could just be the amount of traffic she's getting from HN/Reddit/Twitter/etc. https://news.ycombinator.com/item?id=5400134 https://news.ycombinator.com/item?id=5400134 EDIT: It seems that her personal website (adriarichards.com) is down too.
- Hopka 14y agoHow is putting it behind Cloudflare going to help now that the (alleged) DDoSers have the IP address of the actual server?
- beedogs 14y agomirror the content to a different host. point cloudflare at it. something like that i guess. could be a pain in the ass if you can't get to the original server and you have no other copies of the content, though.
- deleted 14y ago[deleted]
- andyl 14y agoGreat for Adria: more blog hits. Not so great for SendGrid: a PR disaster.
- slig 14y agoEven though I'm a SG customer, I can't help but feel a little schadenfreude. edit: when I made this comment I thought this was a random service failure that would last couple of minutes. More than 1 hour later, I don't think it's that funny anymore as I'm being affected as well.
- efa 14y agoSG customer as well. What's all the negativity towards them? Is there something I should know about them? I wasn't aware of any incidents.
- spindritf 14y agohttps://news.ycombinator.com/item?id=5410515 https://news.ycombinator.com/item?id=5410515
- deleted 14y ago[deleted]
- deleted 14y ago[deleted]
- adambenayoun 14y agoThere's not much negativity toward them just a bit of criticism on how they're handling this outage. As I mentioned in another comment of mine on that thread - they should have reached out to customers and let them know there's an outage (especially if this is an outage that is > 10 minutes). P.S: I still think Sendgrid are awesome and fortunately they'll listen to what we have to say and next outage will be handled differently.
- JPKab 14y agoA female Sendgrid employee overheard a joke mentioning a dongle looking like a phallus, took a picture of the guy, posted it to Twitter, and he got fired from his job without anyone hearing his side of the story. His apologies were ignored. Now the mob is mad at Sendgrid.
- adambenayoun 14y agoSendgrid customer here. Since many services rely on SMTP providers like Sendgrid, they should have a way to notify customers when their server go down and transactional notifications may be disrupted. I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN.
- rdl 14y agoYeah -- outage notifications to a service's users should go out, rather than just being pulled. Also seems like best practice would be to have SPF/etc. entries in place for multiple ESPs, even if you routinely just use one, and be able to switch, for just this reason.
- adambenayoun 14y agoWe do have SPF entries set up for another provider, the main problem is that some services may need to warm up an IP prior sending lot of emails through a backup provider. What it means is that basically you should be using 2 providers at the same time and send 50%/50% on each provider in order to keep your IP warms. While I think big services should account for such scenario, I think 95% of Sendgrid customers are using them to avoid setting up that kind of redundancy. Main question is should I create the same redundancy for my hosting provider, DNS services, CDN etc...
- kstrauser 14y agoWhat do you mean by "to warm up an IP"?
- jrs235 14y agoIf an IP address does not have a history of sending [large amounts of] email it is considered "cold". If that IP address then suddenly begins sending huge numbers of emails, many providers will assume the worst (that the email is being used for SPAM) and either block the IP or mark messages coming from it as SPAM. "Warming up" the IP means sending a smaller number of emails (per time period) and slowly increasing the volume so that Email providers can properly "score" the IP as legit and okay in their systems. Update: For some reason I couldn't reply to your response kstrauser so here is my reply: That works if and when their is trust. First you have trust the buyer isn't going to abuse the system and wreck the IP [reputation] and you have to trust the buyer isn't giving you stolen info. Obviously things one can work through but again, it involves trust. Also the provider has to have warmed up IPs to give out. Which, having warmed up IPs would actually be a great valued add upsell for those that need them! Update 2: Thank you FfejL and symfoniq for explaining it better.
- jneal 14y agoAs a SendGrid customer, I'm certainly glad I visit HN frequently and was able to find out about this. I'm surprised we weren't notified by them directly, though.
- markeganfuller 14y agoWe've never been directly notified by them despite their SMTP breaking a number of times.
- jneal 14y agoI'll just start following them on Twitter then. I've recently started working somewhere new where we use SendGrid, and this is the first outage I've been aware of.
- luney 14y agoI did the same thing. Followed them on twitter just to be aware of the outage status without having to go look for it.
- dasil003 14y agoSince the service is timing out, I think we'd all be best served in this case by having robust internal monitoring.
- scottbruin 14y agoThey tend not to post about interruptions on Twitter. In a lot of ways, they fail as an API-type service provider: * require you to use user:pass instead of keys * no historical service status page * poor notifications about downtime * unclear explanation of uptime
- grey-area 14y agoI'm sad to see that HN is being used as a platform for this sort of demagogy, and sad that stories about this tawdry little tale are indiscriminately voted up above other stories which are far more interesting. DDOS of a mailing service that lots of websites rely on because a completely unrelated company decided to fire someone is not an occasion for lol and schadenfreude as some posters here would have it. As a method of justice it has more in common with a lynch mob than a court of law - this isn't going to get the guy's job back, and it certainly isn't going to teach anyone a lesson, apart from that the internet is fickle, and monumentally stupid. But I very much doubt the people behind this attack are interested in justice or truly care about the man who lost his job, they're just doing it for the lolz and are punishing the internet at large over a silly little dispute at a tech conference. Congratulations to the mob, I guess; it has shown its power, if not any sense of discrimination or proportionality.
- adambenayoun 14y agoI couldn't agree with you more on the subject of what happened with PyCon. I don't think it should be mixed with what is happening now. On the flip side, I'm happy that this story has been up-voted and that it attracted some attention because I was able to know that Sendgrid is down (never received any mail notification and I don't follow them on Twitter).
- untog 14y agoOK, is there any actual evidence that this is a DDOS of SendGrid? Because their status page indicates nothing of the sort, and all I'm seeing is HN jumping to conclusions.
- sp332 14y agoAdria's blog was already under a DDOS, so it's not a huge stretch to think this is too.
- untog 14y agoIt is a huge stretch- at least partly because there is no third party verification that Adria's blog is actually victim to a DDOS attack, as far as I know. I'm not accusing her of lying, just that "a huge, huge number of people reading your blog post" can look a lot like a DDOS attack if you're running an unoptimised version of Wordpress, or similar.
- unhe 14y agoall thnx to adria and her cyber bullying
- aculver 14y agoThis was a Heroku win for us. I just visited their add-ons page, identified a comparable provider, and restored mail in our application with the following: heroku addons:add mailgun:basic heroku addons:docs mailgun:basic vim config/initializers/mail.rb git add config/initializers/mail.rb git commit -m "Switching to back-up mail provider." git push heroku master Since the account creation is all automatic and billing is all through Heroku, I never even had to visit Mailgun's website.
- adambenayoun 14y agoHow do you handle SPF/DKIM and domain keys records? Do you send out a lot of emails? Have you had a mailgun account ready?
- aculver 14y agoI don't know anything about this. If someone could provide some details/resources around this I'm sure it would be valuable to folks right now who need to evaluate the risks of switching providers on a short-term basis. (For us in particular, SendGrid only represents a small amount of the email we send and it's mostly internal emails. The problem for us was that the ActionMailer emails aren't sent in a background process, so this caused a couple request timeouts.)
- swampthing 14y agoIf memory serves, if you can log into Mailgun, it will have some text on how to set up all those things on your dashboard. Congrats on switching - Mailgun has vastly superior deliverability, at least in my observation.
- adambenayoun 14y agoI'd love to hear why you think mailgun is more superior than sendgrid. I'm currently using sendgrid.
- deleted 14y ago[deleted]
- deleted 14y ago[deleted]
- entropyneur 14y agoFirst we replaced SMTP that was distributed and designed to be resilient to outages (i.e. not lose messages) with proprietary HTTP APIs and now we complain that they don't notify us via email when they are down. Nice. I am guilty of using SES myself, but it's sad to see email becoming increasingly centralized.
- symfoniq 14y agoWe use SendGrid for transactional email, but handle (solicited) bulk email in-house. The centralization of email is at least partly due to the fact that ensuring email gets delivered is difficult and time-consuming. While I'm sure a lot of SendGrid's customers don't want (or don't know how) to configure a mail server, there are other customers who know that delivering email isn't as simple as installing Postfix. The rise of centralized email services is the inevitable byproduct.
- entropyneur 14y agoYeah, I know. Not blaming SendGrid in the slightest. Spammers have spoiled it for us all. Freedom disappearing because of a minority of abusers seems to be a common pattern...
- benatkin 14y agoActually SendGrid recommends using SMTP. And their API endpoint is very SMTP-like. As services go, its lock-in seems very minimal.
- uptown 14y agoI'm surprised PlayHaven hasn't shared the blame for their role in this whole thing. Adria, and by-extension, SendGrid shoulder some of the responsibility for what happened, but PlayHaven chose not to back their own employee.
- timjahn 14y agoI'm just tuning in now, as I happened to see this here on HN and my company is a SendGrid customer (used for transactional emails). Am I understanding this right - transactional emails for my company may be interrupted because of some random personal argument between two people?
- thomasvendetta 14y agoNot confirmed, but definitely possible: http://venturebeat.com/2013/03/20/playhaven-developer-fired-for-making-sexual-jokes-after-sendgrids-developer-evangelist-outs-him-on-twitter/ http://venturebeat.com/2013/03/20/playhaven-developer-fired-...
- uptown 14y agoIronic: https://twitter.com/adriarichards/status/312265091791847425 https://twitter.com/adriarichards/status/312265091791847425
- plusbryan 14y agoFWIW, we use Amazon SES and enjoy it immensely.
- efa 14y agoLooks like she got the boot. http://www.facebook.com/SendGrid http://www.facebook.com/SendGrid