4 ms·

Sendgrid customer here. Since many services rely on SMTP providers like Sendgrid, they should have a way to notify customers when their server go down and tran

Sendgrid customer here. Since many services rely on SMTP providers like Sendgrid, they should have a way to notify customers when their server go down and transactional notifications may be disrupted. I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN.

rdl 14y ago
Yeah -- outage notifications to a service's users should go out, rather than just being pulled. Also seems like best practice would be to have SPF/etc. entries in place for multiple ESPs, even if you routinely just use one, and be able to switch, for just this reason.
- adambenayoun 14y ago
  We do have SPF entries set up for another provider, the main problem is that some services may need to warm up an IP prior sending lot of emails through a backup provider. What it means is that basically you should be using 2 providers at the same time and send 50%/50% on each provider in order to keep your IP warms. While I think big services should account for such scenario, I think 95% of Sendgrid customers are using them to avoid setting up that kind of redundancy. Main question is should I create the same redundancy for my hosting provider, DNS services, CDN etc...
  - kstrauser 14y ago
    What do you mean by "to warm up an IP"?
    - jrs235 14y ago
      If an IP address does not have a history of sending [large amounts of] email it is considered "cold". If that IP address then suddenly begins sending huge numbers of emails, many providers will assume the worst (that the email is being used for SPAM) and either block the IP or mark messages coming from it as SPAM. "Warming up" the IP means sending a smaller number of emails (per time period) and slowly increasing the volume so that Email providers can properly "score" the IP as legit and okay in their systems. Update: For some reason I couldn't reply to your response kstrauser so here is my reply: That works if and when their is trust. First you have trust the buyer isn't going to abuse the system and wreck the IP [reputation] and you have to trust the buyer isn't giving you stolen info. Obviously things one can work through but again, it involves trust. Also the provider has to have warmed up IPs to give out. Which, having warmed up IPs would actually be a great valued add upsell for those that need them! Update 2: Thank you FfejL and symfoniq for explaining it better.
      - kstrauser 14y ago
        That's when you physically call your sales rep and explain why your using their services and ask them to whitelist you. I say this from experience. If you called my employer (and any of our competitors, I'm certain) and said "I need to host a mail relay that's ready to go from no traffic to tens of thousands of emails per second, here's my company website, and here's our credit card", we'd be happy to make it work for you. I'm not saying this to advertise but to offer a suggestion: call someone and ask for help. This isn't an unusual need at all and any reputable provider will be quick to help. Short version: you don't have to "warm up an IP" if you do a little advance homework.
        FfejL 14y ago
        That's not what 'warming up an IP' means. If Google (or any other ISP) suddenly sees an big spike in email from @example.com on an IP that @example.com hasn't used previously, Google is much more likely to mark those emails as Spam. So senders need to 'warm up' an IP by sending a small amount of email first, usually for a few days at least. So if you're a SendGrid customer expecting to send 2,000,000 emails today, you can't just switch to a new ESP and send those same emails. Spam rates will go through the roof.
      - symfoniq 14y ago
        jrs235 is referring to IP reputation as measured on the receiving end of the email. In other words, Yahoo or Hotmail might reject your emails if they see unusual activity from an IP address. Until an IP is "established" as having sent a lot of email with low bounce and spam rates, getting bulk email delivered from that IP address will be a frustrating experience.
      - kstrauser 14y ago
        Thanks for the clarification! I'll pass that along to our products department. :-)
        jrs235 14y ago
        Who do you work for?
  - rdl 14y ago
    Most of the really heavy CDN users I know do have multiple CDNs constantly in operation, for contract negotiation reasons if nothing else. DNS is interesting, especially if you do anything location-based. I currently use just CloudFlare, but am not convinced they have enough internal redundancy on DNS, so investigating using Dyn, or self hosting again, or Route53, or some combination. It gets a more complex if you want to use multiple providers doing anything beyond simple DNS though (the DNS protocol itself is totally fine for this, but most of the config management is provider-specific, and using normal dns zone transfers/notifies doesn't really work in this model)
mryan 14y ago
They've been tweeting about it for an hour or so: https://twitter.com/SendGrid https://twitter.com/SendGrid I would expect them to send emails to customers in the event of serious issues though, expecting your customers to learn about problems through Twitter isn't that much better than learning about them from HN.
- nwh 14y ago
  > I would expect them to send emails
  - pytrin 14y ago
    Preferably not with their own SMTP service ;)
- adambenayoun 14y ago
  Critical notification should be sent to me and not pulled by me. Nice addition would be to have an addon to newrelic that would basically poll services/API like sendgrid and allow you to setup notifications for failure or degradation.
whaevr 14y ago
Aaand the PR slide for SendGrid continues. Perfectly valid point though.
- acdha 14y ago
  This is a “PR slide” in the same sense that torching someone's store is PR.
  - whaevr 14y ago
    "I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN." Was what I was replying to, not insinuating the fact that SendGrid's website going down is a "PR Slide"
jlebrech 14y ago
how does one send outage notices if the mail is down?
- kstrauser 14y ago
  Have an offsite cloud server with a mail daemon that's ready to send a few thousand messages to customers when your main server is down.
  - rdl 14y ago
    It might actually be worthwhile for a third-party status site/monitoring service to have a way to send this mail to users. It would be thousands of users per developer tool, but not millions like consumer services, so technically a much simpler service. Letting users set "I must ack notifications of outages or keep pinging me" would make sense, too. I'd probably consider mobile push notifications with confirmation and/or voice or SMS as well. Basically like Nagios, but for third-party components.

rdl 14y ago

Yeah -- outage notifications to a service's users should go out, rather than just being pulled. Also seems like best practice would be to have SPF/etc. entries in place for multiple ESPs, even if you routinely just use one, and be able to switch, for just this reason.

adambenayoun 14y ago
We do have SPF entries set up for another provider, the main problem is that some services may need to warm up an IP prior sending lot of emails through a backup provider. What it means is that basically you should be using 2 providers at the same time and send 50%/50% on each provider in order to keep your IP warms. While I think big services should account for such scenario, I think 95% of Sendgrid customers are using them to avoid setting up that kind of redundancy. Main question is should I create the same redundancy for my hosting provider, DNS services, CDN etc...
- kstrauser 14y ago
  What do you mean by "to warm up an IP"?
  - jrs235 14y ago
    If an IP address does not have a history of sending [large amounts of] email it is considered "cold". If that IP address then suddenly begins sending huge numbers of emails, many providers will assume the worst (that the email is being used for SPAM) and either block the IP or mark messages coming from it as SPAM. "Warming up" the IP means sending a smaller number of emails (per time period) and slowly increasing the volume so that Email providers can properly "score" the IP as legit and okay in their systems. Update: For some reason I couldn't reply to your response kstrauser so here is my reply: That works if and when their is trust. First you have trust the buyer isn't going to abuse the system and wreck the IP [reputation] and you have to trust the buyer isn't giving you stolen info. Obviously things one can work through but again, it involves trust. Also the provider has to have warmed up IPs to give out. Which, having warmed up IPs would actually be a great valued add upsell for those that need them! Update 2: Thank you FfejL and symfoniq for explaining it better.
    - kstrauser 14y ago
      That's when you physically call your sales rep and explain why your using their services and ask them to whitelist you. I say this from experience. If you called my employer (and any of our competitors, I'm certain) and said "I need to host a mail relay that's ready to go from no traffic to tens of thousands of emails per second, here's my company website, and here's our credit card", we'd be happy to make it work for you. I'm not saying this to advertise but to offer a suggestion: call someone and ask for help. This isn't an unusual need at all and any reputable provider will be quick to help. Short version: you don't have to "warm up an IP" if you do a little advance homework.
      - FfejL 14y ago
        That's not what 'warming up an IP' means. If Google (or any other ISP) suddenly sees an big spike in email from @example.com on an IP that @example.com hasn't used previously, Google is much more likely to mark those emails as Spam. So senders need to 'warm up' an IP by sending a small amount of email first, usually for a few days at least. So if you're a SendGrid customer expecting to send 2,000,000 emails today, you can't just switch to a new ESP and send those same emails. Spam rates will go through the roof.
    - symfoniq 14y ago
      jrs235 is referring to IP reputation as measured on the receiving end of the email. In other words, Yahoo or Hotmail might reject your emails if they see unusual activity from an IP address. Until an IP is "established" as having sent a lot of email with low bounce and spam rates, getting bulk email delivered from that IP address will be a frustrating experience.
    - kstrauser 14y ago
      Thanks for the clarification! I'll pass that along to our products department. :-)
      - jrs235 14y ago
        Who do you work for?
- rdl 14y ago
  Most of the really heavy CDN users I know do have multiple CDNs constantly in operation, for contract negotiation reasons if nothing else. DNS is interesting, especially if you do anything location-based. I currently use just CloudFlare, but am not convinced they have enough internal redundancy on DNS, so investigating using Dyn, or self hosting again, or Route53, or some combination. It gets a more complex if you want to use multiple providers doing anything beyond simple DNS though (the DNS protocol itself is totally fine for this, but most of the config management is provider-specific, and using normal dns zone transfers/notifies doesn't really work in this model)

mryan 14y ago

They've been tweeting about it for an hour or so: https://twitter.com/SendGrid https://twitter.com/SendGrid I would expect them to send emails to customers in the event of serious issues though, expecting your customers to learn about problems through Twitter isn't that much better than learning about them from HN.

nwh 14y ago
> I would expect them to send emails
- pytrin 14y ago
  Preferably not with their own SMTP service ;)
adambenayoun 14y ago
Critical notification should be sent to me and not pulled by me. Nice addition would be to have an addon to newrelic that would basically poll services/API like sendgrid and allow you to setup notifications for failure or degradation.

whaevr 14y ago

Aaand the PR slide for SendGrid continues. Perfectly valid point though.

acdha 14y ago
This is a “PR slide” in the same sense that torching someone's store is PR.
- whaevr 14y ago
  "I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN." Was what I was replying to, not insinuating the fact that SendGrid's website going down is a "PR Slide"

jlebrech 14y ago

how does one send outage notices if the mail is down?

kstrauser 14y ago
Have an offsite cloud server with a mail daemon that's ready to send a few thousand messages to customers when your main server is down.
- rdl 14y ago
  It might actually be worthwhile for a third-party status site/monitoring service to have a way to send this mail to users. It would be thousands of users per developer tool, but not millions like consumer services, so technically a much simpler service. Letting users set "I must ack notifications of outages or keep pinging me" would make sense, too. I'd probably consider mobile push notifications with confirmation and/or voice or SMS as well. Basically like Nagios, but for third-party components.