3 ms·

PayFail: PayPal And Its APIs Go Down

22 points by vaksel 17y ago

st3fan 17y ago
This must be some huge corner case failure because I can't imagine that Paypal doesn't have some serious failover/high-availability setup. Or some interesting human error of course :-)
- lanaer 17y ago
  Considering the massive authorize.net outage that occured because of a fire in 1 datacenter not long ago… well, I no longer have faith that large companies have geographic redundancy ;) Edit: or, if they have it, I don’t trust that it is implemented in a way that allows an immediate & transparent (or near-transparent) failover.
  - mbrubeck 17y ago
    Amazon does regular "game day" drills that involve things like unplugging an entire datacenter to simulate an electrical failure. It's the only way to really figure out how things will break.
  - mrdodge 17y ago
    I never have had Chase Paymentech or Bank of America's system go down; but PayPal has gone down many times. Of course since I'm a pro, I have a queue setup for processing transactions and a backup processor, so the users experienced no problems.
zandorg 17y ago
Yup, it took me over 30 minutes to get it to work with an Ebay item I just bought. I was worried it had been hacked. Funnily enough, I checked their RSS feed for what's going on, and it told nothing. Then I checked twitter.com/paypal and that was garbage too. So much for the 'real time web'.
- blhack 17y ago
  I think they're probably more concerned with bringing it back up at the moment than they are with informing people about what is going on. (I understand that there are more than just engineers working there, but I can understand a PHB screaming at somebody to stop jacking around on twitter during a crisis). Either that or they hoped that they could rectify the problem before anybody noticed.
  - potatolicious 17y ago
    This is where your PR department needs to kick in. It's not just engineers that need to go mad right now, PR folk need to join in on the fun.
pyre 17y ago
This isn't news to me. I've had numerous times where I couldn't log into Paypal, but instead would get Paypal 'This is an error, try again later' messages for a couple of hours. I also had an item a couple of years ago that I could not pay for. Paypal would just die in the middle of the transaction with a 500 error. I tried for at least a week to get it to work, and Paypal customer service wasn't any help.
blasdel 17y ago
Starting just before 10 AM it became impossible to authenticate. From about 10:40 to 11:40 they went as far as turning paypal.com into a static page with just this on it: We're sprucing things up right now, but we should be done by 11:20 AM PDT. Want the details? Checkout the PayPal Announcement Board. http://announcements.paypal.com/us/ Sprucing things up? When I finally got back in I noticed some new UI with more gradients+shit. Seems like this is a dev iteration turned cascading failure...
- zandorg 17y ago
  I got the sprucing message too, but it wasn't clear by the time it said (7.20).

22 points by vaksel 17y ago

This must be some huge corner case failure because I can't imagine that Paypal doesn't have some serious failover/high-availability setup. Or some interesting human error of course :-)

lanaer 17y ago
Considering the massive authorize.net outage that occured because of a fire in 1 datacenter not long ago… well, I no longer have faith that large companies have geographic redundancy ;) Edit: or, if they have it, I don’t trust that it is implemented in a way that allows an immediate & transparent (or near-transparent) failover.
- mbrubeck 17y ago
  Amazon does regular "game day" drills that involve things like unplugging an entire datacenter to simulate an electrical failure. It's the only way to really figure out how things will break.
- mrdodge 17y ago
  I never have had Chase Paymentech or Bank of America's system go down; but PayPal has gone down many times. Of course since I'm a pro, I have a queue setup for processing transactions and a backup processor, so the users experienced no problems.

zandorg 17y ago

Yup, it took me over 30 minutes to get it to work with an Ebay item I just bought. I was worried it had been hacked. Funnily enough, I checked their RSS feed for what's going on, and it told nothing. Then I checked twitter.com/paypal and that was garbage too. So much for the 'real time web'.

blhack 17y ago
I think they're probably more concerned with bringing it back up at the moment than they are with informing people about what is going on. (I understand that there are more than just engineers working there, but I can understand a PHB screaming at somebody to stop jacking around on twitter during a crisis). Either that or they hoped that they could rectify the problem before anybody noticed.
- potatolicious 17y ago
  This is where your PR department needs to kick in. It's not just engineers that need to go mad right now, PR folk need to join in on the fun.

pyre 17y ago

This isn't news to me. I've had numerous times where I couldn't log into Paypal, but instead would get Paypal 'This is an error, try again later' messages for a couple of hours. I also had an item a couple of years ago that I could not pay for. Paypal would just die in the middle of the transaction with a 500 error. I tried for at least a week to get it to work, and Paypal customer service wasn't any help.

blasdel 17y ago

Starting just before 10 AM it became impossible to authenticate. From about 10:40 to 11:40 they went as far as turning paypal.com into a static page with just this on it: We're sprucing things up right now, but we should be done by 11:20 AM PDT. Want the details? Checkout the PayPal Announcement Board. http://announcements.paypal.com/us/ Sprucing things up? When I finally got back in I noticed some new UI with more gradients+shit. Seems like this is a dev iteration turned cascading failure...

zandorg 17y ago
I got the sprucing message too, but it wasn't clear by the time it said (7.20).