4 ms·

Twitter experiences major, 2-hour+ outage

15 points by seldo 16y ago

Spoutingshite 16y ago
Never before has a free service been relied upon by so many diverse people, from business people, to students and demonstrators. Long live Twitter!
- humbledrone 16y ago
  google.com
rykov 16y ago
"..failed enhancement of a new approach to timeline caching" I hope this is not a testament to scalability of Cassandra or Redis. Ref: http://engineering.twitter.com/2010/02/link-cassandra-at-twitter.html http://engineering.twitter.com/2010/02/link-cassandra-at-twi... http://github.com/twitter/haplocheirus http://github.com/twitter/haplocheirus
- codahale 16y ago
  At this point, neither Cassandra nor Redis are in the critical path of their timeline service.
Osiris 16y ago
I noticed I kept getting Internal Server Error messages, so I kept trying to post, until I checked my timeline and saw it showed my post 4 times. Then I saw someone I follow post the same message about 20 times in a row. It kind of reminds me of the Win32 exception with a result code of 0, "Operation completed successfully." but backwards.
jazzychad 16y ago
This is a perfect storm. Many twitter clients automatically retry posting tweets when there is a non 200 return code from the Twitter API (including humans posting on the website seeing errors). Unfortunately, Twitter has been returning 5xx codes while successfully posting tweets. This, plus the apparent fact that duplicate tweet detection is failing/turned off means that all of the duplicate posts get through. Mayhem ensues...
- dnsworks 16y ago
  Yes, mayhem. The world as we know it will end! Or, well, some people won't be able to tell everybody else what they're eating for dinner, or they might tell everybody 10 times.
- akent 16y ago
  #firstworldproblems
ErrantX 16y ago
This happened the other day as well (probably on a shorter timescale, I'm not sure). I'm not sure if it was a 5xx error code - it seemed more like connections were hanging. It showed me the danger of working on someone else's platform, cue warning story: I had something that queried Twitter in php (Curl) but it never really occurred to me to time out the connection (this was ages ago, I forgot about the project till the other day). For some reason I assumed the host would kill long running PHP scripts.. Apparently not because my whole account got disabled for too many open connections.... oops.

15 points by seldo 16y ago

Never before has a free service been relied upon by so many diverse people, from business people, to students and demonstrators. Long live Twitter!

humbledrone 16y ago
google.com

rykov 16y ago

"..failed enhancement of a new approach to timeline caching" I hope this is not a testament to scalability of Cassandra or Redis. Ref: http://engineering.twitter.com/2010/02/link-cassandra-at-twitter.html http://engineering.twitter.com/2010/02/link-cassandra-at-twi... http://github.com/twitter/haplocheirus http://github.com/twitter/haplocheirus

codahale 16y ago
At this point, neither Cassandra nor Redis are in the critical path of their timeline service.

Osiris 16y ago

I noticed I kept getting Internal Server Error messages, so I kept trying to post, until I checked my timeline and saw it showed my post 4 times. Then I saw someone I follow post the same message about 20 times in a row. It kind of reminds me of the Win32 exception with a result code of 0, "Operation completed successfully." but backwards.

jazzychad 16y ago

This is a perfect storm. Many twitter clients automatically retry posting tweets when there is a non 200 return code from the Twitter API (including humans posting on the website seeing errors). Unfortunately, Twitter has been returning 5xx codes while successfully posting tweets. This, plus the apparent fact that duplicate tweet detection is failing/turned off means that all of the duplicate posts get through. Mayhem ensues...

dnsworks 16y ago
Yes, mayhem. The world as we know it will end! Or, well, some people won't be able to tell everybody else what they're eating for dinner, or they might tell everybody 10 times.
akent 16y ago
#firstworldproblems

ErrantX 16y ago

This happened the other day as well (probably on a shorter timescale, I'm not sure). I'm not sure if it was a 5xx error code - it seemed more like connections were hanging. It showed me the danger of working on someone else's platform, cue warning story: I had something that queried Twitter in php (Curl) but it never really occurred to me to time out the connection (this was ages ago, I forgot about the project till the other day). For some reason I assumed the host would kill long running PHP scripts.. Apparently not because my whole account got disabled for too many open connections.... oops.