4 ms·
Why RIM still hasn’t found the cause of its world-wide outage
- recoiledsnake 15y agoThe common trend among the outages(atleast Gmail, Amazon EC2) seems to be that the queuing of the transaction messages seems to be overwhelming the network thus further making the attempts at recovery futile. Maybe a solution is to immediately stop all new requests and have servers that record all the transactions to disk instead of redirecting them to overloaded servers.
- acqq 15y agoHow can you make people stop sending messages? How can "recording to disks" make some change? What do you think is being done otherwise than writing some databases?
- gwright 15y agoYou have to build this sort of think into the infrastructure. For example, TCP and Ethernet retransmission strategies and HTTP servers should have reasonable timeouts that result in error responses rather than just letting incomplete HTTP sessions pile up. The same principles can and should be applied to any communication protocol.
- serverascode 15y agoWhile I feel that saying that massive systems fail in unusual ways is accurate, I also am concerned that RIM may not have the same technical prowess as a google or even amazon and yet in a way they are competing at that scale. Further, I guess pulling their users email into their system makes business sense for them, but that has always struck me as an unusual technical choice.
- randomdata 15y agoFurther, I guess pulling their users email into their system makes business sense for them, but that has always struck me as an unusual technical choice. I always thought it was so they could provide mail using a protocol that was optimized for battery life. Constantly polling a mail provider for updates is not exactly a good use of power. They couldn't really expect everyone who runs a mail server to install the necessary infrastructure to support the devices, so the next best thing is to have computers operated by RIM attached to power mains worry about collecting the email and then notify the wireless device only when there are changes. Apple uses the same basic model for their push services.
- wmf 15y agoMany customers already have BES installed. Why doesn't the phone talk to the BES directly? What value is RIM's centralized data center adding?
- westajay 15y agoIn other industries, such as petroleum refining, pipelines and nuclear power, their are structured methodologies for determining root causes. Some of these take into account equipment failure, and modelling is often done on equipment life cycles to determine replacement and inspection schedules in oil refineries. These industries also employ strict management of change processes so that an ad-hoc decision or improvisation doesn't cause an incident. What puzzles me is why you don't see these kind of practices applied in data center operations. Are data centers really more complex.. then say.. a nuclear plant?
- shaggyfrog 15y ago>In other industries, such as petroleum refining, pipelines and nuclear power, their are structured methodologies for determining root causes. I don't see why they would need a domain-specific methodology. Toyota gave us the 5 Whys, which applies nicely to just about any kind problem. http://en.wikipedia.org/wiki/5_Whys http://en.wikipedia.org/wiki/5_Whys
- joezydeco 15y agoMy three year-old daughter uses the same technique, she has no idea who Toyota is. Thankfully the root cause usually ends up being "BECAUSE I SAID SO".
- fleitz 15y agoYes, data centers are more complex than Nuclear power plants. Nuclear power plants generally operate in well-known ways and with few unknown interactions. eg. the work load on a nuclear power plant system does not spike around Christmas the work load on a nuclear power plant system does not change when there is a video spreading virally the nuclear power plant control system does not have to support a diverse eco-system of clients, some of them running hostile code. the nuclear power plant control system does not need to be upgraded very often Modern data centres and the infrastructure that runs on them operate in an environment of constant flux, this makes them vastly more complicated than a nuclear power plant system. Nuclear power plants have a vast number of components and a few very catastrophic failure modes but in essence they're very simple. The components don't change (save for the well known ways in which metal degrades under neutron bombardment), and lots of money has been invested to know exactly how these components act under various stresses. The engineers who operate them operate the same system for years and years becoming experts on how their very well defined system operates and know intimately its exact failure modes, and have a SOP manual and various checklists to quickly pinpoint and eliminate problems. Nuclear power plants are also developed and operated under a very clear singly focused mandate: Prevent at any cost the Nuclear Power plant from entering a catastrophic failure mode.