9 ms·

How a load-balancing bug led to worldwide Chrome crashes

176 points by jpdus 14y ago

hosay123 14y ago
Previous discussion (perplexingly marked dead): http://news.ycombinator.com/item?id=4904125 http://news.ycombinator.com/item?id=4904125
- mattmanser 14y ago
  Worryingly marked [dead] in fact, how did it end up dead? It might not be the greatest article, but it highlights a very real and new point of failure that cloud apps are introducing. I was hoping for a good discussion to check out later.
  - hosay123 14y ago
    Never ascribe to malice that which is adequately explained by incompetence.. :) HN's code is weird in places, e.g. it's possible to accidentally kill your own posts by double-clicking submit (I can't remember why it does this, but there was a sensible-ish explanation)
    - jQueryIsAwesome 14y ago
      Yep, happens the same thing with comments (found out thanks to a mouse malfunction)
  - polyfractal 14y ago
    Probably because of the horrible link-bait title and general inaccuracy of the article. Especially when this link was also on the front-page (a clear, concise description of what happened without the added FUD).
  - eli 14y ago
    People probably flagged it over the crappy title and the fact that much of the comments were meta discussion.
oscargrouch 14y ago
Im pretty unconfortable with my browser to have "hardcoded" code to connect me with one walled cloud.. be it google, microsoft or apple.. wheres is the choice? look like these days using anything software or hardware from the tech giants means to be their pets
- rahoulb 14y ago
  Forgive me if I've misunderstood, but Chrome Sync is off by default, and there are alternatives, such as XMarks, that you can install instead. Same goes for iCloud Sync in Safari and Firefox Sync too.
  - mitchty 14y ago
    Yep, I signed up for lastpass+xmarks for these reasons. I pay them the $20 for mobile access as well, that and I want to support a cross browser solution instead of having everything tied to one vendor.
- philfreo 14y ago
  Perhaps you should use WebKit or Chromium as your browser then.
  - pcwalton 14y ago
    Chromium has the same code.
    - philfreo 14y ago
      Chromium has Chrome Sync??
  - lttlrck 14y ago
    Or perhaps contribute to Chrome development. There seem to be a lot of experts here, I'm sure they'd all be welcomed with open arms.
- Gmo 14y ago
hammock 14y ago
The last discussion was mostly about how the title was linkbait. I want to hear people's opinions on whether they think it's appropriate for a browser (Chrome) to be designed such that it doesn't operate independently- that it can be crashed (or self destruct bug, insert your own word here) by a remote server at any time. To my knowledge, Firefox doesn't do that. Safari doesn't do that. Internet browsers are probably the #1 most important app on a computer these days, browser reliability is vital.
- EricBurnett 14y ago
  In what way was it 'designed' to not operate independently? The browser is - in theory - perfectly fine operating when sync is broken (and in fact wouldn't have triggered this were sync fully unreachable). This was simply a bug in sanitizing input, nothing further. Not different in flavour to input sanitization problems within the javascript engine, which have been known to occur as well.
- prawks 14y ago
  I think cross-device syncing is valuable functionality that I enjoy. Obviously this comes with a risk that if bad data is sent that is unhandle-able by my browser, it may cause a crash. The fact that syncing is elective and toggle-able is a great feature to be included. I think it's a bit sensationalist to still refer to it "being crashed" at any time, rather than saying it "may crash due to a bug".
- maggit 14y ago
  It's a crash bug. Bugs happen. It is not a design flaw. Sure, this specific vulnerability would not be there if the remote sync feature wasn't there, but people like features. Chrome has a pretty good security track record. I'm not worried.
  - MatthewPhillips 14y ago
    It's a design flaw. Should be designed like this: try { // Do syncing things. } catch(...) { // Continuing browsing. }
RobAley 14y ago
Currently getting "500. That’s an error. " page. Seems like even the bug has crashed too...!
andrewcooke 14y ago
on the server side, it looks like the problem could have been avoided with better types - it seem that there was a confusion between status values that can include 0 and those that cannot (alternatively, perhaps better, there was no status for the case where the status was undefined?) and then a hand-written assertion that a particular case could not happen (and so was not tested for). the bug report describes all that, roughly (if i've understood) but doesn't seem to be worried about the higher level issues - the inconsistent types and need for fragile human assertions about type logic. (not java bashing - don't see why this couldn't be solved in java) but i guess this is just a bug report. for an outage like this i suppose there's going to be a major review? is that all internal? would be interesting to watch.
- irahul 14y ago
  > (not java bashing - don't see why this couldn't be solved in java) Chromium is c++. The code diffs in the bug report are c++. Where is Java coming into picture?
  - andrewcooke 14y ago
    oh, sorry - didn't look at code, assumed java from some comment. but either way, same point - when you talk about solving problems with types, people tend to assume you're thinking of haskell or similar, but just because you're not using some super-cool functional language doesn't mean you shouldn't demand all you can get from your type system. (ps i was talking about the code on the server side; not chromium)
    - irahul 14y ago
      > (ps i was talking about the code on the server side; not chromium) Oh I see. > on the server side, it looks like the problem could have been avoided with better types - it seem that there was a confusion between status values that can include 0 and those that cannot (alternatively, perhaps better, there was no status for the case where the status was undefined?) From what I understood, they are talking about protocol buffer types. The sync server sent message to chromium to throttle for all types say A, B, and C. Chromium didn't know about type C, some code returned 0 for unspecified type, another piece of code calculates index based on what is returned in the previous step, and then due to the 0, a negative index was accessed in the bitset leading to out of bound exception. The issue is server sent all types to clients rather than sending only types known to the client, and the client didn't gracefully handle unknown types. I don't think a better type system would have helped the server - that looks like a logic bug, not a typing bug.
mintplant 14y ago
I'm surprised at the level of hyperbole here on this thread.
deleted 14y ago
[deleted]
rachelbythebay 14y ago
Hmm, a load balancing bug which eats several production services for lunch and has a bunch of second-order effects. I wonder if it involves running a script and pushing the output without looking at a "diff" view to see what changed, and they managed to push a config which sent all of the world's traffic to one location. It seems like just the other day when I was thinking about this very thing. http://rachelbythebay.com/w/2012/11/19/lb/ http://rachelbythebay.com/w/2012/11/19/lb/
- deleted 14y ago
  [deleted]
  - rachelbythebay 14y ago
    bns_aggregator.py, not even once?
macrael 14y ago
I don't understand why the crash happened when visiting Gmail if the bug was in Chrome Sync.
kintamanimatt 14y ago
Quick question for those that experienced it: did it take down the whole Chrome process, or just a single tab? I'd be very displeased if that happened again and I lost work as a result.
- paulgb 14y ago
  It took down the whole Chrome process.
  - kintamanimatt 14y ago
    That's grim and completely unacceptable!
    - lttlrck 14y ago
      Switch browsers or Contribute to Chrome development.
      - mh- 14y ago
        you can't contribute to Chrome development unless you work at Google, fwiw.
    - eli 14y ago
      Yeah.... it was a bug. Multiple bugs working in concert, actually.
nfm 14y ago
Had significant issues with the Chrome Web Store yesterday - images not loading from some content servers, extensions failing to install ('The extension file was not a CRX'). I guess this was the cause.
- nfm 14y ago
  "That quota service experienced traffic problems today due to a faulty load balancing configuration change. That change was to a core piece of infrastructure that many services at Google depend on. This means other services may have been affected at the same time, leading to the confounding original title of this bug." Why the downvote?
DigitalSea 14y ago
Bugs are a part of software life. Considering the great track record of Google Chrome, I'm not concerned and the issue was resolved pretty quickly. Even Google developers make mistakes as do the rest of us.

176 points by jpdus 14y ago

Previous discussion (perplexingly marked dead): http://news.ycombinator.com/item?id=4904125 http://news.ycombinator.com/item?id=4904125

mattmanser 14y ago
Worryingly marked [dead] in fact, how did it end up dead? It might not be the greatest article, but it highlights a very real and new point of failure that cloud apps are introducing. I was hoping for a good discussion to check out later.
- hosay123 14y ago
  Never ascribe to malice that which is adequately explained by incompetence.. :) HN's code is weird in places, e.g. it's possible to accidentally kill your own posts by double-clicking submit (I can't remember why it does this, but there was a sensible-ish explanation)
  - jQueryIsAwesome 14y ago
    Yep, happens the same thing with comments (found out thanks to a mouse malfunction)
- polyfractal 14y ago
  Probably because of the horrible link-bait title and general inaccuracy of the article. Especially when this link was also on the front-page (a clear, concise description of what happened without the added FUD).
- eli 14y ago
  People probably flagged it over the crappy title and the fact that much of the comments were meta discussion.

oscargrouch 14y ago

Im pretty unconfortable with my browser to have "hardcoded" code to connect me with one walled cloud.. be it google, microsoft or apple.. wheres is the choice? look like these days using anything software or hardware from the tech giants means to be their pets

rahoulb 14y ago
Forgive me if I've misunderstood, but Chrome Sync is off by default, and there are alternatives, such as XMarks, that you can install instead. Same goes for iCloud Sync in Safari and Firefox Sync too.
- mitchty 14y ago
  Yep, I signed up for lastpass+xmarks for these reasons. I pay them the $20 for mobile access as well, that and I want to support a cross browser solution instead of having everything tied to one vendor.
philfreo 14y ago
Perhaps you should use WebKit or Chromium as your browser then.
- pcwalton 14y ago
  Chromium has the same code.
  - philfreo 14y ago
    Chromium has Chrome Sync??
- lttlrck 14y ago
  Or perhaps contribute to Chrome development. There seem to be a lot of experts here, I'm sure they'd all be welcomed with open arms.
Gmo 14y ago

hammock 14y ago

The last discussion was mostly about how the title was linkbait. I want to hear people's opinions on whether they think it's appropriate for a browser (Chrome) to be designed such that it doesn't operate independently- that it can be crashed (or self destruct bug, insert your own word here) by a remote server at any time. To my knowledge, Firefox doesn't do that. Safari doesn't do that. Internet browsers are probably the #1 most important app on a computer these days, browser reliability is vital.

EricBurnett 14y ago
In what way was it 'designed' to not operate independently? The browser is - in theory - perfectly fine operating when sync is broken (and in fact wouldn't have triggered this were sync fully unreachable). This was simply a bug in sanitizing input, nothing further. Not different in flavour to input sanitization problems within the javascript engine, which have been known to occur as well.
prawks 14y ago
I think cross-device syncing is valuable functionality that I enjoy. Obviously this comes with a risk that if bad data is sent that is unhandle-able by my browser, it may cause a crash. The fact that syncing is elective and toggle-able is a great feature to be included. I think it's a bit sensationalist to still refer to it "being crashed" at any time, rather than saying it "may crash due to a bug".
maggit 14y ago
It's a crash bug. Bugs happen. It is not a design flaw. Sure, this specific vulnerability would not be there if the remote sync feature wasn't there, but people like features. Chrome has a pretty good security track record. I'm not worried.
- MatthewPhillips 14y ago
  It's a design flaw. Should be designed like this: try { // Do syncing things. } catch(...) { // Continuing browsing. }

RobAley 14y ago

Currently getting "500. That’s an error. " page. Seems like even the bug has crashed too...!

andrewcooke 14y ago

on the server side, it looks like the problem could have been avoided with better types - it seem that there was a confusion between status values that can include 0 and those that cannot (alternatively, perhaps better, there was no status for the case where the status was undefined?) and then a hand-written assertion that a particular case could not happen (and so was not tested for). the bug report describes all that, roughly (if i've understood) but doesn't seem to be worried about the higher level issues - the inconsistent types and need for fragile human assertions about type logic. (not java bashing - don't see why this couldn't be solved in java) but i guess this is just a bug report. for an outage like this i suppose there's going to be a major review? is that all internal? would be interesting to watch.

irahul 14y ago
> (not java bashing - don't see why this couldn't be solved in java) Chromium is c++. The code diffs in the bug report are c++. Where is Java coming into picture?
- andrewcooke 14y ago
  oh, sorry - didn't look at code, assumed java from some comment. but either way, same point - when you talk about solving problems with types, people tend to assume you're thinking of haskell or similar, but just because you're not using some super-cool functional language doesn't mean you shouldn't demand all you can get from your type system. (ps i was talking about the code on the server side; not chromium)
  - irahul 14y ago
    > (ps i was talking about the code on the server side; not chromium) Oh I see. > on the server side, it looks like the problem could have been avoided with better types - it seem that there was a confusion between status values that can include 0 and those that cannot (alternatively, perhaps better, there was no status for the case where the status was undefined?) From what I understood, they are talking about protocol buffer types. The sync server sent message to chromium to throttle for all types say A, B, and C. Chromium didn't know about type C, some code returned 0 for unspecified type, another piece of code calculates index based on what is returned in the previous step, and then due to the 0, a negative index was accessed in the bitset leading to out of bound exception. The issue is server sent all types to clients rather than sending only types known to the client, and the client didn't gracefully handle unknown types. I don't think a better type system would have helped the server - that looks like a logic bug, not a typing bug.

mintplant 14y ago

I'm surprised at the level of hyperbole here on this thread.

deleted 14y ago

[deleted]

rachelbythebay 14y ago

Hmm, a load balancing bug which eats several production services for lunch and has a bunch of second-order effects. I wonder if it involves running a script and pushing the output without looking at a "diff" view to see what changed, and they managed to push a config which sent all of the world's traffic to one location. It seems like just the other day when I was thinking about this very thing. http://rachelbythebay.com/w/2012/11/19/lb/ http://rachelbythebay.com/w/2012/11/19/lb/

deleted 14y ago
[deleted]
- rachelbythebay 14y ago
  bns_aggregator.py, not even once?

macrael 14y ago

I don't understand why the crash happened when visiting Gmail if the bug was in Chrome Sync.

kintamanimatt 14y ago

Quick question for those that experienced it: did it take down the whole Chrome process, or just a single tab? I'd be very displeased if that happened again and I lost work as a result.

paulgb 14y ago
It took down the whole Chrome process.
- kintamanimatt 14y ago
  That's grim and completely unacceptable!
  - lttlrck 14y ago
    Switch browsers or Contribute to Chrome development.
    - mh- 14y ago
      you can't contribute to Chrome development unless you work at Google, fwiw.
  - eli 14y ago
    Yeah.... it was a bug. Multiple bugs working in concert, actually.

nfm 14y ago

Had significant issues with the Chrome Web Store yesterday - images not loading from some content servers, extensions failing to install ('The extension file was not a CRX'). I guess this was the cause.

nfm 14y ago
"That quota service experienced traffic problems today due to a faulty load balancing configuration change. That change was to a core piece of infrastructure that many services at Google depend on. This means other services may have been affected at the same time, leading to the confounding original title of this bug." Why the downvote?

DigitalSea 14y ago

Bugs are a part of software life. Considering the great track record of Google Chrome, I'm not concerned and the issue was resolved pretty quickly. Even Google developers make mistakes as do the rest of us.