9 ms·

The bug that almost killed Google's Pac-Man doodle

219 points by steven 11y ago

ilyanep 11y ago
I love these kinds of stories. They're like the programmer's version of a detective story, with the twist that you always somehow learn some weird obscure fact about people or the Internet: - A web browser plug-in loads Google's homepage in the background. - There's a weird standard to writing Polish characters with diacritics. - Some Windows systems carry around this decades-old bitmapped font. I think it's similar to why I love those lists titled "X myths that programmers believe about Y". It's insane how varied the world is and how that comes out when we deal with as many domains as we do when programming.
- TacticalTable 11y ago
  Reminds me of the 500 mile email, my absolute favorite bug story: http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html
  - jacquesm 11y ago
    That's a classic.
- bsimpson 11y ago
  Incidentally, those stories were all told by the OP.
userbinator 11y ago
That reminds me of this, which was even more insidious and probably caused more than a few unnecessary hardware replacements: http://www.techsupportforum.com/forums/f10/strange-mysterious-two-random-scratch-sounds-intermittent-342048.html http://www.techsupportforum.com/forums/f10/strange-mysteriou... http://forums.studentdoctor.net/threads/windows-xp-computer-makes-random-annoying-woosh-woosh-sound.619402/ http://forums.studentdoctor.net/threads/windows-xp-computer-... http://forums.whirlpool.net.au/archive/624611 http://forums.whirlpool.net.au/archive/624611
Johnny555 11y ago
The real problem was autoplaying the sound -- they should not have auto-enabled sound... if they were worried that users wouldn't know there was sound, they could have put in a prominent button to turn it on, but auto-sound sucks. There are plenty of ways to trigger it accidentally, like loading Google, then walking away from your desk, or doing a google search in a presentation and pausing to talk about something. I might expect an autoplaying video with sound at a news or social media site, but I certainly wouldn't expect it from Google.com
teen 11y ago
this story is way too long for the amount of content. tldr; google homepage pacman game auto played sound, and a browser extension in firefox was silently loading google.com.
- groby_b 11y ago
  That's sort of like saying that really, Sherlock Holmes stories are too long, and the Hound of the Baskervilles merely was painted with phosphorus. Telling a story in an entertaining way isn't just listing the facts.
  - gertef 11y ago
    This article was no Doyle.
  - wingerlang 11y ago
    The actual "bug hunt" wasn't really touched upon though: > I don’t remember how exactly we figured it all out
nostrademons 11y ago
I did one of the code reviews for the Pac-Man doodle (and actually, the part I was responsible for reviewing focused specifically on sound, so this was kinda my bad). We had an early sneak-peak: Marcin was a UX designer for what was my 80% project at the time, so he'd passed it around the office looking for feedback. One of the factors we'd identified in the post-mortem was actually the difference in Google culture vs. the rest of the world culture. When we heard strange whirring noises coming from nearby computers in the office, our first impulse was "Woah, that's really cool! We're seriously going to launch this on the home page? Can I help?" While at many other offices, if you hear strange whirring noises from an employee's computer, the reaction is "You're fired!" Similar cultural mismatches have been responsible for a few other gaffes, eg. Google Buzz was a huge hit internally because pretty much everything inside Google is (was?) public anyway and people are very tolerant of different opinions, and it never occurred to us that people could be seriously harmed by others' knowing details of their lives.
- teddyh 11y ago
  > it never occurred to us [Google] that people could be seriously harmed by others' knowing details of their lives. This explains so much. EDIT: A year ago I wrote this, and it seems I was right: > Google is self-selecting to employ those people who are fine with the idea of Google recording everything they do. This does not bode well for the rest of us. (https://news.ycombinator.com/item?id=8591417 https://news.ycombinator.com/item?id=8591417)
  - nostrademons 11y ago
    It actually was far worse in the past than it is now, both because Google management does learn from their mistakes and because Google is much bigger now and so they employ more people from marginalized backgrounds. I'd say that peak "all your data are belong to us" was back in 2010, before the launch of Google Buzz and various location/wifi privacy flaps in Europe. I would be more worried now about the same dynamic existing in every other startup that takes off. When a startup is on a break-out trajectory, it has its pick of applicants from a wide number of elite institutions. Most of them will never have experienced powerlessness or marginalization, because that's what it takes to get into a hot startup. And so they generally won't be able to empathize with what it's like to have arbitrary fact X in your background taken and used against you in devastating ways, even if they do genuinely sympathize.
kentt 11y ago
> The other fun part is that, back in 2010, I also had to reintroduce a bug from the original Pac-Man code… but that’s a whole different article. Any idea what the bug was?
- abrugsch 11y ago
  probably the level 256 rollover glitch or killscreen (the basis for the mobile game pacman 256)
- DanBC 11y ago
  Maybe this? https://news.ycombinator.com/item?id=590109 https://news.ycombinator.com/item?id=590109 Or this? https://news.ycombinator.com/item?id=4464375 https://news.ycombinator.com/item?id=4464375 Or maybe this? https://news.ycombinator.com/item?id=1969347 https://news.ycombinator.com/item?id=1969347 https://hn.algolia.com/?query=pacman%20ghosts&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=pacman%20ghosts&sort=byPopular...
- splonk 11y ago
  IIRC it was the bug that allows you to pass through ghosts. I vaguely recall learning about it because of something the author wrote when this was released. Apparently speedrunners and the like have memorized patterns that exploit this if you can play with perfect timing. Explanation of the bug at http://www.gamasutra.com/view/feature/3938/the_pacman_dossier.php?print=1 http://www.gamasutra.com/view/feature/3938/the_pacman_dossie..., in the section "Just Passing Through".
ph33t 11y ago
I am of the opinion that web sites should never make noise automatically. Never, no matter what site. I often have my sound on without even realizing it. Nothing is more disturbing to me or those near me than me going to a site and having stuff come over the speaker. I almost always close the tab immediately. Please never autoplay anything.
- duaneb 11y ago
  Can't agree more. Also, autoplaying videos—never a good idea.
  - eru 11y ago
    Even on youtube?
    - CamperBob2 11y ago
      Especially on YouTube.
    - duaneb 11y ago
      I mean, how hard is a "youtube autoplay" plugin? What other site even needs that function? Are people really too lazy to press play?
      - palmer_eldritch 11y ago
        Since autoplay is the default right now, you could expect a fair amount of people to think youtube's broken if you changed it...
        duaneb 11y ago
        Somehow I think they'll get over it.
    - ars 11y ago
      Yes. Even youtube. I routinely open a bunch of tabs with interesting things, then read them. Youtube included - I'd like to be able to open a bunch of videos, then watch then one at a time in the tabs. I don't want all of them to start playing at once.
jfoutz 11y ago
I really don't like the last step of the post-mortem. "Who is to blame?" I much much prefer "how did the system fail?" We want engineers trying crazy things and hit bumpers when they, for example, autoplay sound. The alternative is being hesitant about trying crazy things. I mean, yeah, personal responsibility, but google already has a preflight checklist with code reviews and tests and such. Treating it as a systemic feature protects a lot of stuff you want.
- nostrademons 11y ago
  I was a little surprised he added that, because 1.) post-mortems at Google don't blame people, they blame processes, and then they suggest ways those processes could be fixed. 2.) I know Marcin personally and I've never known him to blame anyone either. And indeed, he didn't: the takeaway from the article was that the complexity of the web was at fault. My best guess is that he included that because many other organizations who do post-mortems, who he might want to reach with his writing, do think in terms of "Who is to blame?", and addressing it explicitly may be better than leaving it unanswered.
kbrosnan 11y ago
I remember this day which was fairly close to when I was hired at Mozilla. What this looked like from the Mozilla side several bugs filed [1]. Support forums had hundreds of questions about this [2][3]. The data from input.mozilla.org is gone or archived offline, I recall that blowing up as well. I vaguely recall this doodle causing a graphics crash on a specific set of hardware as well (though maybe that was a different doodle). [1] https://bugzilla.mozilla.org/buglist.cgi?quicksearch=567398%2C567402%2C567406%2C567431%2C567445%2C567453%2C567456 https://bugzilla.mozilla.org/buglist.cgi?quicksearch=567398%... [2] https://encrypted.google.com/search?hl=en&q=site%3Ainput.mozilla.org%20pacman%20coolpreviews#hl=en&q=site:support.mozilla.org+pacman+coolpreviews https://encrypted.google.com/search?hl=en&q=site%3Ainput.moz... [3] https://encrypted.google.com/search?hl=en&q=site%3Amozillazine.org%20pacman%20coolpreviews https://encrypted.google.com/search?hl=en&q=site%3Amozillazi...
jacquesm 11y ago
Computers should be seen, not heard unless explicitly instructed to do so by their users.
benlower 11y ago
I met Marcin at SXSW back in 2011 where he talked about this project (and other doodles). Some more info is here http://searchengineland.com/behind-the-scenes-with-googles-doodlers-68049 http://searchengineland.com/behind-the-scenes-with-googles-d.... Also invited him to come speak at Microsoft to share more about his work (some interesting internal debates about a Googler coming to talk at MSFT ;)
- benlower 11y ago
  Here's a video presentation from I/O that has more background on the doodle https://www.youtube.com/watch?v=ttavBa4giPc&hd=1 https://www.youtube.com/watch?v=ttavBa4giPc&hd=1
sudo_bang_bang 11y ago
Awesome essay! It looks like some version of the source code at least is available here: https://github.com/macek/google_pacman https://github.com/macek/google_pacman
craigds 11y ago
> The day I talked to my father for the last time Was expecting some explaining of this bit, but it's the only time he mentions it. That is indeed a crazy day!
TwoBit 11y ago
It was Google's fault. You should never play sound for a user automatically. I cringed when I read that before the article revealed the problem.
- joemi 11y ago
  Indeed. I'm not really sure why the author went with "the complexity of the web" is the real culprit. It was google, and specifically this guy's team, that was at fault. They put auto-playing sound on one of the most popular webpages in the world, and as he noted, this page never made sound before. The rest of the article was an interesting read, but not fully taking the blame seems wrong to me.
chavesn 11y ago
Where did this title come from? Not only is it not the title of the post in question, but it's quite inaccurate -- the article doesn't even come close to claiming that anything "almost killed" the doodle. With that said, I clicked the click-bait title, and I enjoyed the article. (shrug.)
sbuttgereit 11y ago
An unrelated aside concerning the blog's author, Marcin Wichary. He's also a photographer, including a number of wonderful photos of historical computer equipment. https://www.flickr.com/photos/mwichary/albums/72157629963516325 https://www.flickr.com/photos/mwichary/albums/72157629963516... When I read this blog post, I thought the name was familiar and was very pleased to be reminded of his good work in this area as well.
- deleted 11y ago
  [deleted]
smcl 11y ago
> We initially brushed off these complaints — “tell them to just close the Google homepage” Yep, that sounds like Google support alright.
steven 11y ago
Just sayin' that we see this as first of a series of amazing stories about bugs--ones that are so vexing, with fixes so illuminating, that telling the story casts light beyond the specific problem... If you got 'em, respond to this piece on the page!

219 points by steven 11y ago

I love these kinds of stories. They're like the programmer's version of a detective story, with the twist that you always somehow learn some weird obscure fact about people or the Internet: - A web browser plug-in loads Google's homepage in the background. - There's a weird standard to writing Polish characters with diacritics. - Some Windows systems carry around this decades-old bitmapped font. I think it's similar to why I love those lists titled "X myths that programmers believe about Y". It's insane how varied the world is and how that comes out when we deal with as many domains as we do when programming.

TacticalTable 11y ago
Reminds me of the 500 mile email, my absolute favorite bug story: http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html
- jacquesm 11y ago
  That's a classic.
bsimpson 11y ago
Incidentally, those stories were all told by the OP.

userbinator 11y ago

That reminds me of this, which was even more insidious and probably caused more than a few unnecessary hardware replacements: http://www.techsupportforum.com/forums/f10/strange-mysterious-two-random-scratch-sounds-intermittent-342048.html http://www.techsupportforum.com/forums/f10/strange-mysteriou... http://forums.studentdoctor.net/threads/windows-xp-computer-makes-random-annoying-woosh-woosh-sound.619402/ http://forums.studentdoctor.net/threads/windows-xp-computer-... http://forums.whirlpool.net.au/archive/624611 http://forums.whirlpool.net.au/archive/624611

Johnny555 11y ago

The real problem was autoplaying the sound -- they should not have auto-enabled sound... if they were worried that users wouldn't know there was sound, they could have put in a prominent button to turn it on, but auto-sound sucks. There are plenty of ways to trigger it accidentally, like loading Google, then walking away from your desk, or doing a google search in a presentation and pausing to talk about something. I might expect an autoplaying video with sound at a news or social media site, but I certainly wouldn't expect it from Google.com

teen 11y ago

this story is way too long for the amount of content. tldr; google homepage pacman game auto played sound, and a browser extension in firefox was silently loading google.com.

groby_b 11y ago
That's sort of like saying that really, Sherlock Holmes stories are too long, and the Hound of the Baskervilles merely was painted with phosphorus. Telling a story in an entertaining way isn't just listing the facts.
- gertef 11y ago
  This article was no Doyle.
- wingerlang 11y ago
  The actual "bug hunt" wasn't really touched upon though: > I don’t remember how exactly we figured it all out

nostrademons 11y ago

I did one of the code reviews for the Pac-Man doodle (and actually, the part I was responsible for reviewing focused specifically on sound, so this was kinda my bad). We had an early sneak-peak: Marcin was a UX designer for what was my 80% project at the time, so he'd passed it around the office looking for feedback. One of the factors we'd identified in the post-mortem was actually the difference in Google culture vs. the rest of the world culture. When we heard strange whirring noises coming from nearby computers in the office, our first impulse was "Woah, that's really cool! We're seriously going to launch this on the home page? Can I help?" While at many other offices, if you hear strange whirring noises from an employee's computer, the reaction is "You're fired!" Similar cultural mismatches have been responsible for a few other gaffes, eg. Google Buzz was a huge hit internally because pretty much everything inside Google is (was?) public anyway and people are very tolerant of different opinions, and it never occurred to us that people could be seriously harmed by others' knowing details of their lives.

teddyh 11y ago
> it never occurred to us [Google] that people could be seriously harmed by others' knowing details of their lives. This explains so much. EDIT: A year ago I wrote this, and it seems I was right: > Google is self-selecting to employ those people who are fine with the idea of Google recording everything they do. This does not bode well for the rest of us. (https://news.ycombinator.com/item?id=8591417 https://news.ycombinator.com/item?id=8591417)
- nostrademons 11y ago
  It actually was far worse in the past than it is now, both because Google management does learn from their mistakes and because Google is much bigger now and so they employ more people from marginalized backgrounds. I'd say that peak "all your data are belong to us" was back in 2010, before the launch of Google Buzz and various location/wifi privacy flaps in Europe. I would be more worried now about the same dynamic existing in every other startup that takes off. When a startup is on a break-out trajectory, it has its pick of applicants from a wide number of elite institutions. Most of them will never have experienced powerlessness or marginalization, because that's what it takes to get into a hot startup. And so they generally won't be able to empathize with what it's like to have arbitrary fact X in your background taken and used against you in devastating ways, even if they do genuinely sympathize.

kentt 11y ago

> The other fun part is that, back in 2010, I also had to reintroduce a bug from the original Pac-Man code… but that’s a whole different article. Any idea what the bug was?

abrugsch 11y ago
probably the level 256 rollover glitch or killscreen (the basis for the mobile game pacman 256)
DanBC 11y ago
Maybe this? https://news.ycombinator.com/item?id=590109 https://news.ycombinator.com/item?id=590109 Or this? https://news.ycombinator.com/item?id=4464375 https://news.ycombinator.com/item?id=4464375 Or maybe this? https://news.ycombinator.com/item?id=1969347 https://news.ycombinator.com/item?id=1969347 https://hn.algolia.com/?query=pacman%20ghosts&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=pacman%20ghosts&sort=byPopular...
splonk 11y ago
IIRC it was the bug that allows you to pass through ghosts. I vaguely recall learning about it because of something the author wrote when this was released. Apparently speedrunners and the like have memorized patterns that exploit this if you can play with perfect timing. Explanation of the bug at http://www.gamasutra.com/view/feature/3938/the_pacman_dossier.php?print=1 http://www.gamasutra.com/view/feature/3938/the_pacman_dossie..., in the section "Just Passing Through".

ph33t 11y ago

I am of the opinion that web sites should never make noise automatically. Never, no matter what site. I often have my sound on without even realizing it. Nothing is more disturbing to me or those near me than me going to a site and having stuff come over the speaker. I almost always close the tab immediately. Please never autoplay anything.

duaneb 11y ago
Can't agree more. Also, autoplaying videos—never a good idea.
- eru 11y ago
  Even on youtube?
  - CamperBob2 11y ago
    Especially on YouTube.
  - duaneb 11y ago
    I mean, how hard is a "youtube autoplay" plugin? What other site even needs that function? Are people really too lazy to press play?
    - palmer_eldritch 11y ago
      Since autoplay is the default right now, you could expect a fair amount of people to think youtube's broken if you changed it...
      - duaneb 11y ago
        Somehow I think they'll get over it.
  - ars 11y ago
    Yes. Even youtube. I routinely open a bunch of tabs with interesting things, then read them. Youtube included - I'd like to be able to open a bunch of videos, then watch then one at a time in the tabs. I don't want all of them to start playing at once.

jfoutz 11y ago

I really don't like the last step of the post-mortem. "Who is to blame?" I much much prefer "how did the system fail?" We want engineers trying crazy things and hit bumpers when they, for example, autoplay sound. The alternative is being hesitant about trying crazy things. I mean, yeah, personal responsibility, but google already has a preflight checklist with code reviews and tests and such. Treating it as a systemic feature protects a lot of stuff you want.

nostrademons 11y ago
I was a little surprised he added that, because 1.) post-mortems at Google don't blame people, they blame processes, and then they suggest ways those processes could be fixed. 2.) I know Marcin personally and I've never known him to blame anyone either. And indeed, he didn't: the takeaway from the article was that the complexity of the web was at fault. My best guess is that he included that because many other organizations who do post-mortems, who he might want to reach with his writing, do think in terms of "Who is to blame?", and addressing it explicitly may be better than leaving it unanswered.

kbrosnan 11y ago

I remember this day which was fairly close to when I was hired at Mozilla. What this looked like from the Mozilla side several bugs filed [1]. Support forums had hundreds of questions about this [2][3]. The data from input.mozilla.org is gone or archived offline, I recall that blowing up as well. I vaguely recall this doodle causing a graphics crash on a specific set of hardware as well (though maybe that was a different doodle). [1] https://bugzilla.mozilla.org/buglist.cgi?quicksearch=567398%2C567402%2C567406%2C567431%2C567445%2C567453%2C567456 https://bugzilla.mozilla.org/buglist.cgi?quicksearch=567398%... [2] https://encrypted.google.com/search?hl=en&q=site%3Ainput.mozilla.org%20pacman%20coolpreviews#hl=en&q=site:support.mozilla.org+pacman+coolpreviews https://encrypted.google.com/search?hl=en&q=site%3Ainput.moz... [3] https://encrypted.google.com/search?hl=en&q=site%3Amozillazine.org%20pacman%20coolpreviews https://encrypted.google.com/search?hl=en&q=site%3Amozillazi...

jacquesm 11y ago

Computers should be seen, not heard unless explicitly instructed to do so by their users.

benlower 11y ago

I met Marcin at SXSW back in 2011 where he talked about this project (and other doodles). Some more info is here http://searchengineland.com/behind-the-scenes-with-googles-doodlers-68049 http://searchengineland.com/behind-the-scenes-with-googles-d.... Also invited him to come speak at Microsoft to share more about his work (some interesting internal debates about a Googler coming to talk at MSFT ;)

benlower 11y ago
Here's a video presentation from I/O that has more background on the doodle https://www.youtube.com/watch?v=ttavBa4giPc&hd=1 https://www.youtube.com/watch?v=ttavBa4giPc&hd=1

sudo_bang_bang 11y ago

Awesome essay! It looks like some version of the source code at least is available here: https://github.com/macek/google_pacman https://github.com/macek/google_pacman

craigds 11y ago

> The day I talked to my father for the last time Was expecting some explaining of this bit, but it's the only time he mentions it. That is indeed a crazy day!

TwoBit 11y ago

It was Google's fault. You should never play sound for a user automatically. I cringed when I read that before the article revealed the problem.

joemi 11y ago
Indeed. I'm not really sure why the author went with "the complexity of the web" is the real culprit. It was google, and specifically this guy's team, that was at fault. They put auto-playing sound on one of the most popular webpages in the world, and as he noted, this page never made sound before. The rest of the article was an interesting read, but not fully taking the blame seems wrong to me.

chavesn 11y ago

Where did this title come from? Not only is it not the title of the post in question, but it's quite inaccurate -- the article doesn't even come close to claiming that anything "almost killed" the doodle. With that said, I clicked the click-bait title, and I enjoyed the article. (shrug.)

sbuttgereit 11y ago

An unrelated aside concerning the blog's author, Marcin Wichary. He's also a photographer, including a number of wonderful photos of historical computer equipment. https://www.flickr.com/photos/mwichary/albums/72157629963516325 https://www.flickr.com/photos/mwichary/albums/72157629963516... When I read this blog post, I thought the name was familiar and was very pleased to be reminded of his good work in this area as well.

deleted 11y ago
[deleted]

smcl 11y ago

> We initially brushed off these complaints — “tell them to just close the Google homepage” Yep, that sounds like Google support alright.

steven 11y ago

Just sayin' that we see this as first of a series of amazing stories about bugs--ones that are so vexing, with fixes so illuminating, that telling the story casts light beyond the specific problem... If you got 'em, respond to this piece on the page!