5 ms·

Peter Norvig on Being Wrong

164 points by plinkplonk 16y ago

yarapavan 16y ago
Like this: And, like you say, it does create a very different attitude toward error. If you're a politician, admitting you're wrong is a weakness, but if you're an engineer, you essentially want to be wrong half the time. If you do experiments and you're always right, then you aren't getting enough information out of those experiments. You want your experiment to be like the flip of a coin: You have no idea if it is going to come up heads or tails. You want to not know what the results are going to be.
- houseabsolute 16y ago
  That's idealism talking, but practically speaking experimenters are invested in the outcomes of all but the least interesting experiments. Nobody wants the search feature they spent a month implementing to come back with negative results on evaluation. This is not only true at Google. Some of the stuff in this sounds nice but I feel it lacks some honesty. "What is the thing you have been the most wrong about?" And you say that the worst error in judgment you've made is an off-hand remark in a meeting? Let's be serious here. His call on big mistakes from Google seems good but I doubt Google could have executed on it even if they had known how big Facebook would be beforehand. Big G is not a product company and is best at making products that minimize the time you have to spend using them.
  - plinkplonk 16y ago
    "Some of the stuff in this sounds nice but I feel it lacks some honesty. "What is the thing you have been the most wrong about?" And you say that the worst error in judgment you've made is an off-hand remark in a meeting?" Huh? That isn't how I read it at all. From the article: "Question: What about you yourself—what have you personally been most wrong about? Answer: One thing is how fast things change.[Stuff about meeting here]You think you have this experience—and we talked about how important experience is for having intuitions—but experience can go out of date very quickly." To me he seems to be saying that one of his major mistakes is underestimating how fast things change, and underestimating the erosion of the value of his experience in guiding his intuition (He just finished talking about the importance of intuition for situations where statistics and data can't help/don't exist.) Given that he was speaking about a professional context [1] and the position he occupies at Google (Director of Research), where anticipating change would be presumably part of his job description, that sounded to me like admission of a major lapse, somewhat different from "My biggest mistake was an off hand remark I made at a meeting", as you seemed to imply. He was using the meeting incident as a concrete example of a rather large failing. How is it "dishonest"? [1] The whole interview is about errors in a professional context , about Google and his work there, so I wouldn't expect to hear about whether he got stoned as a teenager and got thrown in jail! ;-)
    - ntoshev 16y ago
      I think it was more of a regret that his intuition about the problem area gets invalidated so quickly. The top technical guys are supposed to have good intuitions about what works and what doesn't; this is a major part of the value they can add. So I take this as a very serious self-criticism.
  - arethuza 16y ago
    "is best at making products that minimize the time you have to spend using them" An excellent observation - I use Google a lot the total amount of time I spend on their sites really is pretty small and the "brand loyalty" for me is almost non existent.
- will_critchlow 16y ago
  This reminded me of information theory. The idea of entropy in information - that if you can predict the next 0 or 1 with better than 50% certainty, you can compress that information (or in this case, you could have designed a better sequence of experiments). I do think it's worth differentiating between types of failure, however. You don't want 50% of your experiments failing in the sense of failing to give you an answer in either direction (i.e. being inconclusive). And this leaves aside the topic (covered in the article) of the more serious kinds of failure - of live code etc. You probably don't want that happening 50% of the time.
  - mturmon 16y ago
    That's precisely where Norvig is coming from. Given their infrastructure, it's entirely possible that some portion of their A/B testing is run this way (i.e., test thresholds chosen to maximize information gain).
  - dkarl 16y ago
    I had a boss who always pushed me away from low-probability experiments. Whenever I suggested trying something that was unlikely to work, he probed to see if the single low-probability experiment could be restructured as a series of 50/50 experiments. It turned a lot of failures into partial successes that could be applied to future problems.
- cabalamat 16y ago
  Surely that's a scientist rather than an engineer? Engineers want to build things that work, and only incidentally care about how the world behaves.
  - billswift 16y ago
    It depends on whether you're a production engineer making things for use (ICs, software, bridges) or a development engineer trying to push the limits of what can be done.
houseabsolute 16y ago
Googlers are fine with admitting the small errors. I had a bug, this design isn't so great, the evals show this change to be a big loss. Those are the easy ones to admit. The big errors -- this product sucks, this idea is bad, eighty percent of the support personell on this piece of infrastructure are uncooperative jerks -- come less easily. Anyway, eng doesn't really run Google any more. PM does. Eng can still push back to some extent and feels important when they control huge budgets for new datacenters. But who designs, specifies and ultimately controls the products for which those datacenters are being allocated? Most everything you see, good or bad, that happens on Google these days came from a PM.
- Luc 16y ago
  What does PM stand for?
  - kqr2 16y ago
    Product Manager.
  - c1sc0 16y ago
    Funny you ask, going through the middle of an organization where ProjM is turned into ProdM right now. As far as I can tell very little changes except the titles on business cards ;-)
    - SkyMarshal 16y ago
      Product Manager sounds better. It makes it sound more like you're responsible for shipping something that generates revenue, rather than overseeing an org-chart rework or some other corporate circlejerk.
    - jeffcoat 16y ago
      The distinction I usually make is that a "project manager" is responsible for scheduling: maintaining a picture of the current state of the project, the current best estimates for when things will happen next, and make sure that everything that must happen before the next release is accounted for somewhere. It's not a "management" role in the traditional sense (people management), but it's not an engineering role, either. A product manager's job is to understand why people are (or are not) buying your product. They need to understand the market, keep track of competing products, and be able to make decisions (or offer advice) about which of a list of potential new features are most likely to make the customer's mouth water. It's fundamentally a marketing role, but, especially with a complicated product, can require a fair amount of technical skill to do. Disclaimer: This is all Big Company language, but not necessarily identical to Google's language.
cake 16y ago
Reminds me of a blog post from J. Fried : http://37signals.com/svn/posts/1555-learning-from-failure-is-overrated http://37signals.com/svn/posts/1555-learning-from-failure-is... I couldn't disagree more with what Fried has to say, which surprised me because it doesn't happen very often, this interview expresses why quite good.
demallien 16y ago
The thing I found most interesting about this interview is just how badly Peter Norvig has failed to understand what Facebook is all about - I don't use Facebook so that friends can give me recommendations about what camera to buy - I use Facebook because knowing what my friend Sally got up to this weekend is far more interesting to me than what Britney Spears has been up to. I might be interesting in knowing what camera she used to take that brilliant photo of Jim under the fireworks, but that's about as close as it gets. This has a huge implication for the value of Facebook - they know who I am, really. They know where I live, my age, my sex, and they can even tell what sort of activities I like. Google tries to guess this information from my page view habits, but they often get it wrong, big time. Here's an example - if I get a dating site ad on Facebook it correctly identifies me as a woman and throws up pictures of guys. If that ad is being served up by Google, I get identified as a man because of all the time I spend on tech sites, so the same ad has pictures of women in it. Facebook gives me ads about clothes, Google almost never does. I suspect that Google's core business is at significant risk here. People are spending more and more time in the Facebook universe, those 'Like' buttons have popped up all over the Web, and that spells trouble for Google. It'll be interesting to see how they respond, but I for one won't be buying any more stock in Google for a while.
- jakevoytko 16y ago
  I think Norvig looks at Facebook as a lost opportunity for data mining: Facebook isn't used as a recommendation engine any more than Google is used as an ad company, yet all user actions always gets piped back to ad results. If John is friends with Sally (who is kind of cute, and kind of a photography geek, and recently bought a good midrange point-and-shoot), John subconsciously wants that brand to show up somewhere in the recommendations, even if it is too expensive. But Google doesn't know that John's friends have bought that camera, because who posts what camera they bought on their Google profile? If Facebook has the ad instead of Google, then Facebook might get the all-important click.
  - msg 16y ago
    That is right. Norvig doesn't see what Facebook is good at and ask how it will make money. He turns the question on its head, looks to a successful business model and asks how Facebook could be used for it. And it is an uphill battle for Facebook to monetize in this fashion. I think what we've seen so far is that Facebook is about creating social spaces, not solving problems, so it really can't be used that effectively as a recommendation engine. Facebook doesn't have a great way to seek rent on these social spaces because they've built their user base on free service. So where is the money going to come from? Probably selling user data to corporations.
protomyth 16y ago
Does it seem like some people would really like to be a social network created by Visa?

164 points by plinkplonk 16y ago

Like this: And, like you say, it does create a very different attitude toward error. If you're a politician, admitting you're wrong is a weakness, but if you're an engineer, you essentially want to be wrong half the time. If you do experiments and you're always right, then you aren't getting enough information out of those experiments. You want your experiment to be like the flip of a coin: You have no idea if it is going to come up heads or tails. You want to not know what the results are going to be.

houseabsolute 16y ago
That's idealism talking, but practically speaking experimenters are invested in the outcomes of all but the least interesting experiments. Nobody wants the search feature they spent a month implementing to come back with negative results on evaluation. This is not only true at Google. Some of the stuff in this sounds nice but I feel it lacks some honesty. "What is the thing you have been the most wrong about?" And you say that the worst error in judgment you've made is an off-hand remark in a meeting? Let's be serious here. His call on big mistakes from Google seems good but I doubt Google could have executed on it even if they had known how big Facebook would be beforehand. Big G is not a product company and is best at making products that minimize the time you have to spend using them.
- plinkplonk 16y ago
  "Some of the stuff in this sounds nice but I feel it lacks some honesty. "What is the thing you have been the most wrong about?" And you say that the worst error in judgment you've made is an off-hand remark in a meeting?" Huh? That isn't how I read it at all. From the article: "Question: What about you yourself—what have you personally been most wrong about? Answer: One thing is how fast things change.[Stuff about meeting here]You think you have this experience—and we talked about how important experience is for having intuitions—but experience can go out of date very quickly." To me he seems to be saying that one of his major mistakes is underestimating how fast things change, and underestimating the erosion of the value of his experience in guiding his intuition (He just finished talking about the importance of intuition for situations where statistics and data can't help/don't exist.) Given that he was speaking about a professional context [1] and the position he occupies at Google (Director of Research), where anticipating change would be presumably part of his job description, that sounded to me like admission of a major lapse, somewhat different from "My biggest mistake was an off hand remark I made at a meeting", as you seemed to imply. He was using the meeting incident as a concrete example of a rather large failing. How is it "dishonest"? [1] The whole interview is about errors in a professional context , about Google and his work there, so I wouldn't expect to hear about whether he got stoned as a teenager and got thrown in jail! ;-)
  - ntoshev 16y ago
    I think it was more of a regret that his intuition about the problem area gets invalidated so quickly. The top technical guys are supposed to have good intuitions about what works and what doesn't; this is a major part of the value they can add. So I take this as a very serious self-criticism.
- arethuza 16y ago
  "is best at making products that minimize the time you have to spend using them" An excellent observation - I use Google a lot the total amount of time I spend on their sites really is pretty small and the "brand loyalty" for me is almost non existent.
will_critchlow 16y ago
This reminded me of information theory. The idea of entropy in information - that if you can predict the next 0 or 1 with better than 50% certainty, you can compress that information (or in this case, you could have designed a better sequence of experiments). I do think it's worth differentiating between types of failure, however. You don't want 50% of your experiments failing in the sense of failing to give you an answer in either direction (i.e. being inconclusive). And this leaves aside the topic (covered in the article) of the more serious kinds of failure - of live code etc. You probably don't want that happening 50% of the time.
- mturmon 16y ago
  That's precisely where Norvig is coming from. Given their infrastructure, it's entirely possible that some portion of their A/B testing is run this way (i.e., test thresholds chosen to maximize information gain).
- dkarl 16y ago
  I had a boss who always pushed me away from low-probability experiments. Whenever I suggested trying something that was unlikely to work, he probed to see if the single low-probability experiment could be restructured as a series of 50/50 experiments. It turned a lot of failures into partial successes that could be applied to future problems.
cabalamat 16y ago
Surely that's a scientist rather than an engineer? Engineers want to build things that work, and only incidentally care about how the world behaves.
- billswift 16y ago
  It depends on whether you're a production engineer making things for use (ICs, software, bridges) or a development engineer trying to push the limits of what can be done.

houseabsolute 16y ago

Googlers are fine with admitting the small errors. I had a bug, this design isn't so great, the evals show this change to be a big loss. Those are the easy ones to admit. The big errors -- this product sucks, this idea is bad, eighty percent of the support personell on this piece of infrastructure are uncooperative jerks -- come less easily. Anyway, eng doesn't really run Google any more. PM does. Eng can still push back to some extent and feels important when they control huge budgets for new datacenters. But who designs, specifies and ultimately controls the products for which those datacenters are being allocated? Most everything you see, good or bad, that happens on Google these days came from a PM.

Luc 16y ago
What does PM stand for?
- kqr2 16y ago
  Product Manager.
- c1sc0 16y ago
  Funny you ask, going through the middle of an organization where ProjM is turned into ProdM right now. As far as I can tell very little changes except the titles on business cards ;-)
  - SkyMarshal 16y ago
    Product Manager sounds better. It makes it sound more like you're responsible for shipping something that generates revenue, rather than overseeing an org-chart rework or some other corporate circlejerk.
  - jeffcoat 16y ago
    The distinction I usually make is that a "project manager" is responsible for scheduling: maintaining a picture of the current state of the project, the current best estimates for when things will happen next, and make sure that everything that must happen before the next release is accounted for somewhere. It's not a "management" role in the traditional sense (people management), but it's not an engineering role, either. A product manager's job is to understand why people are (or are not) buying your product. They need to understand the market, keep track of competing products, and be able to make decisions (or offer advice) about which of a list of potential new features are most likely to make the customer's mouth water. It's fundamentally a marketing role, but, especially with a complicated product, can require a fair amount of technical skill to do. Disclaimer: This is all Big Company language, but not necessarily identical to Google's language.

cake 16y ago

Reminds me of a blog post from J. Fried : http://37signals.com/svn/posts/1555-learning-from-failure-is-overrated http://37signals.com/svn/posts/1555-learning-from-failure-is... I couldn't disagree more with what Fried has to say, which surprised me because it doesn't happen very often, this interview expresses why quite good.

demallien 16y ago

The thing I found most interesting about this interview is just how badly Peter Norvig has failed to understand what Facebook is all about - I don't use Facebook so that friends can give me recommendations about what camera to buy - I use Facebook because knowing what my friend Sally got up to this weekend is far more interesting to me than what Britney Spears has been up to. I might be interesting in knowing what camera she used to take that brilliant photo of Jim under the fireworks, but that's about as close as it gets. This has a huge implication for the value of Facebook - they know who I am, really. They know where I live, my age, my sex, and they can even tell what sort of activities I like. Google tries to guess this information from my page view habits, but they often get it wrong, big time. Here's an example - if I get a dating site ad on Facebook it correctly identifies me as a woman and throws up pictures of guys. If that ad is being served up by Google, I get identified as a man because of all the time I spend on tech sites, so the same ad has pictures of women in it. Facebook gives me ads about clothes, Google almost never does. I suspect that Google's core business is at significant risk here. People are spending more and more time in the Facebook universe, those 'Like' buttons have popped up all over the Web, and that spells trouble for Google. It'll be interesting to see how they respond, but I for one won't be buying any more stock in Google for a while.

jakevoytko 16y ago
I think Norvig looks at Facebook as a lost opportunity for data mining: Facebook isn't used as a recommendation engine any more than Google is used as an ad company, yet all user actions always gets piped back to ad results. If John is friends with Sally (who is kind of cute, and kind of a photography geek, and recently bought a good midrange point-and-shoot), John subconsciously wants that brand to show up somewhere in the recommendations, even if it is too expensive. But Google doesn't know that John's friends have bought that camera, because who posts what camera they bought on their Google profile? If Facebook has the ad instead of Google, then Facebook might get the all-important click.
- msg 16y ago
  That is right. Norvig doesn't see what Facebook is good at and ask how it will make money. He turns the question on its head, looks to a successful business model and asks how Facebook could be used for it. And it is an uphill battle for Facebook to monetize in this fashion. I think what we've seen so far is that Facebook is about creating social spaces, not solving problems, so it really can't be used that effectively as a recommendation engine. Facebook doesn't have a great way to seek rent on these social spaces because they've built their user base on free service. So where is the money going to come from? Probably selling user data to corporations.

protomyth 16y ago

Does it seem like some people would really like to be a social network created by Visa?