3 ms·

> A single number just can't represent the result of something that matters I just fundamentally disagree with this. One of the most famous examples is the you

> A single number just can't represent the result of something that matters I just fundamentally disagree with this. One of the most famous examples is the youtube team working towards a billion watch-hours in a day. Does that not matter to youtube? And if it's impossible for something that matters to be represented by a number, what does matter? I'd love to hear examples of what matters in your companies

mewpmewp2 3y ago
That's a really good example you just brought up, because if your goal is to get billion watch hours a day: 1. Why are you doing this? Is it to get more ad revenue? 2. Considering this metric, how do you know that the extra hours you are getting is the kind of traffic that brings ad revenue? Maybe it's children or folks from poor country. Maybe it's actually costing more than it was previously. 3. Maybe you are getting the same people to watch more, but it won't make them pay more. You are just increasing your server costs. And making those people addicted to your platform. 4. Maybe the algorithm that you produce is actively harmful to people by making them take too much screen-time leading to potential regulations from the government. 5. Maybe the content you provide is of really low value, and in the long run will make your audience really low value. 6. Maybe you just pay too much for that extra traffic, so it's not worth it. Very simple example when doing ads. If my goal is to get 200% more clicks, and I have been so far advertising in US only, I'll just now make the destination to be India instead. I will get tons more clicks, but I will get less payments than before. > And if it's impossible for something that matters to be represented by a number, what does matter? I'd love to hear examples of what matters in your companies If you want a number, profit obviously matters, but everything below that, can mostly be intuitive unless it obviously fits some sort of numerical idea. In such case building a great product matters, but it's much more difficult to measure what makes a great product, so you need this intuitive discussion and people who have great intuition for building a good product. Profit will follow performing this strategy. You should measure, monitor, use data to be informed and analyze results, but you should not have these as primary goals to strive for. You should use this data to improve your intuition.
- RandomLensman 3y ago
  The questions and points are what you should go through before arriving at the billion watch hours.
  - mewpmewp2 3y ago
    Before arriving at the goal, or before setting that goal?
    - RandomLensman 3y ago
      Sorry, I was a bit imprecise there. I meant before setting such a goal ("arriving at that being a goal").
      - mewpmewp2 3y ago
        So to describe the process. We would first brainstorm that watch hours is something we want to potentially be our OKR? And then we brainstorm all the constraints and potential pitfalls and finally we think what the reasonable amount of watch hours is based on those constraints? E.g. we first run by the constraints that these watch hours must still keep the same base level of ad revenue per watch hour, demographics proportions must stay similar, etc? I do feel however despite that there will be quite many different ways to game this number that you couldn't think of before hand. And there's also a spectrum of "level of gaming". There's a spectrum of non obvious gaming to very obvious. So whoever is striving towards that goal will still be incentivised to find the level of gaming that is not obvious enough for everyone, but might still be fundamentally not good. So then you have to spend a lot of your time to make sure this OKR couldn't be gamed, but you still probably won't be able to stop it from being gamed even if you spend very, very much time on preventing it.
        RandomLensman 3y ago
        No, we would figure out what we actually want to achieve (objective) and then (perhaps) arrive at that watch hours might be a useful way to measure our movement towards our objective. If it is too much likely to be gamed than perhaps it is not a good metric or it needs other metrics alongside. If everything we can think of in terms of measuring if we get to our objective is too easily/too much game-able, then perhaps the objective needs revisiting. Having things gamed to the extreme is also not such a big issue usually because in the end you also watch things like profit. During the ZIRP frenzy that got a bit forgotten in some places, so gaming was perhaps more of an issue. You can also switch objectives and measures if they turn out to be wrong/not useful.
tlb 3y ago
Watch-hours obviously matter, but so do other things like "are the best up & coming creators using us?" The danger is the tendency to focus on boosting measurable things at the expense of nonmeasurables. Often the nonmeasurables eventually show up as measurables, but after too long a time delay to fix mistakes. So you could alienate creators while juicing the current numbers, and find a few years later that all the good content has gone elsewhere. The above didn't happen at YouTube, but it's the sort of thing that can happen. An example of something that did happen: Facebook found that rage-bait political news created a lot of clicks, but led to people getting fed up and leaving the platform after a while. By the time that was reflected in the traffic stats, it was too late to get them back. But fed-upness wasn't something they could easily measure, so it got ignored.

mewpmewp2 3y ago

That's a really good example you just brought up, because if your goal is to get billion watch hours a day: 1. Why are you doing this? Is it to get more ad revenue? 2. Considering this metric, how do you know that the extra hours you are getting is the kind of traffic that brings ad revenue? Maybe it's children or folks from poor country. Maybe it's actually costing more than it was previously. 3. Maybe you are getting the same people to watch more, but it won't make them pay more. You are just increasing your server costs. And making those people addicted to your platform. 4. Maybe the algorithm that you produce is actively harmful to people by making them take too much screen-time leading to potential regulations from the government. 5. Maybe the content you provide is of really low value, and in the long run will make your audience really low value. 6. Maybe you just pay too much for that extra traffic, so it's not worth it. Very simple example when doing ads. If my goal is to get 200% more clicks, and I have been so far advertising in US only, I'll just now make the destination to be India instead. I will get tons more clicks, but I will get less payments than before. > And if it's impossible for something that matters to be represented by a number, what does matter? I'd love to hear examples of what matters in your companies If you want a number, profit obviously matters, but everything below that, can mostly be intuitive unless it obviously fits some sort of numerical idea. In such case building a great product matters, but it's much more difficult to measure what makes a great product, so you need this intuitive discussion and people who have great intuition for building a good product. Profit will follow performing this strategy. You should measure, monitor, use data to be informed and analyze results, but you should not have these as primary goals to strive for. You should use this data to improve your intuition.

RandomLensman 3y ago
The questions and points are what you should go through before arriving at the billion watch hours.
- mewpmewp2 3y ago
  Before arriving at the goal, or before setting that goal?
  - RandomLensman 3y ago
    Sorry, I was a bit imprecise there. I meant before setting such a goal ("arriving at that being a goal").
    - mewpmewp2 3y ago
      So to describe the process. We would first brainstorm that watch hours is something we want to potentially be our OKR? And then we brainstorm all the constraints and potential pitfalls and finally we think what the reasonable amount of watch hours is based on those constraints? E.g. we first run by the constraints that these watch hours must still keep the same base level of ad revenue per watch hour, demographics proportions must stay similar, etc? I do feel however despite that there will be quite many different ways to game this number that you couldn't think of before hand. And there's also a spectrum of "level of gaming". There's a spectrum of non obvious gaming to very obvious. So whoever is striving towards that goal will still be incentivised to find the level of gaming that is not obvious enough for everyone, but might still be fundamentally not good. So then you have to spend a lot of your time to make sure this OKR couldn't be gamed, but you still probably won't be able to stop it from being gamed even if you spend very, very much time on preventing it.
      - RandomLensman 3y ago
        No, we would figure out what we actually want to achieve (objective) and then (perhaps) arrive at that watch hours might be a useful way to measure our movement towards our objective. If it is too much likely to be gamed than perhaps it is not a good metric or it needs other metrics alongside. If everything we can think of in terms of measuring if we get to our objective is too easily/too much game-able, then perhaps the objective needs revisiting. Having things gamed to the extreme is also not such a big issue usually because in the end you also watch things like profit. During the ZIRP frenzy that got a bit forgotten in some places, so gaming was perhaps more of an issue. You can also switch objectives and measures if they turn out to be wrong/not useful.

tlb 3y ago

Watch-hours obviously matter, but so do other things like "are the best up & coming creators using us?" The danger is the tendency to focus on boosting measurable things at the expense of nonmeasurables. Often the nonmeasurables eventually show up as measurables, but after too long a time delay to fix mistakes. So you could alienate creators while juicing the current numbers, and find a few years later that all the good content has gone elsewhere. The above didn't happen at YouTube, but it's the sort of thing that can happen. An example of something that did happen: Facebook found that rage-bait political news created a lot of clicks, but led to people getting fed up and leaving the platform after a while. By the time that was reflected in the traffic stats, it was too late to get them back. But fed-upness wasn't something they could easily measure, so it got ignored.