4 ms·

The truth may be banal, but the impact is not. That's the problem with technology. So often, there's no ill intent in the design decisions, but at scale, the

The truth may be banal, but the impact is not. That's the problem with technology. So often, there's no ill intent in the design decisions, but at scale, the effects can be harmful, even massively harmful. > We're optimizing for click through rate. That's it. There's no editorial judgement... Something that all of us as technologists need to learn is that this IS an editorial judgement. We do not get to disclaim responsibility just because we delegated that responsibility to an algorithm. It is we who delegated it, we who chose the algorithm and the metrics, and we who are responsible. "We're optimizing for click through rate" is how we got the proliferation of misinformation on Facebook. It's how we got https://twitter.com/chrislhayes/status/1037831503101579264 https://twitter.com/chrislhayes/status/1037831503101579264. It's how we got Pizzagate. "We're optimizing for click through rate" is simply not good enough in 2019. We could claim to be naive in 2000, perhaps even 2010. But today? We all know we're playing with fire now, and it doesn't much matter that of course we didn't MEAN to burn the house down. What matters is that we don't give lit matches to children and we know how to not set the house on fire.

prlambert 7y ago
I agree and good point on the broader meaning of editorial judgement. If we had seen any negative impact such as the FB/YouTube examples we would absolutely take that into account. I haven't seen anything remotely like yet with regard to smart reply but would want to be the first to see it if it exists.
- zestyping 7y ago
  Your willingness to use "we're optimizing for click through rate" as a defense is quite frightening to me, though. Someone who thinks "We're optimizing for click through rate" is morally neutral is not qualified to be making these decisions, just as someone who thinks giving lit matches to children is morally neutral is not qualified to be a fire marshal. Would you agree?
  - foldingmoney 7y ago
    It's a pretty long journey from 'our algorithm chooses which auto-replies to offer you based on which have the highest probability of being selected' to giving lit matches to children. I'm afraid to ask your opinion of SwiftKey's autocomplete.
    - zestyping 7y ago
      That's not quite what I'm saying, though. My concern is not with stating the fact that the algorithm uses this metric. My concern is with presenting the decision to use this metric as morally neutral. If you think that giving lit matches to children is morally neutral, you probably either don't understand how dangerous fire is or don't understand how unpredictable children can be. That's the point of the analogy.
      - foldingmoney 7y ago
        I don't know what could be more morally neutral than 'we offer the suggestions that you've shown us you're most likely to want to use.' Anything else would be trying to put _their_ words in people's mouths. And let's not lose sight of the fact that this is a tool that offers an automatic one to three-word email response for when you're too lazy to actually reply. The stakes are about as low as they come.
        zestyping 7y ago
        "As low as they come"? You might be forgetting that Gmail has about a billion monthly active users. Scale matters. I'm sure you can think of many products that have fewer users, and features in those products that are less central to them than e-mail composition is to a mail client.
        dTal 7y ago
        Pretty much the entire history of progressivism is a long, slow dawning realization that it's not harmless to treat individuals on the basis of group statistics, for one group category after another. Part of the problem is feedback loops - you can inadvertently amplify subtle perturbations by feeding them back to people. For example, it might seem harmless to subtly nudge a woman away from a career in STEM, on the basis that women are rare in STEM and therefore they probably won't enjoy it. But this action, repeated across millions of people, creates the pattern that you are reacting to. I think the principle probably generalizes to "everyone", not just "women" and "black people". In the case of autocomplete suggestions, they can still cause harm even when they're statistically likely. What do you think Google would autosuggest for "Blacks are...", if it were around in 1950? What would the statistics on the completion of that sentence look like? And would the effect of someone seeing that list be 100% neutral - or would it subtly nudge them? There's nothing obviously harmful with the specifics of GMail's auto-suggestions now. But the principle of 'we offer the suggestions that you've shown us you're most likely to want to use' is not morally neutral.
- zestyping 7y ago
  How would you know? And what would your threshold be? For example, suppose you did a controlled study and found that the new autocomplete UI was presenting users with only "yes" options, X% of the time. Is there a threshold beyond which you would unlaunch or adjust the feature? What if the new UI was actually changing user behaviour, causing them to answer "yes" more often than they otherwise would have? Is there a delta (positive or negative) beyond which you would unlaunch or adjust the feature? If you had to adjust it, how would you know you were adjusting it correctly? What sort of telemetry would you need to put in the product so that you really would know if this was happening? I'm curious if your team thought about these questions before launching, and what ideas came up.
- dmitriid 7y ago
  > If we had seen any negative impact such as the FB/YouTube examples we would absolutely take that into account. Maybe? Maybe not? There’s visible and tangible negative impact from Youtube algorithms (owned, incidentally by the same conpany). And Youtube (or Google? Or Alphabet?) rarely if ever acknowledges and fixes those problems because their algorithms are also optimised for click-through.
kkarakk 7y ago
Pretty sure Gmail is a product not a life giving service, they still do tend to optimise for people continuing to use their service. The societal effects of X technology seems to be the new scare tactic buzzword. It's an automated email response used for emails that don't merit an actual response, the slippery slope fallacy doesn't apply here. I'd apply that instead to their language autocomplete feature(the grayed out text that appears when you start typing and that you can just press tab to insert into your email) which is scary in the way it tries to shepherd your language
fenomas 7y ago
I read GP as saying they watched the system's behavior and decided that going by click rates gave the best results. I'd say that is the editorial judgment, in the sense that you're talking about. If they just set things up to go by click rates and assumed that whatever results occurred must necessarily be preferable, then that would be disclaiming responsibility in the way you describe, but GP's version doesn't sound like that.
- awinter-py 7y ago
  It's not editorial it's meta-editorial (metatorial?). They're picking the method that picks the content, not the content itself. (But this isn't blind -- they're picking the method while watching the output, so it may be a fancy way of picking the exact content you want plus other similar content plus adversarial noise).
aeternus 7y ago
In this case click through rate does seem to be the right measure. The feature is most useful when it is suggesting the most likely replies.
Arnt 7y ago
So what does good mean in this case, then? It's easy to say that something isn't good enough if you don't elaborate on what the axis good-bad is.
deleted 7y ago
[deleted]
- deleted 7y ago
  [deleted]
B-Con 7y ago
It is good enough, 2019 doesn't mean we don't reflect user behavior. Not every recommender is a moral agent. Auto-suggest and conspiracy videos aren't the same thing. And technically, yes, there's not such thing as a non-editorialized recommender, but this feels like a pointless nit since there's a world of difference between trying to be a reflection vs intentionally not reflecting based on a human-made moral judgement. What even is the moral editorialization in auto-suggest? It's obvious we don't want conspiracy videos pushed front and center of educational topics, but how on earth do you balance the number of "yes" vs "no"? That's clearly going to be a completely made up number. If you don't have a clear problem to fight, it's probably best to not editorialize just to say you did.
ljm 7y ago
Totally agreed. My take on it is that all this work on optimising for business metrics is optimising away humanity, and diversity. It almost sounds like human nature gets in the way of revenue, so it has to be manipulated into something more business friendly. We won’t consciously make that choice but it often seems to be the consequence.
- dTal 7y ago
  I think this is exactly right, though it sounds pithy. The same process produces food that is unsubtle, fatty, and highly sugared. It sells really well! But in the long run, it's terrible for us, and also boring - all fast food tastes more or less the same, because the more adventurous it is, the worse it sells. The term frequently used in discourse is "lowest common denominator". It's a question of capitalism optimizing for local optima versus global optima. What were seeing with the internet is, what happens when we apply this process not just to food, but culture itself?
ramblerman 7y ago
A lot of outrage and grandiose claims about misuses of technology, but you didn't really tackle the issue at hand. What dangers do you see in this specific example. People will be too lazy to type no, if they don't agree?

prlambert 7y ago

I agree and good point on the broader meaning of editorial judgement. If we had seen any negative impact such as the FB/YouTube examples we would absolutely take that into account. I haven't seen anything remotely like yet with regard to smart reply but would want to be the first to see it if it exists.

zestyping 7y ago
Your willingness to use "we're optimizing for click through rate" as a defense is quite frightening to me, though. Someone who thinks "We're optimizing for click through rate" is morally neutral is not qualified to be making these decisions, just as someone who thinks giving lit matches to children is morally neutral is not qualified to be a fire marshal. Would you agree?
- foldingmoney 7y ago
  It's a pretty long journey from 'our algorithm chooses which auto-replies to offer you based on which have the highest probability of being selected' to giving lit matches to children. I'm afraid to ask your opinion of SwiftKey's autocomplete.
  - zestyping 7y ago
    That's not quite what I'm saying, though. My concern is not with stating the fact that the algorithm uses this metric. My concern is with presenting the decision to use this metric as morally neutral. If you think that giving lit matches to children is morally neutral, you probably either don't understand how dangerous fire is or don't understand how unpredictable children can be. That's the point of the analogy.
    - foldingmoney 7y ago
      I don't know what could be more morally neutral than 'we offer the suggestions that you've shown us you're most likely to want to use.' Anything else would be trying to put _their_ words in people's mouths. And let's not lose sight of the fact that this is a tool that offers an automatic one to three-word email response for when you're too lazy to actually reply. The stakes are about as low as they come.
      - zestyping 7y ago
        "As low as they come"? You might be forgetting that Gmail has about a billion monthly active users. Scale matters. I'm sure you can think of many products that have fewer users, and features in those products that are less central to them than e-mail composition is to a mail client.
      - dTal 7y ago
        Pretty much the entire history of progressivism is a long, slow dawning realization that it's not harmless to treat individuals on the basis of group statistics, for one group category after another. Part of the problem is feedback loops - you can inadvertently amplify subtle perturbations by feeding them back to people. For example, it might seem harmless to subtly nudge a woman away from a career in STEM, on the basis that women are rare in STEM and therefore they probably won't enjoy it. But this action, repeated across millions of people, creates the pattern that you are reacting to. I think the principle probably generalizes to "everyone", not just "women" and "black people". In the case of autocomplete suggestions, they can still cause harm even when they're statistically likely. What do you think Google would autosuggest for "Blacks are...", if it were around in 1950? What would the statistics on the completion of that sentence look like? And would the effect of someone seeing that list be 100% neutral - or would it subtly nudge them? There's nothing obviously harmful with the specifics of GMail's auto-suggestions now. But the principle of 'we offer the suggestions that you've shown us you're most likely to want to use' is not morally neutral.
zestyping 7y ago
How would you know? And what would your threshold be? For example, suppose you did a controlled study and found that the new autocomplete UI was presenting users with only "yes" options, X% of the time. Is there a threshold beyond which you would unlaunch or adjust the feature? What if the new UI was actually changing user behaviour, causing them to answer "yes" more often than they otherwise would have? Is there a delta (positive or negative) beyond which you would unlaunch or adjust the feature? If you had to adjust it, how would you know you were adjusting it correctly? What sort of telemetry would you need to put in the product so that you really would know if this was happening? I'm curious if your team thought about these questions before launching, and what ideas came up.
dmitriid 7y ago
> If we had seen any negative impact such as the FB/YouTube examples we would absolutely take that into account. Maybe? Maybe not? There’s visible and tangible negative impact from Youtube algorithms (owned, incidentally by the same conpany). And Youtube (or Google? Or Alphabet?) rarely if ever acknowledges and fixes those problems because their algorithms are also optimised for click-through.

kkarakk 7y ago

Pretty sure Gmail is a product not a life giving service, they still do tend to optimise for people continuing to use their service. The societal effects of X technology seems to be the new scare tactic buzzword. It's an automated email response used for emails that don't merit an actual response, the slippery slope fallacy doesn't apply here. I'd apply that instead to their language autocomplete feature(the grayed out text that appears when you start typing and that you can just press tab to insert into your email) which is scary in the way it tries to shepherd your language

fenomas 7y ago

I read GP as saying they watched the system's behavior and decided that going by click rates gave the best results. I'd say that is the editorial judgment, in the sense that you're talking about. If they just set things up to go by click rates and assumed that whatever results occurred must necessarily be preferable, then that would be disclaiming responsibility in the way you describe, but GP's version doesn't sound like that.

awinter-py 7y ago
It's not editorial it's meta-editorial (metatorial?). They're picking the method that picks the content, not the content itself. (But this isn't blind -- they're picking the method while watching the output, so it may be a fancy way of picking the exact content you want plus other similar content plus adversarial noise).

aeternus 7y ago

In this case click through rate does seem to be the right measure. The feature is most useful when it is suggesting the most likely replies.

Arnt 7y ago

So what does good mean in this case, then? It's easy to say that something isn't good enough if you don't elaborate on what the axis good-bad is.

deleted 7y ago

[deleted]

deleted 7y ago
[deleted]

B-Con 7y ago

It is good enough, 2019 doesn't mean we don't reflect user behavior. Not every recommender is a moral agent. Auto-suggest and conspiracy videos aren't the same thing. And technically, yes, there's not such thing as a non-editorialized recommender, but this feels like a pointless nit since there's a world of difference between trying to be a reflection vs intentionally not reflecting based on a human-made moral judgement. What even is the moral editorialization in auto-suggest? It's obvious we don't want conspiracy videos pushed front and center of educational topics, but how on earth do you balance the number of "yes" vs "no"? That's clearly going to be a completely made up number. If you don't have a clear problem to fight, it's probably best to not editorialize just to say you did.

ljm 7y ago

Totally agreed. My take on it is that all this work on optimising for business metrics is optimising away humanity, and diversity. It almost sounds like human nature gets in the way of revenue, so it has to be manipulated into something more business friendly. We won’t consciously make that choice but it often seems to be the consequence.

dTal 7y ago
I think this is exactly right, though it sounds pithy. The same process produces food that is unsubtle, fatty, and highly sugared. It sells really well! But in the long run, it's terrible for us, and also boring - all fast food tastes more or less the same, because the more adventurous it is, the worse it sells. The term frequently used in discourse is "lowest common denominator". It's a question of capitalism optimizing for local optima versus global optima. What were seeing with the internet is, what happens when we apply this process not just to food, but culture itself?

ramblerman 7y ago

A lot of outrage and grandiose claims about misuses of technology, but you didn't really tackle the issue at hand. What dangers do you see in this specific example. People will be too lazy to type no, if they don't agree?