4 ms·

> "But given this has been live for 7 months and is very usable[sic] visible (see this reddit[link] thread where users in our ablation experiment noticed and ca

> "But given this has been live for 7 months and is very usable[sic] visible (see this reddit[link] thread where users in our ablation experiment noticed and called it a bug!)" The reddit link & comment is just... chef's kiss

Fluorescence 3y ago
> ablation experiment What is that? "Ablation" as in removal? Do they AB test removing new features just for data? ... and also monitor SM threads for feedback?
- hyeonwho22 3y ago
  In AI research ablation experiments are comparative tests, but in one branch some feature is removed. Here the Chrome team is saying that removal of the feature (which the Ads team wanted) was instantly noticed by users.
- Atsuii 3y ago
  Yes ablation as in removal. They then refer to using the 3% experiment to estimate out revenue loss, so it appears they were running an AB test where it was removed for 3% of users so they could understand the metric impact of removal. Part of being a decent PM is monitoring SM for any feature talk.
- deleted 3y ago
  [deleted]
harrulous 3y ago
I did some sleuthing, and I found a notable amount of complaints about a "bug" around this time (mid March 2019), after Chrome update 73.0.3683.75 went live. Some examples: [1][2][3] More specifically, starting 14 March, many users noted that the latest Chrome update removed almost everything from the New Tab page (including shortcuts, themes, settings button), leaving a single prominent Google search bar — nothing else! This looks like it was intended to increase the use of the search bar, thus increasing SQV. By prominently displaying it as the only option when using New Tab whilst removing shortcuts, users accessing frequently used sites could only do so via search. I'm not sure if this is exactly Ablation Experiment #7 that Anil was resisting to rollback, but it appears seemingly related to the general ablation experiments performed during this time to increase SQV. Given Jerry eventually suggested a "scrappy tactical" compromise, i.e. "increasing vertical space between the search box/icons/feed on the new tab to make search more prominent", the March "bug" definitely seems to be related to these emails. It is absolutely crazy to see tangible evidence of the pressure from the Ads(?) Team forcing the Chrome team to ablate/degrade functionality for their users. [1] https://support.google.com/chrome/thread/2403795?msgid=2404523 https://support.google.com/chrome/thread/2403795?msgid=24045... [2] https://old.reddit.com/r/chrome/comments/b0ynkq/missing_2_shortcuts_on_new_tab/eii9ccx/ https://old.reddit.com/r/chrome/comments/b0ynkq/missing_2_sh... [3] https://bugs.chromium.org/p/chromium/issues/detail?id=941631 https://bugs.chromium.org/p/chromium/issues/detail?id=941631
- bpye 3y ago
  Additionally, Chrome 69 was released ~7 months before that email [0] - in September 2018. This release added answers to certain queries directly in the Omnibox - and presumably it was that change that impacted ad revenue. [0] - https://venturebeat.com/business/chrome-69-arrives-with-revamped-design-more-powerful-omnibox-and-better-password-manager/ https://venturebeat.com/business/chrome-69-arrives-with-reva...

Fluorescence 3y ago

> ablation experiment What is that? "Ablation" as in removal? Do they AB test removing new features just for data? ... and also monitor SM threads for feedback?

hyeonwho22 3y ago
In AI research ablation experiments are comparative tests, but in one branch some feature is removed. Here the Chrome team is saying that removal of the feature (which the Ads team wanted) was instantly noticed by users.
Atsuii 3y ago
Yes ablation as in removal. They then refer to using the 3% experiment to estimate out revenue loss, so it appears they were running an AB test where it was removed for 3% of users so they could understand the metric impact of removal. Part of being a decent PM is monitoring SM for any feature talk.
deleted 3y ago
[deleted]

harrulous 3y ago

I did some sleuthing, and I found a notable amount of complaints about a "bug" around this time (mid March 2019), after Chrome update 73.0.3683.75 went live. Some examples: [1][2][3] More specifically, starting 14 March, many users noted that the latest Chrome update removed almost everything from the New Tab page (including shortcuts, themes, settings button), leaving a single prominent Google search bar — nothing else! This looks like it was intended to increase the use of the search bar, thus increasing SQV. By prominently displaying it as the only option when using New Tab whilst removing shortcuts, users accessing frequently used sites could only do so via search. I'm not sure if this is exactly Ablation Experiment #7 that Anil was resisting to rollback, but it appears seemingly related to the general ablation experiments performed during this time to increase SQV. Given Jerry eventually suggested a "scrappy tactical" compromise, i.e. "increasing vertical space between the search box/icons/feed on the new tab to make search more prominent", the March "bug" definitely seems to be related to these emails. It is absolutely crazy to see tangible evidence of the pressure from the Ads(?) Team forcing the Chrome team to ablate/degrade functionality for their users. [1] https://support.google.com/chrome/thread/2403795?msgid=2404523 https://support.google.com/chrome/thread/2403795?msgid=24045... [2] https://old.reddit.com/r/chrome/comments/b0ynkq/missing_2_shortcuts_on_new_tab/eii9ccx/ https://old.reddit.com/r/chrome/comments/b0ynkq/missing_2_sh... [3] https://bugs.chromium.org/p/chromium/issues/detail?id=941631 https://bugs.chromium.org/p/chromium/issues/detail?id=941631

bpye 3y ago
Additionally, Chrome 69 was released ~7 months before that email [0] - in September 2018. This release added answers to certain queries directly in the Omnibox - and presumably it was that change that impacted ad revenue. [0] - https://venturebeat.com/business/chrome-69-arrives-with-revamped-design-more-powerful-omnibox-and-better-password-manager/ https://venturebeat.com/business/chrome-69-arrives-with-reva...