12 ms·

> People seem to like it — we are seeing 5,000 hits to our API a minute to serve those cards that show when you hover over any link. Uhm, no. It just means tha

> People seem to like it — we are seeing 5,000 hits to our API a minute to serve those cards that show when you hover over any link. Uhm, no. It just means that people are hovering over links with their mouse. It does not imply any opinion about the previews. > The original idea was conceived four years ago When I was active at Wikipedia/Wikibooks 12 years ago, there was a user script floating around that did the same thing, except I'm not sure if it included an image. (Mediawiki allows a user to define custom CSS and JS that get embedded in every page of their user session.) I don't mean to express dislike or downplay the hard work that went into this feature, just to add some context.

panic 9y ago
> Uhm, no. It just means that people are hovering over links with their mouse. It does not imply any opinion about the previews. When you evaluate features using engagement metrics, there are only two possibilities. If the metric is high, users love the feature. If the metric is low, users don't know the feature exists, and more alerts or "unread" badges must be added to help them learn.
- majewsky 9y ago
  My point is that the metric is bad.
- mappu 9y ago
  Alternatively, if it's a feature you don't like: If the metric is high, users are spending too much time on it. Also, it is responsible for every single frustration that users have with your product. If the metric is low, the feature can be safely removed (see: Windows Start menu).
  - megaman22 9y ago
    > If the metric is low, the feature can be safely removed (see: Windows Start menu). That turned out really well... Whoever is responsible for putting the touch start screen on Windows Server is responsible for 90% of my frustrations...
    - brazzledazzle 9y ago
      The best way to run Windows Server is without a desktop installed, using PowerShell to manage it. Even one-offs. You might know this but it’s worth saying just in case. Also worth noting that some server applications assume/require a desktop during install or operation (usually because they’re trash).
  - chiefalchemist 9y ago
    Yeah. Time on site, for example, might mean they can't find what they need; as opposed to getting what they need and staying for more. Numbers are helpful, but sometimes they can be misinterpreted to mask the why.
- dmitriid 9y ago
  > When you evaluate features using engagement metrics No-no-no. Stop right there. "Engagement metrics" are the worst kind of metrics. Engagement means next to nothing. It just means that a user interacted with something. Was it good? Was it bad? Was it intentional? Was it by mistake? "Engagement metrics" answer none of that. And yet... They are the easiest metrics to collect and too many companies use them as if they were meaningful.
  - mattkrause 9y ago
    If you hadn’t stopped right there, you might have seen the sarcasm in the rest of the post!
  - notriddle 9y ago
    I'm case you were curious, WM has done more user testing than just the engagement metrics: https://m.mediawiki.org/wiki/Special:MyLanguage/Wikimedia_Research/Design_Research/Reading_Team_UX_Research/Hovercards_Usability https://m.mediawiki.org/wiki/Special:MyLanguage/Wikimedia_Re... Not sure what it says about them that they're mentioning worthless numbers instead of the way more useful data they got back during this survey.
- iofiiiiiiiii 9y ago
  But when the feature only needs you to hover over a link to activate (which people do anyway) you have no basis to claim that this is users "liking" the feature - it might just be noise from everyday regular navigation.
  - teddyuk 9y ago
    i was using wikipedia to with my son yesterday for a project and found it really annoying, so i'm in that list of people who hit the api but aren't in the list of people who seem to be "liking it"
- samjbobb 9y ago
  Linked from the article is the final A/B test they ran: https://www.mediawiki.org/wiki/Page_Previews/2017-18_A/B_Tests https://www.mediawiki.org/wiki/Page_Previews/2017-18_A/B_Tes... They had better metrics than just number of API requests
jwilk 9y ago
Also, is 5K hits/minute really that impressive? According to https://stats.wikimedia.org/EN/Sitemap.htm https://stats.wikimedia.org/EN/Sitemap.htm , English Wikipedia gets 88K views per minute.
- foota 9y ago
  I'm thinking it has to be heavily cached.
  - majewsky 9y ago
    "5000 hits to our API per minute" does not say anything about whether they're measuring in front of or behind the cache. For all intents and purposes, the cache's endpoint is the API endpoint.
    - nkozyra 9y ago
      And uncached hits only would be an even more worthless metric. I, too, think either that number is wrong or this wasn't a feature available to all users. I also agree that using this metric as affirmation the feature is liked is dangerous; count me among the many here inadvertantly contributing without enjoying the feature.
- majewsky 9y ago
  That was also my first reaction, but I did not mention it because judging this (in relation to pageviews) would require more knowledge about the audience, e.g. how many users are using a mouse at all (vs. a touch screen, or pure keyboard navigation).
- giancarlostoro 9y ago
  I imagine most people spend more time reading on Wikipedia as opposed to trying to hover over every link available?
  - larkeith 9y ago
    That's less than one API hit per 17 English pages viewed - the overwhelming majority of the time, users do not use this feature at all, based on these numbers. This response is pure sophistry.
fjarlq 9y ago
I expect you're remembering User:Lupin/popups.js[1], which dates back to August 2005. It soon became known as Navigation Popups[2], and is still available today. The main documentation for Page Previews mentions Navigation Popups.[3] [1]: https://en.wikipedia.org/wiki/User:Lupin/popups.js https://en.wikipedia.org/wiki/User:Lupin/popups.js [2]: https://en.wikipedia.org/wiki/Wikipedia:Tools/Navigation_popups https://en.wikipedia.org/wiki/Wikipedia:Tools/Navigation_pop... [3]: https://www.mediawiki.org/wiki/Page_Previews https://www.mediawiki.org/wiki/Page_Previews
KenanSulayman 9y ago
As far as I know the Lupin user-script was the foundation of the PagePreview feature. I guess the Reading group at WikiMedia rewrote the extension as part of MediaWiki core and tried to fix most edge cases on the way.
Crontab 9y ago
I personally do not care for the link preview feature as I am one of those people who like to hover over a link so I can see the url leads (which is also why I detest URL shorteners). I also have to wonder how much bandwidth Wikipedia is going to end up using to display unneeded previews.
- tuukkah 9y ago
  > I personally do not care for the link preview feature as I am one of those people who like to hover over a link so I can see the url leads But that's exactly what these preview popups are for! A bit heavier than the plain URL, but much more useful and friendlier. (EDIT: I see there's a bug in that it displays the preview even if you don't stop the mouse there. Still, this can be lighter than requiring people to click through to the full articles, considering the preview can tell you what you need or that the link is not relevant for you, and slows you going down the rabbit hole to more links. Maybe it should be located at the bottom of the screen like the plain URL popup though?)
- zhte415 9y ago
  > I also have to wonder how much bandwidth Wikipedia is going to end up using to display unneeded previews. This is interesting. There's an external arrow for external links, so I'm still good with this use case for inspecting links; for internal links I've found myself hovering where I previously clicked (and loaded) an additional page at least 50% of the time. Where I'm OK with a quick paragraph description and can then decide if the linked paragraph requires / inspires a deeper read, about 50% of the time. I think this is a net reduction of bandwidth for my personal use-case. Wide A/B testing would reveal more, obviously.
subpixel 9y ago
I do mean to express dislike, as I'm one of the millions of users giving Wikipedia a false metric and sense of satisfaction. I've seen the pop-up hundreds of times, and have considered it a hindrance to my process every single time. Something that just happens accidentally is a bug, no matter how useful it might be if it were not happening accidentally.
- publicfig 9y ago
  You can disable it either from the screen, or if you have an account, permanently from your settings.
- bobbobo 9y ago
  I completely agree with this. When you go from reading the Atlantic, The Economist, The Washington Post... any content site... then begin to read a Wikipedia entry everything you know about reading an article no longer applies. You are now forced to change what you do. This is just the most basic no no in web design - unplanned interactions, forcing a user to interact, forcing a user to actually think about the interface. The developers have just forced every person who wants to consume content on Wikipedia to do it differently than on every other content site. And there are literally countless millions of people who will have no idea how to disable it, who use computers every day but have no idea how to change something. Just an incredible UX failure.
SeanMacConMara 9y ago
>Uhm, no. It just means that people are hovering over links with their mouse. in my case every single time was by accident whilst moving the mouse around, off of other text I was trying to read. frankly, its bloody annoying it covered the text i was reading the search for yet another "disable" setting begins....
- rootlocus 9y ago
  The popup has a little gear icon in the bottom right. You can disable it from there.
  - freeone3000 9y ago
    They put a small click button on the edge of a hover control? really?
    - deleted 9y ago
      [deleted]
    - always_good 9y ago
      I just tested it and it's trivial to click. Are you actually having difficulty or just pretending it's hard so you can be condescending? I think the best place for these controls are usually on the feature itself. I don't want to have to search in some independent preferences UI every I want to see if something can be tweaked, but it makes sense to offer it there as well.
  - SeanMacConMara 9y ago
    cheers!
- jessriedel 9y ago
  > the search for yet another "disable" setting begins.... Your inability to figure out how to disable it was already counted as success! > the rates of disabling the feature are negligibly low — a strong indicator that people find it useful. https://medium.com/freely-sharing-the-sum-of-all-knowledge/wikipedia-page-previews-738cddac7a21 https://medium.com/freely-sharing-the-sum-of-all-knowledge/w...
jdlrobson 9y ago
Hey! (author here) I sincerely regret the use of that statement "people seem to like it" in my post. I've now removed it as I worry this confuses my message so thank you for pointing it out. This really trivializes all the work that went into A/B testing this and how we measured success. I really recommend a read of https://www.mediawiki.org/wiki/Page_Previews#Success_Metrics_and_Feature_Evaluation https://www.mediawiki.org/wiki/Page_Previews#Success_Metrics... Side note: the volume of traffic was also wrong by several magnitudes... actually 0.5 million) My main motivation when writing this post was to share how small changes require magnitudes of effort not to express the merits of the feature. As a developer who works with product owners a lot and often get asked how I can build things quicker (I'm sure many can relate). I wanted to provide something useful to other developers that easy looking things are not actually easy to build, so thanks for flagging. With regards to the user script, yeh that's been around for some time and was the seed for this idea. It's just taken a long time to get that from such a small audience to a mainstream one. It doesn't downplay it in my opinion, just shows how far we've come. Thanks for reading.

panic 9y ago

> Uhm, no. It just means that people are hovering over links with their mouse. It does not imply any opinion about the previews. When you evaluate features using engagement metrics, there are only two possibilities. If the metric is high, users love the feature. If the metric is low, users don't know the feature exists, and more alerts or "unread" badges must be added to help them learn.

majewsky 9y ago
My point is that the metric is bad.
mappu 9y ago
Alternatively, if it's a feature you don't like: If the metric is high, users are spending too much time on it. Also, it is responsible for every single frustration that users have with your product. If the metric is low, the feature can be safely removed (see: Windows Start menu).
- megaman22 9y ago
  > If the metric is low, the feature can be safely removed (see: Windows Start menu). That turned out really well... Whoever is responsible for putting the touch start screen on Windows Server is responsible for 90% of my frustrations...
  - brazzledazzle 9y ago
    The best way to run Windows Server is without a desktop installed, using PowerShell to manage it. Even one-offs. You might know this but it’s worth saying just in case. Also worth noting that some server applications assume/require a desktop during install or operation (usually because they’re trash).
- chiefalchemist 9y ago
  Yeah. Time on site, for example, might mean they can't find what they need; as opposed to getting what they need and staying for more. Numbers are helpful, but sometimes they can be misinterpreted to mask the why.
dmitriid 9y ago
> When you evaluate features using engagement metrics No-no-no. Stop right there. "Engagement metrics" are the worst kind of metrics. Engagement means next to nothing. It just means that a user interacted with something. Was it good? Was it bad? Was it intentional? Was it by mistake? "Engagement metrics" answer none of that. And yet... They are the easiest metrics to collect and too many companies use them as if they were meaningful.
- mattkrause 9y ago
  If you hadn’t stopped right there, you might have seen the sarcasm in the rest of the post!
- notriddle 9y ago
  I'm case you were curious, WM has done more user testing than just the engagement metrics: https://m.mediawiki.org/wiki/Special:MyLanguage/Wikimedia_Research/Design_Research/Reading_Team_UX_Research/Hovercards_Usability https://m.mediawiki.org/wiki/Special:MyLanguage/Wikimedia_Re... Not sure what it says about them that they're mentioning worthless numbers instead of the way more useful data they got back during this survey.
iofiiiiiiiii 9y ago
But when the feature only needs you to hover over a link to activate (which people do anyway) you have no basis to claim that this is users "liking" the feature - it might just be noise from everyday regular navigation.
- teddyuk 9y ago
  i was using wikipedia to with my son yesterday for a project and found it really annoying, so i'm in that list of people who hit the api but aren't in the list of people who seem to be "liking it"
samjbobb 9y ago
Linked from the article is the final A/B test they ran: https://www.mediawiki.org/wiki/Page_Previews/2017-18_A/B_Tests https://www.mediawiki.org/wiki/Page_Previews/2017-18_A/B_Tes... They had better metrics than just number of API requests

jwilk 9y ago

Also, is 5K hits/minute really that impressive? According to https://stats.wikimedia.org/EN/Sitemap.htm https://stats.wikimedia.org/EN/Sitemap.htm , English Wikipedia gets 88K views per minute.

foota 9y ago
I'm thinking it has to be heavily cached.
- majewsky 9y ago
  "5000 hits to our API per minute" does not say anything about whether they're measuring in front of or behind the cache. For all intents and purposes, the cache's endpoint is the API endpoint.
  - nkozyra 9y ago
    And uncached hits only would be an even more worthless metric. I, too, think either that number is wrong or this wasn't a feature available to all users. I also agree that using this metric as affirmation the feature is liked is dangerous; count me among the many here inadvertantly contributing without enjoying the feature.
majewsky 9y ago
That was also my first reaction, but I did not mention it because judging this (in relation to pageviews) would require more knowledge about the audience, e.g. how many users are using a mouse at all (vs. a touch screen, or pure keyboard navigation).
giancarlostoro 9y ago
I imagine most people spend more time reading on Wikipedia as opposed to trying to hover over every link available?
- larkeith 9y ago
  That's less than one API hit per 17 English pages viewed - the overwhelming majority of the time, users do not use this feature at all, based on these numbers. This response is pure sophistry.

fjarlq 9y ago

I expect you're remembering User:Lupin/popups.js[1], which dates back to August 2005. It soon became known as Navigation Popups[2], and is still available today. The main documentation for Page Previews mentions Navigation Popups.[3] [1]: https://en.wikipedia.org/wiki/User:Lupin/popups.js https://en.wikipedia.org/wiki/User:Lupin/popups.js [2]: https://en.wikipedia.org/wiki/Wikipedia:Tools/Navigation_popups https://en.wikipedia.org/wiki/Wikipedia:Tools/Navigation_pop... [3]: https://www.mediawiki.org/wiki/Page_Previews https://www.mediawiki.org/wiki/Page_Previews

KenanSulayman 9y ago

As far as I know the Lupin user-script was the foundation of the PagePreview feature. I guess the Reading group at WikiMedia rewrote the extension as part of MediaWiki core and tried to fix most edge cases on the way.

Crontab 9y ago

I personally do not care for the link preview feature as I am one of those people who like to hover over a link so I can see the url leads (which is also why I detest URL shorteners). I also have to wonder how much bandwidth Wikipedia is going to end up using to display unneeded previews.

tuukkah 9y ago
> I personally do not care for the link preview feature as I am one of those people who like to hover over a link so I can see the url leads But that's exactly what these preview popups are for! A bit heavier than the plain URL, but much more useful and friendlier. (EDIT: I see there's a bug in that it displays the preview even if you don't stop the mouse there. Still, this can be lighter than requiring people to click through to the full articles, considering the preview can tell you what you need or that the link is not relevant for you, and slows you going down the rabbit hole to more links. Maybe it should be located at the bottom of the screen like the plain URL popup though?)
zhte415 9y ago
> I also have to wonder how much bandwidth Wikipedia is going to end up using to display unneeded previews. This is interesting. There's an external arrow for external links, so I'm still good with this use case for inspecting links; for internal links I've found myself hovering where I previously clicked (and loaded) an additional page at least 50% of the time. Where I'm OK with a quick paragraph description and can then decide if the linked paragraph requires / inspires a deeper read, about 50% of the time. I think this is a net reduction of bandwidth for my personal use-case. Wide A/B testing would reveal more, obviously.

subpixel 9y ago

I do mean to express dislike, as I'm one of the millions of users giving Wikipedia a false metric and sense of satisfaction. I've seen the pop-up hundreds of times, and have considered it a hindrance to my process every single time. Something that just happens accidentally is a bug, no matter how useful it might be if it were not happening accidentally.

publicfig 9y ago
You can disable it either from the screen, or if you have an account, permanently from your settings.
bobbobo 9y ago
I completely agree with this. When you go from reading the Atlantic, The Economist, The Washington Post... any content site... then begin to read a Wikipedia entry everything you know about reading an article no longer applies. You are now forced to change what you do. This is just the most basic no no in web design - unplanned interactions, forcing a user to interact, forcing a user to actually think about the interface. The developers have just forced every person who wants to consume content on Wikipedia to do it differently than on every other content site. And there are literally countless millions of people who will have no idea how to disable it, who use computers every day but have no idea how to change something. Just an incredible UX failure.

SeanMacConMara 9y ago

>Uhm, no. It just means that people are hovering over links with their mouse. in my case every single time was by accident whilst moving the mouse around, off of other text I was trying to read. frankly, its bloody annoying it covered the text i was reading the search for yet another "disable" setting begins....

rootlocus 9y ago
The popup has a little gear icon in the bottom right. You can disable it from there.
- freeone3000 9y ago
  They put a small click button on the edge of a hover control? really?
  - deleted 9y ago
    [deleted]
  - always_good 9y ago
    I just tested it and it's trivial to click. Are you actually having difficulty or just pretending it's hard so you can be condescending? I think the best place for these controls are usually on the feature itself. I don't want to have to search in some independent preferences UI every I want to see if something can be tweaked, but it makes sense to offer it there as well.
- SeanMacConMara 9y ago
  cheers!
jessriedel 9y ago
> the search for yet another "disable" setting begins.... Your inability to figure out how to disable it was already counted as success! > the rates of disabling the feature are negligibly low — a strong indicator that people find it useful. https://medium.com/freely-sharing-the-sum-of-all-knowledge/wikipedia-page-previews-738cddac7a21 https://medium.com/freely-sharing-the-sum-of-all-knowledge/w...

jdlrobson 9y ago

Hey! (author here) I sincerely regret the use of that statement "people seem to like it" in my post. I've now removed it as I worry this confuses my message so thank you for pointing it out. This really trivializes all the work that went into A/B testing this and how we measured success. I really recommend a read of https://www.mediawiki.org/wiki/Page_Previews#Success_Metrics_and_Feature_Evaluation https://www.mediawiki.org/wiki/Page_Previews#Success_Metrics... Side note: the volume of traffic was also wrong by several magnitudes... actually 0.5 million) My main motivation when writing this post was to share how small changes require magnitudes of effort not to express the merits of the feature. As a developer who works with product owners a lot and often get asked how I can build things quicker (I'm sure many can relate). I wanted to provide something useful to other developers that easy looking things are not actually easy to build, so thanks for flagging. With regards to the user script, yeh that's been around for some time and was the seed for this idea. It's just taken a long time to get that from such a small audience to a mainstream one. It doesn't downplay it in my opinion, just shows how far we've come. Thanks for reading.