5 ms·

Paper linking frequency of search terms to violence against women retracted

56 points by samizdis 5y ago

Cenk 5y ago
> Using date range commands, the author claimed to compare searches for five months in 2019 to the same five months in 2020 — pre-pandemic to mid-pandemic. They did so not by turning to Google Trends data — the method promoted by the scholars they cited — but by inputting their search phrases with date delimiters using Google search. > They reported the number of hits Google displays at the top of the page as the number of searches made for that search string. >These numbers were exceptionally high because, well, the search phrases were not enclosed in quotation marks. All this in an article arguing for the value of tech-enabled “rapid response” research. A culture of “move fast and break things” is common in Silicon Valley, but academics typically work a bit more slowly and carefully to avoid these kinds of errors.
- tgv 5y ago
  Even then, if it had been executed flawlessly, you can only expect that search term frequency and violence remain correlated ceteris paribus, i.e., when all else is the same. The two time periods are clearly different, so how the hell did it even get published?
- wumpus 5y ago
  The number of hits reported by web-scale engines like Google and Bing are fabrications. When I was the CTO of a full-web search engine startup, I'd get an API access request every couple of months by someone wanting to use hit counts for an academic purpose. I would always them no and why; often they wanted to argue with me. http://searchengineland.com/why-google-cant-count-results-properly-53559 http://searchengineland.com/why-google-cant-count-results-pr... http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/google-result-counts-are-a-meaningless-metric.html http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/goo...
  - azinman2 5y ago
    This was the case when I was at Google in 2008; I’m sure it’s even further fabricated now.
    - throwaway3699 5y ago
      Can you elaborate on what you saw?
      - azinman2 5y ago
        That it’s not based on reality at all. That would likely be far to expensive computationally for any arbitrary query to give an accurate number that isn’t particularly valuable outside of degrees of orders of magnitude, and even that is almost never looked at.
    - Shish2k 5y ago
      I’m curious, what are they based on, and is there any reason to have a number there beyond “people expect a number”?
      - azinman2 5y ago
        It’s basically a random number. Don’t extract any meaning from it.
  - lakis 5y ago
    I've worked to Inktomi/Yahoo search ~2000-~2015. Any query that is accurate across the whole index is going to be very very expensive. In order to return fast results, there are caches and precomputed results in almost every level of a web search query. But an accurate count implies that you will bypass all the caches and count every single document for every single query term. That's very very expensive in both CPU and memory. We had special internal keywords that disabled caches and produced accurate results. All of them came with special warnings that querying too fast with these special options could bring down the whole search engine. A constant worry was the new employee that knew very little, learnt these keywords and then ops were dealing with crashes across ten of thousands of machines due to out of memory. As to the fabrication. The number are not random numbers. They are estimates based on what we think is the best approximation of the real numbers. We had done internal research to compute the best approximation function. Also, it's very difficult to figure out what is real and what is fake from the outside. I was involved in https://www.nytimes.com/2005/08/10/business/worldbusiness/yahoo-says-its-search-index-is-bigger-than-googles.html https://www.nytimes.com/2005/08/10/business/worldbusiness/ya... Yahoo was 100% correct on their claims. I run the queries myself. But external researchers could not verify the claims because all external queries were hitting caches. So they were trying unique queries in order to estimate the index size which end up having a lot of problems.
- croes 5y ago
  By that logic, every site citing the paper makes the situation worse.
tryonenow 5y ago
>How did this article get through peer review? Like the author, the journal’s reviewers and editors seemed to have been glamoured by the shine, tech fetishism, and naive empiricism of even the most poorly executed digital methods — without the methodological humility to work together with colleagues from information science, or at least check in with someone familiar with the basic workings of tools like Google Well, that's what happens when your institution is brimming with ideologues who practice one sided research and immediately praise any results that confirms their political, dogmatic biases. Doubly so when criticizing certain results or topics will get you implicitly or ex-communicated, particularly if you are not part of an approved protected class. The retraction doesn't matter very much, the damage has already been done, and far more eyes will have been exposed to the results than to the retraction.
ezequiel-garzon 5y ago
Regardless of the subject matter, I think it’s good policy to do what arXiv does with retractions: they become a new, empty version, alongside a corresponding explanation, but all previous versions remain on the record.
ec109685 5y ago
I watched this happen recently in real time. This article interpreted a Flurry study wrong and was featured on Tech Meme (its headline has since been softened): https://www.macrumors.com/2021/05/07/most-iphone-users-app-tracking-opt-out/ https://www.macrumors.com/2021/05/07/most-iphone-users-app-t... – "Analytics Suggest 96% of Users Leave App Tracking Disabled in iOS 14.5" Then the game of telephone began as bloggers simply rewrote the article and added their own spin: https://www.techmeme.com/210507/p14#a210507p14 https://www.techmeme.com/210507/p14#a210507p14 - Ars: 96% of US users opt out of app tracking in iOS 14.5, analytics find - iMore: 96% of iPhone users have opted out of app tracking since iOS 14.5 launched - Wccftech: Analytics Reveal 96 Percent of Users Have Disabled iOS 14.5's App Tracking on Their iPhone - Ubergizmo: 96% Of iOS Users Have Opted Out Of App Tracking - … Flurry since updated their study to make it clear what it was measuring and found opt out in rate to be about 24% for prompted users (not the 4% cited): https://www.flurry.com/blog/ios-14-5-opt-in-rate-att-restricted-app-tracking-transparency-worldwide-us-daily-latest-update/ https://www.flurry.com/blog/ios-14-5-opt-in-rate-att-restric... But none of those articles are interested in correction and the initial visceral reaction and herumphing tweets have all since occurred. Hundreds of thousands of people got the wrong idea. What it shows it that there is very little original reporting, people don't ask study authors for clarification, and misinterpreting of data happens all too often.
doggodaddo78 5y ago
Men are bad, even though we need them for homo sapiens sapiens to continue existing without cloning. Let's have a study and write a paper to find that conclusion! We don't need no men! Yeah! Oh wait, we used thin correlation not even matched in time instead measuring causes, effects, and search terms. We're going to have to retract that one. ): But we published! :) Damnit! Find real evidence, or create your own, like honey traps at tech conferences! Okay, let's try those!
- incrudible 5y ago
  The "we need males to continue the species"-argument isn't a very good one to keep men around. Very few men can impregnate an enormous amount of women. You might want keep around a good selection in your breeding zoo, for genetic variance. However, you won't need >50% as nature dictates.
  - throwaway3699 5y ago
    Genetic diversity is critical for the long-term health of a species.
    - doggodaddo78 5y ago
      Exactly. Either mass polyandry or mass polygyny like a few Moulay Ismail Ibn Sharifs / the Dr. Strangelove suggestion are bad ideas that lead to de facto near incest in subsequent generations. Cloning is also a terrible idea because it freezes evolution to changing conditions and creates an existential risk like crop monoculture that could wipe-out the entire species with one pathogen.
croes 5y ago
QED https://news.ycombinator.com/item?id=27241715 https://news.ycombinator.com/item?id=27241715
- alecco 5y ago
  I did not see that yesterday. There should be some indication when something got punished in rankings by flagging but not [flagged] altogether. @dang?

56 points by samizdis 5y ago

Cenk 5y ago

> Using date range commands, the author claimed to compare searches for five months in 2019 to the same five months in 2020 — pre-pandemic to mid-pandemic. They did so not by turning to Google Trends data — the method promoted by the scholars they cited — but by inputting their search phrases with date delimiters using Google search. > They reported the number of hits Google displays at the top of the page as the number of searches made for that search string. >These numbers were exceptionally high because, well, the search phrases were not enclosed in quotation marks. All this in an article arguing for the value of tech-enabled “rapid response” research. A culture of “move fast and break things” is common in Silicon Valley, but academics typically work a bit more slowly and carefully to avoid these kinds of errors.

tgv 5y ago
Even then, if it had been executed flawlessly, you can only expect that search term frequency and violence remain correlated ceteris paribus, i.e., when all else is the same. The two time periods are clearly different, so how the hell did it even get published?
wumpus 5y ago
The number of hits reported by web-scale engines like Google and Bing are fabrications. When I was the CTO of a full-web search engine startup, I'd get an API access request every couple of months by someone wanting to use hit counts for an academic purpose. I would always them no and why; often they wanted to argue with me. http://searchengineland.com/why-google-cant-count-results-properly-53559 http://searchengineland.com/why-google-cant-count-results-pr... http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/google-result-counts-are-a-meaningless-metric.html http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/goo...
- azinman2 5y ago
  This was the case when I was at Google in 2008; I’m sure it’s even further fabricated now.
  - throwaway3699 5y ago
    Can you elaborate on what you saw?
    - azinman2 5y ago
      That it’s not based on reality at all. That would likely be far to expensive computationally for any arbitrary query to give an accurate number that isn’t particularly valuable outside of degrees of orders of magnitude, and even that is almost never looked at.
  - Shish2k 5y ago
    I’m curious, what are they based on, and is there any reason to have a number there beyond “people expect a number”?
    - azinman2 5y ago
      It’s basically a random number. Don’t extract any meaning from it.
- lakis 5y ago
  I've worked to Inktomi/Yahoo search ~2000-~2015. Any query that is accurate across the whole index is going to be very very expensive. In order to return fast results, there are caches and precomputed results in almost every level of a web search query. But an accurate count implies that you will bypass all the caches and count every single document for every single query term. That's very very expensive in both CPU and memory. We had special internal keywords that disabled caches and produced accurate results. All of them came with special warnings that querying too fast with these special options could bring down the whole search engine. A constant worry was the new employee that knew very little, learnt these keywords and then ops were dealing with crashes across ten of thousands of machines due to out of memory. As to the fabrication. The number are not random numbers. They are estimates based on what we think is the best approximation of the real numbers. We had done internal research to compute the best approximation function. Also, it's very difficult to figure out what is real and what is fake from the outside. I was involved in https://www.nytimes.com/2005/08/10/business/worldbusiness/yahoo-says-its-search-index-is-bigger-than-googles.html https://www.nytimes.com/2005/08/10/business/worldbusiness/ya... Yahoo was 100% correct on their claims. I run the queries myself. But external researchers could not verify the claims because all external queries were hitting caches. So they were trying unique queries in order to estimate the index size which end up having a lot of problems.
croes 5y ago
By that logic, every site citing the paper makes the situation worse.

tryonenow 5y ago

>How did this article get through peer review? Like the author, the journal’s reviewers and editors seemed to have been glamoured by the shine, tech fetishism, and naive empiricism of even the most poorly executed digital methods — without the methodological humility to work together with colleagues from information science, or at least check in with someone familiar with the basic workings of tools like Google Well, that's what happens when your institution is brimming with ideologues who practice one sided research and immediately praise any results that confirms their political, dogmatic biases. Doubly so when criticizing certain results or topics will get you implicitly or ex-communicated, particularly if you are not part of an approved protected class. The retraction doesn't matter very much, the damage has already been done, and far more eyes will have been exposed to the results than to the retraction.

ezequiel-garzon 5y ago

Regardless of the subject matter, I think it’s good policy to do what arXiv does with retractions: they become a new, empty version, alongside a corresponding explanation, but all previous versions remain on the record.

ec109685 5y ago

I watched this happen recently in real time. This article interpreted a Flurry study wrong and was featured on Tech Meme (its headline has since been softened): https://www.macrumors.com/2021/05/07/most-iphone-users-app-tracking-opt-out/ https://www.macrumors.com/2021/05/07/most-iphone-users-app-t... – "Analytics Suggest 96% of Users Leave App Tracking Disabled in iOS 14.5" Then the game of telephone began as bloggers simply rewrote the article and added their own spin: https://www.techmeme.com/210507/p14#a210507p14 https://www.techmeme.com/210507/p14#a210507p14 - Ars: 96% of US users opt out of app tracking in iOS 14.5, analytics find - iMore: 96% of iPhone users have opted out of app tracking since iOS 14.5 launched - Wccftech: Analytics Reveal 96 Percent of Users Have Disabled iOS 14.5's App Tracking on Their iPhone - Ubergizmo: 96% Of iOS Users Have Opted Out Of App Tracking - … Flurry since updated their study to make it clear what it was measuring and found opt out in rate to be about 24% for prompted users (not the 4% cited): https://www.flurry.com/blog/ios-14-5-opt-in-rate-att-restricted-app-tracking-transparency-worldwide-us-daily-latest-update/ https://www.flurry.com/blog/ios-14-5-opt-in-rate-att-restric... But none of those articles are interested in correction and the initial visceral reaction and herumphing tweets have all since occurred. Hundreds of thousands of people got the wrong idea. What it shows it that there is very little original reporting, people don't ask study authors for clarification, and misinterpreting of data happens all too often.

doggodaddo78 5y ago

Men are bad, even though we need them for homo sapiens sapiens to continue existing without cloning. Let's have a study and write a paper to find that conclusion! We don't need no men! Yeah! Oh wait, we used thin correlation not even matched in time instead measuring causes, effects, and search terms. We're going to have to retract that one. ): But we published! :) Damnit! Find real evidence, or create your own, like honey traps at tech conferences! Okay, let's try those!

incrudible 5y ago
The "we need males to continue the species"-argument isn't a very good one to keep men around. Very few men can impregnate an enormous amount of women. You might want keep around a good selection in your breeding zoo, for genetic variance. However, you won't need >50% as nature dictates.
- throwaway3699 5y ago
  Genetic diversity is critical for the long-term health of a species.
  - doggodaddo78 5y ago
    Exactly. Either mass polyandry or mass polygyny like a few Moulay Ismail Ibn Sharifs / the Dr. Strangelove suggestion are bad ideas that lead to de facto near incest in subsequent generations. Cloning is also a terrible idea because it freezes evolution to changing conditions and creates an existential risk like crop monoculture that could wipe-out the entire species with one pathogen.

croes 5y ago

QED https://news.ycombinator.com/item?id=27241715 https://news.ycombinator.com/item?id=27241715

alecco 5y ago
I did not see that yesterday. There should be some indication when something got punished in rankings by flagging but not [flagged] altogether. @dang?