5 ms·

> And what about those conjured celebrities I added as a precaution? All five were all scraped right into Google’s search result pages. It provided undeniable p

by mcintyre1994 6y ago

> And what about those conjured celebrities I added as a precaution? All five were all scraped right into Google’s search result pages. It provided undeniable proof that after being turned down, Google simply went ahead and stole the entire database of content CNW took eight years and over a million dollars to build. I remember a few years ago Google had a big blog post about how they’d injected some fake search results to catch Bing scraping like this. It certainly seemed like they thought it was a bad thing back then.

sidibe 6y ago
Recently one of the lyrics sites had a similar complaint that might be the same thing happening in this case (and maybe it was the same issue for Bing's case too). If you specify that you don't want Google to take snippets, but a third party ignores the robots.txt or meta tags and has a permissive one of its own Google can just scrape it from there. A missing detail here is what did the textbox for the fake celebrities' net worth on Google link to?
- jefftk 6y ago
  Playing around with real celebrities: * https://www.google.com/search?q=larry+david+net+worth https://www.google.com/search?q=larry+david+net+worth -> featured snippet from Wikipedia, showing an estimate * https://www.google.com/search?q=bill+gates+net+worth https://www.google.com/search?q=bill+gates+net+worth -> database style response, showing answer with no link
  - deleted 6y ago
    [deleted]
- giarc 6y ago
  I believe it was Genius that was using rare unicode characters in the lyrics that don't really show up when viewing the page. They were able to see that Google had those same characters in their lyric results and therefore Google was just scraping Genius.
  - sidibe 6y ago
    Ah thanks, that's the case. There's definitely some wrongdoing there but it's murky who is responsible since Google actually got the lyrics from LyricFind who stole it from Genius.
    - giarc 6y ago
      Ironically this news came out today. https://techcrunch.com/2020/08/11/court-dismisses-genius-lawsuit-over-lyrics-scraping-by-google/?guccounter=1&guce_referrer=aHR0cHM6Ly9uZXdzLnljb21iaW5hdG9yLmNvbS8&guce_referrer_sig=AQAAAD7S1U9BrD-DNynCtYcUzcy0r0V73lepj3yOoPlMwYevzR7WsiRHR8MdhjvDKnjwQi230hCfyLn6SoiTNUB9HMbASZcO18uaJQsRhWuHXCOvu3XMaHfa0GGU3dLK-YOSt5cbe4ERnmvMDMLeD-3fOTh2WkEdV4R8XlaGh1wHVPMy https://techcrunch.com/2020/08/11/court-dismisses-genius-law...
strgcmc 6y ago
I worked at Microsoft from 2010-2012, and the crux of the issue there was that, for Microsoft users who had the Bing toolbar installed, the toolbar would track what you were visiting in your browser and use that session/journey information to try and improve the relevancy of Bing results. If a lot of people who searched for "best pancake house" (regardless of what search engine you used) end up visiting ihop.com within the same browser session, well then Bing would want to rank ihop.com higher for that search phrase. Google noticed this, and for unique/low-traffic search terms, was able to synthetically generate enough "fake" traffic that the "fake" traffic became the dominating signal for those terms, and therefore Bing started directing users to the fake results. The bad thing here IMO is the level of tracking of users via this toolbar, but fundamentally this seems as bad as any other digital fingerprinting or advertiser tracking as anything else that's become common on the web. This is not to excuse Microsoft for doing a bad thing, but it really had very little to do with "scraping Google", which somehow became the popular media takeaway for this. EDIT: a decent contemporaneous article in Wired: https://www.wired.com/2011/02/bing-copies-google/ https://www.wired.com/2011/02/bing-copies-google/, mentioning how Microsoft was using the clickstream data from the browser/toolbar, not scraping Google results per se
- whalabi 6y ago
  That's the first I've heard of that side of the story, and I'm pretty appalled because everyone I know believes "Bing scraped Google" - which I think was how Google framed it. I wonder how common misunderstandings like this can be prevented or treated.
- mcintyre1994 6y ago
  Thanks for the correction - I never really heard any followup on this at the time and definitely missed this explanation. Obviously the tracking is bad but it's definitely a different thing to the way it was presented as them actually crawling a competitor.

sidibe 6y ago

Recently one of the lyrics sites had a similar complaint that might be the same thing happening in this case (and maybe it was the same issue for Bing's case too). If you specify that you don't want Google to take snippets, but a third party ignores the robots.txt or meta tags and has a permissive one of its own Google can just scrape it from there. A missing detail here is what did the textbox for the fake celebrities' net worth on Google link to?

jefftk 6y ago
Playing around with real celebrities: * https://www.google.com/search?q=larry+david+net+worth https://www.google.com/search?q=larry+david+net+worth -> featured snippet from Wikipedia, showing an estimate * https://www.google.com/search?q=bill+gates+net+worth https://www.google.com/search?q=bill+gates+net+worth -> database style response, showing answer with no link
- deleted 6y ago
  [deleted]
giarc 6y ago
I believe it was Genius that was using rare unicode characters in the lyrics that don't really show up when viewing the page. They were able to see that Google had those same characters in their lyric results and therefore Google was just scraping Genius.
- sidibe 6y ago
  Ah thanks, that's the case. There's definitely some wrongdoing there but it's murky who is responsible since Google actually got the lyrics from LyricFind who stole it from Genius.
  - giarc 6y ago
    Ironically this news came out today. https://techcrunch.com/2020/08/11/court-dismisses-genius-lawsuit-over-lyrics-scraping-by-google/?guccounter=1&guce_referrer=aHR0cHM6Ly9uZXdzLnljb21iaW5hdG9yLmNvbS8&guce_referrer_sig=AQAAAD7S1U9BrD-DNynCtYcUzcy0r0V73lepj3yOoPlMwYevzR7WsiRHR8MdhjvDKnjwQi230hCfyLn6SoiTNUB9HMbASZcO18uaJQsRhWuHXCOvu3XMaHfa0GGU3dLK-YOSt5cbe4ERnmvMDMLeD-3fOTh2WkEdV4R8XlaGh1wHVPMy https://techcrunch.com/2020/08/11/court-dismisses-genius-law...

strgcmc 6y ago

I worked at Microsoft from 2010-2012, and the crux of the issue there was that, for Microsoft users who had the Bing toolbar installed, the toolbar would track what you were visiting in your browser and use that session/journey information to try and improve the relevancy of Bing results. If a lot of people who searched for "best pancake house" (regardless of what search engine you used) end up visiting ihop.com within the same browser session, well then Bing would want to rank ihop.com higher for that search phrase. Google noticed this, and for unique/low-traffic search terms, was able to synthetically generate enough "fake" traffic that the "fake" traffic became the dominating signal for those terms, and therefore Bing started directing users to the fake results. The bad thing here IMO is the level of tracking of users via this toolbar, but fundamentally this seems as bad as any other digital fingerprinting or advertiser tracking as anything else that's become common on the web. This is not to excuse Microsoft for doing a bad thing, but it really had very little to do with "scraping Google", which somehow became the popular media takeaway for this. EDIT: a decent contemporaneous article in Wired: https://www.wired.com/2011/02/bing-copies-google/ https://www.wired.com/2011/02/bing-copies-google/, mentioning how Microsoft was using the clickstream data from the browser/toolbar, not scraping Google results per se

whalabi 6y ago
That's the first I've heard of that side of the story, and I'm pretty appalled because everyone I know believes "Bing scraped Google" - which I think was how Google framed it. I wonder how common misunderstandings like this can be prevented or treated.
mcintyre1994 6y ago
Thanks for the correction - I never really heard any followup on this at the time and definitely missed this explanation. Obviously the tracking is bad but it's definitely a different thing to the way it was presented as them actually crawling a competitor.