11 ms·

Maybe Stackoverflow finally started to aggressively block everything that might "crawl" their data. I'm super tired of all the paraphrased/translated copies in

by chaoz_ 5y ago

Maybe Stackoverflow finally started to aggressively block everything that might "crawl" their data. I'm super tired of all the paraphrased/translated copies in Google results.

1337shadow 5y ago
Which don't even link to original contents to look like their content is actually original... Do something about it google please!
petercooper 5y ago
PSA: Anyone who needs a large amount of Stack Overflow data for whatever (legal) reason, they share dumps at https://archive.org/details/stackexchange https://archive.org/details/stackexchange – and perhaps elsewhere.
- fao_ 5y ago
  The license is CC-BY-SA 4.0, so the sites that host copies of their data verbatim are (arguably) legal.
  - peschkaj 5y ago
    Yup! Mostly legal. We maintain a legal guidance page[1] covering what you can and can't do with the data. I assume our lawyers have been involved, but what do I know? [1]: https://stackoverflow.com/legal/trademark-guidance https://stackoverflow.com/legal/trademark-guidance
  - zozbot234 5y ago
    It depends. The "attribution" requirements can be a bit onerous at times, in ways that might put the most casual and opportunistic reusers of even CC-BY-SA data at some legal risk. (It's of course easier for a legitimate reuser with an actual stake in the data to dot the i's and cross the t's.)
southerntofu 5y ago
The problem is with Google not with StackOverflow letting people read their stuff. A search engine which does not prioritize pages with advertisements and broken Google-powered translations (yes the french version of those SO/GH sites are hilarious) will filter that crap out. I mean anyone who'd like to crawl/scrap a website can do it from home, with the help of friends, or by renting "resential VPN" services for a few bucks a month. Blocking Tor does not prevent scrapping.
- chaoz_ 5y ago
  You will be shocked how many paraphrasing enthusiasts will not finish their implementations, if the first steps require additional expertise / investment. Google should definitely resolve this problem on their side, but language models got huge and complex. It's hard to tell (especially with their scale) if a website if composed from paraphrased content of 20 other websites. I'm kinda shocked how successful those "doorway pages" are at reaching the top and how long they live there.
technion 5y ago
I don't understand how these crawlers even work. Last week I published a PoC for CVE-2022-0316. The next day you Google that CVE, and a whole bunch of foreign language sites with an exact copy of my blog would show up. My own original content wasn't on the first two pages. I hadn't linked it anywhere at all and Google Analytics shows literally 0 viewers. Yet they found it, scraped it, and had winning SEO. Seems fixed today.
- spupe 5y ago
  That's pretty crazy. I think Google is currently losing vs these SEO exploiters, at least for some domains such as programming. The fact things switched within a week suggests they are trying to finetune their algorithm to make it less exploitable again.
- is_true 5y ago
  Your page got "fixed" because probably got some backlinks.
- Godel_unicode 5y ago
  > ...Google Analytics shows literally 0 viewers. This at least makes sense, why run the JavaScript in your scraper? It's easy to identify. What do your webserver access logs say?
  - technion 5y ago
    Oh I get that bit, I just meant that was surprised they were able to find it basically immediately. It's not like my blog is high traffic.
- redwall_hp 5y ago
  Link farming, probably. If you have a network of scraper sites already, you inject links to the posts into other posts. That's one of the motivations behind comment spam as well: it builds links pointing to a site, whether people look at the spam or not. I don't know how much Ye Olde PageRank still factors into Google results, but I'm assuming it's still a significant factor.
bttger 5y ago
In my opinion this is even worse when searching for GitHub issues on Google. I prefer searching for issues on Google but since last year there are so many results of websites that show nothing else than crawled GitHub issues. No idea why they are higher ranked than the original GitHub issues. These results don't link to the original issue on GitHub, are often outdated, have bad UX (e.g. the code snippets), and don't allow participating in the discussion.
- PhantomGremlin 5y ago
  No idea why they are higher ranked than the original GitHub issues. Not to be too cynical about it, but who serves the ads on those sites? If it's Google Ads, then I think you have your answer as to why they are higher ranked.
  - nomilk 5y ago
    Genuinely curious to know now..
  - onionisafruit 5y ago
    My theory is github is getting down ranked due to performance. When you view an issue on github the page gets built from mysql queries and can be slow for large issues. gitmemory and the others just serve the content they scraped, so they load much faster.
    - stjohnswarts 5y ago
      Seems like they would ignore that "performance" parameter for major information sites like github/reddit/stack/etc
  - HWR_14 5y ago
    Or it could be because Google dislikes MS owning so much developer mindspace.
- tessie4 5y ago
  You can use site: or inurl: (or a combination) to constrain the search to GitHub.
deleted 5y ago
[deleted]
oezi 5y ago
There is a ublock origin list which you can import to filter search results: https://github.com/quenhus/uBlock-Origin-dev-filter https://github.com/quenhus/uBlock-Origin-dev-filter
- imglorp 5y ago
  Thank you. There are 322 SO copycats alone.
  - vient 5y ago
    More like 150 but yeah. SO copycat list is taken from https://github.com/arosh/ublacklist-stackoverflow-translation/ https://github.com/arosh/ublacklist-stackoverflow-translatio...
    - tonyedgecombe 5y ago
      It's a shame Google doesn't import that list.
- stjohnswarts 5y ago
  This needs its own blog/HN entry :)
Zobat 5y ago
The Stackoverflow database is available for download, no need to scrape the site. https://www.brentozar.com/archive/2015/10/how-to-download-the-stack-overflow-database-via-bittorrent/ https://www.brentozar.com/archive/2015/10/how-to-download-th...
- dspillett 5y ago
  Is that dump going to be maintained under the new regime? While they can't redact existing data bumps due to the license those are covered by, they don't have to keep publishing updates. If that happens then scraping becomes required for the copy-spam sites and such. Though that is a bit conspiracy-theory-y and I expect blocking Tor is more an issue of a small but very active number of people using it to create accounts to post anonymous spam or abuse. For that, which can be a very real problem, I would suggest a more fine-grained block: stop people from those addresses creating accounts, or posting from relatively new (or recently inactive) accounts, or perhaps somewhere in the middle: prevent connection from Tor addresses posting at all, but still let people use the site. That way you still block the abuse, but have less impact on others accessing via Tor.
  - peschkaj 5y ago
    We intend to keep publishing data dumps and with the same regularity.
  - tenebrisalietum 5y ago
    I'm a 70K rep contributor to a Stack Exchange site. StackExchange is composed of contributions from the Internet at large. I've appreciated over the years how StackExchange took this stewardship very seriously without hoarding it behind paywalls - the ethical thing to do. One of the reasons I was motivated to contribute is because 10-12 years ago, putting tech questions in Google often led to Experts Exchange, a site that takes contributions from the public and makes you pay to see them. (Really need a StackExchange-type site to take on Pinterest now ...) So ... no, they should not stop publishing updates. People need to bookmark sites and go directly to them instead of relying on an increasingly broken search engine that's past its heyday.
xxs 5y ago
> their data The "data" is under creative commons. It does not belong to them.
flawn 5y ago
I feel this one.
userbinator 5y ago
Ironically, you can probably reach the copies via Tor and get the information you want there instead.

by chaoz_ 5y ago

Maybe Stackoverflow finally started to aggressively block everything that might "crawl" their data. I'm super tired of all the paraphrased/translated copies in Google results.

1337shadow 5y ago

Which don't even link to original contents to look like their content is actually original... Do something about it google please!

petercooper 5y ago

PSA: Anyone who needs a large amount of Stack Overflow data for whatever (legal) reason, they share dumps at https://archive.org/details/stackexchange https://archive.org/details/stackexchange – and perhaps elsewhere.

fao_ 5y ago
The license is CC-BY-SA 4.0, so the sites that host copies of their data verbatim are (arguably) legal.
- peschkaj 5y ago
  Yup! Mostly legal. We maintain a legal guidance page[1] covering what you can and can't do with the data. I assume our lawyers have been involved, but what do I know? [1]: https://stackoverflow.com/legal/trademark-guidance https://stackoverflow.com/legal/trademark-guidance
- zozbot234 5y ago
  It depends. The "attribution" requirements can be a bit onerous at times, in ways that might put the most casual and opportunistic reusers of even CC-BY-SA data at some legal risk. (It's of course easier for a legitimate reuser with an actual stake in the data to dot the i's and cross the t's.)

southerntofu 5y ago

The problem is with Google not with StackOverflow letting people read their stuff. A search engine which does not prioritize pages with advertisements and broken Google-powered translations (yes the french version of those SO/GH sites are hilarious) will filter that crap out. I mean anyone who'd like to crawl/scrap a website can do it from home, with the help of friends, or by renting "resential VPN" services for a few bucks a month. Blocking Tor does not prevent scrapping.

chaoz_ 5y ago
You will be shocked how many paraphrasing enthusiasts will not finish their implementations, if the first steps require additional expertise / investment. Google should definitely resolve this problem on their side, but language models got huge and complex. It's hard to tell (especially with their scale) if a website if composed from paraphrased content of 20 other websites. I'm kinda shocked how successful those "doorway pages" are at reaching the top and how long they live there.

technion 5y ago

I don't understand how these crawlers even work. Last week I published a PoC for CVE-2022-0316. The next day you Google that CVE, and a whole bunch of foreign language sites with an exact copy of my blog would show up. My own original content wasn't on the first two pages. I hadn't linked it anywhere at all and Google Analytics shows literally 0 viewers. Yet they found it, scraped it, and had winning SEO. Seems fixed today.

spupe 5y ago
That's pretty crazy. I think Google is currently losing vs these SEO exploiters, at least for some domains such as programming. The fact things switched within a week suggests they are trying to finetune their algorithm to make it less exploitable again.
is_true 5y ago
Your page got "fixed" because probably got some backlinks.
Godel_unicode 5y ago
> ...Google Analytics shows literally 0 viewers. This at least makes sense, why run the JavaScript in your scraper? It's easy to identify. What do your webserver access logs say?
- technion 5y ago
  Oh I get that bit, I just meant that was surprised they were able to find it basically immediately. It's not like my blog is high traffic.
redwall_hp 5y ago
Link farming, probably. If you have a network of scraper sites already, you inject links to the posts into other posts. That's one of the motivations behind comment spam as well: it builds links pointing to a site, whether people look at the spam or not. I don't know how much Ye Olde PageRank still factors into Google results, but I'm assuming it's still a significant factor.

bttger 5y ago

In my opinion this is even worse when searching for GitHub issues on Google. I prefer searching for issues on Google but since last year there are so many results of websites that show nothing else than crawled GitHub issues. No idea why they are higher ranked than the original GitHub issues. These results don't link to the original issue on GitHub, are often outdated, have bad UX (e.g. the code snippets), and don't allow participating in the discussion.

PhantomGremlin 5y ago
No idea why they are higher ranked than the original GitHub issues. Not to be too cynical about it, but who serves the ads on those sites? If it's Google Ads, then I think you have your answer as to why they are higher ranked.
- nomilk 5y ago
  Genuinely curious to know now..
- onionisafruit 5y ago
  My theory is github is getting down ranked due to performance. When you view an issue on github the page gets built from mysql queries and can be slow for large issues. gitmemory and the others just serve the content they scraped, so they load much faster.
  - stjohnswarts 5y ago
    Seems like they would ignore that "performance" parameter for major information sites like github/reddit/stack/etc
- HWR_14 5y ago
  Or it could be because Google dislikes MS owning so much developer mindspace.
tessie4 5y ago
You can use site: or inurl: (or a combination) to constrain the search to GitHub.

deleted 5y ago

[deleted]

oezi 5y ago

There is a ublock origin list which you can import to filter search results: https://github.com/quenhus/uBlock-Origin-dev-filter https://github.com/quenhus/uBlock-Origin-dev-filter

imglorp 5y ago
Thank you. There are 322 SO copycats alone.
- vient 5y ago
  More like 150 but yeah. SO copycat list is taken from https://github.com/arosh/ublacklist-stackoverflow-translation/ https://github.com/arosh/ublacklist-stackoverflow-translatio...
  - tonyedgecombe 5y ago
    It's a shame Google doesn't import that list.
stjohnswarts 5y ago
This needs its own blog/HN entry :)

Zobat 5y ago

The Stackoverflow database is available for download, no need to scrape the site. https://www.brentozar.com/archive/2015/10/how-to-download-the-stack-overflow-database-via-bittorrent/ https://www.brentozar.com/archive/2015/10/how-to-download-th...

dspillett 5y ago
Is that dump going to be maintained under the new regime? While they can't redact existing data bumps due to the license those are covered by, they don't have to keep publishing updates. If that happens then scraping becomes required for the copy-spam sites and such. Though that is a bit conspiracy-theory-y and I expect blocking Tor is more an issue of a small but very active number of people using it to create accounts to post anonymous spam or abuse. For that, which can be a very real problem, I would suggest a more fine-grained block: stop people from those addresses creating accounts, or posting from relatively new (or recently inactive) accounts, or perhaps somewhere in the middle: prevent connection from Tor addresses posting at all, but still let people use the site. That way you still block the abuse, but have less impact on others accessing via Tor.
- peschkaj 5y ago
  We intend to keep publishing data dumps and with the same regularity.
- tenebrisalietum 5y ago
  I'm a 70K rep contributor to a Stack Exchange site. StackExchange is composed of contributions from the Internet at large. I've appreciated over the years how StackExchange took this stewardship very seriously without hoarding it behind paywalls - the ethical thing to do. One of the reasons I was motivated to contribute is because 10-12 years ago, putting tech questions in Google often led to Experts Exchange, a site that takes contributions from the public and makes you pay to see them. (Really need a StackExchange-type site to take on Pinterest now ...) So ... no, they should not stop publishing updates. People need to bookmark sites and go directly to them instead of relying on an increasingly broken search engine that's past its heyday.

xxs 5y ago

> their data The "data" is under creative commons. It does not belong to them.

flawn 5y ago

I feel this one.

userbinator 5y ago

Ironically, you can probably reach the copies via Tor and get the information you want there instead.