4 ms·

Why do AI bots scrape Wikipedia pages instead of downloading the published full database?

nness 1y ago
My guess is that the scraping tools are specialized for web, and creating per-application interfaces isn't cost effective (although you could argue that scraping Wikipedia effectively is definitely worth the effort, but given its all text context with a robust taxonomy/hierarchy, it might be non-issue.) My other thought is that you don't want a link showing you scraped anything... and faking browser traffic might draw less attention.
fzeroracer 1y ago
The rationale I've seen elsewhere is that it saves money. It means you don't need to go to the effort of downloading, storing and updating your copy of the database. You can offload all of the externalities onto whatever site you're scraping.
- danielbln 1y ago
  Man, these companies have bazillions in funding and they can't keep some $100 DB in a closet for that. Smh
  - solarkraft 1y ago
    They could. There’s just no upside in doing so.
    - walterbell 1y ago
      If they destroy the relatively high-trust internet, the low-trust replacement will require digital ID for every client, with non-neutral traffic price varying by {business digital ID, content}. No more free geese, even to check whether there is a golden goose worthy of payment. https://utcc.utoronto.ca/~cks/space/blog/web/WeShouldBlockForSocialReasons?showcomments https://utcc.utoronto.ca/~cks/space/blog/web/WeShouldBlockFo...
SideburnsOfDoom 1y ago
Sheer laziness?
ectospheno 1y ago
Money. One requires you to use your hardware and your developers. The other way doesn’t.
jjtheblunt 1y ago
i tried doing that in summer 2019, and the downloaded formats were at that time proprietary and depended on decoders which were like a tail recursive rabbit hole. in contrast, letting their servers render the content with their proprietary tools yields the sought data, so scraping might be a pragmatic choice still.
NoPicklez 1y ago
Because that would probably require extra work, why do that if it already crawls and scrapes it in the first place
twosdai 1y ago
It's possible that they don't know. I literally didn't know there was a full downloadable db until right now.
- walterbell 1y ago
  Even on offline phones! https://kiwix.org https://kiwix.org

My guess is that the scraping tools are specialized for web, and creating per-application interfaces isn't cost effective (although you could argue that scraping Wikipedia effectively is definitely worth the effort, but given its all text context with a robust taxonomy/hierarchy, it might be non-issue.) My other thought is that you don't want a link showing you scraped anything... and faking browser traffic might draw less attention.

fzeroracer 1y ago

The rationale I've seen elsewhere is that it saves money. It means you don't need to go to the effort of downloading, storing and updating your copy of the database. You can offload all of the externalities onto whatever site you're scraping.

danielbln 1y ago
Man, these companies have bazillions in funding and they can't keep some $100 DB in a closet for that. Smh
- solarkraft 1y ago
  They could. There’s just no upside in doing so.
  - walterbell 1y ago
    If they destroy the relatively high-trust internet, the low-trust replacement will require digital ID for every client, with non-neutral traffic price varying by {business digital ID, content}. No more free geese, even to check whether there is a golden goose worthy of payment. https://utcc.utoronto.ca/~cks/space/blog/web/WeShouldBlockForSocialReasons?showcomments https://utcc.utoronto.ca/~cks/space/blog/web/WeShouldBlockFo...

SideburnsOfDoom 1y ago

Sheer laziness?

ectospheno 1y ago

Money. One requires you to use your hardware and your developers. The other way doesn’t.

jjtheblunt 1y ago

i tried doing that in summer 2019, and the downloaded formats were at that time proprietary and depended on decoders which were like a tail recursive rabbit hole. in contrast, letting their servers render the content with their proprietary tools yields the sought data, so scraping might be a pragmatic choice still.

NoPicklez 1y ago

Because that would probably require extra work, why do that if it already crawls and scrapes it in the first place

twosdai 1y ago

It's possible that they don't know. I literally didn't know there was a full downloadable db until right now.

walterbell 1y ago
Even on offline phones! https://kiwix.org https://kiwix.org