4 ms·

Regarding mwoffliner: Why scrape Wikipedia when you can just download a dump?

• dal 1y ago
  I was thinking the same. It must take much less space in database form than all the html pages.
  • josephg 1y ago
    Its also kind of bad form to scrape a huge website when there's a downloadable dump available. Save yourself, and more importantly wikimedia, a whole lot of bandwidth & CPU cycles.
    • blacksmith_tb 1y ago
      And torrenting the dumps helps distribute them to others as well.
• detaro 1y ago
  If you want to test Mediawiki tooling, wikipedia is good test target, because it uses a lot of the features (unsurprisingly), compared to smaller wikis. (OTOH, the latter often have custom extensions, so it's not quite enough)
  • yreg 1y ago
    Sure, but I understood the parent as saying that the tool primarily serves for scraping Wikipedia.