6 ms·

Remember: there are BitTorrent links that the Wikimedia Foundation gives out of SQL dumps of Wikipedia and the other projects. You can have a copy in case this

Remember: there are BitTorrent links that the Wikimedia Foundation gives out of SQL dumps of Wikipedia and the other projects. You can have a copy in case this happens in your country: https://en.wikipedia.org/wiki/Wikipedia:Database_download#Where_do_I_get_it.3F https://en.wikipedia.org/wiki/Wikipedia:Database_download#Wh... Also, the Kiwix project has a hotspot project that allows you to host ZIM files (dumps of Wikipedia and other CC licensed content, like TED talks and StackOverflow) on a Raspberry Pi, allowing you to share it with others. Setup info here: https://www.kiwix.org/en/downloads/kiwix-hotspot/ https://www.kiwix.org/en/downloads/kiwix-hotspot/

buildbuildbuild 7y ago
Caveat: the last full Kiwix English Wikipedia archive was made in 2018. They could use some help with automating their build process if anyone here has the time.
- voltagex_ 7y ago
  From a cursory glance at the site and source code, it's really hard to see who/what is involved with building an archive. There's automated builds set up for the Pi image itself.
  - pronoiac 7y ago
    Agreed. I can see that other Wikipedia languages are crawled - https://wiki.kiwix.org/wiki/Content_in_all_languages https://wiki.kiwix.org/wiki/Content_in_all_languages shows dozens of updates this week - but the best leads I have involve poking around the openZIM Github org, https://github.com/openzim https://github.com/openzim . There might be a running "zimfarm" somewhere?
    - bshep 7y ago
      Looks like you are right, you may be able to join the farm to help ( i have not tested as I am away from my computer at the moment ) https://github.com/openzim/zimfarm/blob/master/worker/README.md https://github.com/openzim/zimfarm/blob/master/worker/README...
      - voltagex_ 7y ago
        Someone should ask drone.io or packet.net for an Epyc machine to do this.
    - jedieaston 7y ago
      You can build your own ZIMs from any MediaWiki instance using this tool: https://github.com/openzim/mwoffliner https://github.com/openzim/mwoffliner. Maybe it would be worth putting together another zimfarm that is constantly updating.
  - Nemo_bis 7y ago
    The last time I checked, it was more a problem of lacking servers with sufficient resources: https://phabricator.wikimedia.org/T124960 https://phabricator.wikimedia.org/T124960 https://phabricator.wikimedia.org/T219078 https://phabricator.wikimedia.org/T219078 It sure doesn't harm if someone creates their own ZIM files and reports on their results (and/or shares the resulting files).
Metus 7y ago
Assuming that all formats contain the exact same data, i.e. they were generated at the exact same time, which is the (1) most useful for offline viewing (2) most future proof for archival and backup? Is there another, more viable/useful format?
- Nemo_bis 7y ago
  The XML dumps are the most compact and sustainable format in the mid term (let's say decades). https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_... ZIM might be able to survive longer (centuries?) as probably the future will still need some HTML parser, while wikitext parsers or PHP might be long dead, who knows.
  - Metus 7y ago
    Thank you for the answer. So especially for personal use I am better off hoarding the ZIM version, especially considering there is the dedicated Kiwix reader, while I am not aware of a similar tool for the XML dumps.
Avamander 7y ago
I'd actually love to see a fully working IPFS fallback for wikipedia when regular hosting doesn't work. Would it even be possible with ipfs?
- buildbuildbuild 7y ago
  IPFS has a Wikipedia mirror but it is fairly out of date since it is dependent on the Kiwix archive. https://github.com/ipfs/distributed-wikipedia-mirror https://github.com/ipfs/distributed-wikipedia-mirror
- pms 7y ago
  From a comment above: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/ https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m...
ollybritton 7y ago
There's also a read-only IPFS mirror of Wikipedia in English: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/ https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m...
- mpfundstein 7y ago
  I love ipfs. Can this actually be ddos’d as well?
  - smitop 7y ago
    The ipfs.io is just a web-based way to access IPFS, called a gateway. There are a bunch of different gateways. In addition, you can run an IPFS node locally, and then as long as just one node holds the content you're looking for you are looking for you're good. There are also browser extensions to re-write gateway URIs to localhost URIs.
    - alexeldeib 7y ago
      Does this actually answer the question? If there's a node online that means I can reach the content, but would it help with DDOS? Not so sure.
      - nine_k 7y ago
        Ipfs rarely stores 100% of content on one node.
      - whitexn--g28h 7y ago
        A popular IPFS file might be available on thousands of nodes similar to how popular torrents have thousands of seeders. DDoS attack against thousands of servers across multiple countries and networks would be nearly impossible to perform.
        viraptor 7y ago
        It depends on the kind of node storing the data and how many there are. It's likely easier to DDoS 100 people on DSL than a single Wikimedia endpoint.
vonseel 7y ago
Only 85GB uncompressed for current revisions, excluding “user” pages. Not so bad!

buildbuildbuild 7y ago

Caveat: the last full Kiwix English Wikipedia archive was made in 2018. They could use some help with automating their build process if anyone here has the time.

voltagex_ 7y ago
From a cursory glance at the site and source code, it's really hard to see who/what is involved with building an archive. There's automated builds set up for the Pi image itself.
- pronoiac 7y ago
  Agreed. I can see that other Wikipedia languages are crawled - https://wiki.kiwix.org/wiki/Content_in_all_languages https://wiki.kiwix.org/wiki/Content_in_all_languages shows dozens of updates this week - but the best leads I have involve poking around the openZIM Github org, https://github.com/openzim https://github.com/openzim . There might be a running "zimfarm" somewhere?
  - bshep 7y ago
    Looks like you are right, you may be able to join the farm to help ( i have not tested as I am away from my computer at the moment ) https://github.com/openzim/zimfarm/blob/master/worker/README.md https://github.com/openzim/zimfarm/blob/master/worker/README...
    - voltagex_ 7y ago
      Someone should ask drone.io or packet.net for an Epyc machine to do this.
  - jedieaston 7y ago
    You can build your own ZIMs from any MediaWiki instance using this tool: https://github.com/openzim/mwoffliner https://github.com/openzim/mwoffliner. Maybe it would be worth putting together another zimfarm that is constantly updating.
- Nemo_bis 7y ago
  The last time I checked, it was more a problem of lacking servers with sufficient resources: https://phabricator.wikimedia.org/T124960 https://phabricator.wikimedia.org/T124960 https://phabricator.wikimedia.org/T219078 https://phabricator.wikimedia.org/T219078 It sure doesn't harm if someone creates their own ZIM files and reports on their results (and/or shares the resulting files).

Metus 7y ago

Assuming that all formats contain the exact same data, i.e. they were generated at the exact same time, which is the (1) most useful for offline viewing (2) most future proof for archival and backup? Is there another, more viable/useful format?

Nemo_bis 7y ago
The XML dumps are the most compact and sustainable format in the mid term (let's say decades). https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_... ZIM might be able to survive longer (centuries?) as probably the future will still need some HTML parser, while wikitext parsers or PHP might be long dead, who knows.
- Metus 7y ago
  Thank you for the answer. So especially for personal use I am better off hoarding the ZIM version, especially considering there is the dedicated Kiwix reader, while I am not aware of a similar tool for the XML dumps.

Avamander 7y ago

I'd actually love to see a fully working IPFS fallback for wikipedia when regular hosting doesn't work. Would it even be possible with ipfs?

buildbuildbuild 7y ago
IPFS has a Wikipedia mirror but it is fairly out of date since it is dependent on the Kiwix archive. https://github.com/ipfs/distributed-wikipedia-mirror https://github.com/ipfs/distributed-wikipedia-mirror
pms 7y ago
From a comment above: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/ https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m...

ollybritton 7y ago

There's also a read-only IPFS mirror of Wikipedia in English: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/ https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m...

mpfundstein 7y ago
I love ipfs. Can this actually be ddos’d as well?
- smitop 7y ago
  The ipfs.io is just a web-based way to access IPFS, called a gateway. There are a bunch of different gateways. In addition, you can run an IPFS node locally, and then as long as just one node holds the content you're looking for you are looking for you're good. There are also browser extensions to re-write gateway URIs to localhost URIs.
  - alexeldeib 7y ago
    Does this actually answer the question? If there's a node online that means I can reach the content, but would it help with DDOS? Not so sure.
    - nine_k 7y ago
      Ipfs rarely stores 100% of content on one node.
    - whitexn--g28h 7y ago
      A popular IPFS file might be available on thousands of nodes similar to how popular torrents have thousands of seeders. DDoS attack against thousands of servers across multiple countries and networks would be nearly impossible to perform.
      - viraptor 7y ago
        It depends on the kind of node storing the data and how many there are. It's likely easier to DDoS 100 people on DSL than a single Wikimedia endpoint.

vonseel 7y ago

Only 85GB uncompressed for current revisions, excluding “user” pages. Not so bad!