10 ms·

Peerwiki: all of Wikipedia on BitTorrent

88 points by galapago 12y ago

sdfjkl 12y ago
This README is a bit short on telling what it actually does. Anyone?
- CHY872 12y ago
  Pretty sure that it's: When you want to go on a wikipedia article, it requests the file from other peers in the network using the bittorrent protocol. It's a copy of Wikipedia placed on bittorrent, presumably with some semantics for article updates.
  - xnyhps 12y ago
    It looks like it uses a single Wikipedia dump from over a year ago. Would be cool if it supported deltas somehow, so the network doesn't split when a new dump is used.
    - slaman 12y ago
      this breaks the bittorrent.
    - placeybordeaux 12y ago
      Bittorrent doesn't have support for that, but you could publish torrents that just host a full version of all the modified pages.
    - basiliothecat 12y ago
      Found out recently that there's a cool project underway to make a distributed wiki - http://wardcunningham.github.io/ http://wardcunningham.github.io/.
sleepychu 12y ago
Why? Specifically - why would I choose to browse this way over just accessing wikipedia?
- tormeh 12y ago
  You wouldn't ask if you lived in China. Though bittorrent traffic may need to be disguised. Decentralising the internet is generally a great idea.
  - hnyc 12y ago
    Then take a look at Freenet. :P
  - kancer 12y ago
    An interesting point here is that Wikipedia actually works in China, only some articles are censored. I tried to bypass this by typing https://www.wikipedia.org https://www.wikipedia.org into the browser, but it loaded the http version regardless.
- madsravn 12y ago
  I don't think it is so much a question about why. More that is pretty cool idea.
- jekrb 12y ago
  This is just one answer. You know how Wikipedia is always asking for donations (and rightfully so)? Part of those donations go to paying server cost. Well, this cuts the server cost by decentralizing their content.
mafintosh 12y ago
Author here. If anyone is interested I presented this at jsconf.eu last year as part of a BitTorrent talk, https://www.youtube.com/watch?v=BTCsSwCpGP8 https://www.youtube.com/watch?v=BTCsSwCpGP8 (slides: http://mafintosh.github.io/slides/jsconf-2014/jsconf-eu-2014.html http://mafintosh.github.io/slides/jsconf-2014/jsconf-eu-2014...)
- wongarsu 12y ago
  What are the obstacles to running this client-side in the browser? That would lower the barrier to entry for users by a lot. Edit: I just finished watching your talk where you mention that https://github.com/feross/webtorrent https://github.com/feross/webtorrent is already doing that.
freeall 12y ago
The idea is to showcase how a large dataset is pretty good at being shared without having central servers. Here using bittorrent. mafintosh showed how wikipedia could be shared without a central server(s), and instead rely on a network of peers. subtack did something similar, peermaps, which is a showcase on how you can share geo data over bittorrent. Imagine a google maps without a google servers. https://github.com/substack/peermaps https://github.com/substack/peermaps Of course there's many unsolved questions, like "how do you update?", "how do you manage the data?", etc. But the examples are pretty solid.
higherpurpose 12y ago
Is this like Popcorn Time for Wikipedia? I think Bittorrent is also working on a similar project for the whole web, called Maelstrom: http://blog.bittorrent.com/2014/12/10/project-maelstrom-the-internet-we-build-next/ http://blog.bittorrent.com/2014/12/10/project-maelstrom-the-...
- freeall 12y ago
  Closed source?
  - higherpurpose 12y ago
    I think so, yes.
    - freeall 12y ago
      I wouldn't want something like this to be closed source. It's very important that the security of the decentralization can be validated. And in these NSA times it's even more important.
      - wongarsu 12y ago
        If the concept works well, an open source variant is sure to be created precisely for this reason.
  - placeybordeaux 12y ago
    Seems like bittorrent foundation has been doing a lot of closed work source ever since bittorrent has been used across the world and they got very little money out of it. Just my drive by judgement.
belorn 12y ago
Can an observer watch what pages people read?
AndrewDucker 12y ago
Would a blockchain-like solution work here? Where new edits piled up on top of the existing data, constantly sharing it across all servers?
- murbard2 12y ago
  It could but it would be an unnecessary hurdle, because there is no need for a global consensus on a single version of the encyclopedia. Think of it as a git tree, and checkout the branch you like. With a currency, it's imperative that everyone refers to the same branch all the time, not so with an encyclopedia. In practice there would be a few "popular" branches, and one would likely dominate, so that it would be trivial to identify it by relying on a social consensus. Using a blockchain when what you need is a distributed database is overkill.
  - fragmede 12y ago
    It's not imperative that everyone refer to the same branch all the time for currency either. If User A gives Bitcoin to User B, and User C gives Bitcoin to User D, there's no need for them to be on the same branch as the two transactions are unrelated. It's when you need to merge branches back together that things tend to get messy - with currency or an encyclopedia, and everyone being on the same branch all the time is better, for both cases. I can just imagine what the 'popular' branch for an encyclopedia would look like if 4chan set their sites on causing problems for it.
    - murbard2 12y ago
      1) Sure, some operations are commutative and compatible, and you can import them from one branch to another, but in the general case, they aren't 2) Same thing that happens currently on Wikipedia. There's a set of rules that determine what edits are valid and which aren't, and a local client can applies those rules (or not) to determine the HEAD version. There's nothing that prevents a Sybil attack on Wikipedia, but there doesn't seem to be the need for it.
  - wongarsu 12y ago
known 12y ago
Interesting project.
mizzao 12y ago
A Wikipedia hosted in a decentralized manner (i.e. DHT on running computers) that could still be updated in a distributed fashion would really help us maintain that knowledge for the future while not relying Wikimedia's servers to keep running.

88 points by galapago 12y ago

sdfjkl 12y ago

This README is a bit short on telling what it actually does. Anyone?

CHY872 12y ago
Pretty sure that it's: When you want to go on a wikipedia article, it requests the file from other peers in the network using the bittorrent protocol. It's a copy of Wikipedia placed on bittorrent, presumably with some semantics for article updates.
- xnyhps 12y ago
  It looks like it uses a single Wikipedia dump from over a year ago. Would be cool if it supported deltas somehow, so the network doesn't split when a new dump is used.
  - slaman 12y ago
    this breaks the bittorrent.
  - placeybordeaux 12y ago
    Bittorrent doesn't have support for that, but you could publish torrents that just host a full version of all the modified pages.
  - basiliothecat 12y ago
    Found out recently that there's a cool project underway to make a distributed wiki - http://wardcunningham.github.io/ http://wardcunningham.github.io/.

sleepychu 12y ago

Why? Specifically - why would I choose to browse this way over just accessing wikipedia?

tormeh 12y ago
You wouldn't ask if you lived in China. Though bittorrent traffic may need to be disguised. Decentralising the internet is generally a great idea.
- hnyc 12y ago
  Then take a look at Freenet. :P
- kancer 12y ago
  An interesting point here is that Wikipedia actually works in China, only some articles are censored. I tried to bypass this by typing https://www.wikipedia.org https://www.wikipedia.org into the browser, but it loaded the http version regardless.
madsravn 12y ago
I don't think it is so much a question about why. More that is pretty cool idea.
jekrb 12y ago
This is just one answer. You know how Wikipedia is always asking for donations (and rightfully so)? Part of those donations go to paying server cost. Well, this cuts the server cost by decentralizing their content.

mafintosh 12y ago

Author here. If anyone is interested I presented this at jsconf.eu last year as part of a BitTorrent talk, https://www.youtube.com/watch?v=BTCsSwCpGP8 https://www.youtube.com/watch?v=BTCsSwCpGP8 (slides: http://mafintosh.github.io/slides/jsconf-2014/jsconf-eu-2014.html http://mafintosh.github.io/slides/jsconf-2014/jsconf-eu-2014...)

wongarsu 12y ago
What are the obstacles to running this client-side in the browser? That would lower the barrier to entry for users by a lot. Edit: I just finished watching your talk where you mention that https://github.com/feross/webtorrent https://github.com/feross/webtorrent is already doing that.

freeall 12y ago

The idea is to showcase how a large dataset is pretty good at being shared without having central servers. Here using bittorrent. mafintosh showed how wikipedia could be shared without a central server(s), and instead rely on a network of peers. subtack did something similar, peermaps, which is a showcase on how you can share geo data over bittorrent. Imagine a google maps without a google servers. https://github.com/substack/peermaps https://github.com/substack/peermaps Of course there's many unsolved questions, like "how do you update?", "how do you manage the data?", etc. But the examples are pretty solid.

higherpurpose 12y ago

Is this like Popcorn Time for Wikipedia? I think Bittorrent is also working on a similar project for the whole web, called Maelstrom: http://blog.bittorrent.com/2014/12/10/project-maelstrom-the-internet-we-build-next/ http://blog.bittorrent.com/2014/12/10/project-maelstrom-the-...

freeall 12y ago
Closed source?
- higherpurpose 12y ago
  I think so, yes.
  - freeall 12y ago
    I wouldn't want something like this to be closed source. It's very important that the security of the decentralization can be validated. And in these NSA times it's even more important.
    - wongarsu 12y ago
      If the concept works well, an open source variant is sure to be created precisely for this reason.
- placeybordeaux 12y ago
  Seems like bittorrent foundation has been doing a lot of closed work source ever since bittorrent has been used across the world and they got very little money out of it. Just my drive by judgement.

belorn 12y ago

Can an observer watch what pages people read?

AndrewDucker 12y ago

Would a blockchain-like solution work here? Where new edits piled up on top of the existing data, constantly sharing it across all servers?

murbard2 12y ago
It could but it would be an unnecessary hurdle, because there is no need for a global consensus on a single version of the encyclopedia. Think of it as a git tree, and checkout the branch you like. With a currency, it's imperative that everyone refers to the same branch all the time, not so with an encyclopedia. In practice there would be a few "popular" branches, and one would likely dominate, so that it would be trivial to identify it by relying on a social consensus. Using a blockchain when what you need is a distributed database is overkill.
- fragmede 12y ago
  It's not imperative that everyone refer to the same branch all the time for currency either. If User A gives Bitcoin to User B, and User C gives Bitcoin to User D, there's no need for them to be on the same branch as the two transactions are unrelated. It's when you need to merge branches back together that things tend to get messy - with currency or an encyclopedia, and everyone being on the same branch all the time is better, for both cases. I can just imagine what the 'popular' branch for an encyclopedia would look like if 4chan set their sites on causing problems for it.
  - murbard2 12y ago
    1) Sure, some operations are commutative and compatible, and you can import them from one branch to another, but in the general case, they aren't 2) Same thing that happens currently on Wikipedia. There's a set of rules that determine what edits are valid and which aren't, and a local client can applies those rules (or not) to determine the HEAD version. There's nothing that prevents a Sybil attack on Wikipedia, but there doesn't seem to be the need for it.
- wongarsu 12y ago

known 12y ago

Interesting project.

mizzao 12y ago

A Wikipedia hosted in a decentralized manner (i.e. DHT on running computers) that could still be updated in a distributed fashion would really help us maintain that knowledge for the future while not relying Wikimedia's servers to keep running.