7 ms·

Thanks Feross for creating webtorrent. The biggest problem I see with this and also ipfs is that the latency is up to 30seconds even for files that are distribu

Thanks Feross for creating webtorrent. The biggest problem I see with this and also ipfs is that the latency is up to 30seconds even for files that are distributes over many clients. Is there any work in progress to reduce the latency?

diegorbaquero 8y ago
Maintainer of WebTorrent here: the problem is on how WebRTC works and its various settings. Using a good STUN servers with your own tracker server should be the best solution. Using TURN servers also helps peers connect, but then it's no longer real P2P. Current work is being dedicated to updating the code base to ES6 and soon to ES7, which will improve many things and force recent versions of Node/Browsers to be used. See https://github.com/webtorrent/bittorrent-tracker/issues/199 https://github.com/webtorrent/bittorrent-tracker/issues/199
- marknadal 8y ago
  Why would you take perfectly good code that works across many browsers and force breaking changes onto existing NodeJS/Browsers? WebRTC Condit, STUN, TURN, etc. do not require advanced language features.
  - diegorbaquero 8y ago
    Most, if not all, ES6 features are supported where WebRTC is supported, and it has great benefits for both users and developers. async/await and other ES7 have been supported for 14 versions of chrome now, but that change will come later this year, so we'll be supporting 95%+ of the market. Anyone can always use webtorrent as a module and transpile it.
GistNoesis 8y ago
From the theoretical point of view we can get huge improvements both in terms of torrent availability and latency, by making use of "fountain codes". Basically it is a family of error correcting code which splits the data in N+M distinct pieces, and allow you to reconstruct the data once you have any distinct N of these pieces. This allow you to seed any one of these small pieces yet allow the reconstruction of the whole data provided there are enough other people seeding other pieces (Typically number of "seeds" will increase between x100 and x1000 (Note that it obviously don't increase the outgoing bandwidth of your network, but it makes it a lot easier to conserve and distribute rare files). Which also means any of these pieces can help getting the first packets needed to begin streaming which reduces latency. The negative aspects, are : - bigger "torrent" files (because you need to store the hashes of more pieces). - Some time you'll download an extra piece. - More compute power needed on the clients to decode the stream. - More complexity in the code. Why is it not implemented ? Mainly because it is not a standard yet. There is quite a lot of freedom in the ways to implement it, and it will probably fracture the pools with plenty of non compatible protocols. And if you have a new version of the protocol to improve some things you need the seeds to re-encode the data to be protocol compatible. That is also probably a subject which is encumbered by software patents. Economically targeting the fat tail of rare torrents is probably not worth it. It's fun algebra you should check if you don't know it already :)
- elcritch 8y ago
  It’s pretty straightforward math! I did an M:N implementation in Elixir via on shamir’s key sharing [1]. As you say, there’s no real standard for how to pack the hash bytes for even basic Shamir’s and so I followed Hashicorp Vault’s impl and tested it to their test data. Multiplexing an entire file over this wouldn’t be too hard, but the index/hash table would be need a good protocol. Wasn’t there some cloud storage systems tried using M:N at one time.. Plus the recombination computation can get expensive over lots of data. 1: https://github.com/elcritch/keyx https://github.com/elcritch/keyx
- eximius 8y ago
  Decentralized storage is already barely economical compared to centralized storage, if it makes sense at all. While this might reduce latency, it sounds like in practice it'd multiple the redundancy of the system and make it even less cost effective?
  - GistNoesis 8y ago
    Quite the opposite (it's like a RAID storage system), because of the error correcting scheme, you need a lot less storage to provide a good level of redundancy. You only need to store pieces of size 1/N * file size, and as long as you can get N of those pieces you can reconstruct the file. To guarantee a high level of availability, you only need that there are more than N hosts up containing distinct pieces. So typically you will use (~3 * size of file) of storage on your network, instead of (number of seeds * size of file) bits, and still provide with the same level of service as if number of seeds where seeding the whole file. So each node is able to store a lot more pieces of other files.
    - eximius 8y ago
      I might need to look up how they work because that's basically what my next question is: how do the chunks overlap and how does that affect the probability of a file being unrecoverable? If I have a stupid simple scheme where I split a file into 100 pieces and replicate each piece 3 times, I have a 100C1 * p^3 chance of irrecoverability where p is the independent probability of pieces dropping out. I guess I just want to know, given arbitrary p, what the probability of the file being gone is.

diegorbaquero 8y ago

Maintainer of WebTorrent here: the problem is on how WebRTC works and its various settings. Using a good STUN servers with your own tracker server should be the best solution. Using TURN servers also helps peers connect, but then it's no longer real P2P. Current work is being dedicated to updating the code base to ES6 and soon to ES7, which will improve many things and force recent versions of Node/Browsers to be used. See https://github.com/webtorrent/bittorrent-tracker/issues/199 https://github.com/webtorrent/bittorrent-tracker/issues/199

marknadal 8y ago
Why would you take perfectly good code that works across many browsers and force breaking changes onto existing NodeJS/Browsers? WebRTC Condit, STUN, TURN, etc. do not require advanced language features.
- diegorbaquero 8y ago
  Most, if not all, ES6 features are supported where WebRTC is supported, and it has great benefits for both users and developers. async/await and other ES7 have been supported for 14 versions of chrome now, but that change will come later this year, so we'll be supporting 95%+ of the market. Anyone can always use webtorrent as a module and transpile it.

GistNoesis 8y ago

From the theoretical point of view we can get huge improvements both in terms of torrent availability and latency, by making use of "fountain codes". Basically it is a family of error correcting code which splits the data in N+M distinct pieces, and allow you to reconstruct the data once you have any distinct N of these pieces. This allow you to seed any one of these small pieces yet allow the reconstruction of the whole data provided there are enough other people seeding other pieces (Typically number of "seeds" will increase between x100 and x1000 (Note that it obviously don't increase the outgoing bandwidth of your network, but it makes it a lot easier to conserve and distribute rare files). Which also means any of these pieces can help getting the first packets needed to begin streaming which reduces latency. The negative aspects, are : - bigger "torrent" files (because you need to store the hashes of more pieces). - Some time you'll download an extra piece. - More compute power needed on the clients to decode the stream. - More complexity in the code. Why is it not implemented ? Mainly because it is not a standard yet. There is quite a lot of freedom in the ways to implement it, and it will probably fracture the pools with plenty of non compatible protocols. And if you have a new version of the protocol to improve some things you need the seeds to re-encode the data to be protocol compatible. That is also probably a subject which is encumbered by software patents. Economically targeting the fat tail of rare torrents is probably not worth it. It's fun algebra you should check if you don't know it already :)

elcritch 8y ago
It’s pretty straightforward math! I did an M:N implementation in Elixir via on shamir’s key sharing [1]. As you say, there’s no real standard for how to pack the hash bytes for even basic Shamir’s and so I followed Hashicorp Vault’s impl and tested it to their test data. Multiplexing an entire file over this wouldn’t be too hard, but the index/hash table would be need a good protocol. Wasn’t there some cloud storage systems tried using M:N at one time.. Plus the recombination computation can get expensive over lots of data. 1: https://github.com/elcritch/keyx https://github.com/elcritch/keyx
eximius 8y ago
Decentralized storage is already barely economical compared to centralized storage, if it makes sense at all. While this might reduce latency, it sounds like in practice it'd multiple the redundancy of the system and make it even less cost effective?
- GistNoesis 8y ago
  Quite the opposite (it's like a RAID storage system), because of the error correcting scheme, you need a lot less storage to provide a good level of redundancy. You only need to store pieces of size 1/N * file size, and as long as you can get N of those pieces you can reconstruct the file. To guarantee a high level of availability, you only need that there are more than N hosts up containing distinct pieces. So typically you will use (~3 * size of file) of storage on your network, instead of (number of seeds * size of file) bits, and still provide with the same level of service as if number of seeds where seeding the whole file. So each node is able to store a lot more pieces of other files.
  - eximius 8y ago
    I might need to look up how they work because that's basically what my next question is: how do the chunks overlap and how does that affect the probability of a file being unrecoverable? If I have a stupid simple scheme where I split a file into 100 pieces and replicate each piece 3 times, I have a 100C1 * p^3 chance of irrecoverability where p is the independent probability of pieces dropping out. I guess I just want to know, given arbitrary p, what the probability of the file being gone is.