8 ms·

Serverless Sync in Web Apps Using Bit Torrent

256 points by kinlan 10y ago

martinald 10y ago
Very cool. I imagine when bittorrent first starting how cool this would be to do if we had something like webtorrent Two problems: 1) WebTorrent isn't very persistent. It requires people keep the tab open and I don't think the current WebTorrent to existing torrent clients federation really solves anything interesting. Unless you've got high volumes of traffic it doesn't help much. 2) Bandwidth and storage got so cheap. S3 et al is really pricey but you can get insane deals on colo and bare mental boxes these days. Ok, you're not going to get s3 ease of use and reliability, but webtorrent is going to be way worse on both those points.
- niftich 10y ago
  S3 can serve over the Bittorrent protocol though. In fact, there are several viable configurations to bridge between WebTorrent, Bittorrent, and S3. For example, you can have S3 directly serve over Bittorrent and use Amazon as a tracker; or you can have the HTTP URL to S3 serve as a BEP19 webseed, which is now supported in recent WebTorrent.
  - kinlan 10y ago
    /me runs to learn more.
    - diegorbaquero 10y ago
      Check this: https://medium.com/@diegorbaquero/how-to-create-a-swarm-cdn-for-free-with-webtorrent-bfa09d193f71 https://medium.com/@diegorbaquero/how-to-create-a-swarm-cdn-...
      - kinlan 10y ago
        thanks!
- icebraining 10y ago
  It might make sense in situation where you can keep a quorum, ie., when the assets are common for most users (unlike the example given of the recorder) and where users are likely to keep the tab open for some time. A good example might be distribution of assets for a browser game. Colo and bare metal bandwidth might be cheap, but they require knowledge setting up a server and keeping it running, which many programmers can't do reliably.
michaelbuckbee 10y ago
So I only kind of understand magnet links. But it seems like I could create a Chrome plugin that took every page I visited and turned the assets into magnet links and started request/seeding them on pageload. Is that analogous to what they're doing?
- kinlan 10y ago
  You need access to the Blob data to seed that. You could get that by making HTTP requests to each of the resources on the page and then to seed that (however, as soon as you navigate away from the page there are no more seeders so the data is no longer available)
  - michaelbuckbee 10y ago
    Well plugins are powerful things - I bet you could cache blobs in the browser for a bit as long as it was still open.
- tjohns 10y ago
  I think you just described IPFS (http://ipfs.io http://ipfs.io), actually. :)
kodablah 10y ago
Neat use of web torrent. Now if only we could do it with live data instead of persistent data, have built in encryption, and anonymize the peers we'd have the perfect decentralized internet :-)
- bytefactory 10y ago
  You mean something like this? :) https://zeronet.io/ https://zeronet.io/
  - kodablah 10y ago
    Where is the anonymity? It is unreasonable to use TOR in my opinion because despite being only a few exit nodes, it's not truly P2P then (and therefore can be slow). Tribler tried something similar. Someone needs to build an onion router of peers without defined relays or exit nodes (think of it as everyone-is-a-relay TOR, it's one of my future ideas[0]). This will speed things up substantially and can definitely work when you don't have the limitations of TOR like working w/ all TCP connections (e.g. HTTP). Maybe they could make every ZeroNet node a hidden service, I dunno. Also, I have researched a lot about plausible deniability of publicly available yet distributed data. I have never seen a system that stores public data without them (clarity edit: node runners) being able to find out whether they are storing a piece of something they don't like. I started a thread about it on the Maidsafe forum recently [1] 0 - https://github.com/cretz/software-ideas/issues/2 https://github.com/cretz/software-ideas/issues/2 1 - https://forum.safenetwork.io/t/unencrypted-data-question/9698 https://forum.safenetwork.io/t/unencrypted-data-question/969...
    - dublinben 10y ago
      From what you've described, it seems like i2p would satisfy the requirements of your first paragraph and Freenet would satisfy the requirements of the second. Both of these networks have been around for over a decade, so they're not as in vogue as newer projects.
      - kodablah 10y ago
moosingin3space 10y ago
This is really cool. Wonder how hard it would be to develop double-ratcheted multi-user chat or a distributed game using WebTorrent.
zeveb 10y ago
This sounds really cool, but it also sounds like a really easy way for malware (delivered, say, in a drive-by malicious ad) to exfiltrate data from a browser.
- kinlan 10y ago
  In this case, the content is only stored local to the domain - sure someone could put the data that is some sort of Malware as the file that is synced, but they could also HTTP request it too... The theory at least is that the sandbox of the browser will stop it from getting out from there and keep the user safe.
  - kinlan 10y ago
    Worth thinking about more though!
_bdog 10y ago
Now someone could make a Chrome plugin for YouTube, which seamlessly seeds the watched videos and, when it goes offline, loads from Bittorrent.
- anonbanker 10y ago
  Or when the site gets a DMCA takedown request. Or when content is blocked in your country. There seriously needs to be a Youtube killer, and this would be a good starting point.
stirner 10y ago
Why not use IPFS, which is specifically designed for this sort of usage and will be more resilient?
- kinlan 10y ago
  I didn't know about it. I chose webtorrent because I potentially also wanted to have content created from the app to be long term on the torrent network if people are seeding the content.
  - niftich 10y ago
    There was a discussion about the world's oldest torrent, where others and I mused about solutions to the problem of ephemeral seeds: https://news.ycombinator.com/item?id=10962305 https://news.ycombinator.com/item?id=10962305 I suggested that one could maybe bridge Bittorrent and IPFS, then someone suggested that IPFS would be ideal for being the 'canonical' source of the file while the other protocols (HTTP, BT, WT) are merely ways to access the file.
    - toomuchtodo 10y ago
      > then someone suggested that IPFS would be ideal for being the 'canonical' source of the file while the other protocols (HTTP, BT, WT) are merely ways to access the file. Hey! That was me! I still believe its a great idea ;)
    - greglindahl 10y ago
      The Internet Archive seeds all of our items' torrents since 2012. IPFS is a more straightforward way for us to be sure we've got independent seeds of all of our content. Got 25 petabytes of disk?
      - niftich 10y ago
        Uhhh, 20 GB free... I'm just a third-party, but if you were to get in touch with the IPFS team to host a limited amount of IA content as a proof-of-concept, I'm sure it would be a great publicity for both of your organizations.
mdjeu 10y ago
Maybe this a bit tangential, but I was just thinking lately how it would be nice to use some decentralized communication backend to word processing software to enable shared collaborative editing of documents. Sort of like Google Docs, but without a centralized server.
chj 10y ago
It's great to share data, however, how can a torrent solution support services like web API?
- rictic 10y ago
  (This is a good question, but as asked it's so broad that it's akin to asking "What kinds of things does P2P work for?" and we're still discovering more answers to it every year.)
- bbcbasic 10y ago
  I think these services would compliment a torrent solution, e.g. in a video hosting site web-api gives you the metadata (and perhaps the first 5 buffered seconds of the video), and torrent gives you the video.
- geocar 10y ago
  Well, one way I did it was by creating fake torrents with an unlikely info_hash (if you see the all-0xaa's hash on bittorrent crashing your client a few weeks ago, that's me). In this way, I can get use public WebTorrent signalling servers to establish the WebRTC connections between my (non-bittorrent) clients. You can distinguish between clients and servers (if you don't want a P2P model) by having clients claim to need this unlikely info_hash, and servers seed it. In this way, WebTorrent's websockets server won't link clients to other clients.
chj 10y ago
It's great to share data, however, how can a torrent solution support services like web API?
235337 10y ago
Anyone have suggestions on how to block sites like this from downloading without permissions? edit: So blocking XHR requests seems to do the trick.
solidr53 10y ago
I was surprised to find out that the torrent fetching part of the prototype didn't happen in the service worker . Which is obviously the most cool place for it.
pmlnr 10y ago
If there wasn't a 'cloud' in the picture how seeding is done I'd even believe it's serverless. Sorry for being a cynic, but what if there is no tracker available?
- kinlan 10y ago
  It's a very fair point, I chose the word serverless to imply no traditional back-end. One extension that I am looking forward to seeing is the use of DHT in to the WebTorrent that (at least in my head) can eliminate the need for trackers. The important point in my head is that I can distribute audio files to anyone who is seeding client-to-client rather than pass through a traditional server that I would host.
visarga 10y ago
The app is quite interesting and I was amazed until I realized the play button doesn't display the circular progress indicator on time. It has an ugly lag. This is why web apps are not quite there yet. And yet it is amazing they can record, store and sync audio like that. Almost like real apps.
- kinlan 10y ago
  That is me being lazy and showing a demo/concept experience, not because the web platform can't do this.

256 points by kinlan 10y ago

Very cool. I imagine when bittorrent first starting how cool this would be to do if we had something like webtorrent Two problems: 1) WebTorrent isn't very persistent. It requires people keep the tab open and I don't think the current WebTorrent to existing torrent clients federation really solves anything interesting. Unless you've got high volumes of traffic it doesn't help much. 2) Bandwidth and storage got so cheap. S3 et al is really pricey but you can get insane deals on colo and bare mental boxes these days. Ok, you're not going to get s3 ease of use and reliability, but webtorrent is going to be way worse on both those points.

niftich 10y ago
S3 can serve over the Bittorrent protocol though. In fact, there are several viable configurations to bridge between WebTorrent, Bittorrent, and S3. For example, you can have S3 directly serve over Bittorrent and use Amazon as a tracker; or you can have the HTTP URL to S3 serve as a BEP19 webseed, which is now supported in recent WebTorrent.
- kinlan 10y ago
  /me runs to learn more.
  - diegorbaquero 10y ago
    Check this: https://medium.com/@diegorbaquero/how-to-create-a-swarm-cdn-for-free-with-webtorrent-bfa09d193f71 https://medium.com/@diegorbaquero/how-to-create-a-swarm-cdn-...
    - kinlan 10y ago
      thanks!
icebraining 10y ago
It might make sense in situation where you can keep a quorum, ie., when the assets are common for most users (unlike the example given of the recorder) and where users are likely to keep the tab open for some time. A good example might be distribution of assets for a browser game. Colo and bare metal bandwidth might be cheap, but they require knowledge setting up a server and keeping it running, which many programmers can't do reliably.

michaelbuckbee 10y ago

So I only kind of understand magnet links. But it seems like I could create a Chrome plugin that took every page I visited and turned the assets into magnet links and started request/seeding them on pageload. Is that analogous to what they're doing?

kinlan 10y ago
You need access to the Blob data to seed that. You could get that by making HTTP requests to each of the resources on the page and then to seed that (however, as soon as you navigate away from the page there are no more seeders so the data is no longer available)
- michaelbuckbee 10y ago
  Well plugins are powerful things - I bet you could cache blobs in the browser for a bit as long as it was still open.
tjohns 10y ago
I think you just described IPFS (http://ipfs.io http://ipfs.io), actually. :)

kodablah 10y ago

Neat use of web torrent. Now if only we could do it with live data instead of persistent data, have built in encryption, and anonymize the peers we'd have the perfect decentralized internet :-)

bytefactory 10y ago
You mean something like this? :) https://zeronet.io/ https://zeronet.io/
- kodablah 10y ago
  Where is the anonymity? It is unreasonable to use TOR in my opinion because despite being only a few exit nodes, it's not truly P2P then (and therefore can be slow). Tribler tried something similar. Someone needs to build an onion router of peers without defined relays or exit nodes (think of it as everyone-is-a-relay TOR, it's one of my future ideas[0]). This will speed things up substantially and can definitely work when you don't have the limitations of TOR like working w/ all TCP connections (e.g. HTTP). Maybe they could make every ZeroNet node a hidden service, I dunno. Also, I have researched a lot about plausible deniability of publicly available yet distributed data. I have never seen a system that stores public data without them (clarity edit: node runners) being able to find out whether they are storing a piece of something they don't like. I started a thread about it on the Maidsafe forum recently [1] 0 - https://github.com/cretz/software-ideas/issues/2 https://github.com/cretz/software-ideas/issues/2 1 - https://forum.safenetwork.io/t/unencrypted-data-question/9698 https://forum.safenetwork.io/t/unencrypted-data-question/969...
  - dublinben 10y ago
    From what you've described, it seems like i2p would satisfy the requirements of your first paragraph and Freenet would satisfy the requirements of the second. Both of these networks have been around for over a decade, so they're not as in vogue as newer projects.
    - kodablah 10y ago

moosingin3space 10y ago

This is really cool. Wonder how hard it would be to develop double-ratcheted multi-user chat or a distributed game using WebTorrent.

zeveb 10y ago

This sounds really cool, but it also sounds like a really easy way for malware (delivered, say, in a drive-by malicious ad) to exfiltrate data from a browser.

kinlan 10y ago
In this case, the content is only stored local to the domain - sure someone could put the data that is some sort of Malware as the file that is synced, but they could also HTTP request it too... The theory at least is that the sandbox of the browser will stop it from getting out from there and keep the user safe.
- kinlan 10y ago
  Worth thinking about more though!

_bdog 10y ago

Now someone could make a Chrome plugin for YouTube, which seamlessly seeds the watched videos and, when it goes offline, loads from Bittorrent.

anonbanker 10y ago
Or when the site gets a DMCA takedown request. Or when content is blocked in your country. There seriously needs to be a Youtube killer, and this would be a good starting point.

stirner 10y ago

Why not use IPFS, which is specifically designed for this sort of usage and will be more resilient?

kinlan 10y ago
I didn't know about it. I chose webtorrent because I potentially also wanted to have content created from the app to be long term on the torrent network if people are seeding the content.
- niftich 10y ago
  There was a discussion about the world's oldest torrent, where others and I mused about solutions to the problem of ephemeral seeds: https://news.ycombinator.com/item?id=10962305 https://news.ycombinator.com/item?id=10962305 I suggested that one could maybe bridge Bittorrent and IPFS, then someone suggested that IPFS would be ideal for being the 'canonical' source of the file while the other protocols (HTTP, BT, WT) are merely ways to access the file.
  - toomuchtodo 10y ago
    > then someone suggested that IPFS would be ideal for being the 'canonical' source of the file while the other protocols (HTTP, BT, WT) are merely ways to access the file. Hey! That was me! I still believe its a great idea ;)
  - greglindahl 10y ago
    The Internet Archive seeds all of our items' torrents since 2012. IPFS is a more straightforward way for us to be sure we've got independent seeds of all of our content. Got 25 petabytes of disk?
    - niftich 10y ago
      Uhhh, 20 GB free... I'm just a third-party, but if you were to get in touch with the IPFS team to host a limited amount of IA content as a proof-of-concept, I'm sure it would be a great publicity for both of your organizations.

mdjeu 10y ago

Maybe this a bit tangential, but I was just thinking lately how it would be nice to use some decentralized communication backend to word processing software to enable shared collaborative editing of documents. Sort of like Google Docs, but without a centralized server.

chj 10y ago

It's great to share data, however, how can a torrent solution support services like web API?

rictic 10y ago
(This is a good question, but as asked it's so broad that it's akin to asking "What kinds of things does P2P work for?" and we're still discovering more answers to it every year.)
bbcbasic 10y ago
I think these services would compliment a torrent solution, e.g. in a video hosting site web-api gives you the metadata (and perhaps the first 5 buffered seconds of the video), and torrent gives you the video.
geocar 10y ago
Well, one way I did it was by creating fake torrents with an unlikely info_hash (if you see the all-0xaa's hash on bittorrent crashing your client a few weeks ago, that's me). In this way, I can get use public WebTorrent signalling servers to establish the WebRTC connections between my (non-bittorrent) clients. You can distinguish between clients and servers (if you don't want a P2P model) by having clients claim to need this unlikely info_hash, and servers seed it. In this way, WebTorrent's websockets server won't link clients to other clients.

chj 10y ago

It's great to share data, however, how can a torrent solution support services like web API?

235337 10y ago

Anyone have suggestions on how to block sites like this from downloading without permissions? edit: So blocking XHR requests seems to do the trick.

solidr53 10y ago

I was surprised to find out that the torrent fetching part of the prototype didn't happen in the service worker . Which is obviously the most cool place for it.

pmlnr 10y ago

If there wasn't a 'cloud' in the picture how seeding is done I'd even believe it's serverless. Sorry for being a cynic, but what if there is no tracker available?

kinlan 10y ago
It's a very fair point, I chose the word serverless to imply no traditional back-end. One extension that I am looking forward to seeing is the use of DHT in to the WebTorrent that (at least in my head) can eliminate the need for trackers. The important point in my head is that I can distribute audio files to anyone who is seeding client-to-client rather than pass through a traditional server that I would host.

visarga 10y ago

The app is quite interesting and I was amazed until I realized the play button doesn't display the circular progress indicator on time. It has an ugly lag. This is why web apps are not quite there yet. And yet it is amazing they can record, store and sync audio like that. Almost like real apps.

kinlan 10y ago
That is me being lazy and showing a demo/concept experience, not because the web platform can't do this.