6 ms·

Pushing the Limits of IPFS and OrbitDB

104 points by markhenderson 8y ago

etcet 8y ago
Just a warning, the page takes 75 seconds to load.
- throaaway100126 8y ago
  took 5 seconds for me to load on tor, with noscript and ublock on. treat yoself with the world of no js or ads, it's great.
- the_duke 8y ago
  Instant for me.
welder 8y ago
http://webcache.googleusercontent.com/search?q=cache:eh2eZNy8pGcJ:mrh.io/2018-01-24-pushing-limits-ipfs-orbitdb/+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:eh2eZNy...
spiraldancing 8y ago
Original site is inaccessible for me ... is this a temporary issue?
- 3131s 8y ago
  Seems fine now, loaded quickly.
- spiraldancing 8y ago
  Works for me now, too. Temporary hiccup, apparently.
doronaviguy 8y ago
Very interesting stuff , I'm following ipfs for a while now , and using the browser sessions as full nodes is an exciting idea. One thing I wonder about , Is the browser -> ipfs node communication can be done via webrtc (I think the post refer to websocket)? So the data sync can happen between browsers . I did play with webtorrent for a while , and achieved a similar behavior using webrtc . I had one nodejs server , and multiple browser sessions . It worked well for video streaming peer to peer (poc level )
- diggan 8y ago
  Browser IPFS to desktop IPFS communication can happen via websockets or webrtc right now, since those are the transports the browser currently offers us. The stack we're using for networking is libp2p, a peer-to-peer framework, which has bunch of implementations for different transports. You can see the full list of those here: https://libp2p.io/implementations/#transports https://libp2p.io/implementations/#transports Currently, most browser-apps using IPFS prefers websockets as it's using a lot less resources compared to webrtc, but once webrtc-quic[0] is ready for prime-time, it would be easy to write a transport for that and hopefully will be a lot more efficient than the current webrtc implementations. Disclaimer: I work on IPFS/libp2p fulltime - [0] https://w3c.github.io/webrtc-quic/ https://w3c.github.io/webrtc-quic/
aqsheehy 8y ago
Who pays for ipfs?
- erlend_sh 8y ago
  Protocol Labs - https://protocol.ai https://protocol.ai They did a big ICO with Filecoin.
  - hobofan 8y ago
    They pay for (most of) the development and running some public nodes, but in the end running IPFS is up to individual nodes.
- AgentME 8y ago
  It works like torrents: nodes only re-host content that they want to.
realPubkey 8y ago
I doubt the usefullness of orbitDb. Ipfs-nodes only store and distribute content which they have accessed before. When you have a database with user-based data, noone except yourself will distribute it. This means you could have the same distribution with plain webrtc instead of height-ping ipfs.
- haadcode 8y ago
  It's true that availability is not "automatic" as per the software itself (rarely is in P2P networks). To make sure the data stays online (whether IPFS or OrbitDB), one needs to make sure there are enough nodes serving the data. In OrbitDB the replication protocol will make sure the full database is downloaded and served upon opening the database and as all db-data is IPFS-data, the db will be served just like any content on IPFS. So for example, for an application built with OrbitDB and IPFS, one would probably want to run some nodes themselves to "pin" (IPFS parlance) the databases and serve them to the network. Edit: There's an open issue for a "OrbitDB pinning service" (https://github.com/orbitdb/orbit-db/issues/352 https://github.com/orbitdb/orbit-db/issues/352), it's in the works atm.
  - realPubkey 8y ago
    But doesnt this mean I just distribute a big json over ipfs? Where do I need orbitDB for that?
    - haadcode 8y ago
      Not quite. The data structures for the "database" part are more complex than a single json structure. Each update to the database is an "individual json" which get linked together using an append-only log CRDT (https://github.com/orbitdb/ipfs-log https://github.com/orbitdb/ipfs-log) and you'd essentially serve that log. So instead of "one big json", it's "many small jsons" and the state (view) of the database gets calculated from those small jsons.
      - realPubkey 8y ago
konart 8y ago
inaccessible even via VPN for me...
- konart 8y ago
  finally got through after a few minutes.
haadcode 8y ago
The blog post is accessible through IPFS here https://ipfs.io/ipfs/QmY2LufsW3v6AfxTTkp6SGqDGa5AeJhSZXH8RSsdiao4Ds/ https://ipfs.io/ipfs/QmY2LufsW3v6AfxTTkp6SGqDGa5AeJhSZXH8RSs... in case the original link doesn't work.
- fwdpropaganda 8y ago
  Can anyone be bothered to explain to me how this works? I know that IPFS is some sort of distributed content-addressable thingy. So I expected that ipfs.io would deliver some JS to my browser, which would get it to look-up some distributed hash table in order to find peers with the content (a la torrents). However I'm not running JS and that page loads just fine. So how does it work?
  - detaro 8y ago
    ipfs.io is a gateway, fetching the content from the IPFS network and serving it to you over HTTP.
    - fwdpropaganda 8y ago
      So sort of an intermediary between how the internet works these days, and how it should work in IPFS view. Does the "deliverying JS to browser" thing that I described make sense at all?
      - detaro 8y ago
        There's work on running the IPFS protocol over WebRTC, so yes, it makes sense and is another way of enabling normal browsers to access it.
      - Confiks 8y ago
        There is already a Javascript implementation [1], but it is not feasible to setup a full IPFS node in every browser tab that loads an IPFS resource. Instead you can use ipfs-companion [2], a browser extension that lets you easily load IPFS resources in your browser. While it can use the Javascript implementation, this is not recommended as it is currently pretty resource heavy [3]. [1] https://github.com/ipfs/js-ipfs https://github.com/ipfs/js-ipfs [2] https://github.com/ipfs-shipyard/ipfs-companion https://github.com/ipfs-shipyard/ipfs-companion [3] https://github.com/ipfs-shipyard/ipfs-companion/blob/master/docs/node-types.md https://github.com/ipfs-shipyard/ipfs-companion/blob/master/...
twoodfin 8y ago
It’s too bad that the flowering of distributed hash table / content-addressable storage research & development nearly precisely coincided with Internet Explorer’s ossification of web standards, or maybe we’d have had something like IPFS built into every browser 15 years ago.

104 points by markhenderson 8y ago

etcet 8y ago

Just a warning, the page takes 75 seconds to load.

throaaway100126 8y ago
took 5 seconds for me to load on tor, with noscript and ublock on. treat yoself with the world of no js or ads, it's great.
the_duke 8y ago
Instant for me.

welder 8y ago

http://webcache.googleusercontent.com/search?q=cache:eh2eZNy8pGcJ:mrh.io/2018-01-24-pushing-limits-ipfs-orbitdb/+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:eh2eZNy...

spiraldancing 8y ago

Original site is inaccessible for me ... is this a temporary issue?

3131s 8y ago
Seems fine now, loaded quickly.
spiraldancing 8y ago
Works for me now, too. Temporary hiccup, apparently.

doronaviguy 8y ago

Very interesting stuff , I'm following ipfs for a while now , and using the browser sessions as full nodes is an exciting idea. One thing I wonder about , Is the browser -> ipfs node communication can be done via webrtc (I think the post refer to websocket)? So the data sync can happen between browsers . I did play with webtorrent for a while , and achieved a similar behavior using webrtc . I had one nodejs server , and multiple browser sessions . It worked well for video streaming peer to peer (poc level )

diggan 8y ago
Browser IPFS to desktop IPFS communication can happen via websockets or webrtc right now, since those are the transports the browser currently offers us. The stack we're using for networking is libp2p, a peer-to-peer framework, which has bunch of implementations for different transports. You can see the full list of those here: https://libp2p.io/implementations/#transports https://libp2p.io/implementations/#transports Currently, most browser-apps using IPFS prefers websockets as it's using a lot less resources compared to webrtc, but once webrtc-quic[0] is ready for prime-time, it would be easy to write a transport for that and hopefully will be a lot more efficient than the current webrtc implementations. Disclaimer: I work on IPFS/libp2p fulltime - [0] https://w3c.github.io/webrtc-quic/ https://w3c.github.io/webrtc-quic/

aqsheehy 8y ago

Who pays for ipfs?

erlend_sh 8y ago
Protocol Labs - https://protocol.ai https://protocol.ai They did a big ICO with Filecoin.
- hobofan 8y ago
  They pay for (most of) the development and running some public nodes, but in the end running IPFS is up to individual nodes.
AgentME 8y ago
It works like torrents: nodes only re-host content that they want to.

realPubkey 8y ago

I doubt the usefullness of orbitDb. Ipfs-nodes only store and distribute content which they have accessed before. When you have a database with user-based data, noone except yourself will distribute it. This means you could have the same distribution with plain webrtc instead of height-ping ipfs.

haadcode 8y ago
It's true that availability is not "automatic" as per the software itself (rarely is in P2P networks). To make sure the data stays online (whether IPFS or OrbitDB), one needs to make sure there are enough nodes serving the data. In OrbitDB the replication protocol will make sure the full database is downloaded and served upon opening the database and as all db-data is IPFS-data, the db will be served just like any content on IPFS. So for example, for an application built with OrbitDB and IPFS, one would probably want to run some nodes themselves to "pin" (IPFS parlance) the databases and serve them to the network. Edit: There's an open issue for a "OrbitDB pinning service" (https://github.com/orbitdb/orbit-db/issues/352 https://github.com/orbitdb/orbit-db/issues/352), it's in the works atm.
- realPubkey 8y ago
  But doesnt this mean I just distribute a big json over ipfs? Where do I need orbitDB for that?
  - haadcode 8y ago
    Not quite. The data structures for the "database" part are more complex than a single json structure. Each update to the database is an "individual json" which get linked together using an append-only log CRDT (https://github.com/orbitdb/ipfs-log https://github.com/orbitdb/ipfs-log) and you'd essentially serve that log. So instead of "one big json", it's "many small jsons" and the state (view) of the database gets calculated from those small jsons.
    - realPubkey 8y ago

konart 8y ago

inaccessible even via VPN for me...

konart 8y ago
finally got through after a few minutes.

haadcode 8y ago

The blog post is accessible through IPFS here https://ipfs.io/ipfs/QmY2LufsW3v6AfxTTkp6SGqDGa5AeJhSZXH8RSsdiao4Ds/ https://ipfs.io/ipfs/QmY2LufsW3v6AfxTTkp6SGqDGa5AeJhSZXH8RSs... in case the original link doesn't work.

fwdpropaganda 8y ago
Can anyone be bothered to explain to me how this works? I know that IPFS is some sort of distributed content-addressable thingy. So I expected that ipfs.io would deliver some JS to my browser, which would get it to look-up some distributed hash table in order to find peers with the content (a la torrents). However I'm not running JS and that page loads just fine. So how does it work?
- detaro 8y ago
  ipfs.io is a gateway, fetching the content from the IPFS network and serving it to you over HTTP.
  - fwdpropaganda 8y ago
    So sort of an intermediary between how the internet works these days, and how it should work in IPFS view. Does the "deliverying JS to browser" thing that I described make sense at all?
    - detaro 8y ago
      There's work on running the IPFS protocol over WebRTC, so yes, it makes sense and is another way of enabling normal browsers to access it.
    - Confiks 8y ago
      There is already a Javascript implementation [1], but it is not feasible to setup a full IPFS node in every browser tab that loads an IPFS resource. Instead you can use ipfs-companion [2], a browser extension that lets you easily load IPFS resources in your browser. While it can use the Javascript implementation, this is not recommended as it is currently pretty resource heavy [3]. [1] https://github.com/ipfs/js-ipfs https://github.com/ipfs/js-ipfs [2] https://github.com/ipfs-shipyard/ipfs-companion https://github.com/ipfs-shipyard/ipfs-companion [3] https://github.com/ipfs-shipyard/ipfs-companion/blob/master/docs/node-types.md https://github.com/ipfs-shipyard/ipfs-companion/blob/master/...

twoodfin 8y ago

It’s too bad that the flowering of distributed hash table / content-addressable storage research & development nearly precisely coincided with Internet Explorer’s ossification of web standards, or maybe we’d have had something like IPFS built into every browser 15 years ago.