9 ms·

Peer-to-Peer Databases for the Decentralized Web

261 points by fossislife 6y ago

w3clan 6y ago
I think https://gun.eco/docs/ https://gun.eco/docs/ does so much cooler job.
voicedYoda 6y ago
Why do i always feel things like this and ipfs cone back to blockchain? I think the blockchain is a very interesting tool in a limited capacity. Orbitdb and ipfs seem to capitalize on their relationship to it.
- markhenderson 6y ago
  OrbitDB doesn't rely on a blockchain at all
encryptluks2 6y ago
I don't see how people consider this and IPFS decentralized. You still need peers to subscribe and pin, and services like Pinata aren't cheap and still centralized. Is there an example of websites that would continue to run indefinitely if there wasn't a centralization of services with IPFS? I just don't think it is likely unless you start distributing your app that bundles IPFS, so that your users end up hosting it for you. However, if someone new tries it out and only one of your users has your app open you better hope it is being cached by Cloudflare.
- aaomidi 6y ago
  What do you think decentralized means? It means that hosting it isn't the responsibility of a single party. If I put something online, and people think it's interesting and pin it. Even if I go down, the pinned content remains up. Decentralized doesn't magically solve hosting availability. It just removes responsibility from a single party.
  - encryptluks2 6y ago
    I would say there are very few IPFS sites where hosting doesn't still end up being the primary responsibility of a single party. That was the point I was trying to make. If you have to pay someone else to pin it and keep giving them funds then it is still your responsibility. If you have a website that you want to scale and load fast for all your users on IPFS then it will still end up being your responsibility. There may be a few exceptions.
    - aaomidi 6y ago
      There is no technology possible that removes said responsibility. This is how peer to peer systems work. Your criticism is something that no peer to peer system claims to solve.
      - 9000 6y ago
        Actually, I recommend you check out Freenet [0], a project that's 20 years old. In fact, the Wikipedia page explicitly states: > Information flow in Freenet is different from networks like eMule or BitTorrent; in Freenet: > 1. A user wishing to share a file or update a freesite "inserts" the file "to the network" > 2. After "insertion" is finished, the publishing node is free to shut down, because the file is stored in the network. It will remain available for other users whether or not the original publishing node is online. No single node is responsible for the content; instead, it is replicated to many different nodes. [0] https://en.wikipedia.org/wiki/Freenet https://en.wikipedia.org/wiki/Freenet
        octopoc 6y ago
        Aren’t collaborative systems like that susceptible to abuse? E.g., an attacker could upload 1TB of noise and then shut down, reducing the network size by much more than 1TB.
        FeepingCreature 6y ago
        Of course, in Freenet, if a file is not requested often it will quickly fall off the network due to user churn.
- hosh 6y ago
  I'm looking at integrating IPFS with sandstorm (sandstorm.io). The sandstorm (and other homelab/homeserver) community have an interest in self-hosting. IPFS allows people that are interested in seeing something stored to pin it themselves. But yes: bundling IPFS with the app is pretty much what would need to happen. As they are now, neither the IPFS ecosystem or sandstorm are really decentralized -- or maybe I should be more precise: not decentralized in a resilient way. Yet, I think both are great technologies that enable some interesting things. I'm not as interested in the blockchain notion of decentralized though. I'm looking for resiliency to severe netsplits. As a thought experiment, if Earth crosses a coronal mass ejection event and our power and communication grids go down, would we be able stitch things back together again when our grid fractures?
  - secfirstmd 6y ago
    Very cool idea!
  - pm 6y ago
    Curious about the homelab/homeserver community. Are there any sites with further information?
    - hosh 6y ago
      If you reddit, r/selfhosted and r/homelab are a great place to start. I really like the sandstorm.io platform, though it is not currently supported on an rpi. It has the potential as a platform that “grandma” can use. There is a project called HomelabOS. It reminds me a bit of Homebrew.
      - pm 6y ago
        Cheers. I remember hearing SandStorm (the company) went under a couple years ago, and was curious but hadn't explored it in depth. I've been paying attention to CollapseOS as well, though it's a little more apocalyptic in nature. The eventual aim is to be able to provision a set of servers from a thumb drive without Internet (if that's even possible), but I haven't had much time to get far.
- markhenderson 6y ago
  Swarm persistence is hard, if not close to impossible. Better that multiple parties pin the data than just one though. It's a step in the right direction.
  - encryptluks2 6y ago
    Agreed. I started looking into setting up cluster followers, but currently you have to have enough space to pin everything and can't have it only do a percentage.
- t3rr0r 6y ago
  I share your viewpoint and frustration with "decentralized" apps. But as you said it is indeed possible by bundling your app with IPFS and using something like orbitdb/ipfslog for structuring data. I linked this in my other comment, but take a look at this proof of concept: https://www.reddit.com/r/musichoarder/comments/lrqx7m/record_a_distributed_audio_file_system/ https://www.reddit.com/r/musichoarder/comments/lrqx7m/record... At this point the major challenges are with UX, especially for users who are not accustomed to self-hosting. I think we have the technologies and ideas to overcome the rest.
- bawolff 6y ago
  I think that's the big problem with the "decentralized" WWW movement. There are about a billion conflicting definitions of decentralized providing very different properties, and it's never clear which definition people mean.
- saurik 6y ago
  It is decentralized in that anyone can step up and host the content on IPFS and it is considered equivalent and secure. With a normal website, if the canonical host goes down, someone else can host it, but it isn't the same website.
- Geee 6y ago
  IPFS, torrents and many old-school p2p-networks are decentralized to an extent, but they're also non-sybil-resistant and voluntarist, which makes them unsuitable for many use cases. They're not really a part of what is meant by decentralized web. You should take a look at SkyDB to get an idea what's possible with true decentralization: https://blog.sia.tech/skydb-a-mutable-database-for-the-decentralized-web-7170beeaa985 https://blog.sia.tech/skydb-a-mutable-database-for-the-decen...
- amw-zero 6y ago
  I feel like you don't understand the definition of "peer to peer." The fact that you require nodes to ultimate do the hosting of data does not make IPFS centralized.
opendomain 6y ago
This project may be dead There has been no activity in over a year,and the sponsoring company has no presence in several years.
- ampdepolymerase 6y ago
  Gun DB is a good replacement. https://gun.eco/ https://gun.eco/
- markhenderson 6y ago
  It's not dead, just in a bit of a lull due to funding, which is hopefully coming soon. We have a very active Gitter and OpenCollective. Source: I'm one of the core maintainers
  - octopoc 6y ago
    Is there any interest in supporting other languages? Like how libp2p is being gradually ported to Rust, Go, Java. I feel like OrbitDB could become the new data model of the web if it went multi language. Or maybe was compiled to WebAssembly and embeddable using Wasmer.
    - markhenderson 6y ago
      There's definitely interest. Berty has a Go implementation and the idea of a Rust implementation gets tossed around every season. Can Webassembly read IndexedDB yet?
      - octopoc 6y ago
        It seems so: https://github.com/nwestfall/BlazorDB https://github.com/nwestfall/BlazorDB Although this might be using JavaScript interop instead of using WebAssembly directly. Edit: check this out: https://www.ditto.live/ https://www.ditto.live/ It’s a multi language CRDT implementation that seems to use WebAssembly to be available in multiple languages.
- 6y ago
hartator 6y ago
I wonder if it would be possible to build a decentralized search engine with a decentralized crawl on top of it.
- hosh 6y ago
  I've been wondering the same thing too! Maybe if there is a way to share indexes, or some way to put out a call for searches through the DHT, and then the requestor aggregates the results. A couple examples: Web crawler -- seeded through a curated or personal bookmark list. Knowledge base -- the permaculture community have amassed a knowledge base on growing plants: USDA hardiness zones, heat zones, humidity, soil requirements, companion planting relationships, etc. That's something I can see people adding on to, particularly with non-standardized homegrown cultivars and how well they do in their local area.
  - beaconstudios 6y ago
    Limewire had an interesting protocol for exactly this back in the day, over its Gnutella network. I don't know if it scales to search engine sizes (most things don't, especially distributed) but there's at least some prior production grade work in this space. Edit: having just double checked it looks like Gnutella predates limewire; that's just the avenue through which I came across it.
    - deleted 6y ago
      [deleted]
  - lovemenot 6y ago
    >> Maybe if there is a way to share indexes... What does a SE index look like? Any designs in the open?
- emaro 6y ago
  There is one: https://yacy.net/ https://yacy.net/ However it's slow and the last release is from 2016.
lazzlazzlazz 6y ago
IPFS + (Arweave or Filecoin) behind it seems like the best decentralized web stack I've seen yet. I'm not sure how Orbit holds up.
- j0hnM1st 6y ago
  Arweave has production ready releases ?
  - lazzlazzlazz 6y ago
    Yes. Check it out.
endisneigh 6y ago
I wish there was something like this but with Webtorrent - I find torrents easier to grok for whatever reason.
- oscargrouch 6y ago
  I hope i can share something here more on this line pretty soon. Its not just databases and blobs over torrent but a whole web based application platform based on chrome for native applications. The SDK will start with Swift but the C interface can be used by other languages.
oilbagz 6y ago
I like this, but I feel its important to point out that this is effectively unusable by the very people who would benefit from it most: non-web developers. If this were a drop-in C/C++ library, I'd use it immediately in my native apps and adopt it as a widespread platform technology. I want to be able to do this, and if it were truly a working technology, I'd put it in every single app I use. But as a Node/JS library, it is off limits. Please, decentralization guys, consider native languages first, and toy languages next. These technologies are never going to be embraced unless they treat native platforms as first-class systems for their manifestation.
- christophilus 6y ago
  What is the definition of a toy language? That’s a new term for me. I (thankfully) haven’t written C++ in over 20 years, but if I had to drop back down to a low level language, it’d be C or a compile-to-C language. C is the language these things should target, in my opinion.
  - kyleishie 6y ago
    > What is the definition of a toy language? That’s a new term for me. I would imagine most non-compiled, scripting languages. They are ok when building the business logic of an application on top of faster building blocks but not ok for application building blocks such as a database.
  - StavrosK 6y ago
    > What is the definition of a toy language? A language that doesn't require as much hair-pulling to learn, so you can't feel as smug for knowing it.
    - argvargc 6y ago
      So... not Javascript then?
  - bamfloesche 6y ago
    Systems developer here. I made an account to answer this, because the OP is 100% right about this. A toy language is one which you do not allow to escape the play-room/sandbox, you play with it (because hey, you can, so why not..), but you would not actually use it. The reason this is important for decentralization efforts, is that the OS should be doing this. Sure, paint the pretty UI, but expose the entire thing to common OS-level abstractions, please, and then host the scripts, kthxbai .. (A non-toy language is one you are very comfortable shipping. Javascript is that for a lot of people. Not for systems programmers = it is an application execution environment, not a place for things which must operate, as part of the system, without much user hassle ..)
economusty 6y ago
This would be great if it were sqlite3 in the cloud.
mab122 6y ago
An interesting project is https://github.com/dappkit/aviondb https://github.com/dappkit/aviondb which is mongo-db like database on top of orbitdb

261 points by fossislife 6y ago

I think https://gun.eco/docs/ https://gun.eco/docs/ does so much cooler job.

Why do i always feel things like this and ipfs cone back to blockchain? I think the blockchain is a very interesting tool in a limited capacity. Orbitdb and ipfs seem to capitalize on their relationship to it.

markhenderson 6y ago
OrbitDB doesn't rely on a blockchain at all

encryptluks2 6y ago

I don't see how people consider this and IPFS decentralized. You still need peers to subscribe and pin, and services like Pinata aren't cheap and still centralized. Is there an example of websites that would continue to run indefinitely if there wasn't a centralization of services with IPFS? I just don't think it is likely unless you start distributing your app that bundles IPFS, so that your users end up hosting it for you. However, if someone new tries it out and only one of your users has your app open you better hope it is being cached by Cloudflare.

aaomidi 6y ago
What do you think decentralized means? It means that hosting it isn't the responsibility of a single party. If I put something online, and people think it's interesting and pin it. Even if I go down, the pinned content remains up. Decentralized doesn't magically solve hosting availability. It just removes responsibility from a single party.
- encryptluks2 6y ago
  I would say there are very few IPFS sites where hosting doesn't still end up being the primary responsibility of a single party. That was the point I was trying to make. If you have to pay someone else to pin it and keep giving them funds then it is still your responsibility. If you have a website that you want to scale and load fast for all your users on IPFS then it will still end up being your responsibility. There may be a few exceptions.
  - aaomidi 6y ago
    There is no technology possible that removes said responsibility. This is how peer to peer systems work. Your criticism is something that no peer to peer system claims to solve.
    - 9000 6y ago
      Actually, I recommend you check out Freenet [0], a project that's 20 years old. In fact, the Wikipedia page explicitly states: > Information flow in Freenet is different from networks like eMule or BitTorrent; in Freenet: > 1. A user wishing to share a file or update a freesite "inserts" the file "to the network" > 2. After "insertion" is finished, the publishing node is free to shut down, because the file is stored in the network. It will remain available for other users whether or not the original publishing node is online. No single node is responsible for the content; instead, it is replicated to many different nodes. [0] https://en.wikipedia.org/wiki/Freenet https://en.wikipedia.org/wiki/Freenet
      - octopoc 6y ago
        Aren’t collaborative systems like that susceptible to abuse? E.g., an attacker could upload 1TB of noise and then shut down, reducing the network size by much more than 1TB.
      - FeepingCreature 6y ago
        Of course, in Freenet, if a file is not requested often it will quickly fall off the network due to user churn.
hosh 6y ago
I'm looking at integrating IPFS with sandstorm (sandstorm.io). The sandstorm (and other homelab/homeserver) community have an interest in self-hosting. IPFS allows people that are interested in seeing something stored to pin it themselves. But yes: bundling IPFS with the app is pretty much what would need to happen. As they are now, neither the IPFS ecosystem or sandstorm are really decentralized -- or maybe I should be more precise: not decentralized in a resilient way. Yet, I think both are great technologies that enable some interesting things. I'm not as interested in the blockchain notion of decentralized though. I'm looking for resiliency to severe netsplits. As a thought experiment, if Earth crosses a coronal mass ejection event and our power and communication grids go down, would we be able stitch things back together again when our grid fractures?
- secfirstmd 6y ago
  Very cool idea!
- pm 6y ago
  Curious about the homelab/homeserver community. Are there any sites with further information?
  - hosh 6y ago
    If you reddit, r/selfhosted and r/homelab are a great place to start. I really like the sandstorm.io platform, though it is not currently supported on an rpi. It has the potential as a platform that “grandma” can use. There is a project called HomelabOS. It reminds me a bit of Homebrew.
    - pm 6y ago
      Cheers. I remember hearing SandStorm (the company) went under a couple years ago, and was curious but hadn't explored it in depth. I've been paying attention to CollapseOS as well, though it's a little more apocalyptic in nature. The eventual aim is to be able to provision a set of servers from a thumb drive without Internet (if that's even possible), but I haven't had much time to get far.
markhenderson 6y ago
Swarm persistence is hard, if not close to impossible. Better that multiple parties pin the data than just one though. It's a step in the right direction.
- encryptluks2 6y ago
  Agreed. I started looking into setting up cluster followers, but currently you have to have enough space to pin everything and can't have it only do a percentage.
t3rr0r 6y ago
I share your viewpoint and frustration with "decentralized" apps. But as you said it is indeed possible by bundling your app with IPFS and using something like orbitdb/ipfslog for structuring data. I linked this in my other comment, but take a look at this proof of concept: https://www.reddit.com/r/musichoarder/comments/lrqx7m/record_a_distributed_audio_file_system/ https://www.reddit.com/r/musichoarder/comments/lrqx7m/record... At this point the major challenges are with UX, especially for users who are not accustomed to self-hosting. I think we have the technologies and ideas to overcome the rest.
bawolff 6y ago
I think that's the big problem with the "decentralized" WWW movement. There are about a billion conflicting definitions of decentralized providing very different properties, and it's never clear which definition people mean.
saurik 6y ago
It is decentralized in that anyone can step up and host the content on IPFS and it is considered equivalent and secure. With a normal website, if the canonical host goes down, someone else can host it, but it isn't the same website.
Geee 6y ago
IPFS, torrents and many old-school p2p-networks are decentralized to an extent, but they're also non-sybil-resistant and voluntarist, which makes them unsuitable for many use cases. They're not really a part of what is meant by decentralized web. You should take a look at SkyDB to get an idea what's possible with true decentralization: https://blog.sia.tech/skydb-a-mutable-database-for-the-decentralized-web-7170beeaa985 https://blog.sia.tech/skydb-a-mutable-database-for-the-decen...
amw-zero 6y ago
I feel like you don't understand the definition of "peer to peer." The fact that you require nodes to ultimate do the hosting of data does not make IPFS centralized.

opendomain 6y ago

This project may be dead There has been no activity in over a year,and the sponsoring company has no presence in several years.

ampdepolymerase 6y ago
Gun DB is a good replacement. https://gun.eco/ https://gun.eco/
markhenderson 6y ago
It's not dead, just in a bit of a lull due to funding, which is hopefully coming soon. We have a very active Gitter and OpenCollective. Source: I'm one of the core maintainers
- octopoc 6y ago
  Is there any interest in supporting other languages? Like how libp2p is being gradually ported to Rust, Go, Java. I feel like OrbitDB could become the new data model of the web if it went multi language. Or maybe was compiled to WebAssembly and embeddable using Wasmer.
  - markhenderson 6y ago
    There's definitely interest. Berty has a Go implementation and the idea of a Rust implementation gets tossed around every season. Can Webassembly read IndexedDB yet?
    - octopoc 6y ago
      It seems so: https://github.com/nwestfall/BlazorDB https://github.com/nwestfall/BlazorDB Although this might be using JavaScript interop instead of using WebAssembly directly. Edit: check this out: https://www.ditto.live/ https://www.ditto.live/ It’s a multi language CRDT implementation that seems to use WebAssembly to be available in multiple languages.
6y ago

hartator 6y ago

I wonder if it would be possible to build a decentralized search engine with a decentralized crawl on top of it.

hosh 6y ago
I've been wondering the same thing too! Maybe if there is a way to share indexes, or some way to put out a call for searches through the DHT, and then the requestor aggregates the results. A couple examples: Web crawler -- seeded through a curated or personal bookmark list. Knowledge base -- the permaculture community have amassed a knowledge base on growing plants: USDA hardiness zones, heat zones, humidity, soil requirements, companion planting relationships, etc. That's something I can see people adding on to, particularly with non-standardized homegrown cultivars and how well they do in their local area.
- beaconstudios 6y ago
  Limewire had an interesting protocol for exactly this back in the day, over its Gnutella network. I don't know if it scales to search engine sizes (most things don't, especially distributed) but there's at least some prior production grade work in this space. Edit: having just double checked it looks like Gnutella predates limewire; that's just the avenue through which I came across it.
  - deleted 6y ago
    [deleted]
- lovemenot 6y ago
  >> Maybe if there is a way to share indexes... What does a SE index look like? Any designs in the open?
emaro 6y ago
There is one: https://yacy.net/ https://yacy.net/ However it's slow and the last release is from 2016.

lazzlazzlazz 6y ago

IPFS + (Arweave or Filecoin) behind it seems like the best decentralized web stack I've seen yet. I'm not sure how Orbit holds up.

j0hnM1st 6y ago
Arweave has production ready releases ?
- lazzlazzlazz 6y ago
  Yes. Check it out.

endisneigh 6y ago

I wish there was something like this but with Webtorrent - I find torrents easier to grok for whatever reason.

oscargrouch 6y ago
I hope i can share something here more on this line pretty soon. Its not just databases and blobs over torrent but a whole web based application platform based on chrome for native applications. The SDK will start with Swift but the C interface can be used by other languages.

oilbagz 6y ago

I like this, but I feel its important to point out that this is effectively unusable by the very people who would benefit from it most: non-web developers. If this were a drop-in C/C++ library, I'd use it immediately in my native apps and adopt it as a widespread platform technology. I want to be able to do this, and if it were truly a working technology, I'd put it in every single app I use. But as a Node/JS library, it is off limits. Please, decentralization guys, consider native languages first, and toy languages next. These technologies are never going to be embraced unless they treat native platforms as first-class systems for their manifestation.

christophilus 6y ago
What is the definition of a toy language? That’s a new term for me. I (thankfully) haven’t written C++ in over 20 years, but if I had to drop back down to a low level language, it’d be C or a compile-to-C language. C is the language these things should target, in my opinion.
- kyleishie 6y ago
  > What is the definition of a toy language? That’s a new term for me. I would imagine most non-compiled, scripting languages. They are ok when building the business logic of an application on top of faster building blocks but not ok for application building blocks such as a database.
- StavrosK 6y ago
  > What is the definition of a toy language? A language that doesn't require as much hair-pulling to learn, so you can't feel as smug for knowing it.
  - argvargc 6y ago
    So... not Javascript then?
- bamfloesche 6y ago
  Systems developer here. I made an account to answer this, because the OP is 100% right about this. A toy language is one which you do not allow to escape the play-room/sandbox, you play with it (because hey, you can, so why not..), but you would not actually use it. The reason this is important for decentralization efforts, is that the OS should be doing this. Sure, paint the pretty UI, but expose the entire thing to common OS-level abstractions, please, and then host the scripts, kthxbai .. (A non-toy language is one you are very comfortable shipping. Javascript is that for a lot of people. Not for systems programmers = it is an application execution environment, not a place for things which must operate, as part of the system, without much user hassle ..)

economusty 6y ago

This would be great if it were sqlite3 in the cloud.

mab122 6y ago

An interesting project is https://github.com/dappkit/aviondb https://github.com/dappkit/aviondb which is mongo-db like database on top of orbitdb