5 ms·

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think t

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?

gerwim 5y ago
Well there is Dolt: https://www.dolthub.com/ https://www.dolthub.com/
mosselman 5y ago
> (or, updates require re-deployment of full new .sqlite db, together with a new website) Seems like a small price to pay. I can see new stuff living in torrents that then get committed to a sqlite database that is then deployed every now and then.
chrisjc 5y ago
> Something that would have immutable rows Isn't this just a log (or a stream like Kafka or Kinesis)? In fact you might even be able say every database already has this ;) (binlog, oplog, etc) > not the whole db If rows are immutable, what part of the db is left not being immutable? If rows were immutable, doesn't that imply that any existing "ranges, searches, indexes, etc" are immutable too? If you going to the effort of making a decentralized database, why not also decompose all of these parts from one another... no reason the tables (logs), indexes, search, etc have to live in the same database, they could be spun off as completely different parts. Basically a something that indexes logs and then something else that takes those indexes and makes the searchable. In fact this is all the rage right now with centralized databases... all of the work being done with streaming systems just seems to be an effort of decomposing and inverting databases... every old is new again. Anyways, I agree with the idea and don't really know enough about decentralized systems to really understand why such a distributed database can't or hasn't already been built.
oscargrouch 5y ago
My current project can expose a sqlite database over torrent wrapped by the Chrome cache filesystem. The torrent is merkle-tree based, and for each change in the SQLite memory page it also updates the merkle tree giving you a different torrent infohash at the end. So as databases are published together with the applications the initial torrent info is the same, so theres a swarm of initial peers. Once it gets updated and the torrent "turns into another", the solution is to use the RPC api (that my solution provides) over the initial swarm of peers and use some sort of distributed algorithm like raft or gossip to define how to deal with the subsequent changes over the peers. The important part is that all other changed torrents from all the peers are available to all peers in the network, and with a RPC working as a abstract interface to organize them, it will work for a lot of different scenarios.
- smallerfish 5y ago
  That's interesting. Are you aware of the absurd-sql project? It may have some useful stuff that you can leverage. Also, how do peers find each other? And finally, do you intend to open source it?
  - oscargrouch 5y ago
    >That's interesting. Are you aware of the absurd-sql project? No, thanks for pointing that out, will take a look into it. > Also, how do peers find each other? The idea is to use the torrents as a common shareable resource where given the peers have the same interest in that torrent, lets say the torrents works as a "meta-database" with just enough immutable metadata info, giving the developers of that application a list of peers that will have the same RPC service you designed, with a interface that suits your purpose according to your application goals (a distributed Youtube for instance), and giving you know and designed that API yourself whatever you want from each node, lets say a piece of data, be it a file or a database key-value range, you can ask your API for it, combining the torrent peers and whatever distribution combo you need. > And finally, do you intend to open source it? I've just did https://github.com/mumba-org/mumba https://github.com/mumba-org/mumba It's badly documented given i'm on the final touches before a proper launch, but the "documentation" of the storage layer is planned for today, giving how important it is for the whole thing. The first thing i've tried to get right was the storage layer, and the capacity to use mutable sqlite databases over torrent (together with files which are simpler given they are meant to be immutable). Given theres no proper doc yet, i can point out to the source at https://github.com/mumba-org/mumba/tree/main/lib/storage https://github.com/mumba-org/mumba/tree/main/lib/storage where: https://github.com/mumba-org/mumba/tree/main/lib/storage/backend https://github.com/mumba-org/mumba/tree/main/lib/storage/bac... is a modified chrome cache storage layer (which is the real underlying disk storage) that abstract the files and databases storages that from the bit-torrent layer perspective are on the disk. https://github.com/mumba-org/mumba/blob/main/lib/storage/storage.h https://github.com/mumba-org/mumba/blob/main/lib/storage/sto... is the front end and the: https://github.com/mumba-org/mumba/blob/main/lib/storage/torrent.h https://github.com/mumba-org/mumba/blob/main/lib/storage/tor... is the main abstraction that may be a "fileset"(collection of files as in torrent) or a dataset/database, which in this case you can get the sqlite db handle from the torrent object and deal with it as normal database. I've take care to enable a key-value store over the sqlite btree, so both form of databases are possible, a key-value and a normal SQL database. Key-values are important for the distributed case where you may want the nodes to have partial data and abstract a SQL layer (or whatever) on top of the distributed nodes, which is a better solution for distributed storage. For the database distribution, a 64k SQLite memory page maps to a torrent piece of the same size, which can be synchronized over other peers that knows what's the root of the merkle tree of the given database is (you can use the RPC layer to coordinate this or use the bit-torrent DHT updating your slot with the new merkle root) BTW This is what i'm using to distribute the applications, the DHT which points to a "database torrent" which in turn is a index to other files and database torrents. The application owner have write permissions over that particular DHT slot and can change the root merkle anytime the application itself changes (a new version, or some of the assets). What im finishing right now is exactly the higher level layers that automate this whole thing and make it work "under the hood" without the users or applications distributors need to understand how it is implemented. Of course the developers will have access to iterating over the peers of a giving torrent to group them over RPC interfaces, and the idea is also to give access to the DHT so solutions that need a access to mutable ever-changing merkle root is also possible (even though the torrent based solution cover most if not all of the ground), but the idea is to give the devs the tools to be creative about their solutions. (The solution is a whole browser-based UI/web applications platform, that use torrents and the torrent DHT for the p2p storage layer)
rakoo 5y ago
Mutable torrents do exist (https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html). Your "URL" can now be a public key, and clients will fetch the latest version of the torrent.
- ddtaylor 5y ago
  This is actually really neat I wasn't aware of these BEPs. Do you know of any place where they are being used?
  - rakoo 5y ago
    libtorrent has it, webtorrent has been working on it but development of this feature seems to have stalled. That's a real shame, because webtorrent can be used directly from the browser so it would have been very practical to use a single dependency and have pretty much the same functionalities as the article
    - ddtaylor 5y ago
      I am also curious if any services or sites are using it yet.
      - publiush 5y ago
        I actually made a system called federalist [1] that uses mutable torrents. Hope you like it! [1] https://github.com/publiusfederalist/federalist https://github.com/publiusfederalist/federalist
pradn 5y ago
I think it's possible to use IPNS to create a key-value mapping where the value can be changed. https://flyingzumwalt.github.io/ipfs-tutorials/curriculum/files-on-ipfs/modules/update-files/ https://flyingzumwalt.github.io/ipfs-tutorials/curriculum/fi...
all2 5y ago
What about a distributed hash table [0]? Yggdrasil uses it for network routing information [1]. [0] https://en.wikipedia.org/wiki/Distributed_hash_table https://en.wikipedia.org/wiki/Distributed_hash_table [1] https://yggdrasil-network.github.io/2018/07/17/world-tree.html#the-world-tree https://yggdrasil-network.github.io/2018/07/17/world-tree.ht...
- delusional 5y ago
  I think you just reinvented the bit torrent mainline DHT.
imtringued 5y ago
There should be a separation of data and the actual website that does the reading. I once thought about something similar (not for torrents), but having to download the whole database (300MB and more) was a bad idea for UX. So what I propose is this: There is a central source of truth hosted on IPFS via IPNS or mutable torrents. The data format can be whatever is more efficient to update and store, not easiest to read and query. (An event log is fine) Then, there is a reader application, like an RSS reader. The reader application can be a desktop application or it can be a centrally hosted web app that keeps a conventional SQL database behind the scenes. The idea is that there will be multiple reader providers like there are competing email providers.

gerwim 5y ago

Well there is Dolt: https://www.dolthub.com/ https://www.dolthub.com/

mosselman 5y ago

> (or, updates require re-deployment of full new .sqlite db, together with a new website) Seems like a small price to pay. I can see new stuff living in torrents that then get committed to a sqlite database that is then deployed every now and then.

chrisjc 5y ago

> Something that would have immutable rows Isn't this just a log (or a stream like Kafka or Kinesis)? In fact you might even be able say every database already has this ;) (binlog, oplog, etc) > not the whole db If rows are immutable, what part of the db is left not being immutable? If rows were immutable, doesn't that imply that any existing "ranges, searches, indexes, etc" are immutable too? If you going to the effort of making a decentralized database, why not also decompose all of these parts from one another... no reason the tables (logs), indexes, search, etc have to live in the same database, they could be spun off as completely different parts. Basically a something that indexes logs and then something else that takes those indexes and makes the searchable. In fact this is all the rage right now with centralized databases... all of the work being done with streaming systems just seems to be an effort of decomposing and inverting databases... every old is new again. Anyways, I agree with the idea and don't really know enough about decentralized systems to really understand why such a distributed database can't or hasn't already been built.

oscargrouch 5y ago

My current project can expose a sqlite database over torrent wrapped by the Chrome cache filesystem. The torrent is merkle-tree based, and for each change in the SQLite memory page it also updates the merkle tree giving you a different torrent infohash at the end. So as databases are published together with the applications the initial torrent info is the same, so theres a swarm of initial peers. Once it gets updated and the torrent "turns into another", the solution is to use the RPC api (that my solution provides) over the initial swarm of peers and use some sort of distributed algorithm like raft or gossip to define how to deal with the subsequent changes over the peers. The important part is that all other changed torrents from all the peers are available to all peers in the network, and with a RPC working as a abstract interface to organize them, it will work for a lot of different scenarios.

smallerfish 5y ago
That's interesting. Are you aware of the absurd-sql project? It may have some useful stuff that you can leverage. Also, how do peers find each other? And finally, do you intend to open source it?
- oscargrouch 5y ago
  >That's interesting. Are you aware of the absurd-sql project? No, thanks for pointing that out, will take a look into it. > Also, how do peers find each other? The idea is to use the torrents as a common shareable resource where given the peers have the same interest in that torrent, lets say the torrents works as a "meta-database" with just enough immutable metadata info, giving the developers of that application a list of peers that will have the same RPC service you designed, with a interface that suits your purpose according to your application goals (a distributed Youtube for instance), and giving you know and designed that API yourself whatever you want from each node, lets say a piece of data, be it a file or a database key-value range, you can ask your API for it, combining the torrent peers and whatever distribution combo you need. > And finally, do you intend to open source it? I've just did https://github.com/mumba-org/mumba https://github.com/mumba-org/mumba It's badly documented given i'm on the final touches before a proper launch, but the "documentation" of the storage layer is planned for today, giving how important it is for the whole thing. The first thing i've tried to get right was the storage layer, and the capacity to use mutable sqlite databases over torrent (together with files which are simpler given they are meant to be immutable). Given theres no proper doc yet, i can point out to the source at https://github.com/mumba-org/mumba/tree/main/lib/storage https://github.com/mumba-org/mumba/tree/main/lib/storage where: https://github.com/mumba-org/mumba/tree/main/lib/storage/backend https://github.com/mumba-org/mumba/tree/main/lib/storage/bac... is a modified chrome cache storage layer (which is the real underlying disk storage) that abstract the files and databases storages that from the bit-torrent layer perspective are on the disk. https://github.com/mumba-org/mumba/blob/main/lib/storage/storage.h https://github.com/mumba-org/mumba/blob/main/lib/storage/sto... is the front end and the: https://github.com/mumba-org/mumba/blob/main/lib/storage/torrent.h https://github.com/mumba-org/mumba/blob/main/lib/storage/tor... is the main abstraction that may be a "fileset"(collection of files as in torrent) or a dataset/database, which in this case you can get the sqlite db handle from the torrent object and deal with it as normal database. I've take care to enable a key-value store over the sqlite btree, so both form of databases are possible, a key-value and a normal SQL database. Key-values are important for the distributed case where you may want the nodes to have partial data and abstract a SQL layer (or whatever) on top of the distributed nodes, which is a better solution for distributed storage. For the database distribution, a 64k SQLite memory page maps to a torrent piece of the same size, which can be synchronized over other peers that knows what's the root of the merkle tree of the given database is (you can use the RPC layer to coordinate this or use the bit-torrent DHT updating your slot with the new merkle root) BTW This is what i'm using to distribute the applications, the DHT which points to a "database torrent" which in turn is a index to other files and database torrents. The application owner have write permissions over that particular DHT slot and can change the root merkle anytime the application itself changes (a new version, or some of the assets). What im finishing right now is exactly the higher level layers that automate this whole thing and make it work "under the hood" without the users or applications distributors need to understand how it is implemented. Of course the developers will have access to iterating over the peers of a giving torrent to group them over RPC interfaces, and the idea is also to give access to the DHT so solutions that need a access to mutable ever-changing merkle root is also possible (even though the torrent based solution cover most if not all of the ground), but the idea is to give the devs the tools to be creative about their solutions. (The solution is a whole browser-based UI/web applications platform, that use torrents and the torrent DHT for the p2p storage layer)

rakoo 5y ago

Mutable torrents do exist (https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html). Your "URL" can now be a public key, and clients will fetch the latest version of the torrent.

ddtaylor 5y ago
This is actually really neat I wasn't aware of these BEPs. Do you know of any place where they are being used?
- rakoo 5y ago
  libtorrent has it, webtorrent has been working on it but development of this feature seems to have stalled. That's a real shame, because webtorrent can be used directly from the browser so it would have been very practical to use a single dependency and have pretty much the same functionalities as the article
  - ddtaylor 5y ago
    I am also curious if any services or sites are using it yet.
    - publiush 5y ago
      I actually made a system called federalist [1] that uses mutable torrents. Hope you like it! [1] https://github.com/publiusfederalist/federalist https://github.com/publiusfederalist/federalist

pradn 5y ago

I think it's possible to use IPNS to create a key-value mapping where the value can be changed. https://flyingzumwalt.github.io/ipfs-tutorials/curriculum/files-on-ipfs/modules/update-files/ https://flyingzumwalt.github.io/ipfs-tutorials/curriculum/fi...

all2 5y ago

What about a distributed hash table [0]? Yggdrasil uses it for network routing information [1]. [0] https://en.wikipedia.org/wiki/Distributed_hash_table https://en.wikipedia.org/wiki/Distributed_hash_table [1] https://yggdrasil-network.github.io/2018/07/17/world-tree.html#the-world-tree https://yggdrasil-network.github.io/2018/07/17/world-tree.ht...

delusional 5y ago
I think you just reinvented the bit torrent mainline DHT.

imtringued 5y ago

There should be a separation of data and the actual website that does the reading. I once thought about something similar (not for torrents), but having to download the whole database (300MB and more) was a bad idea for UX. So what I propose is this: There is a central source of truth hosted on IPFS via IPNS or mutable torrents. The data format can be whatever is more efficient to update and store, not easiest to read and query. (An event log is fine) Then, there is a reader application, like an RSS reader. The reader application can be a desktop application or it can be a centrally hosted web app that keeps a conventional SQL database behind the scenes. The idea is that there will be multiple reader providers like there are competing email providers.