9 ms·

Yup. Calling it. This is the future, may not seem like it to everyone but this is a part of actually new and extremely useful, passively scalable technology. Im

by pizza 5y ago

Yup. Calling it. This is the future, may not seem like it to everyone but this is a part of actually new and extremely useful, passively scalable technology. Imagine (good) unkillable zombie databases- so long as the name of a piece of data is known (its hash).. someone, somewhere, might make it possible for you to answer your query, without even needing to setup a server setup.. that’s _it_! Not to mention it might be faster than the original source. Pretty damn powerful. The next level above this is a search index that lets the user generate their own results, using their own machine learning algorithm or ranking weights of their own preferences, bc they would have direct access to the DB index and features. People could wrote anti-ad plugins. There could be foss upgrades all the time. Nobody would have to spend a particularly crazy amount of money on storage if they could all just cache the bits they’d needed themselves. Quite remarkable imo!! Whatever ends Google search’s reign, will probably be user-owned in a way that seems a lot like this..

RF_Savage 5y ago
Yep. This very well could be the future for sites with static content that does not change often.
- zackees 5y ago
- encryptluks2 5y ago
  This isn't a static site. It is dynamic requiring a lot of frameworks. I can think of a lot of better ways to accomplish something similar, especially with IPFS.
  - TimJRobinson 5y ago
    By static they mean no server side processing / can be hosted on a CDN. If you know better ways of doing this you should describe them.
a1371 5y ago
So this is the true Web3 then?
- akomtu 5y ago
  No. This thing puts the user first (forgive my corpspeak): it gives us a powerful tool for free. Web3, on the other hand, puts the advertisers first. If this was web3, it would have some useless microtransactions attached to it. I bet that someone behind web3 is thinking hard right now how to stop this torrent thing.
- woah 5y ago
  It is what people have been referring to as "Web3", yes: https://ipfs.io/#how https://ipfs.io/#how
- sergiotapia 5y ago
  Yes. The web3 name is being used by crypto-shysters right now so unfortunately they have the lions share of web3 implementation as they call it. But this (OP) is real web3!
- quickthrower2 5y ago
  I know everyone is fed up with the cliche of web3, but having done some side projects with web3 (as in npm i web3) and ipfs, the posted article here is very exciting. This posted link could be another piece of the jigsaw (Solidity, IPFS, IPNS, ...) that I think will come together to make interesting apps in the future. Solidity doesn't mean necessarily spending $100 to make a function call - there are other chains, off chain stuff being developed, and you could host a private chain for your app. While none of this stuff can do something new you can't do with Postgres - you can create more open and perhaps 'honest' applications where everyone can see the data-engine and understand it. So it is more of a cultural shift. For example, if you make a Twitter this way, you don't need to rely on an API. The data is there for everyone, all of it. For example, take Uniswap. It's not a company like Facebook, it's an open protocol. Swapping tokens is now functionally open source because of that. There is no "Facebook of swapping tokens". And this can be used as a building block for other apps. Not necessarily just "gambling" or "trading" either.
singron 5y ago
Unfortunately this is still killable. Someone needs to periodically publish up to date databases using some outside protocol. Also if you participate in the public DHT, then your participation is public and can have consequences on your real-life legal entity. If you want to use TOR and onion sites, I don't think this really adds anything to those. I think this just helps you optionaly crowdsource bandwidth.
- chhs 5y ago
  It's possible to publish updates to torrent files via the DHT. See http://www.bittorrent.org/beps/bep_0046.html http://www.bittorrent.org/beps/bep_0046.html
- akomtu 5y ago
  With a simple protocol on top it can be quite resilient. The database is updated only by a group of 3 admins, who perform the update once a week. When one admin initiates an update, the other two ask him the passphrase. If the answer is "tomato" (means "I'm compromised"), the update is rejected and they find another admin. Regular users are generally safe: the laws dont punish downloaders, only uploaders. You can expand the admin team to 5 members, so RIAA (or whoever stalks torrent sites) would have to compromise 2 admins simultaneously. Don't forget that the admins can be in different countries, making their legal pursuit a nightmare.
  - c22 5y ago
    Isn't the entire point of bittorrent that downloaders are also uploaders?
    - delusional 5y ago
      That's mostly important due to social factors. Technically you'll find that most torrents are seeded by people who seed a lot, while most users just drive-by leech it. What makes torrents special is that all leechers CAN be seeders, that don't have to be though.
  - anamax 5y ago
wccrawford 5y ago
Isn't that what Perfect Dark, Winny, Share, and other peer to peer sharing systems already are? Why is this more compelling than those?
- nl 5y ago
  It runs in an unmodified browser. That's a very big deal.
bscphil 5y ago
> so long as the name of a piece of data is known (its hash) But that's also the fatal flaw. It's pretty definitionally true of stuff on the web that the person who owns / controls it wants to be able to change / update it. That isn't really supported at all at present for this distribution method, and it's hard to see how it could be in the future without introducing a single choke point. Case in point: the vast majority of torrent traffic is for new torrents, because people want to download the stuff that's just been released, not the stuff that they either already have or could have had months ago. You lose 100% of this traffic with this "p2psearch" method, because the database can't be updated. Or if it can be updated (stick it on a traditional website for people to mirror), you rely on everyone updating to the newest version of the database. It's also incredibly slow compared to traditional sites. It took longer than it took me to type this comment to see a single result for the popular title I tried to search for.
- khimaros 5y ago
  despite the downvotes, there is some substance to this. mutability is useful. however, as implemented, all that is needed is for the user of p2psearch to refresh and the browser to pick up the latest database. i imagine most users are not keeping torrent search open 24/7, so this doesn't seem onerous. it's probably a bit of a process for the host of the frontend to update the database, prepare a new torrent, update the code [0], and then rebuild the bundle regularly, but this could be automated. regardless, it doesn't seem so unreasonable from an end user perspective, and i personally don't mind if my torrent search index is behind by a few days. [0] https://gitlab.com/boredcaveman/p2psearch/-/blob/main/src/MemoryAsyncVFS.js#L13 https://gitlab.com/boredcaveman/p2psearch/-/blob/main/src/Me...
  - bscphil 5y ago
    > it's probably a bit of a process for the host of the frontend to update the database Right, I suppose I didn't put my point very well, but as I see it there are only two options: * Everyone gets their code (with the built-in torrent for the database) from the same source, presumably the source that created / seeds the database. In this case the host is arguably just as vulnerable to the authorities as The Pirate Bay. If p2psearch doesn't make torrenting more resilient than TPB, what purpose does it serve? * One brave citizen creates and hosts a single copy of the database, and it goes viral. Lots of people host it. It never gets updated, because the whole point of using IPFS / torrents is that everything can be addressed statically. In this case the advantage of p2psearch is obvious - it functions as a DHT with a built-in search that is more or less impossible to take down! On the other hand, the lack of mutability greatly reduces the value, since as I said in my OP most torrenting is focused on new releases.
oscargrouch 5y ago
I've built something akin to this, but with the idea of applications distributing API`s where the databases are also torrented but working behind the api's, so that developers can build basically anything. In my case i've implemented a new "browser" based on Chrome that allows this to work, without having to resort to browser-only infrastructure (for instance applications can dodge Javascript and also call RPC api's from other applications directly). The applications and the applications data are distributed over torrent and managed to work together in the same environment as a flock where one app can consume its own apis and also the api's from others. service Search { rpc doQuery(string query) => (array<string> result) // the access to the sqlite db from torrents is encapsulated here } The beauty of this design is that it can also be re-scheduled and have the same request routed to other peers ---
reggieband 5y ago
This idea touches on (or rather, even expands on) an idea I've had brewing for a while. For those unfamiliar, there is a programming language called Unison [1] that has a nice feature where each function is identified by a hash of its AST. I often thought a cluster environment, maybe something BEAM-ish, would be an interesting idea where functions are retrievable based on a pair of (`func-name`, `hash`). You could have some kind of execution environment (maybe a lightweight process) that receives a hash identifying the function to execute and some pointer to the data (probably a URL) to act as input. Given tech like described in the linked article, you could reasonably lookup such a function from a peer-to-peer network. 1. https://www.unison-lang.org/ https://www.unison-lang.org/
- octoberfranklin 5y ago
  This approach to mobile code in functional languages has been tried repeatedly since the 1990s. > each function is identified by a hash of its AST. That encoding for mobile code only works if the expression has no free variables (i.e. is closed). It turns out that it is surprisingly difficult to write code which you can be sure has no free variables at particular points (the "send this code to another machine for execution" points). Especially in functional languages. If you write a higher-order functional, you know nothing about the closedness of its argument. If you require that all function arguments be closed it breaks all the useful functional programming techniques. The only way to make it workable is to check closedness only on those values which need to be mobile. If you do the closedness-check at runtime, toy examples will work fine and the technique looks quite powerful, but once your codebase gets to a meaningful level of complexity it turns into a game of whack-a-mole with closedness heisenbugs. You quickly discover that closedness is data-dependent. In order to check closedness statically you need quite a sophisticated system of modal types. MetaOCaml was the most usable result of all this research: https://okmij.org/ftp/ML/MetaOCaml.html https://okmij.org/ftp/ML/MetaOCaml.html The important upshot here is that this isn't just some check you can slap onto the compiler. The programmer has to think about these types, and craft them carefully, as an integral part of the programming process. Basically you aren't just writing a program, you're writing a program plus a proof that it won't try to mobilize open code. Writing formal machine-checked proofs is not something that most programmers are good at.
joshspankit 5y ago
The media control barons have spent the past decade building legal strategies for taking down promising services. Personally I also feel like this is the future, but only in a future where we dismantle their stranglehold on culture. Anyone who shares content (whether a file or a byte) exposes themselves in one way or another and can be targeted. Anyone else remember the “good old days” when torrent sites stood up in the face of the law and yelled “we host nothing illegal because we host nothing!” and they were right. Then the law firms started joining the swarms to collect IP addresses of everyone that shared with them. Now they rarely bother trying to sue IP addresses. The legal ability is still there, it just turns out it’s not worth the cost to sometimes collect a few hundred dollars from someone who can’t afford it. When we talk about transformative tech, we have to immediately consider the “standard scenarios” because being blind to them is how most technology trips and falls. Pirating (especially live PPV events), illegal porn, utterly violent videos, information on making weapons, letting children access same and/or mixing them in with adults in unsafe and unsupervised spaces, openly anti-government statements in countries where they make people disappear, and so-on. These things happen. And they happen faster and more intensely than most programmers can handle. The only way to keep tools like this viable is to build the tech with those in mind.
Ambolia 5y ago
The current internet is terrible at archiving anything it creates. Dead links, impossible to find videos if they get deleted for some reason, news articles that get memory-holed, easier to find archives of paper newspapers than of online newspapers... That's why so many people tweet screenshots of other tweets rather than linking them directly, the network relationship may break at any moment, so better just to post an image rather than link information. In the Napster/Torrent era once I found a video, document or audio I was interested in, it was very easy to find again without having to worry about archiving it myself.
Beltiras 5y ago
Only feature missing is making the database mutable.
BeFlatXIII 5y ago
Unkillable zombie databases would be an amazing asset for fanfiction & art. Not only does it become slightly more C&D-resistant than normal hosting, it also protects against the far more common scenario: artists and authors blowing their lid and ragequitting by deleting all their work.
IshKebab 5y ago
I don't think so. This still relies on non-distributed infrastructure to host the code and provide the database torrent. Those are not "unkillable". And an inefficient completely immutable database where even individual queries are not private does not sound like it has a lot of uses.
sliken 5y ago
First CAS (content addressable storage) is nothing new. Torrents aren't new and are basically a distribed CAS database. With a magnet link (basically a checksum) you can find those files hosted by any torrent client on the planet. So sure things are "unkillable", but depend on someone/somewhere decided it's worth storing that particular chunk. Seems strange to pair "unkillable" which depends on someone somewhere that "might" do something. Much like how IPFS was easily oversold, great you can find things hosted anywhere on the planet, but if you publish 1M files and expect to magically seem them hosted elsewhere a year later you are likely to be disappointed. I do wonder if it would be a better approach to replace filecoin or similar complex trust relationships with a simple peer to peer trading program. Something along the lines of "Lets trade 128MB", trust but verify, then watch uptime/availability. For clients up less than a month, give them the free 128MB and watch, 1-3 months trust them enough to store data with a 20x replication, 3-12 months 10x replication, over a year 5x, whitelisted peers of friends/family 3x.

by pizza 5y ago

RF_Savage 5y ago

Yep. This very well could be the future for sites with static content that does not change often.

zackees 5y ago
encryptluks2 5y ago
This isn't a static site. It is dynamic requiring a lot of frameworks. I can think of a lot of better ways to accomplish something similar, especially with IPFS.
- TimJRobinson 5y ago
  By static they mean no server side processing / can be hosted on a CDN. If you know better ways of doing this you should describe them.

a1371 5y ago

So this is the true Web3 then?

akomtu 5y ago
No. This thing puts the user first (forgive my corpspeak): it gives us a powerful tool for free. Web3, on the other hand, puts the advertisers first. If this was web3, it would have some useless microtransactions attached to it. I bet that someone behind web3 is thinking hard right now how to stop this torrent thing.
woah 5y ago
It is what people have been referring to as "Web3", yes: https://ipfs.io/#how https://ipfs.io/#how
sergiotapia 5y ago
Yes. The web3 name is being used by crypto-shysters right now so unfortunately they have the lions share of web3 implementation as they call it. But this (OP) is real web3!
quickthrower2 5y ago
I know everyone is fed up with the cliche of web3, but having done some side projects with web3 (as in npm i web3) and ipfs, the posted article here is very exciting. This posted link could be another piece of the jigsaw (Solidity, IPFS, IPNS, ...) that I think will come together to make interesting apps in the future. Solidity doesn't mean necessarily spending $100 to make a function call - there are other chains, off chain stuff being developed, and you could host a private chain for your app. While none of this stuff can do something new you can't do with Postgres - you can create more open and perhaps 'honest' applications where everyone can see the data-engine and understand it. So it is more of a cultural shift. For example, if you make a Twitter this way, you don't need to rely on an API. The data is there for everyone, all of it. For example, take Uniswap. It's not a company like Facebook, it's an open protocol. Swapping tokens is now functionally open source because of that. There is no "Facebook of swapping tokens". And this can be used as a building block for other apps. Not necessarily just "gambling" or "trading" either.

singron 5y ago

Unfortunately this is still killable. Someone needs to periodically publish up to date databases using some outside protocol. Also if you participate in the public DHT, then your participation is public and can have consequences on your real-life legal entity. If you want to use TOR and onion sites, I don't think this really adds anything to those. I think this just helps you optionaly crowdsource bandwidth.

chhs 5y ago
It's possible to publish updates to torrent files via the DHT. See http://www.bittorrent.org/beps/bep_0046.html http://www.bittorrent.org/beps/bep_0046.html
akomtu 5y ago
With a simple protocol on top it can be quite resilient. The database is updated only by a group of 3 admins, who perform the update once a week. When one admin initiates an update, the other two ask him the passphrase. If the answer is "tomato" (means "I'm compromised"), the update is rejected and they find another admin. Regular users are generally safe: the laws dont punish downloaders, only uploaders. You can expand the admin team to 5 members, so RIAA (or whoever stalks torrent sites) would have to compromise 2 admins simultaneously. Don't forget that the admins can be in different countries, making their legal pursuit a nightmare.
- c22 5y ago
  Isn't the entire point of bittorrent that downloaders are also uploaders?
  - delusional 5y ago
    That's mostly important due to social factors. Technically you'll find that most torrents are seeded by people who seed a lot, while most users just drive-by leech it. What makes torrents special is that all leechers CAN be seeders, that don't have to be though.
- anamax 5y ago

wccrawford 5y ago

Isn't that what Perfect Dark, Winny, Share, and other peer to peer sharing systems already are? Why is this more compelling than those?

nl 5y ago
It runs in an unmodified browser. That's a very big deal.

bscphil 5y ago

> so long as the name of a piece of data is known (its hash) But that's also the fatal flaw. It's pretty definitionally true of stuff on the web that the person who owns / controls it wants to be able to change / update it. That isn't really supported at all at present for this distribution method, and it's hard to see how it could be in the future without introducing a single choke point. Case in point: the vast majority of torrent traffic is for new torrents, because people want to download the stuff that's just been released, not the stuff that they either already have or could have had months ago. You lose 100% of this traffic with this "p2psearch" method, because the database can't be updated. Or if it can be updated (stick it on a traditional website for people to mirror), you rely on everyone updating to the newest version of the database. It's also incredibly slow compared to traditional sites. It took longer than it took me to type this comment to see a single result for the popular title I tried to search for.

khimaros 5y ago
despite the downvotes, there is some substance to this. mutability is useful. however, as implemented, all that is needed is for the user of p2psearch to refresh and the browser to pick up the latest database. i imagine most users are not keeping torrent search open 24/7, so this doesn't seem onerous. it's probably a bit of a process for the host of the frontend to update the database, prepare a new torrent, update the code [0], and then rebuild the bundle regularly, but this could be automated. regardless, it doesn't seem so unreasonable from an end user perspective, and i personally don't mind if my torrent search index is behind by a few days. [0] https://gitlab.com/boredcaveman/p2psearch/-/blob/main/src/MemoryAsyncVFS.js#L13 https://gitlab.com/boredcaveman/p2psearch/-/blob/main/src/Me...
- bscphil 5y ago
  > it's probably a bit of a process for the host of the frontend to update the database Right, I suppose I didn't put my point very well, but as I see it there are only two options: * Everyone gets their code (with the built-in torrent for the database) from the same source, presumably the source that created / seeds the database. In this case the host is arguably just as vulnerable to the authorities as The Pirate Bay. If p2psearch doesn't make torrenting more resilient than TPB, what purpose does it serve? * One brave citizen creates and hosts a single copy of the database, and it goes viral. Lots of people host it. It never gets updated, because the whole point of using IPFS / torrents is that everything can be addressed statically. In this case the advantage of p2psearch is obvious - it functions as a DHT with a built-in search that is more or less impossible to take down! On the other hand, the lack of mutability greatly reduces the value, since as I said in my OP most torrenting is focused on new releases.

oscargrouch 5y ago

I've built something akin to this, but with the idea of applications distributing API`s where the databases are also torrented but working behind the api's, so that developers can build basically anything. In my case i've implemented a new "browser" based on Chrome that allows this to work, without having to resort to browser-only infrastructure (for instance applications can dodge Javascript and also call RPC api's from other applications directly). The applications and the applications data are distributed over torrent and managed to work together in the same environment as a flock where one app can consume its own apis and also the api's from others. service Search { rpc doQuery(string query) => (array<string> result) // the access to the sqlite db from torrents is encapsulated here } The beauty of this design is that it can also be re-scheduled and have the same request routed to other peers ---

reggieband 5y ago

This idea touches on (or rather, even expands on) an idea I've had brewing for a while. For those unfamiliar, there is a programming language called Unison [1] that has a nice feature where each function is identified by a hash of its AST. I often thought a cluster environment, maybe something BEAM-ish, would be an interesting idea where functions are retrievable based on a pair of (`func-name`, `hash`). You could have some kind of execution environment (maybe a lightweight process) that receives a hash identifying the function to execute and some pointer to the data (probably a URL) to act as input. Given tech like described in the linked article, you could reasonably lookup such a function from a peer-to-peer network. 1. https://www.unison-lang.org/ https://www.unison-lang.org/

octoberfranklin 5y ago
This approach to mobile code in functional languages has been tried repeatedly since the 1990s. > each function is identified by a hash of its AST. That encoding for mobile code only works if the expression has no free variables (i.e. is closed). It turns out that it is surprisingly difficult to write code which you can be sure has no free variables at particular points (the "send this code to another machine for execution" points). Especially in functional languages. If you write a higher-order functional, you know nothing about the closedness of its argument. If you require that all function arguments be closed it breaks all the useful functional programming techniques. The only way to make it workable is to check closedness only on those values which need to be mobile. If you do the closedness-check at runtime, toy examples will work fine and the technique looks quite powerful, but once your codebase gets to a meaningful level of complexity it turns into a game of whack-a-mole with closedness heisenbugs. You quickly discover that closedness is data-dependent. In order to check closedness statically you need quite a sophisticated system of modal types. MetaOCaml was the most usable result of all this research: https://okmij.org/ftp/ML/MetaOCaml.html https://okmij.org/ftp/ML/MetaOCaml.html The important upshot here is that this isn't just some check you can slap onto the compiler. The programmer has to think about these types, and craft them carefully, as an integral part of the programming process. Basically you aren't just writing a program, you're writing a program plus a proof that it won't try to mobilize open code. Writing formal machine-checked proofs is not something that most programmers are good at.

joshspankit 5y ago

The media control barons have spent the past decade building legal strategies for taking down promising services. Personally I also feel like this is the future, but only in a future where we dismantle their stranglehold on culture. Anyone who shares content (whether a file or a byte) exposes themselves in one way or another and can be targeted. Anyone else remember the “good old days” when torrent sites stood up in the face of the law and yelled “we host nothing illegal because we host nothing!” and they were right. Then the law firms started joining the swarms to collect IP addresses of everyone that shared with them. Now they rarely bother trying to sue IP addresses. The legal ability is still there, it just turns out it’s not worth the cost to sometimes collect a few hundred dollars from someone who can’t afford it. When we talk about transformative tech, we have to immediately consider the “standard scenarios” because being blind to them is how most technology trips and falls. Pirating (especially live PPV events), illegal porn, utterly violent videos, information on making weapons, letting children access same and/or mixing them in with adults in unsafe and unsupervised spaces, openly anti-government statements in countries where they make people disappear, and so-on. These things happen. And they happen faster and more intensely than most programmers can handle. The only way to keep tools like this viable is to build the tech with those in mind.

Ambolia 5y ago

The current internet is terrible at archiving anything it creates. Dead links, impossible to find videos if they get deleted for some reason, news articles that get memory-holed, easier to find archives of paper newspapers than of online newspapers... That's why so many people tweet screenshots of other tweets rather than linking them directly, the network relationship may break at any moment, so better just to post an image rather than link information. In the Napster/Torrent era once I found a video, document or audio I was interested in, it was very easy to find again without having to worry about archiving it myself.

Beltiras 5y ago

Only feature missing is making the database mutable.

BeFlatXIII 5y ago

Unkillable zombie databases would be an amazing asset for fanfiction & art. Not only does it become slightly more C&D-resistant than normal hosting, it also protects against the far more common scenario: artists and authors blowing their lid and ragequitting by deleting all their work.

IshKebab 5y ago

I don't think so. This still relies on non-distributed infrastructure to host the code and provide the database torrent. Those are not "unkillable". And an inefficient completely immutable database where even individual queries are not private does not sound like it has a lot of uses.

sliken 5y ago

First CAS (content addressable storage) is nothing new. Torrents aren't new and are basically a distribed CAS database. With a magnet link (basically a checksum) you can find those files hosted by any torrent client on the planet. So sure things are "unkillable", but depend on someone/somewhere decided it's worth storing that particular chunk. Seems strange to pair "unkillable" which depends on someone somewhere that "might" do something. Much like how IPFS was easily oversold, great you can find things hosted anywhere on the planet, but if you publish 1M files and expect to magically seem them hosted elsewhere a year later you are likely to be disappointed. I do wonder if it would be a better approach to replace filecoin or similar complex trust relationships with a simple peer to peer trading program. Something along the lines of "Lets trade 128MB", trust but verify, then watch uptime/availability. For clients up less than a month, give them the free 128MB and watch, 1-3 months trust them enough to store data with a 20x replication, 3-12 months 10x replication, over a year 5x, whitelisted peers of friends/family 3x.