4 ms·

I am an author of a distributed database (http://gunDB.io/ http://gunDB.io/) and I thought this article was fairly good, but with one grievance. Hashes aren't t

I am an author of a distributed database (http://gunDB.io/ http://gunDB.io/) and I thought this article was fairly good, but with one grievance. Hashes aren't the ultimate solution. Certainly hashes will make things a lot better, but a lot of people I know (including some of the people referenced in the article) are heralding them as the second coming of the web. I disagree because they present more problems, ones that are harder for humans and easier for machines. First off, in order to find the data you practically speaking have to already have the data. Second, if you don't have the data, then you have to know the hash, but how do you know the hash without having had the data OR trusting some authority for the hash? Now you are back to centralization. Third, and most important from my perspective, a hash often times will point to outdated content because by the time you find it it might have changed (this of course is a good thing but can be problematic). I know they addressed this a little bit in the article but it is worth reiterating. The point is that it is hard to do synchronization on hashes and I would argue that sync is the more important problem and that is what I am trying to address in my open source database.

simcop2387 11y ago
Part of the way that FreeNET addresses the dating issue with hashes is to attach a version number to the content. A monotonic integer that goes up each time it's updated. In this case then though the hash can't be represented directly by the content but is instead more like a UUID. Which puts you back at your second point which FreeNET doesn't really have a way to deal with either, so it ends up people trading urls on message boards essentially.
andrewstuart2 11y ago
Well, you're right that it requires centralization, but at least the amount of data you need to retrieve centrally is that much smaller. It would benefit the global infrastructure greatly if I could retrieve websites over LAN if they'd been cached by a computer there. My wife and I visit a lot of the same sites, even see a lot of the same dynamic content (same facebook friends, etc). Surely there's some community effect that would make this a huge bandwidth win.
- nothrabannosir 11y ago
  Careful: this gets you dangerously close to privacy violations. You may not want certain people to know that you are fetching a piece of content. "Hmm, the profile picture for this specific friend loaded suspiciously quickly.." A similar mistake is security updates over bittorrent: a real-time list of vulnerable hosts.
  - andrewstuart2 11y ago
    Argh. Privacy really is such a double-edged sword. You could still keep those bits off of the WAN, though. In a "privacy mode" the gateway could coordinate and throw in an average delay.
nbadg 11y ago
I agree that hash-based addressing creates a bit of a chicken-and-the-egg problem for content discovery, with the exception of, for example, existing links or QR codes. You'll always need some way of looking up the hash for the content you're hoping to find. But that lookup mechanism need not be a single centralized service. In fact, I would make the argument that in many cases, search engines are doing exactly this right now: you have several coexisting, competing, but centralized services, and a user is free to choose between them. It's centralization, sure, but in a much more robust way (though it admittedly could be significantly improved through, for example, a common search API). The only difference is that contemporary search engines are looking up URLs instead of hashes, but given how we're using URLs these days (ex: https://imgur.com/iauyhdf https://imgur.com/iauyhdf), I don't think it's a big distinction. You're also spot-on that dynamic content is the hardest part of hash-based addressing, but the problem is a bit more nuanced IMO. On the one hand, all content, once created, is static. If a thermometer reads 32.8 at 15:40 on 30 July 2015, that data point is fixed in time, immutable. However, we humans think of things far more conceptually, and we build cognitive connections between things. All content is static, but all concepts are dynamic. So the question is then, how do you reconcile those two? The best answer I've personally come across is bindings -- exactly analogous to binding names to variables in Python. Some hash bindings would be static, some bindings dynamic. That also allows you to construct more complicated objects like buffers natively. That's how I'm doing it in the project I'm working on (https://github.com/Muterra/doc-muse https://github.com/Muterra/doc-muse), though the bindings are a relatively new addition that I haven't had time to test yet (or update the documentation, for that matter).
dreamfactory2 11y ago
What about distributed metadata dictionaries/taxonomies describing the content and syncing via something like pubsubhubbub?
nly 11y ago
I think hashed addressing is a lot easier now mobile has increased the appeal of less typing. A user doesn't care whether a QR code contains a URL or a hash.
chmike 11y ago
A hash based distributed database doesn't allow to benefit from the locality effect where user access locally produced data more frequently. There is also the data hosting cost. It seams more fair and logical that data producers pays for the hosting of their data. Otherwise there is a risk of abuse.

simcop2387 11y ago

Part of the way that FreeNET addresses the dating issue with hashes is to attach a version number to the content. A monotonic integer that goes up each time it's updated. In this case then though the hash can't be represented directly by the content but is instead more like a UUID. Which puts you back at your second point which FreeNET doesn't really have a way to deal with either, so it ends up people trading urls on message boards essentially.

andrewstuart2 11y ago

Well, you're right that it requires centralization, but at least the amount of data you need to retrieve centrally is that much smaller. It would benefit the global infrastructure greatly if I could retrieve websites over LAN if they'd been cached by a computer there. My wife and I visit a lot of the same sites, even see a lot of the same dynamic content (same facebook friends, etc). Surely there's some community effect that would make this a huge bandwidth win.

nothrabannosir 11y ago
Careful: this gets you dangerously close to privacy violations. You may not want certain people to know that you are fetching a piece of content. "Hmm, the profile picture for this specific friend loaded suspiciously quickly.." A similar mistake is security updates over bittorrent: a real-time list of vulnerable hosts.
- andrewstuart2 11y ago
  Argh. Privacy really is such a double-edged sword. You could still keep those bits off of the WAN, though. In a "privacy mode" the gateway could coordinate and throw in an average delay.

nbadg 11y ago

I agree that hash-based addressing creates a bit of a chicken-and-the-egg problem for content discovery, with the exception of, for example, existing links or QR codes. You'll always need some way of looking up the hash for the content you're hoping to find. But that lookup mechanism need not be a single centralized service. In fact, I would make the argument that in many cases, search engines are doing exactly this right now: you have several coexisting, competing, but centralized services, and a user is free to choose between them. It's centralization, sure, but in a much more robust way (though it admittedly could be significantly improved through, for example, a common search API). The only difference is that contemporary search engines are looking up URLs instead of hashes, but given how we're using URLs these days (ex: https://imgur.com/iauyhdf https://imgur.com/iauyhdf), I don't think it's a big distinction. You're also spot-on that dynamic content is the hardest part of hash-based addressing, but the problem is a bit more nuanced IMO. On the one hand, all content, once created, is static. If a thermometer reads 32.8 at 15:40 on 30 July 2015, that data point is fixed in time, immutable. However, we humans think of things far more conceptually, and we build cognitive connections between things. All content is static, but all concepts are dynamic. So the question is then, how do you reconcile those two? The best answer I've personally come across is bindings -- exactly analogous to binding names to variables in Python. Some hash bindings would be static, some bindings dynamic. That also allows you to construct more complicated objects like buffers natively. That's how I'm doing it in the project I'm working on (https://github.com/Muterra/doc-muse https://github.com/Muterra/doc-muse), though the bindings are a relatively new addition that I haven't had time to test yet (or update the documentation, for that matter).

dreamfactory2 11y ago

What about distributed metadata dictionaries/taxonomies describing the content and syncing via something like pubsubhubbub?

nly 11y ago

I think hashed addressing is a lot easier now mobile has increased the appeal of less typing. A user doesn't care whether a QR code contains a URL or a hash.

chmike 11y ago

A hash based distributed database doesn't allow to benefit from the locality effect where user access locally produced data more frequently. There is also the data hosting cost. It seams more fair and logical that data producers pays for the hosting of their data. Otherwise there is a risk of abuse.