5 ms·

"the global database can store all functions ever created, without worrying about anyone stepping on anyone else's toes" I've had similar thoughts in the past,

"the global database can store all functions ever created, without worrying about anyone stepping on anyone else's toes" I've had similar thoughts in the past, though my thought was as a function loader for javascript. Why not just experiment with this as library for javascript functions? Downloading the code from the "global database" would be provided by a function: getDataByHash(hash) : Blob To load the code into the return use eval (blech), or back up a step and use script tag + an advanced version of magnet links I'd call snow links: <script src="snow:?xt=urn:sha2:beaaca..."></script> There will have to be a dom-scanning/watching js library that the pages load from a normal url which scans the page for snow links, downloads them by calling getDataByHash(), and then swaps in new URLs using URL.createObjectURL(Blob). This leads me in a somewhat orthogonal direction to your post... Obviously any element in the page that loads resources would benefit from this (link rel=stylesheet, img, iframe, etc) so the document-scanner should work it's magic on them as well. To me the tricky part becomes "the global database." There are several ways to implement it. My thought would be to build it as a DHT on top of Web-RTC. I'd look into webtorrent, it has to scale from very small to very large files. Maybe have multiple different DHTs that the scanner will try. Storing stuff in the DHT ought to be as simple as declaring to the network that data (or solution) of a given hash is known. Clients p2p connect over to the knower (or daisy chain style as a sort of ad-hoc STUN/TURN setup?), and virally spread the data by declaring they too now have the solution for the hash they just downloaded. A CDN can still store a file and can be listed as a fallback provider in the snow url. As an example application: Once this DHT exists and p2p is built on top of it should not be hard to ask peers to run the content of a given SHA and return the result (or they will return it's SHA if it's large, effectively memo-izing the function). Something like: run(hash) : RunOffer Any peer (AWS or Google etc could implement a peer as well) could respond to that with a offer to run the computation for an amount of p2p currency that scales to billions or trillions of tiny transactions per second. ... Anyhow this comment is way too long already, but the ideas keep flowing from here. There are a lot of technical challenges for vital features (like how to register (im)mutable alias for hashes and distribute them to the network without using DNS for authoritative top-level namespacing? how get reliable redundancy guarantees for stored data without resorting to a CDN? What to do about realtime steaming of data where data is produced by an ongoing process? grouping multiple devices for the same user?) All-in-all it seems like being able to get data by the hash of its content from inside of a program (including library code) as easily as loading it from a URL or off the filesystem is pretty useful. It also seems like we can engineer the technology to make this happen, so I think it's inevitable and will happen pretty soon.

derefr 12y ago
> Clients p2p connect over to the knower (or daisy chain style as a sort of ad-hoc STUN/TURN setup?), and virally spread the data by declaring they too now have the solution for the hash they just downloaded. This is, in fact, exactly what Freenet does! It's a global DHT acting as a content-addressible store, where there's no "direct me to X" protocol function, only a "proxy my request for X to whoever you suspect has X, and cache X for yourself when returning it to me" function. (Aside: Freenet also does Tor-like onion-routing stuff between the DHT nodes, which makes it extremely slow to the point that it was discarded by most as a solution for anonymous mesh-networking. But it doesn't have to do that. "Caching forwarding-proxy DHT" and "encrypted onion-routing message transport" are completely orthogonal ideas, which could be stacked, or used separately, on a case by case basis. "Caching forward-proxying DHT" is to the IP layer as "encrypted onion-routing message transport" is to TLS. PersistentTor-over-PlaintextFreenet would be much better than either Tor or Freenet as they are today.) I agree that this could totally be done as a library in pretty much any language that allows for runtime module loading or runtime code evaluation. And, like you say, there are tons of interesting corollaries. I think, though, that to be truly useful, you have to push this sort of idea as low in the stack as possible. Unix established the "file descriptor" metaphor: a seekable stream of blocks-of-octets, some of which (files) existed persistently on disk, some of which (pipes) existed ephemerally between processes and only required resources proportional to the unconsumed buffer, some of which (sockets) existed between processes on entirely separate hosts. Everything in Unix is a file. (Or could be, at least. Unix programmers get "expose this as a file descriptor" right at about the same rate web API designers get REST right.) Unix (and most descendants) are "built on" files. To truly expose the power of a global code-sharing system, you'd need the OS to be "built on" global-DHT-dereferenceable URNs. There would be cryptographic hashes everywhere in the system-call API. Because your data likely wouldn't just be on your computer (just cached there), you'd see cryptographic signatures instead of ownership, and encryption where a regular OS would use ACL-based policies. At about the same level of importance that Unix treats "directories", such an OS would have to treat the concept of a data equivalence-mapping manifests (telling you how to get git objects from packfiles, a movie from a list of torrent chunk hashes, etc.) For any sort of collaborative manipulation of shared state, you'd probably see blocks from little private cryptocurrency block-chains floating about in the DHT, where the "currency" just represents CPU cycles the nodes are willing to donate to one-another's work. And then (like in Plan9's Fossil), you might see regular filesystems and such built on top of this. But they'd only be metaphors—your files would be "on" your disk to about the same degree that a process's stack is "in" L1 cache. Disks would really just be another layer of the memory hierarchy, entirely page-cache; and marking a memory page "required to be persisted" would shove it out to the DHT, not to disk, since your disk isn't highly available. —but. Designing this as an OS for PCs would be silly. It would have much too far to go to catch up with things like Windows and OSX. Much better to design it to run in userspace, or as a Xen image, or on a rump kernel on server hardware somewhere. It'd be an OS for cloud computers, or to be emulated in a background service on a PC and spoken with over a socket, silently installed as a dependency by some shiny new GUI app. Which is, of course, the niches Erlang already fits into. So see above :)

derefr 12y ago

> Clients p2p connect over to the knower (or daisy chain style as a sort of ad-hoc STUN/TURN setup?), and virally spread the data by declaring they too now have the solution for the hash they just downloaded. This is, in fact, exactly what Freenet does! It's a global DHT acting as a content-addressible store, where there's no "direct me to X" protocol function, only a "proxy my request for X to whoever you suspect has X, and cache X for yourself when returning it to me" function. (Aside: Freenet also does Tor-like onion-routing stuff between the DHT nodes, which makes it extremely slow to the point that it was discarded by most as a solution for anonymous mesh-networking. But it doesn't have to do that. "Caching forwarding-proxy DHT" and "encrypted onion-routing message transport" are completely orthogonal ideas, which could be stacked, or used separately, on a case by case basis. "Caching forward-proxying DHT" is to the IP layer as "encrypted onion-routing message transport" is to TLS. PersistentTor-over-PlaintextFreenet would be much better than either Tor or Freenet as they are today.) I agree that this could totally be done as a library in pretty much any language that allows for runtime module loading or runtime code evaluation. And, like you say, there are tons of interesting corollaries. I think, though, that to be truly useful, you have to push this sort of idea as low in the stack as possible. Unix established the "file descriptor" metaphor: a seekable stream of blocks-of-octets, some of which (files) existed persistently on disk, some of which (pipes) existed ephemerally between processes and only required resources proportional to the unconsumed buffer, some of which (sockets) existed between processes on entirely separate hosts. Everything in Unix is a file. (Or could be, at least. Unix programmers get "expose this as a file descriptor" right at about the same rate web API designers get REST right.) Unix (and most descendants) are "built on" files. To truly expose the power of a global code-sharing system, you'd need the OS to be "built on" global-DHT-dereferenceable URNs. There would be cryptographic hashes everywhere in the system-call API. Because your data likely wouldn't just be on your computer (just cached there), you'd see cryptographic signatures instead of ownership, and encryption where a regular OS would use ACL-based policies. At about the same level of importance that Unix treats "directories", such an OS would have to treat the concept of a data equivalence-mapping manifests (telling you how to get git objects from packfiles, a movie from a list of torrent chunk hashes, etc.) For any sort of collaborative manipulation of shared state, you'd probably see blocks from little private cryptocurrency block-chains floating about in the DHT, where the "currency" just represents CPU cycles the nodes are willing to donate to one-another's work. And then (like in Plan9's Fossil), you might see regular filesystems and such built on top of this. But they'd only be metaphors—your files would be "on" your disk to about the same degree that a process's stack is "in" L1 cache. Disks would really just be another layer of the memory hierarchy, entirely page-cache; and marking a memory page "required to be persisted" would shove it out to the DHT, not to disk, since your disk isn't highly available. —but. Designing this as an OS for PCs would be silly. It would have much too far to go to catch up with things like Windows and OSX. Much better to design it to run in userspace, or as a Xen image, or on a rump kernel on server hardware somewhere. It'd be an OS for cloud computers, or to be emulated in a background service on a PC and spoken with over a socket, silently installed as a dependency by some shiny new GUI app. Which is, of course, the niches Erlang already fits into. So see above :)