3 ms·

Pre-teen me was utterly fascinated with Xanadu, but that was before I learned about the computational and storage costs of data structures, and how large-scale

Pre-teen me was utterly fascinated with Xanadu, but that was before I learned about the computational and storage costs of data structures, and how large-scale human interaction with systems evolves. Now, I can't wrap my head around what Nelson promotes and what I extrapolate to a scaled-up global system. I'd really like to know from those more deeply immersed in the implementation of his vision how the following pieces are addressed, because I can't quite reconcile between the dribs and drabs I've pieced together of the internals and what I know of large-scale systems. 1. As near as I can tell, the ZigZag data structure at its core is a type of directed graph. Fast path traversal has always been a challenge for me with these at scale. When I want to pull a list of all referring/predecessor links, I'm either walking the graph, or maintaining a continuously-updated cross-reference lookup. I must be overlooking how he's solved this problem, because the kind of micro-payment environment he envisions seems to me to require knowing this information (is this what he calls a "cell"?) to compute the presented price of a piece of information derived from a foundation of other pieces of information. 2. How does transclusion work for non-text data like audio, video, rich media, chats, or in a generalized way, arbitrary data sets? 3. Was there ever a notion of transclusion quality or security? I don't see any acknowledgement in Nelson's design of the need to address bad actors abusing transclusion, giving rise to equivalents to spammers, trolls, phishers, etc. within the xanadocs environment. 4. What determines a cell? As near as I can tell (and my understanding is admittedly flawed), cells are decided by the author, but I must be wrong here. That seems a high cognitive load to make those decisions, and I don't see any support for crowd-defined cells, nor an API that would facilitate building such an interaction. BTW, for those who lament the quiescence of the Gzz [1] project due to patent protection, it can pick up again now that the patent expired [2]. Whoever picks it up likely should read about the experiences of one of the original Gzz authors [3]. The most programmer-oriented write up of the data structure I could find [4] didn't erase my impression that Nelson's vision was fine for a small group, but doesn't address scaling up to the challenges we see on a global level. It's still fascinating to think about, and I see value within a corporate environment, but I struggle seeing it staying useful in the public realm due to the this-is-why-we-can't-have-nice-things principle. [1] http://www.nongnu.org/gzz/ http://www.nongnu.org/gzz/ [2] https://en.wikipedia.org/wiki/ZigZag_(software)#History https://en.wikipedia.org/wiki/ZigZag_(software)#History [3] http://lambda-the-ultimate.org/node/233#comment-1715 http://lambda-the-ultimate.org/node/233#comment-1715 [4] https://hackernoon.com/an-engineers-guide-to-the-docuverse-d080cdbb73a6 https://hackernoon.com/an-engineers-guide-to-the-docuverse-d...

enkiv2 7y ago
I am the author of [4]. Regarding your first question: ZigZag is not really intended for large shared multi-user stuff. It's not supposed to scale huge. We have completely different data structures for translit (i.e., the hypertext part). ZigZag can be used for fast manipulation of spacial relations (basically, applying multiple kinds of projections to a multi-dimensional directed graph) & we use it under the hood for representing display layout in XanaSpace, but its main use is as a personal mind-mapping utility. It has only extremely tenuous connections to how we do hypertext, and none at all to micropayments. The transcopyright system (i.e., the thing for ownership and micropayments) is based on transclusion. Basically, all that means is that we provide facilities to fetch things piecemeal, and we provide facilities for owners to restrict control over getting usable forms of things. (Specifically, we support applying a one time pad, so that if you've got distributed fetching of real content, you can still require folks to fetch the OTP from a trusted oracle & maybe pay for it too before they can actually decode the data.) Question 2: We can always fall-back to byte offsets. We floated the idea of supporting more user-friendly offset formats for audio & video (specifically adopting ideas from the w3c media fragments spec), specifically because audio & video have complicated compression schemes & finding safe frame boundaries requires deep knowledge of the underlying format. We put that on hold in order to focus on getting text stuff really solid in the UI department. Question 3: People can create spammy documents. They can also transclude from or link to more popular documents in the hopes that their low-quality documents will get more exposure. But, links are not universally resident (which is to say, creating a link to a document doesn't mean that everybody viewing that document immediately sees it). Instead, links get distributed the same way as documents do: by word of mouth (as people recommend to their friends to view particular documents or load up particular sets of links). Links do double-duty as connections between documents and as formatting information (like themes or skins), and collections of links (ODLs) can be passed around separately from documents. (By 'passed around' I generally mean that their addresses get circulated, although we also had the notion of an archive format for passing around collections of documents and links that weren't to be made generally available.) In other words, the spam potential of linking between a popular document and a low-quality document is akin to the spam potential of creating a low-quality website that is hidden from google. Question 4: A cell is a data structure consisting of a pair of associative arrays (pointers to adjacent cells, keyed by strings) & a 'value' (typically a string). Cells are created by the author (or maybe by a program), which is fine, because cells never travel beyond the vicinity of a single machine. (We had a 'slice' mechanism where cells owned by different folks could join up together, but this was never intended to scale beyond small groups.) Again: since ZigZag is not a part of the hypertext/transliterature system, the performance of ZigZag is a lot less important. (Within the translit realm, we had exotic high-performance data structures during the Autodesk / XOC era, which we've basically dropped in favor of simple stuff like spanpointers.) If you're interested in ZigZag, I recommend reading my implementation (https://github.com/enkiv2/misc/blob/master/ds-lib/ZZCell.py https://github.com/enkiv2/misc/blob/master/ds-lib/ZZCell.py). It's cleaner than GZZ & is almost identical to what's still used internally at Xanadu (last I've heard), because this is a rewrite from memory of what we wrote for XanaSpace & when Ted occasionally sends people to me for advice on how to implement new versions of ZigZag I point them to this.

enkiv2 7y ago

I am the author of [4]. Regarding your first question: ZigZag is not really intended for large shared multi-user stuff. It's not supposed to scale huge. We have completely different data structures for translit (i.e., the hypertext part). ZigZag can be used for fast manipulation of spacial relations (basically, applying multiple kinds of projections to a multi-dimensional directed graph) & we use it under the hood for representing display layout in XanaSpace, but its main use is as a personal mind-mapping utility. It has only extremely tenuous connections to how we do hypertext, and none at all to micropayments. The transcopyright system (i.e., the thing for ownership and micropayments) is based on transclusion. Basically, all that means is that we provide facilities to fetch things piecemeal, and we provide facilities for owners to restrict control over getting usable forms of things. (Specifically, we support applying a one time pad, so that if you've got distributed fetching of real content, you can still require folks to fetch the OTP from a trusted oracle & maybe pay for it too before they can actually decode the data.) Question 2: We can always fall-back to byte offsets. We floated the idea of supporting more user-friendly offset formats for audio & video (specifically adopting ideas from the w3c media fragments spec), specifically because audio & video have complicated compression schemes & finding safe frame boundaries requires deep knowledge of the underlying format. We put that on hold in order to focus on getting text stuff really solid in the UI department. Question 3: People can create spammy documents. They can also transclude from or link to more popular documents in the hopes that their low-quality documents will get more exposure. But, links are not universally resident (which is to say, creating a link to a document doesn't mean that everybody viewing that document immediately sees it). Instead, links get distributed the same way as documents do: by word of mouth (as people recommend to their friends to view particular documents or load up particular sets of links). Links do double-duty as connections between documents and as formatting information (like themes or skins), and collections of links (ODLs) can be passed around separately from documents. (By 'passed around' I generally mean that their addresses get circulated, although we also had the notion of an archive format for passing around collections of documents and links that weren't to be made generally available.) In other words, the spam potential of linking between a popular document and a low-quality document is akin to the spam potential of creating a low-quality website that is hidden from google. Question 4: A cell is a data structure consisting of a pair of associative arrays (pointers to adjacent cells, keyed by strings) & a 'value' (typically a string). Cells are created by the author (or maybe by a program), which is fine, because cells never travel beyond the vicinity of a single machine. (We had a 'slice' mechanism where cells owned by different folks could join up together, but this was never intended to scale beyond small groups.) Again: since ZigZag is not a part of the hypertext/transliterature system, the performance of ZigZag is a lot less important. (Within the translit realm, we had exotic high-performance data structures during the Autodesk / XOC era, which we've basically dropped in favor of simple stuff like spanpointers.) If you're interested in ZigZag, I recommend reading my implementation (https://github.com/enkiv2/misc/blob/master/ds-lib/ZZCell.py https://github.com/enkiv2/misc/blob/master/ds-lib/ZZCell.py). It's cleaner than GZZ & is almost identical to what's still used internally at Xanadu (last I've heard), because this is a rewrite from memory of what we wrote for XanaSpace & when Ted occasionally sends people to me for advice on how to implement new versions of ZigZag I point them to this.