10 ms·
“Xanadu Hypertext Documents” architecture and data structures, 2019 edition
- AlbertoGP 5y agoI was asked to resubmit this new edition that I announced in HN as it got finished on 2019-04-25: https://news.ycombinator.com/item?id=19745517 https://news.ycombinator.com/item?id=19745517 The process took around a month in my spare time, and involved copy-and-pasting the OCR plain text, then marked up as DocBook XML, then converted with my own XSLT stylesheets into HTML5. There were many transcription mistakes, some from OCR from the 35 year old daisy-printer monospace printout, some were in that printout because of electrical noise in the printer serial line in 1986. I fixed those I found by hand, then a couple of readers have reported others. Since then I’ve corrected more OCR errors, and polished a bit more the HTML rendition. From the introduction to this edition: “It features an instance of Douglas Engelbart’s implicit links, automatically linking all appearances of certain terms to their definitions elsewhere in this document. This is done through a Javascript function that operates on the HTML content from the web server. It also shows his structural statement numbers, those little labels to the right of each paragraph that allow high resolution linking.” Doug Engelbart was not part of Xanadu, although he was a close friend of Ted Nelson for decades. It’s just that those aspects of his NLS (of 1968 “Mother of all demos” fame) fit well this document, and work relatively well on web browsers. I did try to implement transclusion as an example, but the target web server did not provide a CORS header and browsers refuse to load their content. This document goes into detail about the addressing scheme for documents (https://sentido-labs.com/en/library/201904240732/Xanadu%20Hypertext%20Documents.html#tumblers-and-humbers https://sentido-labs.com/en/library/201904240732/Xanadu%20Hy...), the enfilade data structure (https://sentido-labs.com/en/library/201904240732/Xanadu%20Hypertext%20Documents.html#enfilade https://sentido-labs.com/en/library/201904240732/Xanadu%20Hy...), their then-future plans, etc.
- 0des 5y ago> “It features an instance of Douglas Engelbart’s implicit links, automatically linking all appearances of certain terms to their definitions elsewhere in this document. This is done through a Javascript function that operates on the HTML content from the web server. It also shows his structural statement numbers, those little labels to the right of each paragraph that allow high resolution linking.” Yo this is really cool! I'm glad you came back to update us.
- tomcam 5y agoWow. Thank you very much.
- jimmySixDOF 5y ago>did not provide a CORS header If you only have a specific target your dealing with then maybe handle it by proxy on your own server: [1] https://github.com/Rob--W/cors-anywhere https://github.com/Rob--W/cors-anywhere
- AlbertoGP 5y agoYou are right, and with a server-side component you could fetch arbitrary content, the endpoint could look something like example.com/transclude/a.b.com/section/page?path=2a14n which would make the server download a.b.com/section/page, extract the HTML element which is the 13th ('n') child of the 14th child of the first ('a') child of the second child of the root element, and return it. I’m sure there are implementations of this already.
- jimmySixDOF 5y agohummmm
- kragen 5y agoBTW I've found the codebase in udanax-1999-09-29.tar.gz to be a fairly illuminating complement to this document. It would be pretty hard to read without some of the context in this proposal, but it adds a lot more detail.
- jdougan 5y agoBesides looking at the green codebase, have you found any good expositions on implementing multi-dimensional enfilades?
- kragen 5y agoMarkM lectured me about it one afternoon for an hour or two but we didn't record it. But I haven't actually tried implementing the Ent or other enfilade data structures, so even my perception that the lecture was very informative is suspect.
- jdougan 5y agoThe story of Xanadu, in a nutshell.
- Animats 5y agoIt's very clever, but solves the wrong problem. It's basically doing what a revision control system like Git does, but for a different reason. Everything is pay per view in Xanadu. It's micropayments to the max. Everybody who contributed to the article gets a cut, in proportion to how much content they contributed. So there's all that tracking. Also, pictures? What are those? Audio and video? Don't even think about it. It's an elaborately optimized design that optimizes for the wrong thing. It belongs to the world in which Mead Data Systems and MEDLINE and NEXIS and LEXIS, all pay to search and read systems, dominated information access. If Elsevier had designed the Web, it would work like this. I knew that crowd back when they were doing this. Great people, but too many were of the "everything should be a market" persuasion.
- mattkevan 5y agoThey just need to use some weird crypto token for payments and Web3 will be all over it.
- Meandering 5y ago"The computer world is not just technicality and razzle-dazzle. It is a continual war over software politics and paradigms. With ideas which are still radical, WE FIGHT ON. We hope for vindication, the last laugh, and recognition as an additional standard-- electronic documents with visible connections." - https://xanadu.com/ https://xanadu.com/ Looks like they would fit right into the crypto world..
- AceJohnny2 5y agoPit it against the John Perry Barlowe's EFF Manifesto: https://www.eff.org/cyberspace-independence https://www.eff.org/cyberspace-independence "[...] Cyberspace consists of transactions, relationships, and thought itself, arrayed like a standing wave in the web of our communications. Ours is a world that is both everywhere and nowhere, but it is not where bodies live. We are creating a world that all may enter without privilege or prejudice accorded by race, economic power, military force, or station of birth. We are creating a world where anyone, anywhere may express his or her beliefs, no matter how singular, without fear of being coerced into silence or conformity. Your legal concepts of property, expression, identity, movement, and context do not apply to us. They are all based on matter, and there is no matter here. [...]" RIP
- deleted 5y ago[deleted]
- adamwong246 5y ago``` The current Xanadu backend is constructed using three essential data structures: - granfilade, used to implement the grandmap - poomfilade, used to implement orgls - spanfilade, used to implement the spanmap ``` um, are the folks over at sentido labs ok?
- deleted 5y ago[deleted]
- nineteen999 5y agoThese structures sound like they were named by programmers fleeing from Urbit.
- deleted 5y ago[deleted]
- tomcam 5y agoI will never, ever again complain about the density of a W3C RFP: > The index space used by the granfilade is I-stream tumbler space. The wid of a granfilade crum is a tumbler specifying the span of I-space beneath the crum (i.e., the distance, in tumbler space, from the first to the last bottom crum descended from it). The widdative function is tumbler addition, therefore a crum’s wid is simply the tumbler sum of its children’s wids. Bottom crums have an implicit wid of 0.0.0.0.1 (i.e., spanning no nodes, no accounts, no orgls, no V-spaces and spanning a single atom). Granfilade disps are tumbler offsets in I-space from the parent crum. Somehow I feel like Buckminster Fuller could have written this
- lehi 5y agoJargon game: "Urbit or Xanadu?"
- kragen 5y agoIt might be the case that you could have invented a new kind of publishing without making up a bunch of new jargon to name the concepts required to make it work, but the WWW definitely does not represent a proof that this is possible. Quoting from https://dev.w3.org/html5/spec-LC/elements.html#classes https://dev.w3.org/html5/spec-LC/elements.html#classes: > The style IDL attribute must return a CSSStyleDeclaration whose value represents the declarations specified in the attribute, if present. Mutating the CSSStyleDeclaration object must create a style attribute on the element (if there isn't one already) and then change its value to be a value representing the serialized form of the CSSStyleDeclaration object. The same object must be returned each time. [CSSOM] The WWW-specific jargon here ("IDL", "attribute", "CSSStyleDeclaration", "declarations", "element", and arguably even "object" and "mutate") is only slightly less dense than the bit you quoted. The difference is that, because we've all spent the last quarter century embedded in the world it describes, its vocabulary is familiar to us. On the other hand, because Xanadu failed, its vocabulary is unfamiliar to us, so this passage seems opaque, even where it's doing things like helpfully restating what role a widdative function plays in generalized enfilade theory in case we've forgotten, or restating what the parts of a tumbler are for the same reason. Incidentally, if the W3C spec you're looking at is ECMAScript, some of the authors are the same as the authors of the Xanadu spec. — ⁂ — What's being described in the passage you quoted is a tree structure over a sort of space of strings called "I-space" that is used to name documents or parts of them, similar to how URLs are used in WWW. The strings (tumblers) consist of sequences of nonnegative integers, rather than sequences of letters drawn from some finite alphabet like ASCII or Unicode. Tumblers have a lexicographical ordering on them: 1.2.3.4.5 comes after 1.2.3.4.4 or 1.2.3.3.6089235802, but before 1.2.3.4.6 or 2.0.0.0.0. The tree is sorted by this lexicographical ordering, so in some sense all the data in the granfilade forms one single very long string, with tumblers like 1.2 or 1.2.3.4.5 identifying contiguous subsets of it. But this is not an advantageous way to organize the tree for efficient access, because 1.2.3.4 might be enormous compared to 1.2.3.3 or 1.2.3.5; instead the storage tree is made out of nodes that are about the same size at a given level of the tree, which are called crums. A crum might include all the data from 1.2.3.4.5 to 1.2.3.4.17 (exclusive); by subtracting these two tumblers we get its width ("wid"), which is 0.0.0.0.12, which means that it spans 12 leafnodes ("bottom crums"). If that crum and another crum that covers 1.2.3.4.17 to 1.2.3.4.25 (wid = 0.0.0.0.8) are the two children of a single parent node ("parent crum"), then the parent crum has a wid of 0.0.0.0.12 + 0.0.0.0.8 = 0.0.0.0.20. This leaves us with the question of how to efficiently store this range "1.2.3.4.17 to 1.2.3.4.25". We could of course store it as a 12-byte sequence: 01 02 03 04 11 ff 01 02 03 04 19 ff or something like that. But there is obvious redundancy here, particularly considering the context of walking down the tree ("granfilade") from the parent crum which covers "1.2.3.4.5 to 1.2.3.4.25"; it's more efficient to just store the crum's wid 0.0.0.0.8, as something like 04 08, and its displacement ("disp") relative to the parent crum's starting point, which is 1.2.3.4.17 - 1.2.3.4.5 = 0.0.0.0.12, which is simply the tumbler sum of the wids of its preceding siblings. This representation not only permits us to use less bytes, it also doesn't need to be modified if we graft a whole subtree from one part of the granfilade to another, or even (and I forget if they did this!) transclude it. And it allows us to build a nicely balanced sequentially-written B-tree of crums that guarantees logarithmic-time access to any atom, permits guaranteed crash recovery and even rollback, and doesn't have too much write amplification for localized writes. It doesn't handle large batches of dispersed writes as nicely as a log-structured merge tree does, but I don't think LSM-trees were known yet when they designed the granfilade. Hopefully that clears things up a bit. I don't really understand Xanadu but I have spent a little time digging into it to try to figure it out. — ⁂ — In general enfilade theory there's the possibility of arbitrary sorts of "wids" and "disps" that are computed bottom-up and top-down in the same way: the wid of a node is computed by its widdative function from the wids of its child nodes, while the disp of a node is combined with the effective disp of its parent node to get its effective disp. For example, if your wids and disps are numbers of bytes, you get a rope that efficiently supports indexing by byte offsets. And if you use ordinary vector addition as the dispative function on a tree of nested windows, using the (x, y) offset of each window relative to its parent window as the disp, you can compute the absolute pixel offset for each window. You can almost do hbox/vbox layout this way, too: your wids are (width, height, orientation) triples, and your widdative function is something like (x₁, y₁, H) ⍤ (x₂, y₂, H) = (x₁ + x₂, y₁ ∨ y₂, H) (x₁, y₁, V) ⍤ (x₂, y₂, V) = (x₁ ∨ x₂, y₁ + y₂, V) where ∨ is the pairwise max function; but at some point you need to turn a stack of hboxes into a vbox or vice versa, which either needs an N-ary widdative function (rather than binary) or a postprocessing function used once per node to flip the orientation. As this indicates, the Xanadu folks were very much in love with the mathematical approach to software that seeks maximally general solutions to everything, and I think sometimes that didn't serve them well. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.451.5876&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.45... (Nelson 01999) may be interesting reading.
- kwatsonafter 5y agoI spent a few years trying to build a Xanadu-two-way-hyperlinks-but-with-Javascript kind of a thing and I eventually started to wonder, like, if you watch a lot of Ted's videos he's got this idea of a kind of potential for what computers could be like-- he often makes reference to this experience he had as a kid that motivated his work-- it kind of seems to me that with what's possible today you could make something that goes far beyond either World Wide Web or Xanadu or anything really conceived yet-- something so fluid and adaptable and humane that it becomes an real extension of the fluid way we think with all of the advantages that come from it being a, "computer interface" (ability to recall past states, discrete calculation, ect) The real question is-- given the enormous economic potential for the creation of, "next generation" computers why isn't that we can't create groups like SRI, Xerox PARC, ARPA (..) in the modern age to create modern, "Engelbartesque" problem solving machines? Why is it that people like Ted Nelson and Douglas Engelbar have issues having their ideas realized when by comparison every dogshit computer company in the world has had a metaphorical step up to plate in terms of potential for investment in broad horizon R&D projects? Apple has enough money to, "Manhattan Project" big breakthroughs (so does Google, Microsoft, Meta...) so why don't they? What holds back, "real" innovation? Why the hell can't I buy a Apple Nelson-Engelbartron at Wal-Mart? What gives Tim Cook? It seems like whatever kind of, "held Xanadu back" is the same thing that held back the Engelbart stuff which held back the Alan Kay/Dan Ingalls Scratchland/E-Toys stuff which relatedly holds back the Seymour Papert/LOGO stuff. It seems like neoliberalism creates a kind of, "censorship by appeal to market forces" where there's so much demand for, "meh" (simulations of paper documents/office processes and video games which didn't exist commercially until the passing of the Mansfield Amendment; neat coincidence) that, "big ideas" aren't able to catch on in the nascent culture. If you look at this historically it's really rather unexceptional-- it's generally a small percentage of the total population that is, "highly literate" and this doesn't seem to be any different in the age of personal computers despite the lofty ideals that surround open source and the World Wide Web. Until you can build an, "adult Dynabook" which solves social inequality it seems like it's going to stay the same. Panem et circenses. https://en.wikipedia.org/wiki/Bread_and_circuses https://en.wikipedia.org/wiki/Bread_and_circuses "You program computers for a living and you think this is the best design for universal problem solving machines? Do you have no imagination? No grasp of the history of these machines?!"
- 5350-uiop-1130 5y agoI recommend this documentary "Lo and Behold, Reveries of the Connected World" from Werner Herzog. There is a brief but touching interview with Ted Nelson.