4 ms·

I don't think the computer performance came up in the design of this protocol. After all, one of the foundations of it is _immutable_, append only data structur

by theamk 5y ago

I don't think the computer performance came up in the design of this protocol. After all, one of the foundations of it is _immutable_, append only data structure where all the information is stored (I-stream). Surely it was clear even in 1984 that this can not be achieved with the computers of the day? And if you are postulating unlimited storage, why not put videos there as well?

kragen 5y ago
This document is not a protocol design; it's a data store design. The protocol, Phoebe, is mentioned briefly in §5.2: https://sentido-labs.com/en/library/201904240732/Xanadu%20Hypertext%20Documents.html#implementation-frontend-backend-interface https://sentido-labs.com/en/library/201904240732/Xanadu%20Hy... and §7.4: https://sentido-labs.com/en/library/201904240732/Xanadu%20Hypertext%20Documents.html#future-phoebe https://sentido-labs.com/en/library/201904240732/Xanadu%20Hy... but had not really been thought through yet. You can read the implementations of the client and server in the Udanax Green code I mentioned. The primary concern of this architecture document from beginning to end is how to get adequate performance out of the computers of the day, given the features they considered to be fundamental. BTW, as I understand it, the I-stream isn't the append-only data structure; the granfilade is. They contain mostly the same information, but in different orders, so that you can insert into the I-stream by appending to the granfilade. I don't think it's true that forever storing all the text the system's users would write was infeasible in 01984; the revenues expected from readers would have paid orders of magnitude more than the cost of the space required, even in 01984. A book might be 1 megabyte; you might write another megabyte or two in the process of writing it. Someone writing a book in Xanadu — a level of success the system never in fact achieved — would have to pay to store those 2–3 megabytes forever, perhaps copying them onto a new computer every 6 years, at ⅛ the cost of the previous computer. How much might that cost? I don't have good pricing data from 01984, but I do have here the Sun price list from December 01988; a Sun-3/100 at that time contained a 16.67-MHz 68020, 4–12 MiB of RAM, and if you got it with a disk, it would be a model 514D 141-MiB disk. The 3/140M-4-P5 configuration, with a 19" monitor, cost US$23700. 3 MB of text would have been about 2% of the machine's capacity, and so would have cost about US$237 in 01988. Four years earlier it probably would have cost about US$1000. Then it would have cost another US$125 for new storage in about 01990, US$16 in 01996, US$2 in 02002, etc., a total of a bit under US$1145. If you amortized the initial investment of the space for the book over the same 6 years, it would be about US$190 per year, so the system would be immediately profitable for any book that was expected to produce more than about US$190 of revenue per year across all its readers. A book with 100 readers per year, each paying US$10 to read the book, would earn five times the cost of its storage space. Data transmission was plenty fast enough; people can only read at about 300 baud, or 150 baud with huffman coding. The Hayes Smartmodem 300 was introduced in 01981, and in 01984 1200-baud modems were common and 2400-baud modems were starting to appear. Then the problem would merely be to build data centers fast enough, stuff them full of fileservers, and somehow keep the whole wild system from collapsing in chaos. That wasn't the problem they had, though. The problem that they had was that their system design was so ambitious that they never shipped a working product. As for video, well, even with modern codecs a two-hour-long movie can gobble up over a gigabyte of storage space (US$300 000 at the price I estimated above), and people were paying US$1 for a matinée movie ticket and US$0 to watch pan-scan-converted movies on late-night TV. There was actually a somewhat popular product a few years later to do data backup onto VCR tapes, called ArVid; it could get 4 gigabytes onto a 4-hour tape. (A few people were doing that starting in the early 01980s, but they were the people who had gigabytes of data, so there wasn't a mass-market product.) Although I was watching postage-stamp-sized MPEGs with mpeg_play_motif on SGI workstations in 01993, fullscreen digital video playback didn't become computationally feasible at all for mass-market hardware until the late 01990s; earlier systems like the Dragon's Lair arcade game overlaid the digital content on analog video backdrops. So to deliver text to customers in 01984 and store it forever, you needed 300-baud modems, which were already a mass consumer product, a data center with fileservers, a packet-switching network like Tymnet, and textual data that averaged at least a few readers per month. Tymnet had been a commercial offering since 01972, and this whole package is something companies like The Source, CompuServe, and (as Animats pointed out) the vile Mead Data Central were already doing. To deliver video without modern codecs, you would have needed four orders of magnitude more storage space in your data center, and you would have needed each video customer to either contract a T1 line to their premises or better, or also have the customers store the video data locally, perhaps on a VHS tape, so they could download it at slower than real time. With a 2400-baud modem you would be able to download a few seconds of video per day, though, so maybe you'd need to contract at least a 64-kbps frame relay link from the phone company, so you could download a minute of video per day. Then you could watch a single movie every other month.
- AlbertoGP 5y ago
  > This document is not a protocol design; it's a data store design. The protocol, Phoebe, is mentioned briefly in §5.2 and §7.4 There is a detailed description of it at: http://udanax.xanadu.com/green/febe/index.html http://udanax.xanadu.com/green/febe/index.html Edit: I’ve added this link in a footnote at the end of https://sentido-labs.com/en/library/201904240732/Xanadu%20Hypertext%20Documents.html#6d3 https://sentido-labs.com/en/library/201904240732/Xanadu%20Hy...
  - kragen 5y ago
    Oh thanks! I think this may make the code easier to understand.
- theamk 5y ago
  That's an interesting price analysis! The conclusion I draw from this is that Xanadu is quite a different usage model than the web. The first webpage I remember reading (over dial-up connection to university server) was physics experiments for beginners. The first webpage I made (in the end of 1990's) was to publish plays from the drama club. Those would be be totally infeasible under "every published document must earn money" model. I wonder if Xanadu could ever get completed, who would have used it? It will be probably be a Serious Place full of Professional Authors writing Real Books. Nothing like fun, random places of early internet [0] -- "a guide for backpackers and wilderness trekkers; [...]; a Grateful Dead fan page".... [0] https://www.theatlantic.com/technology/archive/2017/04/a-search-for-the-zombie-websites-of-1995/523848/ https://www.theatlantic.com/technology/archive/2017/04/a-sea...
  - kragen 5y ago
    It's probably true that it would have started with literature expected to have mass appeal ("Real Books") and only gradually moved downmarket. But let's not exaggerate! I'm guessing there were more than two people in the drama club, so it's likely that even your first webpage had more than two readers a month, which was the breakeven point I calculated above for extrapolated 01984 storage pricing. By my extrapolated 01990 pricing it would have been economical at three readers per year, and by my extrapolated 01996 pricing at one reader every 3 years. Since you called it a "webpage" and not a "BBS" or "FTP site" I'm guessing that this was more like 01994 than 01984, so you would have been fine. These numbers are no different for Xanadu and the WWW, it's just that on the WWW they are paid by the publisher (or their friend or university or something), and in Xanadu they would be paid by the users. All this is assuming the value to the reader is about US$3 per megabyte, a number I based on the idea that a book is about a megabyte, people commonly pay about US$10 for a book, and the edit history for the book might be three times the size of the final book. (Maybe people paid US$2.50 for a paperback, while a hardcover might be US$40. Roughly 01984 dollars.) If you were reading 40 hours a week at 350 words per minute at this speed, that works out to US$15 a week, US$66 a month, or US$789 a year. A more reasonable path is to figure that by 01994 people might pay US$20 a month for using the Xanadu service 40 hours a week, with half of that (US$10 a month) flowing to the author and hosting costs. That's only 46¢ per megabyte, but by 01994 storing a megabyte only cost about US$30 instead of US$1000, maybe US$5/year, so anything with more than one reader a month would pay for itself. And if it didn't, well, a play might be 100kB, and maybe the drama club would spend US$3 up front to pay a Xanadu operator to store those 100K for eternity, money they'd get back if people read the play, but if nobody ever read the play on Xanadu then that would just be an expense. I feel like US$3 is not a prohibitive expense for vanity-publishing a play and having it never go out of print. I mean xeroxing it at the library probably cost almost that much. Per copy! (In 01996, a single Moore's-Law doubling later, I bought my first computer. It had an 800MB hard disk and cost me about US$900, half of which was the monitor, and would have been a perfectly capable fileserver even though it was a cheap IBM PC clone instead of a Sun workstation. So my US$30/MB for 01994 might be too conservative. Maybe it would be more like US$3/MB. And by that time gzip and 14.4kbps modems were mainstream, dropping costs further.) Now, it's possible that authors (or the Xanadu operating company) would have been too greedy, and charged much higher prices than that, which would have had the result you describe. (There's some evidence that people did shift from things like CompuServe and AOL over to the WWW because publishing on the WWW was cheaper.) Or it's possible that they would have subsidized the initial wide dissemination of their work, as we in fact did with the WWW, so as to build up a customer base.

by theamk 5y ago

kragen 5y ago

This document is not a protocol design; it's a data store design. The protocol, Phoebe, is mentioned briefly in §5.2: https://sentido-labs.com/en/library/201904240732/Xanadu%20Hypertext%20Documents.html#implementation-frontend-backend-interface https://sentido-labs.com/en/library/201904240732/Xanadu%20Hy... and §7.4: https://sentido-labs.com/en/library/201904240732/Xanadu%20Hypertext%20Documents.html#future-phoebe https://sentido-labs.com/en/library/201904240732/Xanadu%20Hy... but had not really been thought through yet. You can read the implementations of the client and server in the Udanax Green code I mentioned. The primary concern of this architecture document from beginning to end is how to get adequate performance out of the computers of the day, given the features they considered to be fundamental. BTW, as I understand it, the I-stream isn't the append-only data structure; the granfilade is. They contain mostly the same information, but in different orders, so that you can insert into the I-stream by appending to the granfilade. I don't think it's true that forever storing all the text the system's users would write was infeasible in 01984; the revenues expected from readers would have paid orders of magnitude more than the cost of the space required, even in 01984. A book might be 1 megabyte; you might write another megabyte or two in the process of writing it. Someone writing a book in Xanadu — a level of success the system never in fact achieved — would have to pay to store those 2–3 megabytes forever, perhaps copying them onto a new computer every 6 years, at ⅛ the cost of the previous computer. How much might that cost? I don't have good pricing data from 01984, but I do have here the Sun price list from December 01988; a Sun-3/100 at that time contained a 16.67-MHz 68020, 4–12 MiB of RAM, and if you got it with a disk, it would be a model 514D 141-MiB disk. The 3/140M-4-P5 configuration, with a 19" monitor, cost US$23700. 3 MB of text would have been about 2% of the machine's capacity, and so would have cost about US$237 in 01988. Four years earlier it probably would have cost about US$1000. Then it would have cost another US$125 for new storage in about 01990, US$16 in 01996, US$2 in 02002, etc., a total of a bit under US$1145. If you amortized the initial investment of the space for the book over the same 6 years, it would be about US$190 per year, so the system would be immediately profitable for any book that was expected to produce more than about US$190 of revenue per year across all its readers. A book with 100 readers per year, each paying US$10 to read the book, would earn five times the cost of its storage space. Data transmission was plenty fast enough; people can only read at about 300 baud, or 150 baud with huffman coding. The Hayes Smartmodem 300 was introduced in 01981, and in 01984 1200-baud modems were common and 2400-baud modems were starting to appear. Then the problem would merely be to build data centers fast enough, stuff them full of fileservers, and somehow keep the whole wild system from collapsing in chaos. That wasn't the problem they had, though. The problem that they had was that their system design was so ambitious that they never shipped a working product. As for video, well, even with modern codecs a two-hour-long movie can gobble up over a gigabyte of storage space (US$300 000 at the price I estimated above), and people were paying US$1 for a matinée movie ticket and US$0 to watch pan-scan-converted movies on late-night TV. There was actually a somewhat popular product a few years later to do data backup onto VCR tapes, called ArVid; it could get 4 gigabytes onto a 4-hour tape. (A few people were doing that starting in the early 01980s, but they were the people who had gigabytes of data, so there wasn't a mass-market product.) Although I was watching postage-stamp-sized MPEGs with mpeg_play_motif on SGI workstations in 01993, fullscreen digital video playback didn't become computationally feasible at all for mass-market hardware until the late 01990s; earlier systems like the Dragon's Lair arcade game overlaid the digital content on analog video backdrops. So to deliver text to customers in 01984 and store it forever, you needed 300-baud modems, which were already a mass consumer product, a data center with fileservers, a packet-switching network like Tymnet, and textual data that averaged at least a few readers per month. Tymnet had been a commercial offering since 01972, and this whole package is something companies like The Source, CompuServe, and (as Animats pointed out) the vile Mead Data Central were already doing. To deliver video without modern codecs, you would have needed four orders of magnitude more storage space in your data center, and you would have needed each video customer to either contract a T1 line to their premises or better, or also have the customers store the video data locally, perhaps on a VHS tape, so they could download it at slower than real time. With a 2400-baud modem you would be able to download a few seconds of video per day, though, so maybe you'd need to contract at least a 64-kbps frame relay link from the phone company, so you could download a minute of video per day. Then you could watch a single movie every other month.

AlbertoGP 5y ago
> This document is not a protocol design; it's a data store design. The protocol, Phoebe, is mentioned briefly in §5.2 and §7.4 There is a detailed description of it at: http://udanax.xanadu.com/green/febe/index.html http://udanax.xanadu.com/green/febe/index.html Edit: I’ve added this link in a footnote at the end of https://sentido-labs.com/en/library/201904240732/Xanadu%20Hypertext%20Documents.html#6d3 https://sentido-labs.com/en/library/201904240732/Xanadu%20Hy...
- kragen 5y ago
  Oh thanks! I think this may make the code easier to understand.
theamk 5y ago
That's an interesting price analysis! The conclusion I draw from this is that Xanadu is quite a different usage model than the web. The first webpage I remember reading (over dial-up connection to university server) was physics experiments for beginners. The first webpage I made (in the end of 1990's) was to publish plays from the drama club. Those would be be totally infeasible under "every published document must earn money" model. I wonder if Xanadu could ever get completed, who would have used it? It will be probably be a Serious Place full of Professional Authors writing Real Books. Nothing like fun, random places of early internet [0] -- "a guide for backpackers and wilderness trekkers; [...]; a Grateful Dead fan page".... [0] https://www.theatlantic.com/technology/archive/2017/04/a-search-for-the-zombie-websites-of-1995/523848/ https://www.theatlantic.com/technology/archive/2017/04/a-sea...
- kragen 5y ago
  It's probably true that it would have started with literature expected to have mass appeal ("Real Books") and only gradually moved downmarket. But let's not exaggerate! I'm guessing there were more than two people in the drama club, so it's likely that even your first webpage had more than two readers a month, which was the breakeven point I calculated above for extrapolated 01984 storage pricing. By my extrapolated 01990 pricing it would have been economical at three readers per year, and by my extrapolated 01996 pricing at one reader every 3 years. Since you called it a "webpage" and not a "BBS" or "FTP site" I'm guessing that this was more like 01994 than 01984, so you would have been fine. These numbers are no different for Xanadu and the WWW, it's just that on the WWW they are paid by the publisher (or their friend or university or something), and in Xanadu they would be paid by the users. All this is assuming the value to the reader is about US$3 per megabyte, a number I based on the idea that a book is about a megabyte, people commonly pay about US$10 for a book, and the edit history for the book might be three times the size of the final book. (Maybe people paid US$2.50 for a paperback, while a hardcover might be US$40. Roughly 01984 dollars.) If you were reading 40 hours a week at 350 words per minute at this speed, that works out to US$15 a week, US$66 a month, or US$789 a year. A more reasonable path is to figure that by 01994 people might pay US$20 a month for using the Xanadu service 40 hours a week, with half of that (US$10 a month) flowing to the author and hosting costs. That's only 46¢ per megabyte, but by 01994 storing a megabyte only cost about US$30 instead of US$1000, maybe US$5/year, so anything with more than one reader a month would pay for itself. And if it didn't, well, a play might be 100kB, and maybe the drama club would spend US$3 up front to pay a Xanadu operator to store those 100K for eternity, money they'd get back if people read the play, but if nobody ever read the play on Xanadu then that would just be an expense. I feel like US$3 is not a prohibitive expense for vanity-publishing a play and having it never go out of print. I mean xeroxing it at the library probably cost almost that much. Per copy! (In 01996, a single Moore's-Law doubling later, I bought my first computer. It had an 800MB hard disk and cost me about US$900, half of which was the monitor, and would have been a perfectly capable fileserver even though it was a cheap IBM PC clone instead of a Sun workstation. So my US$30/MB for 01994 might be too conservative. Maybe it would be more like US$3/MB. And by that time gzip and 14.4kbps modems were mainstream, dropping costs further.) Now, it's possible that authors (or the Xanadu operating company) would have been too greedy, and charged much higher prices than that, which would have had the result you describe. (There's some evidence that people did shift from things like CompuServe and AOL over to the WWW because publishing on the WWW was cheaper.) Or it's possible that they would have subsidized the initial wide dissemination of their work, as we in fact did with the WWW, so as to build up a customer base.