4 ms·

> Alexandria on your hard drive That image brings to mind: How much space would you need to store every out-of-copyright book written in English?

dredmorbius 5y ago
There are a few estimates of total books ever published. Those range from the 40 million or so within the US Library of Congress collection to a Google estimate from a few years back of 140 million books existing, with about 2--5 million new ones published per year. Bowker, the US issuer of ISBNs, has registered roughly 300k new titles annually, since at least the 1950s, and beginning in the aughts, about 1--2 million "nontraditional" (self-published) titles. In straight text, assuming 250 pages of 250 words at 8 bytes/word, 40 million books works out to about 20 TB. This is a likely close to the lower bound as there are texts with image and other content which bump this value up. It works out to about 0.5 MB/book, which in my experience is too low. At 150 million books, we'd get 75 TB, and at 200 million, 100 TB. Nice round numbers.... A more realistic per-book size is 5 MB. This is typical of a mostly-text, formatted PDF. At 40, 150, and 200 million texts: 250, 750, and 1,000 TB. A data-heavy / graphics-heavy book might weigh in at 20--100 MB, though those tend to be less common. Scanned-in books with both page images and OCR text typically run about 5--10 MB. The largest I've seen (a copy of Lyell's Geography was >300 MB. Apple has offered iPads with > 1 TB onboard storage for at least several years. This seems to be the higher end of available tablet storage. (For some reason, the Cloud-and-Surveillance-oriented Google skimps with many devices still at 16--32 MB. Remarkable, which also went Cloud, tops out at 16 MB.) If fully devoted to texts at 5 MB each, that would offer storage for up to 200,000 documents, roughly the size of a well-stocked municipal public library. As to how many public-domain works might exist, the 1927 Report of the Librarian of Congress notes that the Library had assigned 4,503,164 copyrights over the preceding 57 years (presumably 1869--1926), the time in which such registrations were the work of the Library. Those works should all presently be in the Public Domain, and would total 22.5 TB at 5 MB/work. Annual report of the Librarian of Congress yr.1927, Washington, Library of Congress; for sale by the Supt. of Docs., U.S. Govt. Print. Off., p. 22. https://babel.hathitrust.org/cgi/pt?id=uc1.a0013520523&view=1up&seq=42 https://babel.hathitrust.org/cgi/pt?id=uc1.a0013520523&view=... The Library of Congress's total holdings was 3,420,345 million as of end of FY 1926 (p. 23 ibid). Presently, works published prior to 1 January 1927 in the United States are in the public domain. An additional roughly 1 million maps, 1 million volumes and/or pieces of music, and ~500k prints were also in the library's collection as of 1926. A full set of the Librarian's reports, dating to the 1860s, is available at Hathi Trust. From at least the 1950s, annual holdings and aquisitions are broken out by Library of Congress classification, which makes for some interesting data, for those who find that sort of thing interesting. http://catalog.hathitrust.org/Record/000072049 http://catalog.hathitrust.org/Record/000072049
- wolverine876 4y ago
  Thank you! We are not far in time (10 years?) from being able to provide a complete set with every computing device, though we would need them all converted and formatted, etc., and many will (unfortunately) think we should just use the cloud instead. I will add this study published in 2010, which estimated 129 million 'editions' published in all languages. Any text can have multiple editions (I don't recall how 'edition' was defined, but IIRC the study was of published objects not of texts), and "includes serials and sets but excludes kits, mixed media, and periodicals such as newspapers". It also contains a useful description of methods to bulk process bibliographic data. (Though maybe this is the Google study you talked about, because IIRC their data is involved.) J-B Michel, et al, "Quantitative Analysis of Culture Using Millions of Digitized Books", Science (16 Dec 2010) https://www.science.org/doi/10.1126/science.1199644 https://www.science.org/doi/10.1126/science.1199644 By the way, I greatly appreciate your informed comments. It's some of the most substantive and interestin reading on HN.
shakna 5y ago
Whilst the Gutenberg Project doesn't have every single book, it is a substantial effort to compile them. The last time I downloaded the complete English collection, it was around 40Gb in plaintext form, and a little closer to 60Gb in HTML, which should be pretty comparable to ePub.
- bongobingo1 5y ago
  That's a lot of CD's to burn.
  - ZainRiz 5y ago
    Stick them on a single blue ray
- cortesoft 5y ago
  Compressed or uncompressed? Books should compress pretty well
  - shakna 5y ago
    Compressed. Sorry, that's an important detail that slipped my mind. zstandard via the ZIM [0] format. [0] https://wiki.openzim.org/wiki/ZIM_file_format https://wiki.openzim.org/wiki/ZIM_file_format
wodenokoto 5y ago
There’s a joke I futurama where they go to the library and it’s just two discs: fiction and non-fiction. Honestly I wouldn’t be surprised if the modern version is just two SD cards. At least if you only want the text.
harshreality 5y ago
Alexandria was strongly anti-copyright. Not that copyright existed, but they allegedly forced visitors to donate copies of any books they had to the library (scribes copied them and gave back the copies). As of last year, someone on reddit determined that l--g-- had 2.7 TB of fiction and 40 TB of nonfiction, and double that size of scientific articles and magazines. It's missing a fair amount of books, which would make the true total size larger, but it also has a lot of duplicates and scans that are unnecessarily large compared to true ebooks, which would make the true total size smaller.
- zrobotics 5y ago
  In an age where books had to be hand copied, copyright really doesn't even make sense to apply as a concept. They weren't anti-copyright, since that concept doesn't even make sense from their viewpoint. The physical book itself was what was valuable, and wasn't separable from the 'work' of the original author. Since they were craft-made objects, the skill of the scribe & bookbinder was as important as the original text. This is much later historically, but in medieval conception 2 identical texts could be worth vastly different amounts based on the quality of the binding and the illumination in the book. This is an example of why applying modern thinking in a historical concept can lead us astray. The Library of Alexandria wasn't pro or anti copyright; that makes as much sense as saying they were pro x86 and anti ARM. It just wasn't even a concept that had been developed. Edit: upon rereading, my tone seemed a bit combative. Not trying to call anyone out, just wanted to point out that applying modern sensibilities to history can cause distortions. It is a pet peeve of mine that people consider ancient people's stupid, and this is one of the major reasons why. We forget that we have the benefit of thousands of generations of previous people improving their understanding of the world, and what seems obvious to us is only obvious in hindsight with the benefit of that prior knowledge.
  - harshreality 5y ago
    I agree there were all kinds of other considerations. The education required to write, and the cost of hand-copying books, meant that the market was nonexistent and those who wrote anything worthwhile were intellectuals and had some other job or patronage. The cost of contacting most authors might've been impractical, even more than the cost of the scribes to copy books, even in cases where the authors were still alive. Still, there was a thriving intellectual culture, and nobody thought about this modern "moral right of the author" to extract rent from the transmission of knowledge. By modern standards, there's hardly anything more anti-copyright than forcing someone to let you copy their personal library. It disrespects not only the modern rent-extraction copyright by the author, but also the idea that written material in any form is anyone's property at all.

dredmorbius 5y ago

There are a few estimates of total books ever published. Those range from the 40 million or so within the US Library of Congress collection to a Google estimate from a few years back of 140 million books existing, with about 2--5 million new ones published per year. Bowker, the US issuer of ISBNs, has registered roughly 300k new titles annually, since at least the 1950s, and beginning in the aughts, about 1--2 million "nontraditional" (self-published) titles. In straight text, assuming 250 pages of 250 words at 8 bytes/word, 40 million books works out to about 20 TB. This is a likely close to the lower bound as there are texts with image and other content which bump this value up. It works out to about 0.5 MB/book, which in my experience is too low. At 150 million books, we'd get 75 TB, and at 200 million, 100 TB. Nice round numbers.... A more realistic per-book size is 5 MB. This is typical of a mostly-text, formatted PDF. At 40, 150, and 200 million texts: 250, 750, and 1,000 TB. A data-heavy / graphics-heavy book might weigh in at 20--100 MB, though those tend to be less common. Scanned-in books with both page images and OCR text typically run about 5--10 MB. The largest I've seen (a copy of Lyell's Geography was >300 MB. Apple has offered iPads with > 1 TB onboard storage for at least several years. This seems to be the higher end of available tablet storage. (For some reason, the Cloud-and-Surveillance-oriented Google skimps with many devices still at 16--32 MB. Remarkable, which also went Cloud, tops out at 16 MB.) If fully devoted to texts at 5 MB each, that would offer storage for up to 200,000 documents, roughly the size of a well-stocked municipal public library. As to how many public-domain works might exist, the 1927 Report of the Librarian of Congress notes that the Library had assigned 4,503,164 copyrights over the preceding 57 years (presumably 1869--1926), the time in which such registrations were the work of the Library. Those works should all presently be in the Public Domain, and would total 22.5 TB at 5 MB/work. Annual report of the Librarian of Congress yr.1927, Washington, Library of Congress; for sale by the Supt. of Docs., U.S. Govt. Print. Off., p. 22. https://babel.hathitrust.org/cgi/pt?id=uc1.a0013520523&view=1up&seq=42 https://babel.hathitrust.org/cgi/pt?id=uc1.a0013520523&view=... The Library of Congress's total holdings was 3,420,345 million as of end of FY 1926 (p. 23 ibid). Presently, works published prior to 1 January 1927 in the United States are in the public domain. An additional roughly 1 million maps, 1 million volumes and/or pieces of music, and ~500k prints were also in the library's collection as of 1926. A full set of the Librarian's reports, dating to the 1860s, is available at Hathi Trust. From at least the 1950s, annual holdings and aquisitions are broken out by Library of Congress classification, which makes for some interesting data, for those who find that sort of thing interesting. http://catalog.hathitrust.org/Record/000072049 http://catalog.hathitrust.org/Record/000072049

wolverine876 4y ago
Thank you! We are not far in time (10 years?) from being able to provide a complete set with every computing device, though we would need them all converted and formatted, etc., and many will (unfortunately) think we should just use the cloud instead. I will add this study published in 2010, which estimated 129 million 'editions' published in all languages. Any text can have multiple editions (I don't recall how 'edition' was defined, but IIRC the study was of published objects not of texts), and "includes serials and sets but excludes kits, mixed media, and periodicals such as newspapers". It also contains a useful description of methods to bulk process bibliographic data. (Though maybe this is the Google study you talked about, because IIRC their data is involved.) J-B Michel, et al, "Quantitative Analysis of Culture Using Millions of Digitized Books", Science (16 Dec 2010) https://www.science.org/doi/10.1126/science.1199644 https://www.science.org/doi/10.1126/science.1199644 By the way, I greatly appreciate your informed comments. It's some of the most substantive and interestin reading on HN.

shakna 5y ago

Whilst the Gutenberg Project doesn't have every single book, it is a substantial effort to compile them. The last time I downloaded the complete English collection, it was around 40Gb in plaintext form, and a little closer to 60Gb in HTML, which should be pretty comparable to ePub.

bongobingo1 5y ago
That's a lot of CD's to burn.
- ZainRiz 5y ago
  Stick them on a single blue ray
cortesoft 5y ago
Compressed or uncompressed? Books should compress pretty well
- shakna 5y ago
  Compressed. Sorry, that's an important detail that slipped my mind. zstandard via the ZIM [0] format. [0] https://wiki.openzim.org/wiki/ZIM_file_format https://wiki.openzim.org/wiki/ZIM_file_format

wodenokoto 5y ago

There’s a joke I futurama where they go to the library and it’s just two discs: fiction and non-fiction. Honestly I wouldn’t be surprised if the modern version is just two SD cards. At least if you only want the text.

harshreality 5y ago

Alexandria was strongly anti-copyright. Not that copyright existed, but they allegedly forced visitors to donate copies of any books they had to the library (scribes copied them and gave back the copies). As of last year, someone on reddit determined that l--g-- had 2.7 TB of fiction and 40 TB of nonfiction, and double that size of scientific articles and magazines. It's missing a fair amount of books, which would make the true total size larger, but it also has a lot of duplicates and scans that are unnecessarily large compared to true ebooks, which would make the true total size smaller.

zrobotics 5y ago
In an age where books had to be hand copied, copyright really doesn't even make sense to apply as a concept. They weren't anti-copyright, since that concept doesn't even make sense from their viewpoint. The physical book itself was what was valuable, and wasn't separable from the 'work' of the original author. Since they were craft-made objects, the skill of the scribe & bookbinder was as important as the original text. This is much later historically, but in medieval conception 2 identical texts could be worth vastly different amounts based on the quality of the binding and the illumination in the book. This is an example of why applying modern thinking in a historical concept can lead us astray. The Library of Alexandria wasn't pro or anti copyright; that makes as much sense as saying they were pro x86 and anti ARM. It just wasn't even a concept that had been developed. Edit: upon rereading, my tone seemed a bit combative. Not trying to call anyone out, just wanted to point out that applying modern sensibilities to history can cause distortions. It is a pet peeve of mine that people consider ancient people's stupid, and this is one of the major reasons why. We forget that we have the benefit of thousands of generations of previous people improving their understanding of the world, and what seems obvious to us is only obvious in hindsight with the benefit of that prior knowledge.
- harshreality 5y ago
  I agree there were all kinds of other considerations. The education required to write, and the cost of hand-copying books, meant that the market was nonexistent and those who wrote anything worthwhile were intellectuals and had some other job or patronage. The cost of contacting most authors might've been impractical, even more than the cost of the scribes to copy books, even in cases where the authors were still alive. Still, there was a thriving intellectual culture, and nobody thought about this modern "moral right of the author" to extract rent from the transmission of knowledge. By modern standards, there's hardly anything more anti-copyright than forcing someone to let you copy their personal library. It disrespects not only the modern rent-extraction copyright by the author, but also the idea that written material in any form is anyone's property at all.