3 ms·

OK, what's the napkin math?

int_19h 4y ago
I don't know about "all the knowledge ever", but to give you a baseline, the entire Wikipedia with images but without editing history, as last archived by Kiwix in May, 2022, is 90 Gb. A web server can run just fine on e.g. Raspberry Pi Zero W ($10), exposing any such content to any smartphone etc able to connect to it via WiFi (Kiwix sells preconfigured SD cards for their content, even). So, assuming that most people already have a phone or a laptop or even something like a Kindle, the only non-negligible cost here is storage. And a 1 Tb SD card can be had for under $150 right now. So if anything, I think OP is overly conservative, given that their estimate was "$200 per person". Unless that counts the devices used to consume the content, and not just storage and the server.
- visarga 4y ago
  There are some language models like DeepMind RETRO that can make use of a 1TB text collection after the model is trained. The idea is to chunk up the text and make the blocks searchable with a neural embedding index. When you ask a question to the model, it first searches for relevant information and then adds it to the prompt. The result - you can get GPT-3 quality on a 25x smaller model. That means you could have your own GPT-3. Another advantage is that you can reference the sources and the generated text is more grounded. An open source project could make a "search-engine-language-model" and in turn make this library much more accessible. A recent article related to this: http://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html http://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html
  - Vetch 4y ago
    The interesting aspect of GPT-3+ is their reasoning like behavior such as Chains of thought or "step by step" generation. There's no evidence RETRO as is has sufficient capacity for the actually interesting behaviors.
    - visarga 4y ago
      I agree, that's why I asked the authors the same question : https://twitter.com/visarga/status/1543826941052166144 https://twitter.com/visarga/status/1543826941052166144
dredmorbius 4y ago
I've specced this out a few times, it's an interesting exercise, and it depends on what "all human knowledge" is. From the perspective of how much you individually could read in a lifetime: that's about 4,000 weeks (to roughly age 80-ish, and presuming you're not already reading at birth). Multiply that by the number of books you plan on reading a week. You'll probably read fewer than 4,000 books over your entire life. Many people read few if any books after graduating secondary school, and even a highly-motivated reader might be challenged to crack, say, 40,000 (ten per week for life). The US Library of Congress has the world's largest book collection at about 40 million distinct titles.[1] There are another 130 million total catalogued items, including photographs, films, audio recordings, maps, pamphlets, and other items. Not all are textual. I'll stick to books. At 4,000 per lifetime, you'd need 10,000 people simply to read all of the LoC's collection at a reasonable pace. (By comparison, at the turn of the 20th century, the LoC's annual report to Congress noted that its cataloguing department could handle about 3,000 books per cataloguer per year, or a pace of about 60/week or 15/day. Over a 40-year career, a cataloguer might handle, if not read completely, 120,000 books.) As a rough rule of thumb, a digitised ebook in PDF format runs about 5 MB. Every single book in the Library of Congress in digital format would occupy about 200 TB of storage. Current disk prices are running about $5 -- $20 / TB. 200 TB in raw disk would set you back $1,000 -- $4,000. Figure 2-4x multiplier for a disk storage system all told, and it's still roughly $4k -- $16k to have as local storage every last book in the US Library of Congress. That's well within scope of a moderately wealthy US household budget, and would be reasonable to consider for a small-town city library. That's the technical storage cost, obviously not the rights or aquisition costs for the materials. The Library of Congress's budget is about $800 million/yr. Those 4,000 books you might read in a lifetime? They'd fit on about 20 GB worth of disk. If you're a 10x reader, 200 GB, and a truly dedicated 100x reader, about 2 TB. Those are well within the range of present desktop / laptop disk allocations, and represent a few tens of dollars of storage expense. The Internet Archive computes $2/GB for storage in perpetuity. That's roughly 400 books worth of storage. But a small household NAS and a very modest server platform (most routers can effectively operate as media servers) could rival a mid-sized city library for about $100 or less in actual hardware outlay. Those prices are falling by half about every 18 to 36 months, as they have been for decades. Books are not all published content, and all published content is not all human knowledge. But standard published books are a good proxy for total cultural knowledge, and in all likelihood, then some. ________________________________ Notes: 1. See: https://www.loc.gov/about/general-information/ https://www.loc.gov/about/general-information/ That includes about 25 million books in the main collection, and 15 million in "nonclassified print collections, including books in large type and raised characters, incunabula (books printed before 1501), monographs and serials, music, bound newspapers, pamphlets, technical reports and other printed material". I suspect you could roughly halve my estimates above based on 25 million vs. 40 million volumes, as many of the nonclassified works may duplicate the main collection.

I don't know about "all the knowledge ever", but to give you a baseline, the entire Wikipedia with images but without editing history, as last archived by Kiwix in May, 2022, is 90 Gb. A web server can run just fine on e.g. Raspberry Pi Zero W ($10), exposing any such content to any smartphone etc able to connect to it via WiFi (Kiwix sells preconfigured SD cards for their content, even). So, assuming that most people already have a phone or a laptop or even something like a Kindle, the only non-negligible cost here is storage. And a 1 Tb SD card can be had for under $150 right now. So if anything, I think OP is overly conservative, given that their estimate was "$200 per person". Unless that counts the devices used to consume the content, and not just storage and the server.

visarga 4y ago
There are some language models like DeepMind RETRO that can make use of a 1TB text collection after the model is trained. The idea is to chunk up the text and make the blocks searchable with a neural embedding index. When you ask a question to the model, it first searches for relevant information and then adds it to the prompt. The result - you can get GPT-3 quality on a 25x smaller model. That means you could have your own GPT-3. Another advantage is that you can reference the sources and the generated text is more grounded. An open source project could make a "search-engine-language-model" and in turn make this library much more accessible. A recent article related to this: http://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html http://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html
- Vetch 4y ago
  The interesting aspect of GPT-3+ is their reasoning like behavior such as Chains of thought or "step by step" generation. There's no evidence RETRO as is has sufficient capacity for the actually interesting behaviors.
  - visarga 4y ago
    I agree, that's why I asked the authors the same question : https://twitter.com/visarga/status/1543826941052166144 https://twitter.com/visarga/status/1543826941052166144

dredmorbius 4y ago

I've specced this out a few times, it's an interesting exercise, and it depends on what "all human knowledge" is. From the perspective of how much you individually could read in a lifetime: that's about 4,000 weeks (to roughly age 80-ish, and presuming you're not already reading at birth). Multiply that by the number of books you plan on reading a week. You'll probably read fewer than 4,000 books over your entire life. Many people read few if any books after graduating secondary school, and even a highly-motivated reader might be challenged to crack, say, 40,000 (ten per week for life). The US Library of Congress has the world's largest book collection at about 40 million distinct titles.[1] There are another 130 million total catalogued items, including photographs, films, audio recordings, maps, pamphlets, and other items. Not all are textual. I'll stick to books. At 4,000 per lifetime, you'd need 10,000 people simply to read all of the LoC's collection at a reasonable pace. (By comparison, at the turn of the 20th century, the LoC's annual report to Congress noted that its cataloguing department could handle about 3,000 books per cataloguer per year, or a pace of about 60/week or 15/day. Over a 40-year career, a cataloguer might handle, if not read completely, 120,000 books.) As a rough rule of thumb, a digitised ebook in PDF format runs about 5 MB. Every single book in the Library of Congress in digital format would occupy about 200 TB of storage. Current disk prices are running about $5 -- $20 / TB. 200 TB in raw disk would set you back $1,000 -- $4,000. Figure 2-4x multiplier for a disk storage system all told, and it's still roughly $4k -- $16k to have as local storage every last book in the US Library of Congress. That's well within scope of a moderately wealthy US household budget, and would be reasonable to consider for a small-town city library. That's the technical storage cost, obviously not the rights or aquisition costs for the materials. The Library of Congress's budget is about $800 million/yr. Those 4,000 books you might read in a lifetime? They'd fit on about 20 GB worth of disk. If you're a 10x reader, 200 GB, and a truly dedicated 100x reader, about 2 TB. Those are well within the range of present desktop / laptop disk allocations, and represent a few tens of dollars of storage expense. The Internet Archive computes $2/GB for storage in perpetuity. That's roughly 400 books worth of storage. But a small household NAS and a very modest server platform (most routers can effectively operate as media servers) could rival a mid-sized city library for about $100 or less in actual hardware outlay. Those prices are falling by half about every 18 to 36 months, as they have been for decades. Books are not all published content, and all published content is not all human knowledge. But standard published books are a good proxy for total cultural knowledge, and in all likelihood, then some. ________________________________ Notes: 1. See: https://www.loc.gov/about/general-information/ https://www.loc.gov/about/general-information/ That includes about 25 million books in the main collection, and 15 million in "nonclassified print collections, including books in large type and raised characters, incunabula (books printed before 1501), monographs and serials, music, bound newspapers, pamphlets, technical reports and other printed material". I suspect you could roughly halve my estimates above based on 25 million vs. 40 million volumes, as many of the nonclassified works may duplicate the main collection.