3 ms·
I don't know about "all the knowledge ever", but to give you a baseline, the entire Wikipedia with images but without editing history, as last archived by Kiwix
by int_19h 4y ago
I don't know about "all the knowledge ever", but to give you a baseline, the entire Wikipedia with images but without editing history, as last archived by Kiwix in May, 2022, is 90 Gb.
A web server can run just fine on e.g. Raspberry Pi Zero W ($10), exposing any such content to any smartphone etc able to connect to it via WiFi (Kiwix sells preconfigured SD cards for their content, even). So, assuming that most people already have a phone or a laptop or even something like a Kindle, the only non-negligible cost here is storage. And a 1 Tb SD card can be had for under $150 right now.
So if anything, I think OP is overly conservative, given that their estimate was "$200 per person". Unless that counts the devices used to consume the content, and not just storage and the server.
- visarga 4y agoThere are some language models like DeepMind RETRO that can make use of a 1TB text collection after the model is trained. The idea is to chunk up the text and make the blocks searchable with a neural embedding index. When you ask a question to the model, it first searches for relevant information and then adds it to the prompt. The result - you can get GPT-3 quality on a 25x smaller model. That means you could have your own GPT-3. Another advantage is that you can reference the sources and the generated text is more grounded. An open source project could make a "search-engine-language-model" and in turn make this library much more accessible. A recent article related to this: http://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html http://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html
- Vetch 4y agoThe interesting aspect of GPT-3+ is their reasoning like behavior such as Chains of thought or "step by step" generation. There's no evidence RETRO as is has sufficient capacity for the actually interesting behaviors.
- visarga 4y agoI agree, that's why I asked the authors the same question : https://twitter.com/visarga/status/1543826941052166144 https://twitter.com/visarga/status/1543826941052166144