Google: "Yah, we did that 20 years ago on mechanical hard drives." Seriously though: Google built realtime indexing a very long time ago. I co-implemented a s

Google: "Yah, we did that 20 years ago on mechanical hard drives." Seriously though: Google built realtime indexing a very long time ago. I co-implemented a small-scale (like 100k pages) full text search engine about 20 years ago with a lot of inspiration from the 1998 paper "The Anatomy of a Large-Scale Hypertextual Web Search Engine". I had always assumed Google used 2-3 layers sort of like in Hierarchical storage management (HSM); fresh data stored in RAM and older data stored on HDDs, then combining them during the query step. I was itching to have a go at implementing that, but it wasn't really required for our use case.