4 ms·
The problem is that newer versions have fewer links to them, so they seem less authoritative. I have a plan for some heuristics that will detect version numbers
by wolfgang42 4y ago
The problem is that newer versions have fewer links to them, so they seem less authoritative. I have a plan for some heuristics that will detect version numbers in URLs and collapse them into a single result with a version picker in case you want an older version.
The server is an HP Microserver Gen8 (purchased on eBay), with an “Intel(R) Pentium(R) CPU G2020T @ 2.50GHz” and 16GB of RAM. The production index is 70GB, and I also have a 1TB spinning rust disk that I use for scratch space and raw data.
- ColonelPhantom 4y ago> The problem is that newer versions have fewer links to them, so they seem less authoritative. Ah, that makes a lot of sense actually, thanks for explaining! The heuristics idea sounds neat. Sounds like it's running on surprisingly little hardware. Is the index stored on a separate SSD?
- wolfgang42 4y ago> Sounds like it's running on surprisingly little hardware. Modern computers (even ones from 2013) are really fast. I implemented PageRank from the original paper: I have about the same size of index as early Google, and what Page and Brin’s server could compute in a week with carefully optimized C code, mine can do in less than 5 minutes (!) with a bit of JavaScript. > Is the index stored on a separate SSD? No, it’s just on the same disk with the rest of the system. My experience has been that pretty much everything that Feep! search does is CPU-bound on this machine; in fact I suspect (though I haven’t tried it) that the index could even be on the HDD and the only difference would be a few extra milliseconds when serving search results.