3 ms·

Seems neat, but trying to find Django documentation is not ideal: searching for "django prefetch" has GitHub repos as the first two results, and the third resul

Seems neat, but trying to find Django documentation is not ideal: searching for "django prefetch" has GitHub repos as the first two results, and the third result is official Django documentation but for an ancient version, something that annoys me about other search engines too. Out of curiousity, what kind of hardware are you running this on? I can imagine that you'd need a lot of storage to store the index, but the size of plain text can often be surprisingly small.

wolfgang42 4y ago
The problem is that newer versions have fewer links to them, so they seem less authoritative. I have a plan for some heuristics that will detect version numbers in URLs and collapse them into a single result with a version picker in case you want an older version. The server is an HP Microserver Gen8 (purchased on eBay), with an “Intel(R) Pentium(R) CPU G2020T @ 2.50GHz” and 16GB of RAM. The production index is 70GB, and I also have a 1TB spinning rust disk that I use for scratch space and raw data.
- ColonelPhantom 4y ago
  > The problem is that newer versions have fewer links to them, so they seem less authoritative. Ah, that makes a lot of sense actually, thanks for explaining! The heuristics idea sounds neat. Sounds like it's running on surprisingly little hardware. Is the index stored on a separate SSD?
  - wolfgang42 4y ago
    > Sounds like it's running on surprisingly little hardware. Modern computers (even ones from 2013) are really fast. I implemented PageRank from the original paper: I have about the same size of index as early Google, and what Page and Brin’s server could compute in a week with carefully optimized C code, mine can do in less than 5 minutes (!) with a bit of JavaScript. > Is the index stored on a separate SSD? No, it’s just on the same disk with the rest of the system. My experience has been that pretty much everything that Feep! search does is CPU-bound on this machine; in fact I suspect (though I haven’t tried it) that the index could even be on the HDD and the only difference would be a few extra milliseconds when serving search results.

wolfgang42 4y ago

The problem is that newer versions have fewer links to them, so they seem less authoritative. I have a plan for some heuristics that will detect version numbers in URLs and collapse them into a single result with a version picker in case you want an older version. The server is an HP Microserver Gen8 (purchased on eBay), with an “Intel(R) Pentium(R) CPU G2020T @ 2.50GHz” and 16GB of RAM. The production index is 70GB, and I also have a 1TB spinning rust disk that I use for scratch space and raw data.

ColonelPhantom 4y ago
> The problem is that newer versions have fewer links to them, so they seem less authoritative. Ah, that makes a lot of sense actually, thanks for explaining! The heuristics idea sounds neat. Sounds like it's running on surprisingly little hardware. Is the index stored on a separate SSD?
- wolfgang42 4y ago
  > Sounds like it's running on surprisingly little hardware. Modern computers (even ones from 2013) are really fast. I implemented PageRank from the original paper: I have about the same size of index as early Google, and what Page and Brin’s server could compute in a week with carefully optimized C code, mine can do in less than 5 minutes (!) with a bit of JavaScript. > Is the index stored on a separate SSD? No, it’s just on the same disk with the rest of the system. My experience has been that pretty much everything that Feep! search does is CPU-bound on this machine; in fact I suspect (though I haven’t tried it) that the index could even be on the HDD and the only difference would be a few extra milliseconds when serving search results.