4 ms·
hmm...this 4 Billion queries number comes from where? Looking at Stackexchange most of the sites have less than a million (possible?) queries(questions). Stack
by hos234 7y ago
hmm...this 4 Billion queries number comes from where?
Looking at Stackexchange most of the sites have less than a million (possible?) queries(questions). Stackoverflow, probably the largest, has 18 million questions. Wikipedia article count is about 5 million.
Is the assumption that people are coming up with 1000 different ways to query every possible result endpoint?
- gravadlax 7y agoHi, I am one of the authors of this post. The number 4 billion is made-up. It was chosen in order to illustrate the scale of the problem we are trying to solve. And it also happens to be just below the maximum number possible to store in an unsigned 4 byte integer. This is of course beneficial when trying to build an index with a small footprint. With more queries, we would have to consider using long integers (8 bytes) or some custom type with e.g. 5 bytes. If you're interested in where the queries come from, you might find these previous posts interesting: https://0x65.dev/blog/2019-12-05/a-new-search-engine.html https://0x65.dev/blog/2019-12-05/a-new-search-engine.html https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch.html https://0x65.dev/blog/2019-12-06/building-a-search-engine-fr...
- elcritch 7y agoA thought regarding the 4 byte vs 8byte. Both of those will be highly optimizable by custom/manual vector instructions, like the modern AVX512 vector instructions. Likely someone on your team thought of that or does that but in the off chance you haven’t looked into it yet it might prove very useful! Thanks for the posts, they’re fascinating and the idea of more alternative search engines would be great. Best of luck!
- hos234 7y agoThanks for these writeups. This is a very inspiring project.
- detaro 7y agoWhy would Stackoverflow and Wikipedia cover all queries for a web search?