5 ms·

Interesting. As you scale from 100 to 200 to 300 servers, how does that translate into differences in the product? More pages indexed? More frequent indexing? F

• ricardo81 6y ago
  Hello, Mojeek back end dev here, I mostly work with Marc on the engine and backend services. More pages indexed, yes, as there's a finite/specific amount stored on each server. Also for redundancy to back up sections of the index in case of any hardware failure on any one disk/server, you can fall back to the other server containing that slice of the index. It also raises new problems as the server count grows. One way of doing things is for one server to ask all the other servers to return their 1000 best results for a query. Over time, that can amount to a considerable amount of bandwidth and sorting for that single server to do. We'd recently made a change to proxy requests more along a per-rack level and let them sort the results and send back. This improved our median search time dramatically and is in good timing for our next batch of 100 servers. We noticed a visible improvement in response times via a 3rd party that uses our API for results. https://www.etools.ch/searchInfo.do https://www.etools.ch/searchInfo.do