6 ms·

This is a search engine indexing the internet on a mariadb database hosted on consumer hardware maintained by a single person as a hobby and it does not suffer

by slim 5y ago

This is a search engine indexing the internet on a mariadb database hosted on consumer hardware maintained by a single person as a hobby and it does not suffer from HN hug of death

swyx 5y ago
how on earth do you index so much on consumer hardware? my frontend developer mind is blown.
- foxfluff 5y ago
  Wait till you learn that modern CPUs run billions of cycles per second. With multiple cores in parallel! And they can reach transfer rates of tens of gigabytes per second to RAM, or around a terabyte per second into L3.
  - ThalesX 5y ago
    And then you add a single HTTP request and everything tones down to the speed of the web. Or I/O. Or DB call.
    - goodpoint 5y ago
      More like: you add some javascript library and now the browser needs 5 seconds to run 10 MB of javascript.
    - knuthsat 5y ago
      You can still support millions of these requests per second if you just bake all of the dependencies directly in a small OS running on your fastest raspberry pi.
- paxys 5y ago
  Consumer hardware today is simply what was cutting-edge and crazy expensive 5 years ago.
marginalia_nu 5y ago
Huh. I've gone through the logs, and at most I got about 2 search queries per second as a sustained load over several hours. It's calmed down to about 0.5-1 QPS now. This is ignoring other page loads. Not bad, if I do say so myself.
- slim 5y ago
  It's a statement to how much other websites are bad
  - marginalia_nu 5y ago
    Do I wonder how much cookies play into it. Sure I have small page loads and reasonably optimized code, but I also don't use any cookies. I'd imagine having to keep track of upwards of a million session objects would be a struggle for almost any web server.