4 ms·

When I was a teenager I made a search engine and database from scratch, not knowing anything about either. I just wanted to see how far I could get, how fast I

When I was a teenager I made a search engine and database from scratch, not knowing anything about either. I just wanted to see how far I could get, how fast I could make it, returning somewhat relevant search results. It's actually very easy to make a fast basic database and search engine, even as an amateur programmer. As long as you understand basic CS algorithms and how to exploit the operating system and hardware, you can put one together in a month or two. Speed is not bad even with high-level languages; something like 250K QPS, back in 2003, on a laptop. Scalability isn't much of an issue either if you shard it. Indexing, locking, and consistency are more complicated than the storage and retrieval parts. The big problem to overcome is the subjective nature of search. What do I really want to find? How do I find something when I don't know what I'm looking for? How do I get around people trying to game the system? How do I handle complex queries and datasets? That's when it gets orders of magnitude harder.

gomezjdaniel 3y ago
It is open sourced anywhere? I am curious
paxys 3y ago
When building a search engine the biggest problem isn't QPS but the size of the data set you are indexing. If your search structures can fit in memory of a single machine then you can serve effective infinite QPS with ~zero latency. When it grows beyond that you have to get creative, and that's when all the different tradeoffs come into the picture.
vlovich123 3y ago
250k RPS seems like a lot when SQLite can’t do that today and I feel like inverted indices are more expensive (eg RocksDB gets to ~130k RPS but I think their numbers are on beefier hardware than my laptop or my machine isn’t set up well). Are you sure you wrote a general purpose database and those numbers are correct? I’m curious what you did to outperform these engines (I agree it’s possible, but I’m curious what techniques you employed)
- lionkor 3y ago
  I'd guess: - everything was in RAM, - it was written in a compiled language, and - probably skipped worrying about too many crazy edge cases
  - thumbuddy 3y ago
    Same guesses I'd also add, single client without network connections. Anyone who says "writing a database from scratch is easy" makes me sweat profusely.
    - hot_gril 3y ago
      Especially not easy if you actually implement read/write transactions (=> concurrency control, WAL, ...)

gomezjdaniel 3y ago

It is open sourced anywhere? I am curious

paxys 3y ago

When building a search engine the biggest problem isn't QPS but the size of the data set you are indexing. If your search structures can fit in memory of a single machine then you can serve effective infinite QPS with ~zero latency. When it grows beyond that you have to get creative, and that's when all the different tradeoffs come into the picture.

vlovich123 3y ago

250k RPS seems like a lot when SQLite can’t do that today and I feel like inverted indices are more expensive (eg RocksDB gets to ~130k RPS but I think their numbers are on beefier hardware than my laptop or my machine isn’t set up well). Are you sure you wrote a general purpose database and those numbers are correct? I’m curious what you did to outperform these engines (I agree it’s possible, but I’m curious what techniques you employed)

lionkor 3y ago
I'd guess: - everything was in RAM, - it was written in a compiled language, and - probably skipped worrying about too many crazy edge cases
- thumbuddy 3y ago
  Same guesses I'd also add, single client without network connections. Anyone who says "writing a database from scratch is easy" makes me sweat profusely.
  - hot_gril 3y ago
    Especially not easy if you actually implement read/write transactions (=> concurrency control, WAL, ...)