3 ms·
"How are you handling ACID guarantees?" Maybe I'm missing something but I don't see how ACID plays into things. You don't imagine the search engine is putting
by abstractbill 19y ago
"How are you handling ACID guarantees?"
Maybe I'm missing something but I don't see how ACID plays into things. You don't imagine the search engine is putting stuff in to the database do you?
From the link above, "Do any of these with your "big hash table" in less than 1 second"
1 second is eternity as far as my search engine is concerned. If I can't handle a request in less than a couple of milliseconds, I'd say I've failed. I've used this same approach in the past on a search engine that indexed 14 million records btw, and I didn't fail ;-)
- dfranke 19y ago14 million? Okay, that's not very much. A RAM-based approach is fine there.
- abstractbill 19y agoHeh, fair enough. Do many startups really have more than a few tens of millions of records to search though?
- dfranke 19y agoI can't speak statistically, but the one I work for now does and the one I'm about to start will too (but in the latter case I'm writing a DBMS, so that's kind of a pathological example). If all your content is human-generated and O(n) in size, then 10^7 is a lot. The big numbers come when your data is produced by a simulation, or when you're doing OLAPey things with O(n^2)-sized intermediate results. Real-life example: keeping a log of all users, the IPs they've logged in from, and when, and looking for evidence of sockpuppet accounts.
- nostrademons 19y agoDaily quote data for the stock exchanges is about 50 GB now. If you do any sort of intraday analysis, you're looking at terabytes of data...