4 ms·

Here we go again. I'll play my traditional opening move: "How are you handling ACID guarantees?" and then I'll direct you here: http://news.ycombinator.com/

• abstractbill 19y ago
  "How are you handling ACID guarantees?" Maybe I'm missing something but I don't see how ACID plays into things. You don't imagine the search engine is putting stuff in to the database do you? From the link above, "Do any of these with your "big hash table" in less than 1 second" 1 second is eternity as far as my search engine is concerned. If I can't handle a request in less than a couple of milliseconds, I'd say I've failed. I've used this same approach in the past on a search engine that indexed 14 million records btw, and I didn't fail ;-)
  • dfranke 19y ago
    14 million? Okay, that's not very much. A RAM-based approach is fine there.
    • abstractbill 19y ago
      Heh, fair enough. Do many startups really have more than a few tens of millions of records to search though?
      • dfranke 19y ago
        I can't speak statistically, but the one I work for now does and the one I'm about to start will too (but in the latter case I'm writing a DBMS, so that's kind of a pathological example). If all your content is human-generated and O(n) in size, then 10^7 is a lot. The big numbers come when your data is produced by a simulation, or when you're doing OLAPey things with O(n^2)-sized intermediate results. Real-life example: keeping a log of all users, the IPs they've logged in from, and when, and looking for evidence of sockpuppet accounts.
        • neilc 19y ago
          I'd be curious to know why you're writing a DBMS from scratch. That seems like kind of an odd thing to do, from a value-for-effort POV, in this day and age.
          • dfranke 19y ago
            Not from scratch. I'm building it on top of Hadoop and HBase.
      • nostrademons 19y ago
        Daily quote data for the stock exchanges is about 50 GB now. If you do any sort of intraday analysis, you're looking at terabytes of data...
• henning 19y ago
  Well, Berkeley DB is totally ACID and it's fast as shit. But BDB isn't a database, it's more of a storage engine, which is why complex queries of the kind you like to cherrypick are obviously a poor fit for it.
  • dfranke 19y ago
    I'd call it a database, just not a relational one. It has everything except a query compiler.