5 ms·

Thanks for the good-luck wishes. Obviously it's a big undertaking, but we're seeing very promising results, so it's time to go all-in. Take that as you will :

Thanks for the good-luck wishes. Obviously it's a big undertaking, but we're seeing very promising results, so it's time to go all-in. Take that as you will :-) Incidentally, Oracle and IBM both have clustered database products, though they can certainly be improved upon.

jacques_chester 15y ago
Oracle and IBM do have clustered stuff and it's amazing, but it still tends to assume closely linked networks of high-reliability hardware and have trouble with very large, slowly linked networks of low-reliability hardware. You might find Greenplum and VoltDB to be more interesting models to study. The key question is: are you aiming at OLTP or OLAP? Because, as those boring mainframe-era guys have discovered, it's difficult to serve two very different purposes.
- justinsb 15y ago
  OLTP. I personally believe Hadoop has got the OLAP problem-space covered.
  - jacques_chester 15y ago
    What are your most important non-functional requirements? In particular, how do you see the tradeoff between durability and performance?
    - justinsb 15y ago
      You've totally hit the nail on the head there. Up until a month ago, it was a fairly obsessive compulsion around performance. Durability and reliability are features of the design, so it was about getting performance under the design. However, the AWS outage has (we hope) put reliability back on the map. So maybe now we don't have to be the fastest database, if we're the fastest reliable database. That's a liberating change of perspective, which has been behind many of the recent advances.
      - jacques_chester 15y ago
        Does durability absolutely trump performance, or are you prepared to trade off? For example, if durability is an absolute must (as it is for current databases), then your main bottlenecks would be a) spinning rust and b) network traffic to distribute copies of the data throughout the network. SSDs will help this a lot, but already you're back where almost everyone has already started. If you're prepared to accept highly replicated in-memory copies as "durable", you can already make your overall system hundreds or even thousands of times faster. So how do you define durability? 1 copy on 1 disk? x copies on y disks in the same computer? x copies in y disks in different computers? x copies in RAM of y different computers?
- ChuckMcM 15y ago
  This was the perspective that I have as well, when I was with Network Appliance we spent a lot of time dealing with folks who wanted bullet proof relational databases running on clusters, NetApp provided an extremely reliable storage platform and a unique ability to 'snapshot' instantly the current state of the storage used by the data base so that you could run various experiments against that data without huge copies etc. There were multiply redundant switches and massive investments in transactional data recovery. The big finance guys were particularly intimidating when they talked about several hundred billion dollars in transactions in a 24 hour period, that is $4,000 - $5,000 per mS, or 4 to 5 million dollars per second. Combined with the whole finance game of not confirming receipt until the last possible moment to keep your liabilities managable, ouch. I would love to see a relational database emerge that could do ACID on an unreliable cloud, and I think the folks who achieve that should become gazillionaires. I was just noting that a whole lot of money and research has already been poured into that hole. We're still waiting to here the 'thump' of it hitting the bottom. :-) It totally rocks that Fathom is taking on that challenge.

jacques_chester 15y ago

Oracle and IBM do have clustered stuff and it's amazing, but it still tends to assume closely linked networks of high-reliability hardware and have trouble with very large, slowly linked networks of low-reliability hardware. You might find Greenplum and VoltDB to be more interesting models to study. The key question is: are you aiming at OLTP or OLAP? Because, as those boring mainframe-era guys have discovered, it's difficult to serve two very different purposes.

justinsb 15y ago
OLTP. I personally believe Hadoop has got the OLAP problem-space covered.
- jacques_chester 15y ago
  What are your most important non-functional requirements? In particular, how do you see the tradeoff between durability and performance?
  - justinsb 15y ago
    You've totally hit the nail on the head there. Up until a month ago, it was a fairly obsessive compulsion around performance. Durability and reliability are features of the design, so it was about getting performance under the design. However, the AWS outage has (we hope) put reliability back on the map. So maybe now we don't have to be the fastest database, if we're the fastest reliable database. That's a liberating change of perspective, which has been behind many of the recent advances.
    - jacques_chester 15y ago
      Does durability absolutely trump performance, or are you prepared to trade off? For example, if durability is an absolute must (as it is for current databases), then your main bottlenecks would be a) spinning rust and b) network traffic to distribute copies of the data throughout the network. SSDs will help this a lot, but already you're back where almost everyone has already started. If you're prepared to accept highly replicated in-memory copies as "durable", you can already make your overall system hundreds or even thousands of times faster. So how do you define durability? 1 copy on 1 disk? x copies on y disks in the same computer? x copies in y disks in different computers? x copies in RAM of y different computers?
ChuckMcM 15y ago
This was the perspective that I have as well, when I was with Network Appliance we spent a lot of time dealing with folks who wanted bullet proof relational databases running on clusters, NetApp provided an extremely reliable storage platform and a unique ability to 'snapshot' instantly the current state of the storage used by the data base so that you could run various experiments against that data without huge copies etc. There were multiply redundant switches and massive investments in transactional data recovery. The big finance guys were particularly intimidating when they talked about several hundred billion dollars in transactions in a 24 hour period, that is $4,000 - $5,000 per mS, or 4 to 5 million dollars per second. Combined with the whole finance game of not confirming receipt until the last possible moment to keep your liabilities managable, ouch. I would love to see a relational database emerge that could do ACID on an unreliable cloud, and I think the folks who achieve that should become gazillionaires. I was just noting that a whole lot of money and research has already been poured into that hole. We're still waiting to here the 'thump' of it hitting the bottom. :-) It totally rocks that Fathom is taking on that challenge.