6 ms·

So long AWS, and thanks for all the fish: FathomDB (W08) focusing on its new DB

28 points by justinsb 15y ago

Joakal 15y ago
This author blames Amazon for a poor SLA yet continues using their service? What cloud provider offers a more reasonable SLA?
- justinsb 15y ago
  I'm the author. I think you're conflating the SLA with the guarantees the cloud offers: one of the points I tried to make was that an SLA doesn't offer you any real guarantees; they only really benefit the service provider. If every SLA gave you a "50 times" penalty, as FathomDB is paying, then they would be interesting, because at that level the provider has to take uptime seriously. In terms of who offers a better cloud, clouds based on OpenStack are 'the great hope'. IaaS is about commoditizing infrastructure, so open source seems necessary; rather than AWS's very closed, secretive approach.
  - Joakal 15y ago
    I should have rephrased better; "What provider offers a SLA that takes IaaS uptime more seriously than Amazon?"
  - mtrimpe 15y ago
    Given that this is the first blog post and you still seem to be in private beta... how many customers do you actually have now? One? ;)
    - justinsb 15y ago
      Note to self: must blog more :-)
  - sunchild 15y ago
    I deal with SLAs in my job all the time. After over a decade negotiating uptime SLAs, I'm firmly convinced that the best protection against downtime is a huge, vocal userbase, and the best remedy for downtime is an unqualified termination right and refund of prepaid fees. An SLA (even with severe penalties) is certainly setting up the right incentives on paper, but in practice it has little effect on the results. Why? Because enforcing a promise on a piece of paper is bothersome, expensive and sours the relationship. Also, saying you'll pay 50x penalty is only half the story. What are the triggering events that entitle customers to payment? Is the SLA measured end-to-end (i.e., from the user's endpoint to yours)? Is the SLA penalty cumulative? Can your customers actually profit from your downtime (i.e., you would end up owing them more cash than they paid you)? Without more details, your post suggests that each fraction of a second of downtime experienced by any single user of your service entitles them to a credit of 50x what they paid for that fraction of a second. Is that your policy?
    - justinsb 15y ago
      Thanks for the expert input. This is exactly why we don't have a SLA at FathomDB, because I don't think that they're worth the paper they're written on. It seems like you almost agree (?) Most cloud providers (including FathomDB) charge by the hour anyway, so you have termination rights for any reason, and there are no prepayments to refund. We think that 'doing the right thing' is the only workable policy, and it shouldn't require an outcry from a vocal userbase. We should have handled the outage better, and we're making a big payment to demonstrate that. We're not going to let any new customers use a system where we can't be confident in its uptime (MySQL on AWS); we're building a new system in which we can be confident.
      - sunchild 15y ago
        Dead on. I went back and re-read the article, and yes – we are in agreement. It's about doing whatever it takes to keep the relationship with your customers on an even keel. I couldn't agree more that SLAs are generally bad for customers, even though the perception is the opposite. I can't tell you how many times I've had a client demand an SLA from their vendor, and when that SLA arrives, it reads like a commitment, but actually it's impossible to fail. And these almost always are wrapped in "exclusive remedy" language, which quite literally means the customer would be better off without it. The devil is in the details. Cheers!
ChuckMcM 15y ago
This reads poorly. FathomDB customers saw outages because the implementation Fathom did using MySQL on AWS went out when AWS went out. So in conclusion Fathom is going to build something which Oracle, IBM, SyBase, and every other database company in the history of the planet has tried and failed to do, build a relational database on a distributed infrastucture that is more reliable than the underlying infrastructure? Uh, good luck with that. Seriously, its Turing Prize material if you succeed. I was thinking the punch line would be "Gee Netflix uses AWS and they didn't go down so we're going to more what they did." I guess they are going to compete with MongoDB and Riak.
- justinsb 15y ago
  Thanks for the good-luck wishes. Obviously it's a big undertaking, but we're seeing very promising results, so it's time to go all-in. Take that as you will :-) Incidentally, Oracle and IBM both have clustered database products, though they can certainly be improved upon.
  - jacques_chester 15y ago
    Oracle and IBM do have clustered stuff and it's amazing, but it still tends to assume closely linked networks of high-reliability hardware and have trouble with very large, slowly linked networks of low-reliability hardware. You might find Greenplum and VoltDB to be more interesting models to study. The key question is: are you aiming at OLTP or OLAP? Because, as those boring mainframe-era guys have discovered, it's difficult to serve two very different purposes.
    - justinsb 15y ago
      OLTP. I personally believe Hadoop has got the OLAP problem-space covered.
      - jacques_chester 15y ago
        What are your most important non-functional requirements? In particular, how do you see the tradeoff between durability and performance?
unshift 15y ago
sorry, this is probably tangential, but how is the AWS outage any different from any other hardware outage, regardless of whether or not you actually own the hardware? why does it seem that everyone expects 100% uptime from a VM just because it's in "the cloud"? shouldn't "the cloud" be used to make fault-tolerance even easier, because you have access to multiple geographic regions and multiple providers with little fuss? they're still computers, and they're still bound to go down occasionally. i don't see how running leased VMs somehow absolves you of doing basic operations work and guarantees a bulletproof experience. regardless of where you host it, writing a nicely distributed and fault-tolerant system is and always has been difficult. the only part that's gotten easier is finding rack space.
- justinsb 15y ago
  With physical hardware, the basic failure characteristics etc are well known. When the hardware is virtual or abstracted as it is on the cloud, you have no choice but to rely on the promises made by the provider. If you can't build a reliable system on the abstraction provided by the cloud, you can't reliably use the cloud. But with a good cloud, you can architect a reasonable solution based on the provider's promises. However, if the provider doesn't keep those promises, then all your hard work and calculations go totally out the window. At that point, you have to figure out a way to run a database on a system that effectively offers you no guarantees. That's what we're working on.
  - dialtone 15y ago
    For your application though a failure is a failure in any scenario. EBS went down in one zone and was slow in a couple others for a few hours, it's a bad downtime but some services survived the problem relatively unscathed. This suggests that something could have been done to avoid any trouble at all. If the datacenter you are in has a slight conditioning problem this summer and 10% of your drives breaks down due to excessive heat, how quickly will you be able to re-provision the data center? About a week after AWS outage the italian ISP Aruba had a UPS failure due to a fire in the UPS room and the entire datacenter switched off automatically during the night. For the following 8 hours that datacenter was off for every customer. How would your new solution handle such a situation? Designing a hot standby replication solution for MySQL/PostgreSQL that works across regions seems easier to me rather than implementing a database from scratch that should solve a very complex problem.
soamv 15y ago
Can you actually tell us a bit about the new DB you're building? (What is it designed to run on, what kind of failures are you aiming to tolerate, and so on.)
- justinsb 15y ago
  Sorry, not too many details yet - let us finish it first :-) Commodity hardware though, and quorum-style failure-tolerance characteristics.

28 points by justinsb 15y ago

This author blames Amazon for a poor SLA yet continues using their service? What cloud provider offers a more reasonable SLA?

justinsb 15y ago
I'm the author. I think you're conflating the SLA with the guarantees the cloud offers: one of the points I tried to make was that an SLA doesn't offer you any real guarantees; they only really benefit the service provider. If every SLA gave you a "50 times" penalty, as FathomDB is paying, then they would be interesting, because at that level the provider has to take uptime seriously. In terms of who offers a better cloud, clouds based on OpenStack are 'the great hope'. IaaS is about commoditizing infrastructure, so open source seems necessary; rather than AWS's very closed, secretive approach.
- Joakal 15y ago
  I should have rephrased better; "What provider offers a SLA that takes IaaS uptime more seriously than Amazon?"
- mtrimpe 15y ago
  Given that this is the first blog post and you still seem to be in private beta... how many customers do you actually have now? One? ;)
  - justinsb 15y ago
    Note to self: must blog more :-)
- sunchild 15y ago
  I deal with SLAs in my job all the time. After over a decade negotiating uptime SLAs, I'm firmly convinced that the best protection against downtime is a huge, vocal userbase, and the best remedy for downtime is an unqualified termination right and refund of prepaid fees. An SLA (even with severe penalties) is certainly setting up the right incentives on paper, but in practice it has little effect on the results. Why? Because enforcing a promise on a piece of paper is bothersome, expensive and sours the relationship. Also, saying you'll pay 50x penalty is only half the story. What are the triggering events that entitle customers to payment? Is the SLA measured end-to-end (i.e., from the user's endpoint to yours)? Is the SLA penalty cumulative? Can your customers actually profit from your downtime (i.e., you would end up owing them more cash than they paid you)? Without more details, your post suggests that each fraction of a second of downtime experienced by any single user of your service entitles them to a credit of 50x what they paid for that fraction of a second. Is that your policy?
  - justinsb 15y ago
    Thanks for the expert input. This is exactly why we don't have a SLA at FathomDB, because I don't think that they're worth the paper they're written on. It seems like you almost agree (?) Most cloud providers (including FathomDB) charge by the hour anyway, so you have termination rights for any reason, and there are no prepayments to refund. We think that 'doing the right thing' is the only workable policy, and it shouldn't require an outcry from a vocal userbase. We should have handled the outage better, and we're making a big payment to demonstrate that. We're not going to let any new customers use a system where we can't be confident in its uptime (MySQL on AWS); we're building a new system in which we can be confident.
    - sunchild 15y ago
      Dead on. I went back and re-read the article, and yes – we are in agreement. It's about doing whatever it takes to keep the relationship with your customers on an even keel. I couldn't agree more that SLAs are generally bad for customers, even though the perception is the opposite. I can't tell you how many times I've had a client demand an SLA from their vendor, and when that SLA arrives, it reads like a commitment, but actually it's impossible to fail. And these almost always are wrapped in "exclusive remedy" language, which quite literally means the customer would be better off without it. The devil is in the details. Cheers!

ChuckMcM 15y ago

This reads poorly. FathomDB customers saw outages because the implementation Fathom did using MySQL on AWS went out when AWS went out. So in conclusion Fathom is going to build something which Oracle, IBM, SyBase, and every other database company in the history of the planet has tried and failed to do, build a relational database on a distributed infrastucture that is more reliable than the underlying infrastructure? Uh, good luck with that. Seriously, its Turing Prize material if you succeed. I was thinking the punch line would be "Gee Netflix uses AWS and they didn't go down so we're going to more what they did." I guess they are going to compete with MongoDB and Riak.

justinsb 15y ago
Thanks for the good-luck wishes. Obviously it's a big undertaking, but we're seeing very promising results, so it's time to go all-in. Take that as you will :-) Incidentally, Oracle and IBM both have clustered database products, though they can certainly be improved upon.
- jacques_chester 15y ago
  Oracle and IBM do have clustered stuff and it's amazing, but it still tends to assume closely linked networks of high-reliability hardware and have trouble with very large, slowly linked networks of low-reliability hardware. You might find Greenplum and VoltDB to be more interesting models to study. The key question is: are you aiming at OLTP or OLAP? Because, as those boring mainframe-era guys have discovered, it's difficult to serve two very different purposes.
  - justinsb 15y ago
    OLTP. I personally believe Hadoop has got the OLAP problem-space covered.
    - jacques_chester 15y ago
      What are your most important non-functional requirements? In particular, how do you see the tradeoff between durability and performance?

unshift 15y ago

sorry, this is probably tangential, but how is the AWS outage any different from any other hardware outage, regardless of whether or not you actually own the hardware? why does it seem that everyone expects 100% uptime from a VM just because it's in "the cloud"? shouldn't "the cloud" be used to make fault-tolerance even easier, because you have access to multiple geographic regions and multiple providers with little fuss? they're still computers, and they're still bound to go down occasionally. i don't see how running leased VMs somehow absolves you of doing basic operations work and guarantees a bulletproof experience. regardless of where you host it, writing a nicely distributed and fault-tolerant system is and always has been difficult. the only part that's gotten easier is finding rack space.

justinsb 15y ago
With physical hardware, the basic failure characteristics etc are well known. When the hardware is virtual or abstracted as it is on the cloud, you have no choice but to rely on the promises made by the provider. If you can't build a reliable system on the abstraction provided by the cloud, you can't reliably use the cloud. But with a good cloud, you can architect a reasonable solution based on the provider's promises. However, if the provider doesn't keep those promises, then all your hard work and calculations go totally out the window. At that point, you have to figure out a way to run a database on a system that effectively offers you no guarantees. That's what we're working on.
- dialtone 15y ago
  For your application though a failure is a failure in any scenario. EBS went down in one zone and was slow in a couple others for a few hours, it's a bad downtime but some services survived the problem relatively unscathed. This suggests that something could have been done to avoid any trouble at all. If the datacenter you are in has a slight conditioning problem this summer and 10% of your drives breaks down due to excessive heat, how quickly will you be able to re-provision the data center? About a week after AWS outage the italian ISP Aruba had a UPS failure due to a fire in the UPS room and the entire datacenter switched off automatically during the night. For the following 8 hours that datacenter was off for every customer. How would your new solution handle such a situation? Designing a hot standby replication solution for MySQL/PostgreSQL that works across regions seems easier to me rather than implementing a database from scratch that should solve a very complex problem.

soamv 15y ago

Can you actually tell us a bit about the new DB you're building? (What is it designed to run on, what kind of failures are you aiming to tolerate, and so on.)

justinsb 15y ago
Sorry, not too many details yet - let us finish it first :-) Commodity hardware though, and quorum-style failure-tolerance characteristics.