5 ms·

One line summary: “99.999% availability and strong consistency — without compromising latency” (I work at G)

by theDoug 9y ago

jfoutz 9y ago
5 minutes a year is really impressive.
- MichaelRenor 9y ago
  Especially considering that all of AWS database solutions require a 20 minute maintenance window per week!
  - acchow 9y ago
    What are they doing during those 20 minutes? Define "maintenance window" - the DB becomes read-only? Or unreachable? This sounds insane to me.
    - matt_wulfeck 9y ago
      Not usually, but it sometimes will go offline for 5 or 10 minutes or more. It fails over essentially.
    - ReidZB 9y ago
      Actions that require database downtime/restarts can happen during the maintenance window. For example, if you change a DB parameter that requires a database restart, you can tell AWS to restart the DB during the next maintenance window. Same goes for DB engine upgrades and the like. If you're running anything production-worthy, you'll be using one of AWS's multi-AZ/failover solutions for your production RDS instances. In that case, the database will perform a failover in those instances so there isn't downtime.
      - falcolas 9y ago
        That depends on the parameter being changed; not all parameter changes are compatible with block level replication. Such as the MySQL InnoDB log file size, something that needs to be changed on every instance.
      - snoman 9y ago
        > In that case, the database will perform a failover in those instances so there isn't downtime. Have you done that? I have and there's been downtime every time (~5m).
        deleted 9y ago
        [deleted]
  - ReidZB 9y ago
    That doesn't mean they're unavailable for 20 minutes weekly, probably because of the nature of AWS's multi-AZ RDS setup. At our shop, we haven't had any problems with RDS maintenance at all -- and we would indeed get paged for even 1 minute of database downtime, much less 20 (!). That said, we don't have that sort of monitoring in place for any of the test/staging DBs (which are not multi-AZ).
  - BillinghamJ 9y ago
    That's a window when they _can_ do maintenance, it's pretty rare for them to actually do it.
  - gtirloni 9y ago
    The SLA document [0] says 99.95% so perhaps you meant per month? 0 - https://aws.amazon.com/rds/sla/ https://aws.amazon.com/rds/sla/
    - MichaelRenor 9y ago
      You see the trick here, that's 99.95% outside of their regularly scheduled maintenance!
      - gtirloni 9y ago
        Oh! Now I see. I haven't used RDS for anything critical yet but thought it'd be a no-brainer. insert mind expanding gif
        MichaelRenor 9y ago
        In practice it works just fine. They just make you give them permission to blow your app up for twenty minutes a week (and wink and say maybe they won't actually do it this week).
    - d4rti 9y ago
      With respect to SLA's the two important things in my opinion are: * How precisely is it measured? * What happens if it is not met? As observed by the other posters, scheduled maintenance doesn't count.
  - Androider 9y ago
    The RDS service is literally scripted EC2 instances running MySQL/Postgres/SQL Server etc. Anything you do in the RDS console/API runs (or schedules) scripts on EC2 instances. If you would have downtime on any maintenance of your self-managed DB instance (say, upgrading the DB engine), so will RDS. Some people get disillusioned at this fact, thinking RDS was something more magical. But AWS does offer the Multi-AZ option, which does the maintenance on the standby, fails over, and then performs the same on the main instance. It's effectively transparent. RDS also makes backups, restores (including point in time), encryption using KMS etc. really easy. If you don't have a full-time DBA, RDS provides for a very easy to use DB with virtually no maintenance required, but Multi-AZ is absolutely required for any kind of production deployment. AWS's answer to Cloud Spanner is much more likely to be a future cross-region replicating version of DynamoDB or Aurora, not RDS.
    - snoman 9y ago
      As far as Multi-AZ is concerned, it's a bit of a wash. A month ago when the east coast had its issues, Multi-AZ didn't do anything. Every time I need to reboot an instance, with failover it is down/inaccessible for the same amount of time as if I had just rebooted it plainly. I'm sure there's scenarios where it's worked well for people, but I've never seen it happen.
      - brianwawok 9y ago
        I see multi region saves many disasters. Multi AZ seems a waste. What percent of failures only hit 1 AZ? Your loss is for sure (higher latency, bandwidth cost).... unclear what the real world gain is of multi AZ.
- theDoug 9y ago
  I’m in marketing and not here to pump our own tires, but Spanner is /the/ database beneath so much of Google. We trust and rely on it, so it’s very exciting to me to see other big companies who need mission-critical /and/ global scale performance do the same.
  - Filligree 9y ago
    I'd love to use it, but I can't reasonably champion Google's services at work when I'm not allowed to use it for side projects. When is it going to be available for non-companies in Ireland?
    - deleted 9y ago
      [deleted]
    - boulos 9y ago
      It's about VAT collection. If you sign up and warrant that you do it yourself, you could click the button. However, your note reminded me to kick off a specific email about Ireland as a possibility. Disclosure: I work on Google Cloud, but I'm definitely not a lawyer, tax lawyer, accountant or any such financial type.
- bradleyjg 9y ago
  It is, but the details of the SLA aren't that great. https://cloud.google.com/spanner/sla https://cloud.google.com/spanner/sla "'Downtime' means, with respect to any Production-Grade Cloud Spanner Instance, more than a five percent Error Rate for the instance. Downtime is measured based on server side Error Rate. 'Downtime Period' means a period of five consecutive minutes of Downtime with a minimum of 60 requests per minute. Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods." For monthly downtown between 99.0% and 99.999% the customer is credited with 10% of the monthly bill. The customer is required to track the uptime and request an SLA credit.
  - dxhdr 9y ago
    > Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods. That's unfortunate. The most frustrating part about cloud services are exactly these kinds intermittent performance and responsiveness issues.
    - falcolas 9y ago
      It probably won't count network downtime either, since it's measured at the server, not the client. Even if your client is in GAE...
    - snoman 9y ago
      That's like saying 'the most frustrating part about cloud services is that they use the internet.' You're right, but that's basically 'by-design' as far as the internet is concerned.
    - puzzle 9y ago
      Part of the problem is that computing the downtime at a higher resolution, in a way that scales from low-traffic users (the quoted minimum of 60 requests per minute is also not a coincidence) to very large ones, is not trivial. You have to consider the rate at which the raw data is collected from the replicas ---and the jitter across all of them!-- then the rate at which that data is aggregated, and so on. (The above assumes that Cloud Spanner uses Borgmon for monitoring.) Any hiccups in the collection itself will result in less noise if you look at 5m vs e.g. 1m. You could increase the sampling rate, but then you're making your DB replicas spend more precious cycles on work that is not serving queries or replicating data.
nailer 9y ago
That really should be in the title of the article.

One line summary: “99.999% availability and strong consistency — without compromising latency” (I work at G)

jfoutz 9y ago

5 minutes a year is really impressive.

MichaelRenor 9y ago
Especially considering that all of AWS database solutions require a 20 minute maintenance window per week!
- acchow 9y ago
  What are they doing during those 20 minutes? Define "maintenance window" - the DB becomes read-only? Or unreachable? This sounds insane to me.
  - matt_wulfeck 9y ago
    Not usually, but it sometimes will go offline for 5 or 10 minutes or more. It fails over essentially.
  - ReidZB 9y ago
    Actions that require database downtime/restarts can happen during the maintenance window. For example, if you change a DB parameter that requires a database restart, you can tell AWS to restart the DB during the next maintenance window. Same goes for DB engine upgrades and the like. If you're running anything production-worthy, you'll be using one of AWS's multi-AZ/failover solutions for your production RDS instances. In that case, the database will perform a failover in those instances so there isn't downtime.
    - falcolas 9y ago
      That depends on the parameter being changed; not all parameter changes are compatible with block level replication. Such as the MySQL InnoDB log file size, something that needs to be changed on every instance.
    - snoman 9y ago
      > In that case, the database will perform a failover in those instances so there isn't downtime. Have you done that? I have and there's been downtime every time (~5m).
      - deleted 9y ago
        [deleted]
- ReidZB 9y ago
  That doesn't mean they're unavailable for 20 minutes weekly, probably because of the nature of AWS's multi-AZ RDS setup. At our shop, we haven't had any problems with RDS maintenance at all -- and we would indeed get paged for even 1 minute of database downtime, much less 20 (!). That said, we don't have that sort of monitoring in place for any of the test/staging DBs (which are not multi-AZ).
- BillinghamJ 9y ago
  That's a window when they _can_ do maintenance, it's pretty rare for them to actually do it.
- gtirloni 9y ago
  The SLA document [0] says 99.95% so perhaps you meant per month? 0 - https://aws.amazon.com/rds/sla/ https://aws.amazon.com/rds/sla/
  - MichaelRenor 9y ago
    You see the trick here, that's 99.95% outside of their regularly scheduled maintenance!
    - gtirloni 9y ago
      Oh! Now I see. I haven't used RDS for anything critical yet but thought it'd be a no-brainer. insert mind expanding gif
      - MichaelRenor 9y ago
        In practice it works just fine. They just make you give them permission to blow your app up for twenty minutes a week (and wink and say maybe they won't actually do it this week).
  - d4rti 9y ago
    With respect to SLA's the two important things in my opinion are: * How precisely is it measured? * What happens if it is not met? As observed by the other posters, scheduled maintenance doesn't count.
- Androider 9y ago
  The RDS service is literally scripted EC2 instances running MySQL/Postgres/SQL Server etc. Anything you do in the RDS console/API runs (or schedules) scripts on EC2 instances. If you would have downtime on any maintenance of your self-managed DB instance (say, upgrading the DB engine), so will RDS. Some people get disillusioned at this fact, thinking RDS was something more magical. But AWS does offer the Multi-AZ option, which does the maintenance on the standby, fails over, and then performs the same on the main instance. It's effectively transparent. RDS also makes backups, restores (including point in time), encryption using KMS etc. really easy. If you don't have a full-time DBA, RDS provides for a very easy to use DB with virtually no maintenance required, but Multi-AZ is absolutely required for any kind of production deployment. AWS's answer to Cloud Spanner is much more likely to be a future cross-region replicating version of DynamoDB or Aurora, not RDS.
  - snoman 9y ago
    As far as Multi-AZ is concerned, it's a bit of a wash. A month ago when the east coast had its issues, Multi-AZ didn't do anything. Every time I need to reboot an instance, with failover it is down/inaccessible for the same amount of time as if I had just rebooted it plainly. I'm sure there's scenarios where it's worked well for people, but I've never seen it happen.
    - brianwawok 9y ago
      I see multi region saves many disasters. Multi AZ seems a waste. What percent of failures only hit 1 AZ? Your loss is for sure (higher latency, bandwidth cost).... unclear what the real world gain is of multi AZ.
theDoug 9y ago
I’m in marketing and not here to pump our own tires, but Spanner is /the/ database beneath so much of Google. We trust and rely on it, so it’s very exciting to me to see other big companies who need mission-critical /and/ global scale performance do the same.
- Filligree 9y ago
  I'd love to use it, but I can't reasonably champion Google's services at work when I'm not allowed to use it for side projects. When is it going to be available for non-companies in Ireland?
  - deleted 9y ago
    [deleted]
  - boulos 9y ago
    It's about VAT collection. If you sign up and warrant that you do it yourself, you could click the button. However, your note reminded me to kick off a specific email about Ireland as a possibility. Disclosure: I work on Google Cloud, but I'm definitely not a lawyer, tax lawyer, accountant or any such financial type.
bradleyjg 9y ago
It is, but the details of the SLA aren't that great. https://cloud.google.com/spanner/sla https://cloud.google.com/spanner/sla "'Downtime' means, with respect to any Production-Grade Cloud Spanner Instance, more than a five percent Error Rate for the instance. Downtime is measured based on server side Error Rate. 'Downtime Period' means a period of five consecutive minutes of Downtime with a minimum of 60 requests per minute. Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods." For monthly downtown between 99.0% and 99.999% the customer is credited with 10% of the monthly bill. The customer is required to track the uptime and request an SLA credit.
- dxhdr 9y ago
  > Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods. That's unfortunate. The most frustrating part about cloud services are exactly these kinds intermittent performance and responsiveness issues.
  - falcolas 9y ago
    It probably won't count network downtime either, since it's measured at the server, not the client. Even if your client is in GAE...
  - snoman 9y ago
    That's like saying 'the most frustrating part about cloud services is that they use the internet.' You're right, but that's basically 'by-design' as far as the internet is concerned.
  - puzzle 9y ago
    Part of the problem is that computing the downtime at a higher resolution, in a way that scales from low-traffic users (the quoted minimum of 60 requests per minute is also not a coincidence) to very large ones, is not trivial. You have to consider the rate at which the raw data is collected from the replicas ---and the jitter across all of them!-- then the rate at which that data is aggregated, and so on. (The above assumes that Cloud Spanner uses Borgmon for monitoring.) Any hiccups in the collection itself will result in less noise if you look at 5m vs e.g. 1m. You could increase the sampling rate, but then you're making your DB replicas spend more precious cycles on work that is not serving queries or replicating data.

nailer 9y ago

That really should be in the title of the article.