6 ms·

This seems like another reason to not rely on Amazon-specific services, other than the obvious vendor lock-in. At least in the event of an instance outage you

This seems like another reason to not rely on Amazon-specific services, other than the obvious vendor lock-in. At least in the event of an instance outage you could conceivably migrate off Amazon to another VPS provider. No one using DynamoDB has an alternative.

jroid 11y ago
You still depend on some provider. Or, are you talking about multi cloud installations ?
alexbilbie 11y ago
DynamoDB now support cross-region replication [0] so you can build more resilient applications with it [0] http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.CrossRegionRepl.html http://docs.aws.amazon.com/amazondynamodb/latest/developergu...
- andrewchilds 11y ago
  I don't think cross-region replication would've helped in this case: "The replica tables are intended to serve as read-only copies of the data; however, it is possible to write data to a replica table. If you write data to a replica, those changes will not be propagated to the master, or to any other replicas."
  - abalone 11y ago
    You could at least fall back to a read-only mode. That could be very helpful, compared to going down completely.
  - dialtone 11y ago
    You can relatively trivially build multi-master cross-region replication in DynamoDB by using kinesis and writing to kinesis instead of DynamoDB directly. On the consuming end of Kinesis you then fan out to all the DynamoDB (or whatever other database you want to use) regions[1]. Admittedly this only works with some relatively relaxed constraints on the latency you can see, intra region latency can go up to 1 second although rarely, while cross-region is around 3 seconds. An important role is also played by the structure of your objects and how accepting they are of concurrent updates coming from different regions (which is the main reason why the default replication in DynamoDB is not multi-master). [1]: http://tech.adroll.com/blog/data/2015/06/26/kinesis.html http://tech.adroll.com/blog/data/2015/06/26/kinesis.html
rue 11y ago
Alternatively, run a fallback data store.
njharman 11y ago
Source for service with better uptime than Amazon?
- mattkrea 11y ago
  I believe he meant things like using RabbitMQ instead of SQS (SQS isn't very good anyway IMHO) and things like that. More services running on plain old instances rather than AWS specific APIs, Azure specific APIs, etc. This prevents you from getting too locked in and losing control over your application. I honestly can't believe that Netflix can't even load their home page without DynamoDB and all this other stuff. Edit: Looks like Netflix is back online (10:53AM EST)
  - nasalgoat 11y ago
    Yes, exactly. It's tempting to drink the Kool-Aid on the various services but the more you rely on them, the less flexible you become and more open to overall service failures like this one. You can take advantage of the benefits of EC2 without relying on the other services. I mean, if even Netflix can't stay up during this, what hope does a startup have?
    - ukd1 11y ago
      Netflix -IS- up, and working fine. Pretty amazing, imho.
      - deleted 11y ago
        [deleted]
      - toomuchtodo 11y ago
        http://downrightnow.com/netflix http://downrightnow.com/netflix
      - mattkrea 11y ago
        https://imgur.com/NNYLUZK https://imgur.com/NNYLUZK
acdha 11y ago
I don't disagree that you would want to have a rough idea of what migrating off of DynamoDB would require but wouldn't the easier step be using redundancy across regions first? Most of the sites which have suffered downtime due to AWS outages have been operating in only a single region (or even AZ!) and adding the extra level isolation is usually going to be a lot easier than dealing with multiple vendors or having to maintain more of your infrastructure directly.
- speleding 11y ago
  In this case AWS sign in is down (perhaps it depends on Dynamo), so redundancy across regions can only help you if fail over is fully automatic which has it's own problems (split brain syndrome...)
  - acdha 11y ago
    AWS sign-in appears to be working but in general it's true that you need some sort of automatic failover. However, isn't that true for every option other than accepting downtime? It doesn't seem like an AWS-specific challenge other except at the implementation level.
  - sitharus 11y ago
    One of the first things Amazon tell you about AWS is the API is king - the console is hosted in us-east-1 but the API is hosted in each region independently. You don't need automatic failover, but you do need your own scripts to talk to the AWS API. The whole point of AWS is automation.
- zurn 11y ago
  There are many high profile companies down (airbnb, IMDb, tinder, ...) so apparently this is not so straightforward.
  - peterjancelis 11y ago
    I don't see airbnb or IMDB being down. If there's some advice on what small time apps (like mine) can do to get thing up again sooner, please let me know.
mark_l_watson 11y ago
I don't much use it anymore but this is something I always liked about AppEngine: baked into the pricing is automatic replication over three regions. Serious AWS web apps are also distributed across multiple regions/availability zones, but many companies don't do that. It would be good for hosting services like Heroku, which I use, would replicate across zones also. To be fare to Amazon, they tell you you architect and built across availability zones.
- hox 11y ago
  dynamo db is affected in the entire region, however, not just a single AZ.
  - DinkyG 11y ago
    That's why parent post mentioned multiple regions. No other AWS region was affected...
jbeda 11y ago
Very good point. I wrote a blog post on this topic to try and characterize different types of lock-in. http://www.eightypercent.net/post/types-of-lock-in.html http://www.eightypercent.net/post/types-of-lock-in.html This was as also part of the strategy when we decided to open source kubernetes. Having an open alternative made the commercial offering (Google Container Engine -- GKE) much stronger because of the reduced dev lock-in.
onetom 11y ago
I would highly recommend http://www.datomic.com http://www.datomic.com as a database for multiple reasons, but one of them is that fact that it's totally portable between DynamoDB, PostgreSQL, MySQL, etc. You can create a Datomic backup from any of these databases and restore them into a different one with the exact same semantics.
- jsprogrammer 11y ago
  Isn't datomic closed source?
  - nadams 11y ago
    That's what it looks like. And after 10 seconds of reading I couldn't tell what it actually does.
  - onetom 11y ago
    It is closed source, just like DynamoDB. However - unlike DynamoDB - it has a quite usable free version and an even better starter version, which are good enough for production use. (The DDB local version is really only good for development) By production use, I mean for example a non-replicated MongoDB/PgSQL/MySQL, which is what I see from many smaller companies... They would be already really better of with Datomic. If you really need more features, like high availability and transparent query caching with memcached, then the cheapest paid version has a one-time cost of $3000: http://www.datomic.com/pricing.html http://www.datomic.com/pricing.html You can easily save that amount of money by not spending 3-6 months developing tons of unnecessary queries, try to maintain that code and try to optimize its performance for scale... On the "open-sourceness" note, I feel obliged to mention https://github.com/tonsky/datascript https://github.com/tonsky/datascript which IS open-source, and has a Datomic-like interface, but it's "just" an in-memory implementation. But we are getting off-topic here I guess...
dialtone 11y ago
This really is the first major failure of DynamoDB since when I started to use it, when it was in beta over 4 years ago. Not sure many companies would be able to run a multi-million QPS scale database with such a great availability record. In any case we just moved traffic away from us-east-1 and into us-west-2 and all is fine still so you should design at least your critical systems for regional failures.
faizalzakaria 11y ago
Kinda agree, same goes to AWS tools, Opsworks screwed us up back then due to AWS outage, and decided to decouple as much as possible from Opsworks.
cle 11y ago
How would this have helped? In the event of an AWS outage, you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure. This outage lasted a few hours, wouldn't it take you that long to switch providers anyway? If you don't abdicate responsibility for maintaining the infrastructure, you lose a lot of the operational benefits of using a cloud service...in general, I think it makes a lot more sense to embrace whatever cloud provider you're using (provided you made a good choice) so that you can focus your efforts on your product and not on maintaining infrastructure.
- kordless 11y ago
  > you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure No, they are fixing their infrastructure. The point here is that all single provider systems are destined for periodic failure. Not relying on one single provider is, in theory, a service's means of providing higher reliability. This is the general argument for regions and availability zones on Amazon, but that relies on trusting there is no single point of failure with the system (i.e. inside Amazon). I've worked for several companies who run the majority of their services on AWS, yet maintain functional systems on other providers in the case of an AWS outage.
- rdtsc 11y ago
  > wouldn't it take you that long to switch providers anyway? It might, or might not, depends how you built it. > you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure. And it still was out for hours... You might see it as a great success, but you can also interpret it as "what good is it, even if with so many people behind the scenes they still failed for hours...". The problem is that because that services ends up being embedded in so many services and products, half the internet ends being down. Even Amazon dogfoods their own stuff so services rely on each other. Dynamo is down, maybe SQS will be down or analytics as well.
deleted 11y ago
[deleted]
DinkyG 11y ago
DynamoDb was only down in only one region, for the first time in years. It's hardly a reason to migrate off, especially with DynamoDb still being available in every single other region throughout. It's simpler to fail over to another AWS region than to an entirely different cloud provider with a different API.

jroid 11y ago

You still depend on some provider. Or, are you talking about multi cloud installations ?

alexbilbie 11y ago

DynamoDB now support cross-region replication [0] so you can build more resilient applications with it [0] http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.CrossRegionRepl.html http://docs.aws.amazon.com/amazondynamodb/latest/developergu...

andrewchilds 11y ago
I don't think cross-region replication would've helped in this case: "The replica tables are intended to serve as read-only copies of the data; however, it is possible to write data to a replica table. If you write data to a replica, those changes will not be propagated to the master, or to any other replicas."
- abalone 11y ago
  You could at least fall back to a read-only mode. That could be very helpful, compared to going down completely.
- dialtone 11y ago
  You can relatively trivially build multi-master cross-region replication in DynamoDB by using kinesis and writing to kinesis instead of DynamoDB directly. On the consuming end of Kinesis you then fan out to all the DynamoDB (or whatever other database you want to use) regions[1]. Admittedly this only works with some relatively relaxed constraints on the latency you can see, intra region latency can go up to 1 second although rarely, while cross-region is around 3 seconds. An important role is also played by the structure of your objects and how accepting they are of concurrent updates coming from different regions (which is the main reason why the default replication in DynamoDB is not multi-master). [1]: http://tech.adroll.com/blog/data/2015/06/26/kinesis.html http://tech.adroll.com/blog/data/2015/06/26/kinesis.html

rue 11y ago

Alternatively, run a fallback data store.

njharman 11y ago

Source for service with better uptime than Amazon?

mattkrea 11y ago
I believe he meant things like using RabbitMQ instead of SQS (SQS isn't very good anyway IMHO) and things like that. More services running on plain old instances rather than AWS specific APIs, Azure specific APIs, etc. This prevents you from getting too locked in and losing control over your application. I honestly can't believe that Netflix can't even load their home page without DynamoDB and all this other stuff. Edit: Looks like Netflix is back online (10:53AM EST)
- nasalgoat 11y ago
  Yes, exactly. It's tempting to drink the Kool-Aid on the various services but the more you rely on them, the less flexible you become and more open to overall service failures like this one. You can take advantage of the benefits of EC2 without relying on the other services. I mean, if even Netflix can't stay up during this, what hope does a startup have?
  - ukd1 11y ago
    Netflix -IS- up, and working fine. Pretty amazing, imho.
    - deleted 11y ago
      [deleted]
    - toomuchtodo 11y ago
      http://downrightnow.com/netflix http://downrightnow.com/netflix
    - mattkrea 11y ago
      https://imgur.com/NNYLUZK https://imgur.com/NNYLUZK

acdha 11y ago

I don't disagree that you would want to have a rough idea of what migrating off of DynamoDB would require but wouldn't the easier step be using redundancy across regions first? Most of the sites which have suffered downtime due to AWS outages have been operating in only a single region (or even AZ!) and adding the extra level isolation is usually going to be a lot easier than dealing with multiple vendors or having to maintain more of your infrastructure directly.

speleding 11y ago
In this case AWS sign in is down (perhaps it depends on Dynamo), so redundancy across regions can only help you if fail over is fully automatic which has it's own problems (split brain syndrome...)
- acdha 11y ago
  AWS sign-in appears to be working but in general it's true that you need some sort of automatic failover. However, isn't that true for every option other than accepting downtime? It doesn't seem like an AWS-specific challenge other except at the implementation level.
- sitharus 11y ago
  One of the first things Amazon tell you about AWS is the API is king - the console is hosted in us-east-1 but the API is hosted in each region independently. You don't need automatic failover, but you do need your own scripts to talk to the AWS API. The whole point of AWS is automation.
zurn 11y ago
There are many high profile companies down (airbnb, IMDb, tinder, ...) so apparently this is not so straightforward.
- peterjancelis 11y ago
  I don't see airbnb or IMDB being down. If there's some advice on what small time apps (like mine) can do to get thing up again sooner, please let me know.

mark_l_watson 11y ago

I don't much use it anymore but this is something I always liked about AppEngine: baked into the pricing is automatic replication over three regions. Serious AWS web apps are also distributed across multiple regions/availability zones, but many companies don't do that. It would be good for hosting services like Heroku, which I use, would replicate across zones also. To be fare to Amazon, they tell you you architect and built across availability zones.

hox 11y ago
dynamo db is affected in the entire region, however, not just a single AZ.
- DinkyG 11y ago
  That's why parent post mentioned multiple regions. No other AWS region was affected...

jbeda 11y ago

Very good point. I wrote a blog post on this topic to try and characterize different types of lock-in. http://www.eightypercent.net/post/types-of-lock-in.html http://www.eightypercent.net/post/types-of-lock-in.html This was as also part of the strategy when we decided to open source kubernetes. Having an open alternative made the commercial offering (Google Container Engine -- GKE) much stronger because of the reduced dev lock-in.

onetom 11y ago

I would highly recommend http://www.datomic.com http://www.datomic.com as a database for multiple reasons, but one of them is that fact that it's totally portable between DynamoDB, PostgreSQL, MySQL, etc. You can create a Datomic backup from any of these databases and restore them into a different one with the exact same semantics.

jsprogrammer 11y ago
Isn't datomic closed source?
- nadams 11y ago
  That's what it looks like. And after 10 seconds of reading I couldn't tell what it actually does.
- onetom 11y ago
  It is closed source, just like DynamoDB. However - unlike DynamoDB - it has a quite usable free version and an even better starter version, which are good enough for production use. (The DDB local version is really only good for development) By production use, I mean for example a non-replicated MongoDB/PgSQL/MySQL, which is what I see from many smaller companies... They would be already really better of with Datomic. If you really need more features, like high availability and transparent query caching with memcached, then the cheapest paid version has a one-time cost of $3000: http://www.datomic.com/pricing.html http://www.datomic.com/pricing.html You can easily save that amount of money by not spending 3-6 months developing tons of unnecessary queries, try to maintain that code and try to optimize its performance for scale... On the "open-sourceness" note, I feel obliged to mention https://github.com/tonsky/datascript https://github.com/tonsky/datascript which IS open-source, and has a Datomic-like interface, but it's "just" an in-memory implementation. But we are getting off-topic here I guess...

dialtone 11y ago

This really is the first major failure of DynamoDB since when I started to use it, when it was in beta over 4 years ago. Not sure many companies would be able to run a multi-million QPS scale database with such a great availability record. In any case we just moved traffic away from us-east-1 and into us-west-2 and all is fine still so you should design at least your critical systems for regional failures.

faizalzakaria 11y ago

Kinda agree, same goes to AWS tools, Opsworks screwed us up back then due to AWS outage, and decided to decouple as much as possible from Opsworks.

cle 11y ago

How would this have helped? In the event of an AWS outage, you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure. This outage lasted a few hours, wouldn't it take you that long to switch providers anyway? If you don't abdicate responsibility for maintaining the infrastructure, you lose a lot of the operational benefits of using a cloud service...in general, I think it makes a lot more sense to embrace whatever cloud provider you're using (provided you made a good choice) so that you can focus your efforts on your product and not on maintaining infrastructure.

kordless 11y ago
> you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure No, they are fixing their infrastructure. The point here is that all single provider systems are destined for periodic failure. Not relying on one single provider is, in theory, a service's means of providing higher reliability. This is the general argument for regions and availability zones on Amazon, but that relies on trusting there is no single point of failure with the system (i.e. inside Amazon). I've worked for several companies who run the majority of their services on AWS, yet maintain functional systems on other providers in the case of an AWS outage.
rdtsc 11y ago
> wouldn't it take you that long to switch providers anyway? It might, or might not, depends how you built it. > you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure. And it still was out for hours... You might see it as a great success, but you can also interpret it as "what good is it, even if with so many people behind the scenes they still failed for hours...". The problem is that because that services ends up being embedded in so many services and products, half the internet ends being down. Even Amazon dogfoods their own stuff so services rely on each other. Dynamo is down, maybe SQS will be down or analytics as well.

deleted 11y ago

[deleted]

DinkyG 11y ago

DynamoDb was only down in only one region, for the first time in years. It's hardly a reason to migrate off, especially with DynamoDb still being available in every single other region throughout. It's simpler to fail over to another AWS region than to an entirely different cloud provider with a different API.