3 ms·

Pretty impressive, but what does "no system-wide downtime" mean? So there was local downtime?

by pors 15y ago

benjaminwootton 15y ago
Well it's a distributed data store so I guess at times they took down certain nodes for upgrades or fixes etc then bought them back up afterwards where they became consistent again. This probably wouldn't have had any user impact. These distributed data stores are so appealing from the perspective of uptime and stability. Needing to upgrade the DB or take cold backups or run massive schema migrations is the reason for lots of scheduled downtime. And removing the huge whopping single point of failure from your system that you would have with a traditional RDBMS can only do good things for your stability.
- pors 15y ago
  Yeah, good point, makes sense
jfoster 15y ago
There absolutely has been App Engine downtime in the past year. They seem to be claiming that the datastore specifically (an App Engine service) hasn't had any downtime. It's not really a good measurement of App Engine's reliability as a whole. Example: http://googleappengine.blogspot.com/2011/07/java-app-engine-outage-july-14-2011.html http://googleappengine.blogspot.com/2011/07/java-app-engine-...
- latchkey 15y ago
  "no HR datastore apps were affected." Sure, it is a small point, but it actually matters a lot. At this point everyone hosting a real service that cares about uptime should be on HR. I had someone from google tell me that GAE lost an entire datacenter and nobody noticed.
vosper 15y ago
Yes, there have been plenty of instances of downtime aside from the HR datastore - e.g. memcache and task queue have both gone down simultaneously and for more than an hour, which can be hugely disruptive but don't count as datastore outage. Also, there are lots of instances of localized downtime that doesn't make it into the global system status - you only have to hang out in the IRC room to see people coming in with "is anyone else seeing really slow startup times" or "is anyone else not seeing logs updating for the last hour".

Pretty impressive, but what does "no system-wide downtime" mean? So there was local downtime?

by pors 15y ago

Pretty impressive, but what does "no system-wide downtime" mean? So there was local downtime?

Well it's a distributed data store so I guess at times they took down certain nodes for upgrades or fixes etc then bought them back up afterwards where they became consistent again. This probably wouldn't have had any user impact. These distributed data stores are so appealing from the perspective of uptime and stability. Needing to upgrade the DB or take cold backups or run massive schema migrations is the reason for lots of scheduled downtime. And removing the huge whopping single point of failure from your system that you would have with a traditional RDBMS can only do good things for your stability.

pors 15y ago
Yeah, good point, makes sense

jfoster 15y ago

There absolutely has been App Engine downtime in the past year. They seem to be claiming that the datastore specifically (an App Engine service) hasn't had any downtime. It's not really a good measurement of App Engine's reliability as a whole. Example: http://googleappengine.blogspot.com/2011/07/java-app-engine-outage-july-14-2011.html http://googleappengine.blogspot.com/2011/07/java-app-engine-...

latchkey 15y ago
"no HR datastore apps were affected." Sure, it is a small point, but it actually matters a lot. At this point everyone hosting a real service that cares about uptime should be on HR. I had someone from google tell me that GAE lost an entire datacenter and nobody noticed.

vosper 15y ago

Yes, there have been plenty of instances of downtime aside from the HR datastore - e.g. memcache and task queue have both gone down simultaneously and for more than an hour, which can be hugely disruptive but don't count as datastore outage. Also, there are lots of instances of localized downtime that doesn't make it into the global system status - you only have to hang out in the IRC room to see people coming in with "is anyone else seeing really slow startup times" or "is anyone else not seeing logs updating for the last hour".