4 ms·

Announcing the High Replication Datastore for App Engine

62 points by peter123 16y ago

piotrSikora 16y ago
Comparison between Master/Slave (current) and High Replication (just announced) Datastores: http://code.google.com/appengine/docs/python/datastore/hr/ http://code.google.com/appengine/docs/python/datastore/hr/
mmastrac 16y ago
Glad to see this live! We've been testing this for the last couple of months at http://gri.pe http://gri.pe without any issues. It basically means you don't have to worry about datastore maintenance, ever. There hasn't been a significant speed hit for us at all. I highly recommend choosing this option when creating a new AppEngine project. It's more expensive than a standard datastore, but we pay virtually nothing for storage anyways. Thanks to Ikai/Kevin and the rest of the HR datastore folks who made this happen.
- ludwigvan 16y ago
  This is slightly off-topic, but here in Turkey, ghs.google.com is blocked for some reason; and Google App Engine projects with custom domains like http://gri.pe http://gri.pe are blocked as well. (Blogspot blogs with custom domains are blocked too.) So, you should be aware of that if you wonder why you get no hits from Turkey.
  - mmastrac 16y ago
    Whoa. I didn't know that. Thanks!
  - ZoFreX 16y ago
    It's because they block things overzealously. When one video on YouTube offended them, they blocked all of YouTube. Some site running on GAE offended them, so the whole of ghs.google.com got blocked.
rbranson 16y ago
How are they providing consistent get/put/delete in the event of network partition? Is it CP?
- powera 16y ago
  The basic approach is CP; if there is a sufficient network partition, writes will fail. But as long as a majority of the network is connected together, that portion of the servers will be able to function normally (though there may be some extra latency); and traffic can be routed away from the unavailable node.
- brown9-2 16y ago
  I would assume that if one of their datacenters became partitioned from the others in the group, then they would stop serving requests out of that DC entirely. This would alleviate the concern for network partition-tolerance, wouldn't it?
  - rbranson 16y ago
    The problem is that C systems have masters. What happens when the master becomes network isolated? The election and recovery process is tricky. If 100% consistency is assumed, there MUST an outage period until this is recognized and corrected.
    - powera 16y ago
      If you require a read/write to talk to a majority of the replicas, you don't need to have a specific replica as the master to maintain consistency.
marcc 16y ago
That explains the 30 minute outage earlier today!
- powera 16y ago
  While the HR datastore is designed so events like that won't happen (and it was not affected by said outage), there's no correlation between this launch and the datastore unavailability earlier today.
grandalf 16y ago
Can this be selected only for some entity groups?
- mmastrac 16y ago
  No, it's all or none, configured at the appid level.
babyshake 16y ago
They say you can only activate this for new apps. But there's no way AFAIK to do this while keeping the same app id. There should definitely be a way to do a one-time upgrade from master/slave to this new datastore without having to mess around with moving our data from one app to another.
- powera 16y ago
  Well, if you use Google Apps and your own domain, you can point the domain at the new app after copying data over. http://code.google.com/appengine/docs/domain.html http://code.google.com/appengine/docs/domain.html Being able to do a migration online and for the same appspot.com domain is something we'd like to do, but I don't have any timeline as for when that will be available.
  - richardw 16y ago
    Unless you use HTTPS, which currently depends on the appspot domain. In any case, thanks! Very welcome upgrade.
strlen 16y ago
Does this perform Paxos on every single transaction (wouldn't that be expensive?) or is Paxos used to elect a write coordinator replica and ensure a consistent view of the cluster to guarantee serializable consistency? If it's former, can we get more details? This would be quite similar to the work Daniel Abadi had been doing at Yale at parallelizing distributed commit (e.g., Paxos) to maintain high throughput despite increased latency.
- rbranson 16y ago
  I'm going to assume the later. Why would they need to perform Paxos on every single transaction when just using it to elect a master gets 99.9% of the benefits? That would be consistent with the implementation of BigTable in general. EDIT: The higher write latency and 3x CPU cost is probably because it uses synchronous replication to write to all of the slaves before acknowledging the transaction.
  - strlen 16y ago
    > I'm going to assume the later. Why would they need to perform Paxos on every I'm going to assume the later. Why would they need to perform Paxos on every single transaction when just using it to elect a master gets 99.9% of the benefits? That would be consistent with the implementation of BigTable in general. That's my thought as well: essentially they have a bigtable cluster replicated in multiple datacenters, with WALs being shipped between them. That's consistent with the way Google builds systems and with recent papers suggests that correlated failures are more common than one thinks and going to a different datacenter is a wiser strategy than increasing replication factor within a single datacenter.
  - powera 16y ago
    A reasonable thought, but wrong. The existing datastore has basically the 99.9% of the benefits you describe. Unfortunately, 99.9% availability (especially when a single datastore request may touch 10 things and thus be at 99% availability) isn't good enough. One of the major problems with having a single bigtable is partial outages. These can either be the major events like there were today, with 10% or more of the cell having issues, or it can be just one server in the bigtable having performance issues. The key to having higher availability with the High Replication datastore is that if certain entities are slow or unavailable in one datacenter, we can rely on the performance of other datacenters to limit the latency.
Maro 16y ago
You can get scalable Paxos replication for yourself when ScalienDB comes out: http://github.com/scalien/scaliendb http://github.com/scalien/scaliendb (My product.)

62 points by peter123 16y ago

Comparison between Master/Slave (current) and High Replication (just announced) Datastores: http://code.google.com/appengine/docs/python/datastore/hr/ http://code.google.com/appengine/docs/python/datastore/hr/

mmastrac 16y ago

Glad to see this live! We've been testing this for the last couple of months at http://gri.pe http://gri.pe without any issues. It basically means you don't have to worry about datastore maintenance, ever. There hasn't been a significant speed hit for us at all. I highly recommend choosing this option when creating a new AppEngine project. It's more expensive than a standard datastore, but we pay virtually nothing for storage anyways. Thanks to Ikai/Kevin and the rest of the HR datastore folks who made this happen.

ludwigvan 16y ago
This is slightly off-topic, but here in Turkey, ghs.google.com is blocked for some reason; and Google App Engine projects with custom domains like http://gri.pe http://gri.pe are blocked as well. (Blogspot blogs with custom domains are blocked too.) So, you should be aware of that if you wonder why you get no hits from Turkey.
- mmastrac 16y ago
  Whoa. I didn't know that. Thanks!
- ZoFreX 16y ago
  It's because they block things overzealously. When one video on YouTube offended them, they blocked all of YouTube. Some site running on GAE offended them, so the whole of ghs.google.com got blocked.

rbranson 16y ago

How are they providing consistent get/put/delete in the event of network partition? Is it CP?

powera 16y ago
The basic approach is CP; if there is a sufficient network partition, writes will fail. But as long as a majority of the network is connected together, that portion of the servers will be able to function normally (though there may be some extra latency); and traffic can be routed away from the unavailable node.
brown9-2 16y ago
I would assume that if one of their datacenters became partitioned from the others in the group, then they would stop serving requests out of that DC entirely. This would alleviate the concern for network partition-tolerance, wouldn't it?
- rbranson 16y ago
  The problem is that C systems have masters. What happens when the master becomes network isolated? The election and recovery process is tricky. If 100% consistency is assumed, there MUST an outage period until this is recognized and corrected.
  - powera 16y ago
    If you require a read/write to talk to a majority of the replicas, you don't need to have a specific replica as the master to maintain consistency.

marcc 16y ago

That explains the 30 minute outage earlier today!

powera 16y ago
While the HR datastore is designed so events like that won't happen (and it was not affected by said outage), there's no correlation between this launch and the datastore unavailability earlier today.

grandalf 16y ago

Can this be selected only for some entity groups?

mmastrac 16y ago
No, it's all or none, configured at the appid level.

babyshake 16y ago

They say you can only activate this for new apps. But there's no way AFAIK to do this while keeping the same app id. There should definitely be a way to do a one-time upgrade from master/slave to this new datastore without having to mess around with moving our data from one app to another.

powera 16y ago
Well, if you use Google Apps and your own domain, you can point the domain at the new app after copying data over. http://code.google.com/appengine/docs/domain.html http://code.google.com/appengine/docs/domain.html Being able to do a migration online and for the same appspot.com domain is something we'd like to do, but I don't have any timeline as for when that will be available.
- richardw 16y ago
  Unless you use HTTPS, which currently depends on the appspot domain. In any case, thanks! Very welcome upgrade.

strlen 16y ago

Does this perform Paxos on every single transaction (wouldn't that be expensive?) or is Paxos used to elect a write coordinator replica and ensure a consistent view of the cluster to guarantee serializable consistency? If it's former, can we get more details? This would be quite similar to the work Daniel Abadi had been doing at Yale at parallelizing distributed commit (e.g., Paxos) to maintain high throughput despite increased latency.

rbranson 16y ago
I'm going to assume the later. Why would they need to perform Paxos on every single transaction when just using it to elect a master gets 99.9% of the benefits? That would be consistent with the implementation of BigTable in general. EDIT: The higher write latency and 3x CPU cost is probably because it uses synchronous replication to write to all of the slaves before acknowledging the transaction.
- strlen 16y ago
  > I'm going to assume the later. Why would they need to perform Paxos on every I'm going to assume the later. Why would they need to perform Paxos on every single transaction when just using it to elect a master gets 99.9% of the benefits? That would be consistent with the implementation of BigTable in general. That's my thought as well: essentially they have a bigtable cluster replicated in multiple datacenters, with WALs being shipped between them. That's consistent with the way Google builds systems and with recent papers suggests that correlated failures are more common than one thinks and going to a different datacenter is a wiser strategy than increasing replication factor within a single datacenter.
- powera 16y ago
  A reasonable thought, but wrong. The existing datastore has basically the 99.9% of the benefits you describe. Unfortunately, 99.9% availability (especially when a single datastore request may touch 10 things and thus be at 99% availability) isn't good enough. One of the major problems with having a single bigtable is partial outages. These can either be the major events like there were today, with 10% or more of the cell having issues, or it can be just one server in the bigtable having performance issues. The key to having higher availability with the High Replication datastore is that if certain entities are slow or unavailable in one datacenter, we can rely on the performance of other datacenters to limit the latency.

Maro 16y ago

You can get scalable Paxos replication for yourself when ScalienDB comes out: http://github.com/scalien/scaliendb http://github.com/scalien/scaliendb (My product.)