5 ms·

Fair enough; I think it depends where you draw the line between key/value store and database. Both of those depend on other distributed storage systems under t

Fair enough; I think it depends where you draw the line between key/value store and database. Both of those depend on other distributed storage systems under the hood, as far as I am aware? Or is Datastore an end to end system? I know Firebase was backed by MongoDB.

elvinyung 10y ago
If I remember correctly, Datastore is basically a thin layer on top of Megastore[1] (aka the precursor to Spanner). 1: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36971.pdf https://static.googleusercontent.com/media/research.google.c...
- itcmcgrath 10y ago
  It's a lot more than a thin layer, but like most systems at Google is a specialized layer that utilizes built on top of more fundamental building blocks that are designed and proven to do a particular job really well. In this case Megastore provides the underly multi-region/datacenter K-V replication services. All the database features like secondary & composite indexes, query language, multi-tenancy support, PAYG model, etc, etc, are built in the Cloud Datastore layer.
  - elvinyung 10y ago
    > secondary & composite indexes Interesting, so you don't use Megastore's indexes?
    - wsh91 10y ago
      As you noted earlier, Megastore has a schema and we don't, so we have our own index implementation, yes. :)
wsh91 10y ago
Datastore runs on top of Megastore [1]. You can find out more about our data model here [2], but it's definitely not limited to key-value data. Our end users don't have to think much about our storage system, though, if we're doing our jobs right. :) [1] https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/ https://cloud.google.com/datastore/docs/articles/balancing-s... [2] https://cloud.google.com/datastore/docs/concepts/entities https://cloud.google.com/datastore/docs/concepts/entities
- elvinyung 10y ago
  An interesting observation that I can't seem to "un-observe" is that Megastore is actually a lot more like MongoDB than one would expect. Both ostensibly work best when the application fits a hierarchical data model (entity groups vs. documents), and provide out-of-the-box strongly-consistent transactions for a single entity group. MongoDB feels like schemaless Megastore.
  - wsh91 10y ago
    :) Here's a paper with which you might already be familiar, but it's one of the citations for the Megastore paper: http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pdf http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pd.... You'll probably enjoy it (if you haven't already!).

elvinyung 10y ago

If I remember correctly, Datastore is basically a thin layer on top of Megastore[1] (aka the precursor to Spanner). 1: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36971.pdf https://static.googleusercontent.com/media/research.google.c...

itcmcgrath 10y ago
It's a lot more than a thin layer, but like most systems at Google is a specialized layer that utilizes built on top of more fundamental building blocks that are designed and proven to do a particular job really well. In this case Megastore provides the underly multi-region/datacenter K-V replication services. All the database features like secondary & composite indexes, query language, multi-tenancy support, PAYG model, etc, etc, are built in the Cloud Datastore layer.
- elvinyung 10y ago
  > secondary & composite indexes Interesting, so you don't use Megastore's indexes?
  - wsh91 10y ago
    As you noted earlier, Megastore has a schema and we don't, so we have our own index implementation, yes. :)

wsh91 10y ago

Datastore runs on top of Megastore [1]. You can find out more about our data model here [2], but it's definitely not limited to key-value data. Our end users don't have to think much about our storage system, though, if we're doing our jobs right. :) [1] https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/ https://cloud.google.com/datastore/docs/articles/balancing-s... [2] https://cloud.google.com/datastore/docs/concepts/entities https://cloud.google.com/datastore/docs/concepts/entities

elvinyung 10y ago
An interesting observation that I can't seem to "un-observe" is that Megastore is actually a lot more like MongoDB than one would expect. Both ostensibly work best when the application fits a hierarchical data model (entity groups vs. documents), and provide out-of-the-box strongly-consistent transactions for a single entity group. MongoDB feels like schemaless Megastore.
- wsh91 10y ago
  :) Here's a paper with which you might already be familiar, but it's one of the citations for the Megastore paper: http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pdf http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pd.... You'll probably enjoy it (if you haven't already!).