7 ms·

Spanner: Becoming a SQL System [pdf]

162 points by elvinyung 9y ago

elvinyung 9y ago
This "modern" Spanner feels very different from the one we saw in 2012 [1]. Some interesting takeaways: * There is a native SQL interface in Spanner, rather than relying on a separate upper-layer SQL layer, a la F1 [2] * Spanner is no longer on top of Bigtable! Instead, the storage engine seems to be a heavily modified Bigtable with a column-oriented file format * Data is resharded frequently and concurrently with other operations -- the shard layout is abstracted away from the query plan using the "distributed union" operator * Possible explanation for why Spanner doesn't support SQL DML writes: writes are required to be the last step of a transaction, and there is currently no support for reading uncommitted writes (this is in contrast to F1, which does support DML) * Spanner supports full-text search (!) [1] https://research.google.com/archive/spanner.html https://research.google.com/archive/spanner.html [2] https://research.google.com/pubs/pub41344.html https://research.google.com/pubs/pub41344.html
- iambvk 9y ago
  Did not digest the paper yet. Does it provide nested transactions? My understanding from 2012 paper was that it doesn't support nested transactions, even at the storage layer. Can anybody provide insider-knowledge if it was even a requested feature from Google devs internally?
  - elvinyung 9y ago
    Since writes are required to be the last step of a transaction, I suspect there wouldn't be much of a point in having nested transactions.
  - sanxiyn 9y ago
    From the paper: > Overcoming this limitation requires supporting reading the uncommitted results within an active transaction. While we have seen comparatively little demand for this feature internally, supporting such semantics improves compatibility with other SQL systems and their ecosystems and is on our long-term radar. No internal demand, but on "long-term radar".
gravypod 9y ago
Is Spanner free/open source software? Can we look at the code?
- tav 9y ago
  No, but it is available as part of Google's cloud services [0]. Particularly noteworthy is the fact that they offer an SLA with 99.99% availability. Even crazier is their pending multi-region version due later this year which will come with a 99.999% SLA!! [0] https://cloud.google.com/spanner/ https://cloud.google.com/spanner/
- kyrra 9y ago
  SaaS. Even if it was open sourced, as the original paper calls out, there is specialized hardware involved for keep clocks in sync[0]. If you want something open source, cockroachdb is the closest right now. https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo...
  - aliasaria 9y ago
    There is also TiDB https://github.com/pingcap/tidb/blob/master/README.md https://github.com/pingcap/tidb/blob/master/README.md
    - forgot-my-pw 9y ago
      With MySQL-compatible, this sounds really promising: https://www.pingcap.com/doc-mysql-compatibility https://www.pingcap.com/doc-mysql-compatibility
  - derefr 9y ago
    Specialized, but not proprietary. Spanner presumably uses off-the-shelf Chip-Scale Atomic Clock (CSAC) modules with custom system-level integration. But you can buy CSACs yourself—mounted on PCIe cards—for your own data center. (Every site I can find is of the "request a quote" variety where prices are concerned. They were apparently $1500 apiece in 2011, and have probably come down since then.) But you don't need an atomic clock to get Spanner's guarantees. CSACs are more convenient in terms of setup and requirements—but GPS clock-sources will do just fine for Spanner's 6ms quantum. You can buy a commercial-off-the-shelf GPS NTP appliance (https://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/network-appliances-servers/syncserver/syncserver-s650 https://www.microsemi.com/products/timing-synchronization-sy...) and run signal lines from it to all your machines; or cobble a similar solution together yourself, in true HAM fashion, using a GPS antenna + a Linux box + a UHF Software-Defined Radio card (i.e. a TV-tuner card) + GPSd + NTPd. (Or or you could buy cheap USB GPS receivers and hook them up to each and every server in your DC—but you'd need a very thin ceiling for that to work.) Amusingly, the practicality of that last approach also means that you could run Spanner just fine on a cluster of Android phones. :)
speedplane 9y ago
I was at the Google Cloud conference a few months ago and spoke to a few engineers there about Spanner. When I asked them about how it would affect their other storage options (e.g., datastore, cloud sql, etc.), they said that over time, all of their internal storage systems would be moved over to Spanner. His words were "you'll be using Spanner whether you know it or not." This engineer was clearly a cheerleader for the product, so I'm dubious as to whether that will actually happen, but it's clear that they have quite a bit of confidence in it.
- deleted 9y ago
  [deleted]
- Artemis2 9y ago
  That sounds a lot like the announcement of Azure Cosmos DB: https://docs.microsoft.com/en-us/azure/cosmos-db/introduction https://docs.microsoft.com/en-us/azure/cosmos-db/introductio... > we made the service available externally to all Azure Developers in the form of Azure DocumentDB. Azure Cosmos DB is the next big leap in the evolution of DocumentDB and we are now making it available for you to use. As a part of this release of Azure Cosmos DB, DocumentDB customers (with their data) are automatically Azure Cosmos DB customers. From my understanding, DocumentDB ≈ Dynamo, while Cosmos DB would be closer to Cloud Spanner.
- pacala 9y ago
  Google's game is planet scale systems. Cloud SQL is woefully inadequate, basically a single machine DB with replication. Datastore is built on Megastore, which is the precursor to Spanner, largely confined to a handful of nearly located datacenters [think US-East]. Google's internal data will be stored on / migrated to Spanner, I expect sooner rather than later. We'll be using Spanner whenever we'll be using Google. Furthermore, it's likely we'll use Spanner whenever we use a planet scale system built on GCP, say Spotify.
guilt 9y ago
Did they just write it during CockroachDB's release? So sketchy.
- deleted 9y ago
  [deleted]
rusanu 9y ago
Tangential. For me the fascinating thing is looking at the list of authors to recognize so many from the 2005-2012 Microsoft SQL Server team. Folk I know personally as exceptional performers. Same when I look at Aurora papers. I see this as the result of Ballmer's famous HR initiatives and the massive brain drain that occurred at Microsoft around 2010-ish.
- pavlov 9y ago
  Can you tell some more about the HR initiatives? (I would just google it, but a bit unclear what I should be looking for.)
  - rusanu 9y ago
    You can point back to the 2004 benefits overhaul [0], the famous Towels story [1], low compensation rates compared to Google, Amazon and Facebook [2]. Just go over minimsft.blogspot.com posts at the time. Add to this the lack of vision and direction, catastrophic acquisitions, dismal flagship product releases. At the time there were running jokes about the Inbox filling up with "After 15 years, is time to send that email" subject lines... [0] http://old.seattletimes.com/html/businesstechnology/2001938654_microsoft26.html [1] http://www.zdnet.com/article/microsoft-brings-back-the-towels-5000148135/ [2] http://minimsft.blogspot.com/2006/03/internal-microsoft-compensation.html
sheeshkebab 9y ago
Is anyone outside of Google or msft using these proprietary k/v/sql/Json databases? As a developer I can't bring myself to code against something I can't install on my laptop. And as enterprises would go, they won't use any data store that can't run their financials/hr/ldap/sharepoint thing. So, who uses them and for what?
- tyingq 9y ago
  Your viewpoint is shared by many, but there are lots of enterprises using proprietary cloud features. They either use an abstraction layer for running on a laptop, or otherwise a CI process that kicks off dev instances and test cases on demand, forcing you to be online when you check things in. That's not terribly new. Teams have had to find solutions/standins for things like AWS load balancers, lambdas, certificate servers, etc. Cloud spanner, being fairly new and unusual (SQL, but no INSERT/UPDATE), though, doesn't yet have a big name customer. Jda.com and quizlet.com were their reference customers.

162 points by elvinyung 9y ago

This "modern" Spanner feels very different from the one we saw in 2012 [1]. Some interesting takeaways: * There is a native SQL interface in Spanner, rather than relying on a separate upper-layer SQL layer, a la F1 [2] * Spanner is no longer on top of Bigtable! Instead, the storage engine seems to be a heavily modified Bigtable with a column-oriented file format * Data is resharded frequently and concurrently with other operations -- the shard layout is abstracted away from the query plan using the "distributed union" operator * Possible explanation for why Spanner doesn't support SQL DML writes: writes are required to be the last step of a transaction, and there is currently no support for reading uncommitted writes (this is in contrast to F1, which does support DML) * Spanner supports full-text search (!) [1] https://research.google.com/archive/spanner.html https://research.google.com/archive/spanner.html [2] https://research.google.com/pubs/pub41344.html https://research.google.com/pubs/pub41344.html

iambvk 9y ago
Did not digest the paper yet. Does it provide nested transactions? My understanding from 2012 paper was that it doesn't support nested transactions, even at the storage layer. Can anybody provide insider-knowledge if it was even a requested feature from Google devs internally?
- elvinyung 9y ago
  Since writes are required to be the last step of a transaction, I suspect there wouldn't be much of a point in having nested transactions.
- sanxiyn 9y ago
  From the paper: > Overcoming this limitation requires supporting reading the uncommitted results within an active transaction. While we have seen comparatively little demand for this feature internally, supporting such semantics improves compatibility with other SQL systems and their ecosystems and is on our long-term radar. No internal demand, but on "long-term radar".

gravypod 9y ago

Is Spanner free/open source software? Can we look at the code?

tav 9y ago
No, but it is available as part of Google's cloud services [0]. Particularly noteworthy is the fact that they offer an SLA with 99.99% availability. Even crazier is their pending multi-region version due later this year which will come with a 99.999% SLA!! [0] https://cloud.google.com/spanner/ https://cloud.google.com/spanner/
kyrra 9y ago
SaaS. Even if it was open sourced, as the original paper calls out, there is specialized hardware involved for keep clocks in sync[0]. If you want something open source, cockroachdb is the closest right now. https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo...
- aliasaria 9y ago
  There is also TiDB https://github.com/pingcap/tidb/blob/master/README.md https://github.com/pingcap/tidb/blob/master/README.md
  - forgot-my-pw 9y ago
    With MySQL-compatible, this sounds really promising: https://www.pingcap.com/doc-mysql-compatibility https://www.pingcap.com/doc-mysql-compatibility
- derefr 9y ago
  Specialized, but not proprietary. Spanner presumably uses off-the-shelf Chip-Scale Atomic Clock (CSAC) modules with custom system-level integration. But you can buy CSACs yourself—mounted on PCIe cards—for your own data center. (Every site I can find is of the "request a quote" variety where prices are concerned. They were apparently $1500 apiece in 2011, and have probably come down since then.) But you don't need an atomic clock to get Spanner's guarantees. CSACs are more convenient in terms of setup and requirements—but GPS clock-sources will do just fine for Spanner's 6ms quantum. You can buy a commercial-off-the-shelf GPS NTP appliance (https://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/network-appliances-servers/syncserver/syncserver-s650 https://www.microsemi.com/products/timing-synchronization-sy...) and run signal lines from it to all your machines; or cobble a similar solution together yourself, in true HAM fashion, using a GPS antenna + a Linux box + a UHF Software-Defined Radio card (i.e. a TV-tuner card) + GPSd + NTPd. (Or or you could buy cheap USB GPS receivers and hook them up to each and every server in your DC—but you'd need a very thin ceiling for that to work.) Amusingly, the practicality of that last approach also means that you could run Spanner just fine on a cluster of Android phones. :)

speedplane 9y ago

I was at the Google Cloud conference a few months ago and spoke to a few engineers there about Spanner. When I asked them about how it would affect their other storage options (e.g., datastore, cloud sql, etc.), they said that over time, all of their internal storage systems would be moved over to Spanner. His words were "you'll be using Spanner whether you know it or not." This engineer was clearly a cheerleader for the product, so I'm dubious as to whether that will actually happen, but it's clear that they have quite a bit of confidence in it.

deleted 9y ago
[deleted]
Artemis2 9y ago
That sounds a lot like the announcement of Azure Cosmos DB: https://docs.microsoft.com/en-us/azure/cosmos-db/introduction https://docs.microsoft.com/en-us/azure/cosmos-db/introductio... > we made the service available externally to all Azure Developers in the form of Azure DocumentDB. Azure Cosmos DB is the next big leap in the evolution of DocumentDB and we are now making it available for you to use. As a part of this release of Azure Cosmos DB, DocumentDB customers (with their data) are automatically Azure Cosmos DB customers. From my understanding, DocumentDB ≈ Dynamo, while Cosmos DB would be closer to Cloud Spanner.
pacala 9y ago
Google's game is planet scale systems. Cloud SQL is woefully inadequate, basically a single machine DB with replication. Datastore is built on Megastore, which is the precursor to Spanner, largely confined to a handful of nearly located datacenters [think US-East]. Google's internal data will be stored on / migrated to Spanner, I expect sooner rather than later. We'll be using Spanner whenever we'll be using Google. Furthermore, it's likely we'll use Spanner whenever we use a planet scale system built on GCP, say Spotify.

guilt 9y ago

Did they just write it during CockroachDB's release? So sketchy.

deleted 9y ago
[deleted]

rusanu 9y ago

Tangential. For me the fascinating thing is looking at the list of authors to recognize so many from the 2005-2012 Microsoft SQL Server team. Folk I know personally as exceptional performers. Same when I look at Aurora papers. I see this as the result of Ballmer's famous HR initiatives and the massive brain drain that occurred at Microsoft around 2010-ish.

pavlov 9y ago
Can you tell some more about the HR initiatives? (I would just google it, but a bit unclear what I should be looking for.)
- rusanu 9y ago
  You can point back to the 2004 benefits overhaul [0], the famous Towels story [1], low compensation rates compared to Google, Amazon and Facebook [2]. Just go over minimsft.blogspot.com posts at the time. Add to this the lack of vision and direction, catastrophic acquisitions, dismal flagship product releases. At the time there were running jokes about the Inbox filling up with "After 15 years, is time to send that email" subject lines... [0] http://old.seattletimes.com/html/businesstechnology/2001938654_microsoft26.html [1] http://www.zdnet.com/article/microsoft-brings-back-the-towels-5000148135/ [2] http://minimsft.blogspot.com/2006/03/internal-microsoft-compensation.html

sheeshkebab 9y ago

Is anyone outside of Google or msft using these proprietary k/v/sql/Json databases? As a developer I can't bring myself to code against something I can't install on my laptop. And as enterprises would go, they won't use any data store that can't run their financials/hr/ldap/sharepoint thing. So, who uses them and for what?

tyingq 9y ago
Your viewpoint is shared by many, but there are lots of enterprises using proprietary cloud features. They either use an abstraction layer for running on a laptop, or otherwise a CI process that kicks off dev instances and test cases on demand, forcing you to be online when you check things in. That's not terribly new. Teams have had to find solutions/standins for things like AWS load balancers, lambdas, certificate servers, etc. Cloud spanner, being fairly new and unusual (SQL, but no INSERT/UPDATE), though, doesn't yet have a big name customer. Jda.com and quizlet.com were their reference customers.