5 ms·

Is Spanner free/open source software? Can we look at the code?

tav 9y ago
No, but it is available as part of Google's cloud services [0]. Particularly noteworthy is the fact that they offer an SLA with 99.99% availability. Even crazier is their pending multi-region version due later this year which will come with a 99.999% SLA!! [0] https://cloud.google.com/spanner/ https://cloud.google.com/spanner/
kyrra 9y ago
SaaS. Even if it was open sourced, as the original paper calls out, there is specialized hardware involved for keep clocks in sync[0]. If you want something open source, cockroachdb is the closest right now. https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo...
- aliasaria 9y ago
  There is also TiDB https://github.com/pingcap/tidb/blob/master/README.md https://github.com/pingcap/tidb/blob/master/README.md
  - forgot-my-pw 9y ago
    With MySQL-compatible, this sounds really promising: https://www.pingcap.com/doc-mysql-compatibility https://www.pingcap.com/doc-mysql-compatibility
- derefr 9y ago
  Specialized, but not proprietary. Spanner presumably uses off-the-shelf Chip-Scale Atomic Clock (CSAC) modules with custom system-level integration. But you can buy CSACs yourself—mounted on PCIe cards—for your own data center. (Every site I can find is of the "request a quote" variety where prices are concerned. They were apparently $1500 apiece in 2011, and have probably come down since then.) But you don't need an atomic clock to get Spanner's guarantees. CSACs are more convenient in terms of setup and requirements—but GPS clock-sources will do just fine for Spanner's 6ms quantum. You can buy a commercial-off-the-shelf GPS NTP appliance (https://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/network-appliances-servers/syncserver/syncserver-s650 https://www.microsemi.com/products/timing-synchronization-sy...) and run signal lines from it to all your machines; or cobble a similar solution together yourself, in true HAM fashion, using a GPS antenna + a Linux box + a UHF Software-Defined Radio card (i.e. a TV-tuner card) + GPSd + NTPd. (Or or you could buy cheap USB GPS receivers and hook them up to each and every server in your DC—but you'd need a very thin ceiling for that to work.) Amusingly, the practicality of that last approach also means that you could run Spanner just fine on a cluster of Android phones. :)
  - dgacmu 9y ago
    CSACs seem unnecessary. Within a few racks, PTP (IEEE 1588) can do a pretty decent job of getting things synchronized to tens of microseconds. Requires that your ToRs and NICs support it, but that's not a very onerous requirement, particularly if you're GoogleBookSoft. This means you only need a few time receivers for a datacenter, along with careful monitoring and implementation of your time distribution, but that happens through your Ethernet switches. More: http://www.ni.com/newsletter/50130/en/ http://www.ni.com/newsletter/50130/en/ But you need your datacenters to stay synced even if you lose GPS. You can use an atomic clock for this (Cs or Rb). But, for the rest of us, a good GPS-disciplined double-oven crystal oscillator (XO) can get you within the range needed for Spanner, IIRC spanner's time sync requirements. For example, this little one: https://www.microsemi.com/document-portal/doc_download/133412-gps-2500-and-gps-2550 https://www.microsemi.com/document-portal/doc_download/13341... will do +- 7 microseconds / 24h holdover. ("holdover" == operating when it's lost GPS).
    - tyingq 9y ago
      Sounds like spanner can't tolerate much drift: "The most serious problem would be if a local clock’s drift were greater than 200us/sec: that would break assumptions made by TrueTime."[1] [1] http://www.bluetreble.com/2015/10/time-travel/ http://www.bluetreble.com/2015/10/time-travel/
  - iambvk 9y ago
    > But you don't need an atomic clock to get Spanner's guarantees. I always hear this from CockroachDB folks and fans, but no details. What are the downsides? As per my understanding, if a server/replica could not catch up wrt time (i.e. goes out of sync), it can be identified and usually be marked as a temporary-replica-failure -- which is the maximum extent that Hybrid Logical Clocks can help us with. Rest of the system has to deal with consequences: A new replica must take it's place to keep-up the fault-tolerance-level and start making-a-new-copy -- which is in the order-of the amount of data stored in the out-of-sync replica. I assume this puts lot different design requirements on the storage-engine; also, making-a-copy and cancel-copying operations would eat network traffic, thus throughput. IIRC Google Spanner uses atomic clocks only on few severs per data-center because cross data-center latencies are much higher and erratic (over internet). So CockroadDB has much higher rate of temporary-failures due to out of time-sync and associated downsides. It would be helpful if CockroachDB guys could shed some light on this.
- necubi 9y ago
  The bigger issue is that you need Google's incredible inter-DC networking, which in practice makes partitions very rare. Eric Brewer (author of CAP theorem) lays out here [0] how Spanner relies on those networking guarantees to be effectively CA. Google's inter-dc traffic flows entirely on private links rather than on the public internet, which is very hard for any other company to match on a global scale. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45855.pdf https://static.googleusercontent.com/media/research.google.c...
  - lobster_johnson 9y ago
    If you run your code on Google Cloud Platform, you should get the same benefits, no?
    - aoeuasdf1 9y ago
      Good point
deleted 9y ago
[deleted]
slackingoff2017 9y ago
This is part of a worrying new trend. Increasingly you can't buy software anymore, only rent. Innovation is being kept from scrutiny hidden behind closed doors. The kind of thing patents were meant to prevent back when the system wasn't broken. Google is one of the better players in this regard, at least telling the world what they're up to. Try to figure out how something like Amazon's systems work and you'll run into a deafening wall of silence. Funny that we're so willing to trust these "clouds" when we know next to nothing about their internal workings. I don't think the honeymoon will last forever. Somebody will eventually abuse their position and within a few years everyone will be "on prem" again.
- elvinyung 9y ago
  Interestingly, Amazon is also presenting a paper this year at SIGMOD, about Aurora: http://dl.acm.org/authorize?N37778 http://dl.acm.org/authorize?N37778 If my memory serves, it's the second paper they've ever published (the other one being the famous Dynamo paper[1]). [1] http://cloudgroup.neu.edu.cn/papers/cloud%20data%20storage/dynamo-sosp-2007.pdf http://cloudgroup.neu.edu.cn/papers/cloud%20data%20storage/d...
- brandur 9y ago
  > This is part of a worrying new trend. Increasingly you can't buy software anymore, only rent. More or less agreed, but you'd probably have to concede that the existence of papers like this one and some of Google's other Spanner publications are admirable; they're being more open about the system's design than they have to be. Of course Google knows that the system's secret sauce is not the concept itself, but the cost of its implementation, the infrastructural harness to support it, and the resources to reliably operate it. Even with a rough understanding on how Spanner works, it's still going to be difficult to ever migrate off it for practical reasons alone — who else is going to be able to build and run an alternative? I have huge respect for what the Spanner team is doing, but this is a reason that Citus [1] is also very interesting to me right now. You could conceivably start out with nothing but Postgres and migrate into a Citus cluster when (and if) you need to. If a point comes where you realize that you need out, you could either (1) see if you can scale back down to simple Postgres, (2) host your own cluster on your own infrastructure with the Citus source code, or (3) migrate onto your own Postgres sharding scheme à la Instagram. At no point do you lock yourself into custom GRPC APIs which are going to be ~impossible to get off of. GCP and AWS provide hugely useful foundational IaaS, but they're incentivized to move beyond that layer and provide more custom solutions that (1) provide better margins, and (2) lock you into their services. As people and companies building on top of these clouds, we should be looking for whatever opportunities we can to keep our stacks generic so that AWS <-> GCP <-> Azure migrations are possible, even if a last resort. [1] https://www.citusdata.com/product/cloud https://www.citusdata.com/product/cloud

tav 9y ago

No, but it is available as part of Google's cloud services [0]. Particularly noteworthy is the fact that they offer an SLA with 99.99% availability. Even crazier is their pending multi-region version due later this year which will come with a 99.999% SLA!! [0] https://cloud.google.com/spanner/ https://cloud.google.com/spanner/

kyrra 9y ago

SaaS. Even if it was open sourced, as the original paper calls out, there is specialized hardware involved for keep clocks in sync[0]. If you want something open source, cockroachdb is the closest right now. https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo...

aliasaria 9y ago
There is also TiDB https://github.com/pingcap/tidb/blob/master/README.md https://github.com/pingcap/tidb/blob/master/README.md
- forgot-my-pw 9y ago
  With MySQL-compatible, this sounds really promising: https://www.pingcap.com/doc-mysql-compatibility https://www.pingcap.com/doc-mysql-compatibility
derefr 9y ago
Specialized, but not proprietary. Spanner presumably uses off-the-shelf Chip-Scale Atomic Clock (CSAC) modules with custom system-level integration. But you can buy CSACs yourself—mounted on PCIe cards—for your own data center. (Every site I can find is of the "request a quote" variety where prices are concerned. They were apparently $1500 apiece in 2011, and have probably come down since then.) But you don't need an atomic clock to get Spanner's guarantees. CSACs are more convenient in terms of setup and requirements—but GPS clock-sources will do just fine for Spanner's 6ms quantum. You can buy a commercial-off-the-shelf GPS NTP appliance (https://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/network-appliances-servers/syncserver/syncserver-s650 https://www.microsemi.com/products/timing-synchronization-sy...) and run signal lines from it to all your machines; or cobble a similar solution together yourself, in true HAM fashion, using a GPS antenna + a Linux box + a UHF Software-Defined Radio card (i.e. a TV-tuner card) + GPSd + NTPd. (Or or you could buy cheap USB GPS receivers and hook them up to each and every server in your DC—but you'd need a very thin ceiling for that to work.) Amusingly, the practicality of that last approach also means that you could run Spanner just fine on a cluster of Android phones. :)
- dgacmu 9y ago
  CSACs seem unnecessary. Within a few racks, PTP (IEEE 1588) can do a pretty decent job of getting things synchronized to tens of microseconds. Requires that your ToRs and NICs support it, but that's not a very onerous requirement, particularly if you're GoogleBookSoft. This means you only need a few time receivers for a datacenter, along with careful monitoring and implementation of your time distribution, but that happens through your Ethernet switches. More: http://www.ni.com/newsletter/50130/en/ http://www.ni.com/newsletter/50130/en/ But you need your datacenters to stay synced even if you lose GPS. You can use an atomic clock for this (Cs or Rb). But, for the rest of us, a good GPS-disciplined double-oven crystal oscillator (XO) can get you within the range needed for Spanner, IIRC spanner's time sync requirements. For example, this little one: https://www.microsemi.com/document-portal/doc_download/133412-gps-2500-and-gps-2550 https://www.microsemi.com/document-portal/doc_download/13341... will do +- 7 microseconds / 24h holdover. ("holdover" == operating when it's lost GPS).
  - tyingq 9y ago
    Sounds like spanner can't tolerate much drift: "The most serious problem would be if a local clock’s drift were greater than 200us/sec: that would break assumptions made by TrueTime."[1] [1] http://www.bluetreble.com/2015/10/time-travel/ http://www.bluetreble.com/2015/10/time-travel/
- iambvk 9y ago
  > But you don't need an atomic clock to get Spanner's guarantees. I always hear this from CockroachDB folks and fans, but no details. What are the downsides? As per my understanding, if a server/replica could not catch up wrt time (i.e. goes out of sync), it can be identified and usually be marked as a temporary-replica-failure -- which is the maximum extent that Hybrid Logical Clocks can help us with. Rest of the system has to deal with consequences: A new replica must take it's place to keep-up the fault-tolerance-level and start making-a-new-copy -- which is in the order-of the amount of data stored in the out-of-sync replica. I assume this puts lot different design requirements on the storage-engine; also, making-a-copy and cancel-copying operations would eat network traffic, thus throughput. IIRC Google Spanner uses atomic clocks only on few severs per data-center because cross data-center latencies are much higher and erratic (over internet). So CockroadDB has much higher rate of temporary-failures due to out of time-sync and associated downsides. It would be helpful if CockroachDB guys could shed some light on this.
necubi 9y ago
The bigger issue is that you need Google's incredible inter-DC networking, which in practice makes partitions very rare. Eric Brewer (author of CAP theorem) lays out here [0] how Spanner relies on those networking guarantees to be effectively CA. Google's inter-dc traffic flows entirely on private links rather than on the public internet, which is very hard for any other company to match on a global scale. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45855.pdf https://static.googleusercontent.com/media/research.google.c...
- lobster_johnson 9y ago
  If you run your code on Google Cloud Platform, you should get the same benefits, no?
  - aoeuasdf1 9y ago
    Good point

deleted 9y ago

[deleted]

slackingoff2017 9y ago

This is part of a worrying new trend. Increasingly you can't buy software anymore, only rent. Innovation is being kept from scrutiny hidden behind closed doors. The kind of thing patents were meant to prevent back when the system wasn't broken. Google is one of the better players in this regard, at least telling the world what they're up to. Try to figure out how something like Amazon's systems work and you'll run into a deafening wall of silence. Funny that we're so willing to trust these "clouds" when we know next to nothing about their internal workings. I don't think the honeymoon will last forever. Somebody will eventually abuse their position and within a few years everyone will be "on prem" again.

elvinyung 9y ago
Interestingly, Amazon is also presenting a paper this year at SIGMOD, about Aurora: http://dl.acm.org/authorize?N37778 http://dl.acm.org/authorize?N37778 If my memory serves, it's the second paper they've ever published (the other one being the famous Dynamo paper[1]). [1] http://cloudgroup.neu.edu.cn/papers/cloud%20data%20storage/dynamo-sosp-2007.pdf http://cloudgroup.neu.edu.cn/papers/cloud%20data%20storage/d...
brandur 9y ago
> This is part of a worrying new trend. Increasingly you can't buy software anymore, only rent. More or less agreed, but you'd probably have to concede that the existence of papers like this one and some of Google's other Spanner publications are admirable; they're being more open about the system's design than they have to be. Of course Google knows that the system's secret sauce is not the concept itself, but the cost of its implementation, the infrastructural harness to support it, and the resources to reliably operate it. Even with a rough understanding on how Spanner works, it's still going to be difficult to ever migrate off it for practical reasons alone — who else is going to be able to build and run an alternative? I have huge respect for what the Spanner team is doing, but this is a reason that Citus [1] is also very interesting to me right now. You could conceivably start out with nothing but Postgres and migrate into a Citus cluster when (and if) you need to. If a point comes where you realize that you need out, you could either (1) see if you can scale back down to simple Postgres, (2) host your own cluster on your own infrastructure with the Citus source code, or (3) migrate onto your own Postgres sharding scheme à la Instagram. At no point do you lock yourself into custom GRPC APIs which are going to be ~impossible to get off of. GCP and AWS provide hugely useful foundational IaaS, but they're incentivized to move beyond that layer and provide more custom solutions that (1) provide better margins, and (2) lock you into their services. As people and companies building on top of these clouds, we should be looking for whatever opportunities we can to keep our stacks generic so that AWS <-> GCP <-> Azure migrations are possible, even if a last resort. [1] https://www.citusdata.com/product/cloud https://www.citusdata.com/product/cloud