5 ms·

SaaS. Even if it was open sourced, as the original paper calls out, there is specialized hardware involved for keep clocks in sync[0]. If you want something op

by kyrra 9y ago

SaaS. Even if it was open sourced, as the original paper calls out, there is specialized hardware involved for keep clocks in sync[0]. If you want something open source, cockroachdb is the closest right now. https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo...

aliasaria 9y ago
There is also TiDB https://github.com/pingcap/tidb/blob/master/README.md https://github.com/pingcap/tidb/blob/master/README.md
- forgot-my-pw 9y ago
  With MySQL-compatible, this sounds really promising: https://www.pingcap.com/doc-mysql-compatibility https://www.pingcap.com/doc-mysql-compatibility
derefr 9y ago
Specialized, but not proprietary. Spanner presumably uses off-the-shelf Chip-Scale Atomic Clock (CSAC) modules with custom system-level integration. But you can buy CSACs yourself—mounted on PCIe cards—for your own data center. (Every site I can find is of the "request a quote" variety where prices are concerned. They were apparently $1500 apiece in 2011, and have probably come down since then.) But you don't need an atomic clock to get Spanner's guarantees. CSACs are more convenient in terms of setup and requirements—but GPS clock-sources will do just fine for Spanner's 6ms quantum. You can buy a commercial-off-the-shelf GPS NTP appliance (https://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/network-appliances-servers/syncserver/syncserver-s650 https://www.microsemi.com/products/timing-synchronization-sy...) and run signal lines from it to all your machines; or cobble a similar solution together yourself, in true HAM fashion, using a GPS antenna + a Linux box + a UHF Software-Defined Radio card (i.e. a TV-tuner card) + GPSd + NTPd. (Or or you could buy cheap USB GPS receivers and hook them up to each and every server in your DC—but you'd need a very thin ceiling for that to work.) Amusingly, the practicality of that last approach also means that you could run Spanner just fine on a cluster of Android phones. :)
- dgacmu 9y ago
  CSACs seem unnecessary. Within a few racks, PTP (IEEE 1588) can do a pretty decent job of getting things synchronized to tens of microseconds. Requires that your ToRs and NICs support it, but that's not a very onerous requirement, particularly if you're GoogleBookSoft. This means you only need a few time receivers for a datacenter, along with careful monitoring and implementation of your time distribution, but that happens through your Ethernet switches. More: http://www.ni.com/newsletter/50130/en/ http://www.ni.com/newsletter/50130/en/ But you need your datacenters to stay synced even if you lose GPS. You can use an atomic clock for this (Cs or Rb). But, for the rest of us, a good GPS-disciplined double-oven crystal oscillator (XO) can get you within the range needed for Spanner, IIRC spanner's time sync requirements. For example, this little one: https://www.microsemi.com/document-portal/doc_download/133412-gps-2500-and-gps-2550 https://www.microsemi.com/document-portal/doc_download/13341... will do +- 7 microseconds / 24h holdover. ("holdover" == operating when it's lost GPS).
  - tyingq 9y ago
    Sounds like spanner can't tolerate much drift: "The most serious problem would be if a local clock’s drift were greater than 200us/sec: that would break assumptions made by TrueTime."[1] [1] http://www.bluetreble.com/2015/10/time-travel/ http://www.bluetreble.com/2015/10/time-travel/
- iambvk 9y ago
  > But you don't need an atomic clock to get Spanner's guarantees. I always hear this from CockroachDB folks and fans, but no details. What are the downsides? As per my understanding, if a server/replica could not catch up wrt time (i.e. goes out of sync), it can be identified and usually be marked as a temporary-replica-failure -- which is the maximum extent that Hybrid Logical Clocks can help us with. Rest of the system has to deal with consequences: A new replica must take it's place to keep-up the fault-tolerance-level and start making-a-new-copy -- which is in the order-of the amount of data stored in the out-of-sync replica. I assume this puts lot different design requirements on the storage-engine; also, making-a-copy and cancel-copying operations would eat network traffic, thus throughput. IIRC Google Spanner uses atomic clocks only on few severs per data-center because cross data-center latencies are much higher and erratic (over internet). So CockroadDB has much higher rate of temporary-failures due to out of time-sync and associated downsides. It would be helpful if CockroachDB guys could shed some light on this.
necubi 9y ago
The bigger issue is that you need Google's incredible inter-DC networking, which in practice makes partitions very rare. Eric Brewer (author of CAP theorem) lays out here [0] how Spanner relies on those networking guarantees to be effectively CA. Google's inter-dc traffic flows entirely on private links rather than on the public internet, which is very hard for any other company to match on a global scale. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45855.pdf https://static.googleusercontent.com/media/research.google.c...
- lobster_johnson 9y ago
  If you run your code on Google Cloud Platform, you should get the same benefits, no?
  - aoeuasdf1 9y ago
    Good point

by kyrra 9y ago

aliasaria 9y ago

There is also TiDB https://github.com/pingcap/tidb/blob/master/README.md https://github.com/pingcap/tidb/blob/master/README.md

forgot-my-pw 9y ago
With MySQL-compatible, this sounds really promising: https://www.pingcap.com/doc-mysql-compatibility https://www.pingcap.com/doc-mysql-compatibility

derefr 9y ago

Specialized, but not proprietary. Spanner presumably uses off-the-shelf Chip-Scale Atomic Clock (CSAC) modules with custom system-level integration. But you can buy CSACs yourself—mounted on PCIe cards—for your own data center. (Every site I can find is of the "request a quote" variety where prices are concerned. They were apparently $1500 apiece in 2011, and have probably come down since then.) But you don't need an atomic clock to get Spanner's guarantees. CSACs are more convenient in terms of setup and requirements—but GPS clock-sources will do just fine for Spanner's 6ms quantum. You can buy a commercial-off-the-shelf GPS NTP appliance (https://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/network-appliances-servers/syncserver/syncserver-s650 https://www.microsemi.com/products/timing-synchronization-sy...) and run signal lines from it to all your machines; or cobble a similar solution together yourself, in true HAM fashion, using a GPS antenna + a Linux box + a UHF Software-Defined Radio card (i.e. a TV-tuner card) + GPSd + NTPd. (Or or you could buy cheap USB GPS receivers and hook them up to each and every server in your DC—but you'd need a very thin ceiling for that to work.) Amusingly, the practicality of that last approach also means that you could run Spanner just fine on a cluster of Android phones. :)

dgacmu 9y ago
CSACs seem unnecessary. Within a few racks, PTP (IEEE 1588) can do a pretty decent job of getting things synchronized to tens of microseconds. Requires that your ToRs and NICs support it, but that's not a very onerous requirement, particularly if you're GoogleBookSoft. This means you only need a few time receivers for a datacenter, along with careful monitoring and implementation of your time distribution, but that happens through your Ethernet switches. More: http://www.ni.com/newsletter/50130/en/ http://www.ni.com/newsletter/50130/en/ But you need your datacenters to stay synced even if you lose GPS. You can use an atomic clock for this (Cs or Rb). But, for the rest of us, a good GPS-disciplined double-oven crystal oscillator (XO) can get you within the range needed for Spanner, IIRC spanner's time sync requirements. For example, this little one: https://www.microsemi.com/document-portal/doc_download/133412-gps-2500-and-gps-2550 https://www.microsemi.com/document-portal/doc_download/13341... will do +- 7 microseconds / 24h holdover. ("holdover" == operating when it's lost GPS).
- tyingq 9y ago
  Sounds like spanner can't tolerate much drift: "The most serious problem would be if a local clock’s drift were greater than 200us/sec: that would break assumptions made by TrueTime."[1] [1] http://www.bluetreble.com/2015/10/time-travel/ http://www.bluetreble.com/2015/10/time-travel/
iambvk 9y ago
> But you don't need an atomic clock to get Spanner's guarantees. I always hear this from CockroachDB folks and fans, but no details. What are the downsides? As per my understanding, if a server/replica could not catch up wrt time (i.e. goes out of sync), it can be identified and usually be marked as a temporary-replica-failure -- which is the maximum extent that Hybrid Logical Clocks can help us with. Rest of the system has to deal with consequences: A new replica must take it's place to keep-up the fault-tolerance-level and start making-a-new-copy -- which is in the order-of the amount of data stored in the out-of-sync replica. I assume this puts lot different design requirements on the storage-engine; also, making-a-copy and cancel-copying operations would eat network traffic, thus throughput. IIRC Google Spanner uses atomic clocks only on few severs per data-center because cross data-center latencies are much higher and erratic (over internet). So CockroadDB has much higher rate of temporary-failures due to out of time-sync and associated downsides. It would be helpful if CockroachDB guys could shed some light on this.

necubi 9y ago

The bigger issue is that you need Google's incredible inter-DC networking, which in practice makes partitions very rare. Eric Brewer (author of CAP theorem) lays out here [0] how Spanner relies on those networking guarantees to be effectively CA. Google's inter-dc traffic flows entirely on private links rather than on the public internet, which is very hard for any other company to match on a global scale. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45855.pdf https://static.googleusercontent.com/media/research.google.c...

lobster_johnson 9y ago
If you run your code on Google Cloud Platform, you should get the same benefits, no?
- aoeuasdf1 9y ago
  Good point