7 ms·

Avoid Vendor Lock-In Risk with Multi-Cloud Deployments

153 points by nate_stewart 9y ago

daxorid 9y ago
I can't find any documentation on how, exactly, ACID is achieved across all these partitions. It's impossible to reason about performance characteristics in the absence of documentation. For example, if they use two-phase commit, what is the granularity of locks ( table, row, etc )during an open transaction? Table locks during two-phase commit with, say, 50ms coast-to-coast latency would be an immediate dealkiller. Or, if it's eventually consistent as most asynchronously distributed DBs are, they can't claim AC. I really wish Cockroach had better documentation - not so much how to use it, but how it works so as to let engineers reason about performance characteristics.
- deleted 9y ago
  [deleted]
- andreimatei1 9y ago
  Besides the blog, you can find interesting stuff in the source repo, in the technotes and RFC dirs: https://github.com/cockroachdb/cockroach/tree/master/docs/tech-notes https://github.com/cockroachdb/cockroach/tree/master/docs/te... https://github.com/cockroachdb/cockroach/tree/master/docs/RFCS https://github.com/cockroachdb/cockroach/tree/master/docs/RF... There's also a bunch of technical tech-talks linked from our company's website: https://www.cockroachlabs.com/community/tech-talks/ https://www.cockroachlabs.com/community/tech-talks/
- chubot 9y ago
  Not much detail here, but: https://www.cockroachlabs.com/docs/stable/frequently-asked-questions.html#how-is-cockroachdb-both-highly-available-and-strongly-consistent https://www.cockroachlabs.com/docs/stable/frequently-asked-q... Last I heard CockroachDB is like "Spanner without atomic clocks". Not sure if that helps but at least the WAN latency is not an afterthought. https://www.cockroachlabs.com/docs/stable/frequently-asked-questions.html#do-transactions-in-cockroachdb-guarantee-acid-semantics https://www.cockroachlabs.com/docs/stable/frequently-asked-q... https://www.cockroachlabs.com/docs/stable/frequently-asked-questions.html#when-is-cockroachdb-not-a-good-choice https://www.cockroachlabs.com/docs/stable/frequently-asked-q...
- knz42 9y ago
  (working for cockroach labs) As explained in the blog (as linked by another response) and the documentation (https://www.cockroachlabs.com/docs/stable/strong-consistency.html https://www.cockroachlabs.com/docs/stable/strong-consistency... and https://www.cockroachlabs.com/docs/stable/distributed-transactions.html https://www.cockroachlabs.com/docs/stable/distributed-transa...) CockroachDB does not use locks, and write intents in the concurrency control scheme are expressed by K/V pair i.e. per table cell. Meanwhile, I acknowledge that CockroachDB does not yet convey well its performance model so that engineers can get an intuition about it. There is documentation, but it's either too detailed (like the design doc) or too high-level (most of the docs site). This needs a new sort of documentation, which we are working on.
  - daxorid 9y ago
    Thanks!
  - eternalban 9y ago
    From the first link: > Employs Raft, a popular successor to Paxos. I don't believe that is correct and that is not a good meme to promote. Successor implies the 'deprecation' of Paxos, which is certainly not the case. For the architectural sweet spot where both protocols can be used, it is fair to say Raft provides the more accessible approach, and that for those who insist on rolling their own consensus mechanism. Paxos and Raft serve distinct architectural concerns in the shared context of providing a distributed state with formally defined consistency guarantees. And to the point, it is entirely reasonable to see a distributed system that utilizes both to address system component specific concerns. CockroachDB, ironically (in context of your OP), in fact could benefit from a vendor lock-in were the candidate cloud provider to provide atomic clocks and GPS cards in their offerings.
    - ddorian43 9y ago
      What are the GPS cards for ?
      - infogulch 9y ago
- occultist_throw 9y ago
  The trend Ive been seeing is "AP with eventual C". It comoes down to the idea that we have a planet, with datacenters in locations on said planet. Light down fiber or copper only goes so fast. The "eventual consistency" part is simply acknowledging that we can't beat C when our databases are in North America, South America, Europe, Africa, Australia, and Asia. One-way comms takes time. 2 way comms, even more so. The tradeoff is, that changes can come in, and 'eventually' will propagate to the rest of the DB nodes. It does sound not ideal initially, however this type of operation would allow a localDB running as AP to sync when back online. The examples I'm deriving come from recent DBs, like GUN http://gun.js.org/#step1 http://gun.js.org/#step1
  - dastbe 9y ago
    cockroachdb and spanner are "CP with high probability of A". There are situations where the db (or possibly a subset, haven't read too deeply into the papers) can become unavailable due to a loss of quorum, but these situations are highly unlikely.
ddorian43 9y ago
Still waiting for some benchmarks by the team. What I would be curious, is, how much faster the db would be if it was in c++/rust instead of java/go. I know bigtable claims 3x faster than hbase. Scylla claims ~10x (though also different architecture) faster than Cassandra. Trinity ~2x compared to Lucene. etc etc. Also: incoming comments about the name of the db.
- jrs95 9y ago
  Agreed. I love Go, but a database is one of the things in the stack that ought to be as optimized as possible in my opinion. C++ still seems preferable in these types of applications. At the same time, if the underlying architecture is enough of an improvement over older systems, it's still an upgrade.
  - zzzcpan 9y ago
    You can't optimize anything with C++ if your database is stuck waiting for consensus with nodes 100 ms away. Close to the metal and low level languages are really poor choices here, even Go is too low level for CP databases.
    - ddorian43 9y ago
      Yes, but you'll have lower overhead on each query/request and more concurrency ? For each query the bottleneck is the network which you can't fix, but you can fix the 2,3,etc bottlenecks which are inside 1 machine (cpu/ram/disk).
      - jrs95 9y ago
        Yeah...I'm not really aware of a higher level language that mystically makes networking faster. As far as I'm aware, the most efficient network protocols tend to be implemented at a low level, too.
  - wiremine 9y ago
    > C++ still seems preferable in these types of applications Agreed, but until there are some published performance metrics, I think it's an assumption more than a truth. Premature optimization...
mslate 9y ago
Haha cloud lock-in is what they think I'm worried about?
- TruthSHIFT 9y ago
  So... What are you worried about?
  - mslate 9y ago
    Database lockin. Guess that was not clear enough.
    - rrdharan 9y ago
      In theory the OSS-nature of CockroachDB helps with that part.
    - andreimatei1 9y ago
      FWIW, CRDB should be a good fit for people worried about database lock-in. You'd only be "locked in" if you discover that we're much better than the alternatives :) CRDB uses the standard SQL language. Also, CRDB speaks the PostgreSQL wire protocol; for using CRDB you'd be using the Postgres drivers for different languages. So, more or less, any application that can work with us should also work with Postgres (and vice versa). Moreover, the Postgres drivers generally respect other higher interfaces (odbc, jdbc) so, more or less, a CRDB application should also work with any other established SQL database (and vice versa). CRDB can also export your data in formats easily importable by other SQL databases.
- wiremine 9y ago
  It's a valid concern for business. A lot of my customers go to some pretty interesting lengths to make sure they're not locked into a single vendor's stack.
meddlepal 9y ago
I'd cloud vendor lock-in a serious concern for most businesses? It always feels like a requirement that originates from the technical side rather than the business side. But it never feels like it's that important actually.
- ddorian43 9y ago
  It is important for those not on the cloud: dedicated servers and colo.
  - rconti 9y ago
    You should pluralize that 'colo' then. I've been through datacenter power outages, refusal to renew, and difficulty in getting circuits in a timely manner in multiple Tier 4 datacenters. Self-hosting is only as good as your planning, strategy, and execution, which is much the same as the cloud world. You're probably safer running a single EC2 instance than running a single dedicated server in some random colo, but certainly running your own geographically distributed infrastructure is likely better than relying on a single AWS availability zone (for example). However, it's a great idea to open up peoples' eyes to the idea of how being on a major cloudy provider can make you collateral damage of an attack that never would have targeted you if you were in DIY mode.
- manigandham 9y ago
  It's a overblown concern for many, considering that the major clouds are massive companies that will likely be around far longer than most of their customers (or will acquire them). Every new client actually contributes to the longevity of a cloud provider. There's also lots of competition to offer similar features and many services can be mixed/matched easily (eg: we run on GCP but use Azure event hubs).
- scaryclam 9y ago
  We have a bunch of work that requires large, process heavy, calculations, so yes, vendor lock-in is a concern for us as it can mean either huge cost increases, if the vendor changes pricing for the required instances upwards, or huge savings if a competitor can do the same computations for less. We solve this problem by making sure we can trash servers and spin them up elsewhere if needs be (mostly via terraform etc). I don't think this should apply to anyone running some hosts for web applications though.
ycmimi 9y ago
It's like operating systems. In the end, you will have a limited number of options, aws, gcp, and azure, like windows, mac and linux.
- JohnnyConatus 9y ago
  Yup. And the amount of work it takes to be competent in just one (such as AWS) will make it nearly impossible / not worth it for most people to run cross-cloud.
sengork 9y ago
For anyone interested in multi cloud deployments at the IaaS layer I recommend looking at https://libcloud.apache.org/ https://libcloud.apache.org/
- deleted 9y ago
  [deleted]
tyingq 9y ago
I'm not convinced this helps much with lock in. Assuming you want to do things like deployments, scheduled backups, monitoring, etc, you're either: a) Using "lock in" features like Cloud Formation, Lamba, Cloudwatch, etc, to do so, such that you use some of the "value" of the cloud. ==or== b) Rolling your own solutions for these peripheral functions (deployment, backup, monitoring, etc) and running them on the bare VPS instances. In which case...why are you on this expensive cloud? You can get a VPS with lower bandwidth costs elsewhere. Basically being sort of "half in" on the cloud, where you use it as a glorified VPS makes no sense to me. Either be all the way in, or all the way out.
- jchw 9y ago
  Alternatively, you can use tools like Kubernetes and Terraform as abstractions around cloud environments while still being able to utilize them fairly effectively. Kubernetes is a clever product for Google to release, because it makes migrating loads to GCP much easier if you already run them in Kubernetes.
  - tyingq 9y ago
    Similar decision for me though. If you want k8s, use Google or Azure, where they take care of it for you. K8s on AWS, at least right now, means you are in the business of being a plumber. For example, if I remember right, the solid k8s ingress controller means ELB, which has issues like the need for cache warming. ALB is better, but has only bleeding edge support as an ingress controller in K8s. Similar for getting it installed on AWS in the first place. Rackspace or Digital Ocean or similar is a cheaper better place if you want to roll your own k8s solution.
    - jchw 9y ago
      I'm hosting K8S on AWS in production. Kops takes care of most of the details, though I _do_ understand the low level details of what's going on. Gotta be able to debug somehow. Google Cloud Platform is definitely cheaper than AWS, and their hosting of Kubernetes seals the deal for me. The only problem? Legacy. If I want to link two virtual machines across cloud environments, it introduces somewhat unacceptable latency, and throughput will be less reliable. So connecting to things like extremely busy Redis or AMQP clusters is not really as safe and downtime is more likely. Still, I don't feel like Kubernetes is terrible on AWS. It works fairly well. I'm not using Ingress resources right now, just a bunch of services with their own ELBs.
caleblloyd 9y ago
I think this is a poorly written blog post that completely ignores the challenges and difficulties that arise when going multi-cloud. Multi-cloud introduces huge amounts of latency, requires additional load balancing and routing logic, requires twice as much administration, etc. Multi-cloud is a huge decision that need lots of planning and application level support. I'd go so far as to say it is an order of magnitude easier to design an application that can be moved between clouds easily than it is to design a multi-cloud application. This still defeats vendor lock-in. I think CockroachDB is interesting tech, but this blog post reads like a pretty empty marketing piece.
- jchanimal 9y ago
  It’s true that operating your own cluster (of any technology) spanning cloud providers is not for the faint of heart. However, at FaunaDB we’ve found performance of multi-cloud globally consistent operations to be acceptable. We currently run across AWS and GCP, with Azure on the way, and we have customers running their own clusters in hybrid configurations.

153 points by nate_stewart 9y ago

I can't find any documentation on how, exactly, ACID is achieved across all these partitions. It's impossible to reason about performance characteristics in the absence of documentation. For example, if they use two-phase commit, what is the granularity of locks ( table, row, etc )during an open transaction? Table locks during two-phase commit with, say, 50ms coast-to-coast latency would be an immediate dealkiller. Or, if it's eventually consistent as most asynchronously distributed DBs are, they can't claim AC. I really wish Cockroach had better documentation - not so much how to use it, but how it works so as to let engineers reason about performance characteristics.

deleted 9y ago
[deleted]
andreimatei1 9y ago
Besides the blog, you can find interesting stuff in the source repo, in the technotes and RFC dirs: https://github.com/cockroachdb/cockroach/tree/master/docs/tech-notes https://github.com/cockroachdb/cockroach/tree/master/docs/te... https://github.com/cockroachdb/cockroach/tree/master/docs/RFCS https://github.com/cockroachdb/cockroach/tree/master/docs/RF... There's also a bunch of technical tech-talks linked from our company's website: https://www.cockroachlabs.com/community/tech-talks/ https://www.cockroachlabs.com/community/tech-talks/
chubot 9y ago
Not much detail here, but: https://www.cockroachlabs.com/docs/stable/frequently-asked-questions.html#how-is-cockroachdb-both-highly-available-and-strongly-consistent https://www.cockroachlabs.com/docs/stable/frequently-asked-q... Last I heard CockroachDB is like "Spanner without atomic clocks". Not sure if that helps but at least the WAN latency is not an afterthought. https://www.cockroachlabs.com/docs/stable/frequently-asked-questions.html#do-transactions-in-cockroachdb-guarantee-acid-semantics https://www.cockroachlabs.com/docs/stable/frequently-asked-q... https://www.cockroachlabs.com/docs/stable/frequently-asked-questions.html#when-is-cockroachdb-not-a-good-choice https://www.cockroachlabs.com/docs/stable/frequently-asked-q...
knz42 9y ago
(working for cockroach labs) As explained in the blog (as linked by another response) and the documentation (https://www.cockroachlabs.com/docs/stable/strong-consistency.html https://www.cockroachlabs.com/docs/stable/strong-consistency... and https://www.cockroachlabs.com/docs/stable/distributed-transactions.html https://www.cockroachlabs.com/docs/stable/distributed-transa...) CockroachDB does not use locks, and write intents in the concurrency control scheme are expressed by K/V pair i.e. per table cell. Meanwhile, I acknowledge that CockroachDB does not yet convey well its performance model so that engineers can get an intuition about it. There is documentation, but it's either too detailed (like the design doc) or too high-level (most of the docs site). This needs a new sort of documentation, which we are working on.
- daxorid 9y ago
  Thanks!
- eternalban 9y ago
  From the first link: > Employs Raft, a popular successor to Paxos. I don't believe that is correct and that is not a good meme to promote. Successor implies the 'deprecation' of Paxos, which is certainly not the case. For the architectural sweet spot where both protocols can be used, it is fair to say Raft provides the more accessible approach, and that for those who insist on rolling their own consensus mechanism. Paxos and Raft serve distinct architectural concerns in the shared context of providing a distributed state with formally defined consistency guarantees. And to the point, it is entirely reasonable to see a distributed system that utilizes both to address system component specific concerns. CockroachDB, ironically (in context of your OP), in fact could benefit from a vendor lock-in were the candidate cloud provider to provide atomic clocks and GPS cards in their offerings.
  - ddorian43 9y ago
    What are the GPS cards for ?
    - infogulch 9y ago
occultist_throw 9y ago
The trend Ive been seeing is "AP with eventual C". It comoes down to the idea that we have a planet, with datacenters in locations on said planet. Light down fiber or copper only goes so fast. The "eventual consistency" part is simply acknowledging that we can't beat C when our databases are in North America, South America, Europe, Africa, Australia, and Asia. One-way comms takes time. 2 way comms, even more so. The tradeoff is, that changes can come in, and 'eventually' will propagate to the rest of the DB nodes. It does sound not ideal initially, however this type of operation would allow a localDB running as AP to sync when back online. The examples I'm deriving come from recent DBs, like GUN http://gun.js.org/#step1 http://gun.js.org/#step1
- dastbe 9y ago
  cockroachdb and spanner are "CP with high probability of A". There are situations where the db (or possibly a subset, haven't read too deeply into the papers) can become unavailable due to a loss of quorum, but these situations are highly unlikely.

ddorian43 9y ago

Still waiting for some benchmarks by the team. What I would be curious, is, how much faster the db would be if it was in c++/rust instead of java/go. I know bigtable claims 3x faster than hbase. Scylla claims ~10x (though also different architecture) faster than Cassandra. Trinity ~2x compared to Lucene. etc etc. Also: incoming comments about the name of the db.

jrs95 9y ago
Agreed. I love Go, but a database is one of the things in the stack that ought to be as optimized as possible in my opinion. C++ still seems preferable in these types of applications. At the same time, if the underlying architecture is enough of an improvement over older systems, it's still an upgrade.
- zzzcpan 9y ago
  You can't optimize anything with C++ if your database is stuck waiting for consensus with nodes 100 ms away. Close to the metal and low level languages are really poor choices here, even Go is too low level for CP databases.
  - ddorian43 9y ago
    Yes, but you'll have lower overhead on each query/request and more concurrency ? For each query the bottleneck is the network which you can't fix, but you can fix the 2,3,etc bottlenecks which are inside 1 machine (cpu/ram/disk).
    - jrs95 9y ago
      Yeah...I'm not really aware of a higher level language that mystically makes networking faster. As far as I'm aware, the most efficient network protocols tend to be implemented at a low level, too.
- wiremine 9y ago
  > C++ still seems preferable in these types of applications Agreed, but until there are some published performance metrics, I think it's an assumption more than a truth. Premature optimization...

mslate 9y ago

Haha cloud lock-in is what they think I'm worried about?

TruthSHIFT 9y ago
So... What are you worried about?
- mslate 9y ago
  Database lockin. Guess that was not clear enough.
  - rrdharan 9y ago
    In theory the OSS-nature of CockroachDB helps with that part.
  - andreimatei1 9y ago
    FWIW, CRDB should be a good fit for people worried about database lock-in. You'd only be "locked in" if you discover that we're much better than the alternatives :) CRDB uses the standard SQL language. Also, CRDB speaks the PostgreSQL wire protocol; for using CRDB you'd be using the Postgres drivers for different languages. So, more or less, any application that can work with us should also work with Postgres (and vice versa). Moreover, the Postgres drivers generally respect other higher interfaces (odbc, jdbc) so, more or less, a CRDB application should also work with any other established SQL database (and vice versa). CRDB can also export your data in formats easily importable by other SQL databases.
wiremine 9y ago
It's a valid concern for business. A lot of my customers go to some pretty interesting lengths to make sure they're not locked into a single vendor's stack.

meddlepal 9y ago

I'd cloud vendor lock-in a serious concern for most businesses? It always feels like a requirement that originates from the technical side rather than the business side. But it never feels like it's that important actually.

ddorian43 9y ago
It is important for those not on the cloud: dedicated servers and colo.
- rconti 9y ago
  You should pluralize that 'colo' then. I've been through datacenter power outages, refusal to renew, and difficulty in getting circuits in a timely manner in multiple Tier 4 datacenters. Self-hosting is only as good as your planning, strategy, and execution, which is much the same as the cloud world. You're probably safer running a single EC2 instance than running a single dedicated server in some random colo, but certainly running your own geographically distributed infrastructure is likely better than relying on a single AWS availability zone (for example). However, it's a great idea to open up peoples' eyes to the idea of how being on a major cloudy provider can make you collateral damage of an attack that never would have targeted you if you were in DIY mode.
manigandham 9y ago
It's a overblown concern for many, considering that the major clouds are massive companies that will likely be around far longer than most of their customers (or will acquire them). Every new client actually contributes to the longevity of a cloud provider. There's also lots of competition to offer similar features and many services can be mixed/matched easily (eg: we run on GCP but use Azure event hubs).
scaryclam 9y ago
We have a bunch of work that requires large, process heavy, calculations, so yes, vendor lock-in is a concern for us as it can mean either huge cost increases, if the vendor changes pricing for the required instances upwards, or huge savings if a competitor can do the same computations for less. We solve this problem by making sure we can trash servers and spin them up elsewhere if needs be (mostly via terraform etc). I don't think this should apply to anyone running some hosts for web applications though.

ycmimi 9y ago

It's like operating systems. In the end, you will have a limited number of options, aws, gcp, and azure, like windows, mac and linux.

JohnnyConatus 9y ago
Yup. And the amount of work it takes to be competent in just one (such as AWS) will make it nearly impossible / not worth it for most people to run cross-cloud.

sengork 9y ago

For anyone interested in multi cloud deployments at the IaaS layer I recommend looking at https://libcloud.apache.org/ https://libcloud.apache.org/

deleted 9y ago
[deleted]

tyingq 9y ago

I'm not convinced this helps much with lock in. Assuming you want to do things like deployments, scheduled backups, monitoring, etc, you're either: a) Using "lock in" features like Cloud Formation, Lamba, Cloudwatch, etc, to do so, such that you use some of the "value" of the cloud. ==or== b) Rolling your own solutions for these peripheral functions (deployment, backup, monitoring, etc) and running them on the bare VPS instances. In which case...why are you on this expensive cloud? You can get a VPS with lower bandwidth costs elsewhere. Basically being sort of "half in" on the cloud, where you use it as a glorified VPS makes no sense to me. Either be all the way in, or all the way out.

jchw 9y ago
Alternatively, you can use tools like Kubernetes and Terraform as abstractions around cloud environments while still being able to utilize them fairly effectively. Kubernetes is a clever product for Google to release, because it makes migrating loads to GCP much easier if you already run them in Kubernetes.
- tyingq 9y ago
  Similar decision for me though. If you want k8s, use Google or Azure, where they take care of it for you. K8s on AWS, at least right now, means you are in the business of being a plumber. For example, if I remember right, the solid k8s ingress controller means ELB, which has issues like the need for cache warming. ALB is better, but has only bleeding edge support as an ingress controller in K8s. Similar for getting it installed on AWS in the first place. Rackspace or Digital Ocean or similar is a cheaper better place if you want to roll your own k8s solution.
  - jchw 9y ago
    I'm hosting K8S on AWS in production. Kops takes care of most of the details, though I _do_ understand the low level details of what's going on. Gotta be able to debug somehow. Google Cloud Platform is definitely cheaper than AWS, and their hosting of Kubernetes seals the deal for me. The only problem? Legacy. If I want to link two virtual machines across cloud environments, it introduces somewhat unacceptable latency, and throughput will be less reliable. So connecting to things like extremely busy Redis or AMQP clusters is not really as safe and downtime is more likely. Still, I don't feel like Kubernetes is terrible on AWS. It works fairly well. I'm not using Ingress resources right now, just a bunch of services with their own ELBs.

caleblloyd 9y ago

I think this is a poorly written blog post that completely ignores the challenges and difficulties that arise when going multi-cloud. Multi-cloud introduces huge amounts of latency, requires additional load balancing and routing logic, requires twice as much administration, etc. Multi-cloud is a huge decision that need lots of planning and application level support. I'd go so far as to say it is an order of magnitude easier to design an application that can be moved between clouds easily than it is to design a multi-cloud application. This still defeats vendor lock-in. I think CockroachDB is interesting tech, but this blog post reads like a pretty empty marketing piece.

jchanimal 9y ago
It’s true that operating your own cluster (of any technology) spanning cloud providers is not for the faint of heart. However, at FaunaDB we’ve found performance of multi-cloud globally consistent operations to be acceptable. We currently run across AWS and GCP, with Azure on the way, and we have customers running their own clusters in hybrid configurations.