8 ms·

How to escape lock-in with a multi-cloud stack

145 points by mikecb 10y ago

ipsin 10y ago
The blog entry covers a lot of options, but as a cheapskate who wants to play with cloud infrastructure, I'm wondering if there are any good resources for pricing comparison? I could use AWS or Google, but if I'm just keeping a toy cloud service going. I'm not particularly worried about reliability, and I don't want to spend $30+/month.
- ohyes 10y ago
  Spot instances are pretty good for cheapskates who want to play with cloud stuff.
  - mdaniel 10y ago
    And they would quickly demonstrate the value Kubernetes brings to the table when the spots are terminated out from underneath your app (yes, provided the Kubernetes API and etcd are running on the survivors ;-) )
- divbit 10y ago
  Digital ocean droplets are great for a $5/month cloud for toy projects. Extremely easy setup. I use it for running an instance of "https://thelounge.github.io https://thelounge.github.io " and a few other projects which essentially pays for itself since it's replacing a $5/month irccloud membership. I don't know, aws or google cloud or azure might have something similar, but the elastic pricing style options were less appealing for my use case when I was checking out the options.
- deleted 10y ago
  [deleted]
- manmtstream 10y ago
  OVH is really cheap and good. https://ovh.com https://ovh.com
  - pmontra 10y ago
    Agreed, but a word of caution if you're working with single VPSes. When a VPS has problems don't waste time to open a ticket to their support. They were never able to fix anything on my VPSes the 2 or 3 times I had problems since I started using them in December 2010. Try to reboot the VPS from the web manager. If that doesn't fix the problem create another VPS immediately and let the original one expire at the end of the period you paid for. I suggest to pay per month and to have an automated installation script. Make backups for the data and make sure they can be transferred to the new VPS. OVH has other offerings such has cloud and private cloud services. I never used them so the support there could be better.
    - kuschku 10y ago
      Another tip: don't ask for support via a ticket, but just call them. That works a lot better. (At least their German support does)
- theDoug 10y ago
  I also recommend Digital Ocean for quick/cheap personal toy services, but for projects of enduring duration and reliability pricing calculators are totally public for AWS[1] and GCP[2]. I haven't seen any direct comparison calculators as everyone's particular mileage/needs may vary. The good news is that at scale, the days of compute, storage, and data management becoming easy to use utilities to plug into are here, or near. The bad news is it still feels a bit too 'big' to plug small lamps into the main AWS or Cloud Platform grids, but I suspect those days are numbered as well. [1]: https://calculator.s3.amazonaws.com/index.html https://calculator.s3.amazonaws.com/index.html [2]: https://cloud.google.com/products/calculator/ https://cloud.google.com/products/calculator/ (Disclaimer: I’m a Google employee, dealing with Cloud, loving all options, but especially being able to have options.)
  - deleted 10y ago
    [deleted]
- pella 10y ago
  scalaway is also perfect for personal projects https://www.scaleway.com/pricing/ https://www.scaleway.com/pricing/ - €5.99/Month (4 x86 64bit Cores; 4GB Memory; 100GB SSD Disk ;1 Flexible public IPv4 )
- kasbah 10y ago
  Do you know about the AWS free tier? You can get an EC2 micro instance for free for 1 year. When it runs out it might be worth switching to Digital Ocean.
- kyrra 10y ago
  Most cloud providers have hourly billing (if not smaller). So you can spin up some instances of what you want to try and will only be billed a small amount. For Google Cloud, a compute engine f1-micro instance is only $4.09/month. Standing it up for only 24 hours is $0.19. As well, both Amazon and Google have budgets you can set on your account that can notify you if you go over some thresholds you setup. This can help prevent you from spending too much money.
piotrkaminski 10y ago
After all the noise about Firebase at the last Google I/O, it's odd that the article doesn't mention it at all. I guess its proprietary API and undocumented wire protocol with no open source alternatives would be off message...
- zwily 10y ago
  There are few things in life that would excite me more than Firebase going open source. Sad, I know. :/
- jephir 10y ago
  http://horizon.io/ http://horizon.io/ While Firebase is proprietary, you can export your database as JSON and import into RethinkDB.
  - piotrkaminski 10y ago
    And then rewrite your entire app against a new API. No thanks.
  - piotrkaminski 10y ago
    To be fair, I think Horizon is one of the more promising Firebase alternatives. However, it has completely different semantics, and is quite immature at this point (look at the limitations: http://horizon.io/docs/limitations/ http://horizon.io/docs/limitations/). But if Firebase remains proprietary and Horizon continues to improve I could certainly imagine that rewriting my app will eventually start to look attractive.
  - asadlionpk 10y ago
    I wish this was true. I am locked-in with Firebase and would love to self-host now that I have resources. Sadly I will have to rewrite the API. It's not that Firebase is not great, I just want to take control back.
- colemickens 10y ago
  Given that this blog post is about having portable workloads and Firebase is specifically not portable, isn't it obvious that talking about Firebase would be "off message"?
deleted 10y ago
[deleted]
harlowja 10y ago
Well that's odd, they sort of forget to mention the whole premise of (all the projects composed in) openstack (which is to provide the same APIs across providers, vendors... by its very nature escaping lock-in by doing this). Also sadly (and maybe I'm taking this wrong) but calling the holy land 'Google Infrastructure For Everyone Else' seems condescending ...
- deleted 10y ago
  [deleted]
- notliketherest 10y ago
  It also sounds super googley when they say "Some awesome people have started calling it .... " Really gives me the creeps when people talk like that.
  - milesward 10y ago
    Author here: I think Alex Polvi is, well, awesome.
- milesward 10y ago
  Author here: yeahhhh we don't know what to call it. CoreOS folks said this and it sorta made sense, we call out the name ookyness right in the post. AsKSHBDDVBTFRcsM seemed, well, dumb. Ideas?
  - ignoramous 10y ago
    I'll give you an idea: Let the PR do the talking.
  - harlowja 10y ago
    Call it 'googles view of a stack', at least that doesn't make it sound like an elitist view on (one of many) versions of a stack (that yes is just one version of many stacks that have and do work for the <wider world>).
jssmith 10y ago
Google's support for a multi-cloud vision would be a lot more meaningful if they were offering the open source alternatives as part of their cloud platform and contributing to their development. Google has done so in a few key cases (Kubernetes and Tensorflow), but in other areas it's hard to get comfortable with the belief that there will be smooth transition to other infrastructure.
- troymc 10y ago
  You've got a point. BigTable is an example; they've never open-sourced it and instead other OSS projects (notably the Hadoop ones) were inspired by it. Other examples are GFS, Colossus, Spanner and F1. The OSS analogue of GFS is HDFS (AFAIK), but I know of nothing equivalent to the others. (CockroachDB is a bit like Spanner, but is much more pragmatic about its clocks.)
xorgar831 10y ago
Many companies would be better off however using cloud services since the "lockin" still costs them less than the overhead of running xyz service them self. Lock-in is only important if you have a cheaper way to run something, not if you're hiding costs in IT salaries for operations. :)
- stephenr 10y ago
  > Lock-in is only important if you have a cheaper way to run something False. Lock-in is a business risk, and anyone who thinks it isn't is kidding themselves. If you are dependent on a single vendor for a non-standardised service, you are also at risk if the vendor goes out of business, if the vendor decides to raise prices significantly, if the vendor decides to discontinue that service, or even if the vendor decides to change the mode of operation for that service. You are also beholden to that vendor's internal procedures, and tech decisions, which aren't always in your best interests. You may remember last September when there was a fucking massive AWS outage, where customers couldn't access the console, EC2 instances wouldn't spin up, etc. The fault for all of that, was a network disruption which caused DynamoDB to flip it's shit and make "the worlds biggest cloud provider" into the worlds biggest brick shitting machine. They've probably made changes so that type of error is less likely in future, but that doesn't matter because you don't get to make that decision if you're a single vendor customer on AWS. Whatever choices they make, they make for you. > hiding costs in IT salaries for operations Here's a pro-tip for you: use of AWS/Azure/Google Cloud/etc doesn't absolve you of the need for operations staff. If you think your single nodejs developer is equally skilled and has enough time to setup and maintain your infrastructure just because it's dynamic and has a browser interface, I have a great investment offer for you, in magic beans futures.
  - FooHentai 10y ago
    All of this is accurate. Also on the infrastructure deployment and maintenance side, virtualization already vacated the need for any ordinary organization to spend time worrying about infrastructure. You just rent colo and slap hypervisors in. There are still organizations messing around with overly complex internal server systems because they have bad admins or are heavy in technical debt, but these are no longer the rule.
quickben 10y ago
Gentlemen, I would like to interrupt this "google promotes open-source anti lock-in" PR party, and ask for a minute of silence for the recently departed GoogleCode. Their actions speak louder than their PR blogs. Edit: spelling
- MikeKusold 10y ago
  When Google Code shutdown, people just needed to add another git remote and push their code up there. There wasn't any vendor lock-in there either. The Go community managed to migrate everything to GitHub without any real losses.
- johncolanduoni 10y ago
  Or maybe they shut it down because most project's pages had been voluntarily reduced to a link to GitHub? What about that explanation rings false?
- jpatokal 10y ago
  Um, Google Code was comprehensively obsoleted by Github. Google moved all their open source code to there, and provided a truckload of tools to help everybody else migrate to various other repos as well. http://google-opensource.blogspot.com.au/2015/03/farewell-to-google-code.html http://google-opensource.blogspot.com.au/2015/03/farewell-to...
- strcat 10y ago
  There's already an open standard for the core functionality of code hosting sites: Git. Projects migrate between GitHub, GitLab, Bitbucket and other alternatives all the time by simply changing the Git remote. There's some lock-in via extras like issue trackers and wikis but that's not difficult to migrate, especially with Markdown as a common standard so markup can be preserved. I'm not really sure why you think that's relevant to this. Google Code simply failed to compete with the alternatives and was obsoleted. It has nothing to do with lock-in.
  - deleted 10y ago
    [deleted]
mattbillenstein 10y ago
I think the key thing is data interchange so you can get off of product X when you have business reasons to do so without running into a huge data migration project. Standard SQL databases you can use pretty much the same tooling as if you were running it yourself. Cloud storage, you can download/upload to a new thing, but it may take some time and cost some bandwidth. BigQuery/Redshift you should be storing your data as .json.gz on S3/GCS as a backup anyway, so an export shouldn't be needed -- you'd just import the cloud storage data into another system. I did a smallish BigTable dump a couple months ago -- kinda a PITA to do -- I don't recommend it; maybe I was doing it wrong, I'm not that familiar with hbase and it was a one-off thing. I also did a rather small scale data migration out of app engine (~100GB) awhile back -- total PITA -- datastore dumps are protobuf, so you need to build special tooling to import the data into another database. And generally, I don't use the other services to avoid lock-in. DNS round-robin+nginx for load balancing and ssl termination, Saltstack/Ansible deploying to Ubuntu VMs for config management and deployment. I run the databases myself so I can run it on any cloud I want -- or localhost; having a full dev environment you can create and destroy on your own machine using the same tools that you deploy to is really nice. This also enables doing things like running dev workloads on cheaper clouds (Linode) while running production in a VPC on AWS.
itaysk 10y ago
As someone who worked with Google from a customer perspective, they take pride in the fact that their big data (BigTable, BigQuery, DataFlow) is proprietary and superior the their OSS alternatives (I believe they actually are). I am not saying that they should OSS it, if they build something so awesome that differentiate them, they should know how to monetize it. But at least don't come off with a post like this.
- jpatokal 10y ago
  Dataflow is the hosted version of Apache Beam, which is OSS. http://beam.incubator.apache.org/ http://beam.incubator.apache.org/ BigTable is compatible with Apache HBase API, so it's trivial to swap out. https://cloud.google.com/bigtable/docs/bigtable-and-hbase https://cloud.google.com/bigtable/docs/bigtable-and-hbase Even BigQuery now has a standard SQL interface, although it's still Beta: https://cloud.google.com/bigquery/sql-reference/ https://cloud.google.com/bigquery/sql-reference/ Working at Google Cloud myself, I find the company philosophy to be to provide standard interfaces to superior implementations. This keeps us honest, because if the door to leave is open, then you have to actually want to stay. (Standard disclaimer: my opinion, not the company's.)
jakozaur 10y ago
It takes a lot of engineering time and effort to be able to switch between mentioned alternatives. E.g. Druid and Drill vs. BigQuery. This article is more probably to convince ppl to use Google Cloud rather than how to run multicloud strategy.
- stephenr 10y ago
  ding ding ding, we have a winner. If their "cloud" offerings were all just hosted instances of open source, there would be no lock-in, but they would have no 'competitive edge'. I think this is more about letting Google fanboy dev's sell higher-ups on GCP because "hey look they're not trying to lock us in like AWS is!"
  - milesward 10y ago
    OP/fanboy here; i'm interested in your thoughts on our products like Cloud BigTable: OSS (HBase) API, but unique goog performance bits under the hood. Valid? How can we best move the state of the art forward?
    - jakozaur 10y ago
      Though I agree some Google products are great, but cloud industry lacks standardized APIs allowing to switch between products. E.g. in the past we had OpenGL, PC which spur a lot of innovation and created whole industries. Common subset that allows seamless migrations. E.g. I wish we have standard for big table like databases: 1. Google Cloud BigTable. 2. DynamoDB. 3. HBase/Cassandra.
      - milesward 10y ago
        I'm with you! We didn't want to ship yet another one so Cloud BigTable uses the HBase API: https://cloud.google.com/bigtable/docs/bigtable-and-hbase https://cloud.google.com/bigtable/docs/bigtable-and-hbase
        10y ago
niemyjski 10y ago
This is one of the main reasons we built https://github.com/exceptionless/Foundatio https://github.com/exceptionless/Foundatio as we didn't want to be locked into azure or aws and have a great dev experience.
jacques_chester 10y ago
I'm surprised they didn't mention Cloud Foundry, which runs on OpenStack, vSphere, AWS, Azure and ... GCP. It's owned and managed by an independent foundation and is the most mature of the open PaaSes, having a head start of several years. I guess they see it as a competitor to Kubernetes. Or, plausibly, Google is very large and the engineers who are working on Cloud Foundry integration with GCP are not known to Miles Ward, Head of Global Solutions. I work for Pivotal, we donate the majority of engineering to Cloud Foundry. We make our money selling a distribution, Pivotal CF. Cloud Foundry's ability to switch cloud providers, or to take a cloud inhouse, and in future to do so seamlessly, is very attractive to our customers. IaaS providers remind them the bad old days of RDBMS and middleware lockin and they do not want to go back. As for the services, it's good that Google are pointing to open alternatives.
mikecb 10y ago
I don't really understand why the title was changed to this from my original or the title of the post, since it is less informative and more inflammatory...
- brazzledazzle 10y ago
  It looks like the current title is in the URL but not on the page itself. Maybe someone at Google changed it but retained the URL for obvious reasons.

145 points by mikecb 10y ago

ipsin 10y ago

The blog entry covers a lot of options, but as a cheapskate who wants to play with cloud infrastructure, I'm wondering if there are any good resources for pricing comparison? I could use AWS or Google, but if I'm just keeping a toy cloud service going. I'm not particularly worried about reliability, and I don't want to spend $30+/month.

ohyes 10y ago
Spot instances are pretty good for cheapskates who want to play with cloud stuff.
- mdaniel 10y ago
  And they would quickly demonstrate the value Kubernetes brings to the table when the spots are terminated out from underneath your app (yes, provided the Kubernetes API and etcd are running on the survivors ;-) )
divbit 10y ago
Digital ocean droplets are great for a $5/month cloud for toy projects. Extremely easy setup. I use it for running an instance of "https://thelounge.github.io https://thelounge.github.io " and a few other projects which essentially pays for itself since it's replacing a $5/month irccloud membership. I don't know, aws or google cloud or azure might have something similar, but the elastic pricing style options were less appealing for my use case when I was checking out the options.
deleted 10y ago
[deleted]
manmtstream 10y ago
OVH is really cheap and good. https://ovh.com https://ovh.com
- pmontra 10y ago
  Agreed, but a word of caution if you're working with single VPSes. When a VPS has problems don't waste time to open a ticket to their support. They were never able to fix anything on my VPSes the 2 or 3 times I had problems since I started using them in December 2010. Try to reboot the VPS from the web manager. If that doesn't fix the problem create another VPS immediately and let the original one expire at the end of the period you paid for. I suggest to pay per month and to have an automated installation script. Make backups for the data and make sure they can be transferred to the new VPS. OVH has other offerings such has cloud and private cloud services. I never used them so the support there could be better.
  - kuschku 10y ago
    Another tip: don't ask for support via a ticket, but just call them. That works a lot better. (At least their German support does)
theDoug 10y ago
I also recommend Digital Ocean for quick/cheap personal toy services, but for projects of enduring duration and reliability pricing calculators are totally public for AWS[1] and GCP[2]. I haven't seen any direct comparison calculators as everyone's particular mileage/needs may vary. The good news is that at scale, the days of compute, storage, and data management becoming easy to use utilities to plug into are here, or near. The bad news is it still feels a bit too 'big' to plug small lamps into the main AWS or Cloud Platform grids, but I suspect those days are numbered as well. [1]: https://calculator.s3.amazonaws.com/index.html https://calculator.s3.amazonaws.com/index.html [2]: https://cloud.google.com/products/calculator/ https://cloud.google.com/products/calculator/ (Disclaimer: I’m a Google employee, dealing with Cloud, loving all options, but especially being able to have options.)
- deleted 10y ago
  [deleted]
pella 10y ago
scalaway is also perfect for personal projects https://www.scaleway.com/pricing/ https://www.scaleway.com/pricing/ - €5.99/Month (4 x86 64bit Cores; 4GB Memory; 100GB SSD Disk ;1 Flexible public IPv4 )
kasbah 10y ago
Do you know about the AWS free tier? You can get an EC2 micro instance for free for 1 year. When it runs out it might be worth switching to Digital Ocean.
kyrra 10y ago
Most cloud providers have hourly billing (if not smaller). So you can spin up some instances of what you want to try and will only be billed a small amount. For Google Cloud, a compute engine f1-micro instance is only $4.09/month. Standing it up for only 24 hours is $0.19. As well, both Amazon and Google have budgets you can set on your account that can notify you if you go over some thresholds you setup. This can help prevent you from spending too much money.

piotrkaminski 10y ago

After all the noise about Firebase at the last Google I/O, it's odd that the article doesn't mention it at all. I guess its proprietary API and undocumented wire protocol with no open source alternatives would be off message...

zwily 10y ago
There are few things in life that would excite me more than Firebase going open source. Sad, I know. :/
jephir 10y ago
http://horizon.io/ http://horizon.io/ While Firebase is proprietary, you can export your database as JSON and import into RethinkDB.
- piotrkaminski 10y ago
  And then rewrite your entire app against a new API. No thanks.
- piotrkaminski 10y ago
  To be fair, I think Horizon is one of the more promising Firebase alternatives. However, it has completely different semantics, and is quite immature at this point (look at the limitations: http://horizon.io/docs/limitations/ http://horizon.io/docs/limitations/). But if Firebase remains proprietary and Horizon continues to improve I could certainly imagine that rewriting my app will eventually start to look attractive.
- asadlionpk 10y ago
  I wish this was true. I am locked-in with Firebase and would love to self-host now that I have resources. Sadly I will have to rewrite the API. It's not that Firebase is not great, I just want to take control back.
colemickens 10y ago
Given that this blog post is about having portable workloads and Firebase is specifically not portable, isn't it obvious that talking about Firebase would be "off message"?

deleted 10y ago

[deleted]

harlowja 10y ago

Well that's odd, they sort of forget to mention the whole premise of (all the projects composed in) openstack (which is to provide the same APIs across providers, vendors... by its very nature escaping lock-in by doing this). Also sadly (and maybe I'm taking this wrong) but calling the holy land 'Google Infrastructure For Everyone Else' seems condescending ...

deleted 10y ago
[deleted]
notliketherest 10y ago
It also sounds super googley when they say "Some awesome people have started calling it .... " Really gives me the creeps when people talk like that.
- milesward 10y ago
  Author here: I think Alex Polvi is, well, awesome.
milesward 10y ago
Author here: yeahhhh we don't know what to call it. CoreOS folks said this and it sorta made sense, we call out the name ookyness right in the post. AsKSHBDDVBTFRcsM seemed, well, dumb. Ideas?
- ignoramous 10y ago
  I'll give you an idea: Let the PR do the talking.
- harlowja 10y ago
  Call it 'googles view of a stack', at least that doesn't make it sound like an elitist view on (one of many) versions of a stack (that yes is just one version of many stacks that have and do work for the <wider world>).

jssmith 10y ago

Google's support for a multi-cloud vision would be a lot more meaningful if they were offering the open source alternatives as part of their cloud platform and contributing to their development. Google has done so in a few key cases (Kubernetes and Tensorflow), but in other areas it's hard to get comfortable with the belief that there will be smooth transition to other infrastructure.

troymc 10y ago
You've got a point. BigTable is an example; they've never open-sourced it and instead other OSS projects (notably the Hadoop ones) were inspired by it. Other examples are GFS, Colossus, Spanner and F1. The OSS analogue of GFS is HDFS (AFAIK), but I know of nothing equivalent to the others. (CockroachDB is a bit like Spanner, but is much more pragmatic about its clocks.)

xorgar831 10y ago

Many companies would be better off however using cloud services since the "lockin" still costs them less than the overhead of running xyz service them self. Lock-in is only important if you have a cheaper way to run something, not if you're hiding costs in IT salaries for operations. :)

stephenr 10y ago
> Lock-in is only important if you have a cheaper way to run something False. Lock-in is a business risk, and anyone who thinks it isn't is kidding themselves. If you are dependent on a single vendor for a non-standardised service, you are also at risk if the vendor goes out of business, if the vendor decides to raise prices significantly, if the vendor decides to discontinue that service, or even if the vendor decides to change the mode of operation for that service. You are also beholden to that vendor's internal procedures, and tech decisions, which aren't always in your best interests. You may remember last September when there was a fucking massive AWS outage, where customers couldn't access the console, EC2 instances wouldn't spin up, etc. The fault for all of that, was a network disruption which caused DynamoDB to flip it's shit and make "the worlds biggest cloud provider" into the worlds biggest brick shitting machine. They've probably made changes so that type of error is less likely in future, but that doesn't matter because you don't get to make that decision if you're a single vendor customer on AWS. Whatever choices they make, they make for you. > hiding costs in IT salaries for operations Here's a pro-tip for you: use of AWS/Azure/Google Cloud/etc doesn't absolve you of the need for operations staff. If you think your single nodejs developer is equally skilled and has enough time to setup and maintain your infrastructure just because it's dynamic and has a browser interface, I have a great investment offer for you, in magic beans futures.
- FooHentai 10y ago
  All of this is accurate. Also on the infrastructure deployment and maintenance side, virtualization already vacated the need for any ordinary organization to spend time worrying about infrastructure. You just rent colo and slap hypervisors in. There are still organizations messing around with overly complex internal server systems because they have bad admins or are heavy in technical debt, but these are no longer the rule.

quickben 10y ago

Gentlemen, I would like to interrupt this "google promotes open-source anti lock-in" PR party, and ask for a minute of silence for the recently departed GoogleCode. Their actions speak louder than their PR blogs. Edit: spelling

MikeKusold 10y ago
When Google Code shutdown, people just needed to add another git remote and push their code up there. There wasn't any vendor lock-in there either. The Go community managed to migrate everything to GitHub without any real losses.
johncolanduoni 10y ago
Or maybe they shut it down because most project's pages had been voluntarily reduced to a link to GitHub? What about that explanation rings false?
jpatokal 10y ago
Um, Google Code was comprehensively obsoleted by Github. Google moved all their open source code to there, and provided a truckload of tools to help everybody else migrate to various other repos as well. http://google-opensource.blogspot.com.au/2015/03/farewell-to-google-code.html http://google-opensource.blogspot.com.au/2015/03/farewell-to...
strcat 10y ago
There's already an open standard for the core functionality of code hosting sites: Git. Projects migrate between GitHub, GitLab, Bitbucket and other alternatives all the time by simply changing the Git remote. There's some lock-in via extras like issue trackers and wikis but that's not difficult to migrate, especially with Markdown as a common standard so markup can be preserved. I'm not really sure why you think that's relevant to this. Google Code simply failed to compete with the alternatives and was obsoleted. It has nothing to do with lock-in.
- deleted 10y ago
  [deleted]

mattbillenstein 10y ago

I think the key thing is data interchange so you can get off of product X when you have business reasons to do so without running into a huge data migration project. Standard SQL databases you can use pretty much the same tooling as if you were running it yourself. Cloud storage, you can download/upload to a new thing, but it may take some time and cost some bandwidth. BigQuery/Redshift you should be storing your data as .json.gz on S3/GCS as a backup anyway, so an export shouldn't be needed -- you'd just import the cloud storage data into another system. I did a smallish BigTable dump a couple months ago -- kinda a PITA to do -- I don't recommend it; maybe I was doing it wrong, I'm not that familiar with hbase and it was a one-off thing. I also did a rather small scale data migration out of app engine (~100GB) awhile back -- total PITA -- datastore dumps are protobuf, so you need to build special tooling to import the data into another database. And generally, I don't use the other services to avoid lock-in. DNS round-robin+nginx for load balancing and ssl termination, Saltstack/Ansible deploying to Ubuntu VMs for config management and deployment. I run the databases myself so I can run it on any cloud I want -- or localhost; having a full dev environment you can create and destroy on your own machine using the same tools that you deploy to is really nice. This also enables doing things like running dev workloads on cheaper clouds (Linode) while running production in a VPC on AWS.

itaysk 10y ago

As someone who worked with Google from a customer perspective, they take pride in the fact that their big data (BigTable, BigQuery, DataFlow) is proprietary and superior the their OSS alternatives (I believe they actually are). I am not saying that they should OSS it, if they build something so awesome that differentiate them, they should know how to monetize it. But at least don't come off with a post like this.

jpatokal 10y ago
Dataflow is the hosted version of Apache Beam, which is OSS. http://beam.incubator.apache.org/ http://beam.incubator.apache.org/ BigTable is compatible with Apache HBase API, so it's trivial to swap out. https://cloud.google.com/bigtable/docs/bigtable-and-hbase https://cloud.google.com/bigtable/docs/bigtable-and-hbase Even BigQuery now has a standard SQL interface, although it's still Beta: https://cloud.google.com/bigquery/sql-reference/ https://cloud.google.com/bigquery/sql-reference/ Working at Google Cloud myself, I find the company philosophy to be to provide standard interfaces to superior implementations. This keeps us honest, because if the door to leave is open, then you have to actually want to stay. (Standard disclaimer: my opinion, not the company's.)

jakozaur 10y ago

It takes a lot of engineering time and effort to be able to switch between mentioned alternatives. E.g. Druid and Drill vs. BigQuery. This article is more probably to convince ppl to use Google Cloud rather than how to run multicloud strategy.

stephenr 10y ago
ding ding ding, we have a winner. If their "cloud" offerings were all just hosted instances of open source, there would be no lock-in, but they would have no 'competitive edge'. I think this is more about letting Google fanboy dev's sell higher-ups on GCP because "hey look they're not trying to lock us in like AWS is!"
- milesward 10y ago
  OP/fanboy here; i'm interested in your thoughts on our products like Cloud BigTable: OSS (HBase) API, but unique goog performance bits under the hood. Valid? How can we best move the state of the art forward?
  - jakozaur 10y ago
    Though I agree some Google products are great, but cloud industry lacks standardized APIs allowing to switch between products. E.g. in the past we had OpenGL, PC which spur a lot of innovation and created whole industries. Common subset that allows seamless migrations. E.g. I wish we have standard for big table like databases: 1. Google Cloud BigTable. 2. DynamoDB. 3. HBase/Cassandra.
    - milesward 10y ago
      I'm with you! We didn't want to ship yet another one so Cloud BigTable uses the HBase API: https://cloud.google.com/bigtable/docs/bigtable-and-hbase https://cloud.google.com/bigtable/docs/bigtable-and-hbase
      - 10y ago

niemyjski 10y ago

This is one of the main reasons we built https://github.com/exceptionless/Foundatio https://github.com/exceptionless/Foundatio as we didn't want to be locked into azure or aws and have a great dev experience.

jacques_chester 10y ago

I'm surprised they didn't mention Cloud Foundry, which runs on OpenStack, vSphere, AWS, Azure and ... GCP. It's owned and managed by an independent foundation and is the most mature of the open PaaSes, having a head start of several years. I guess they see it as a competitor to Kubernetes. Or, plausibly, Google is very large and the engineers who are working on Cloud Foundry integration with GCP are not known to Miles Ward, Head of Global Solutions. I work for Pivotal, we donate the majority of engineering to Cloud Foundry. We make our money selling a distribution, Pivotal CF. Cloud Foundry's ability to switch cloud providers, or to take a cloud inhouse, and in future to do so seamlessly, is very attractive to our customers. IaaS providers remind them the bad old days of RDBMS and middleware lockin and they do not want to go back. As for the services, it's good that Google are pointing to open alternatives.

mikecb 10y ago

I don't really understand why the title was changed to this from my original or the title of the post, since it is less informative and more inflammatory...

brazzledazzle 10y ago
It looks like the current title is in the URL but not on the page itself. Maybe someone at Google changed it but retained the URL for obvious reasons.