10 ms·

I've been quite frustrated with the google cloud platform, Just take a look at their APIs for AppEngine, CloudSQL, GCE and the sort, Its pathetic compared to th

by mataug 11y ago

I've been quite frustrated with the google cloud platform, Just take a look at their APIs for AppEngine, CloudSQL, GCE and the sort, Its pathetic compared to their direct competitor AWS. Lets compare trying to do create an RDS instance on AWS vs Creating one on CloudSQL AWS: 1. Get AccessToken and AccessSecret from IAM 2. pip install boto 3. conn = boto.rds.connect_to_region("us-west-2") 4. db = conn.create_dbinstance("db-master-1", 10, 'db.m1.small', 'root', 'hunter2') Done ! Google 1. Get a Client Id and Client Secret 2. pip install google-api-python-client 3. Go through the OAuth Flow and run a server locally to capture the access token 4. Use the discovery api to generate a service object. Good Luck finding this in the documentation 5. use the uninspectable service object to create a cloudsql instance. The reason I don't have code for steps 3,4 and 5 is because I gave up after wasting time trying to figure this out. My point is that they've gotten into the habit of doing half assed work so I have no hopes that the've improved this time. Practically no way to automate this The only way to use this would be from the horribly slow GUIs that google provides. EDIT: I ended up using google cloud sdk cli and running the automations with subprocess.check_output(['gcloud', 'sql', 'instance' ... ])

erlichmen 11y ago
You are doing it wrong: 1. curl https://sdk.cloud.google.com | bash 2. gcloud auth login (one time thing) 3. gcloud sql instances create db-master-1
- dantheman 11y ago
  This is a bad practice: curl * | bash and should be discouraged. It's easy but it's like saying, hey I'm logged in as root all the time - it's easier.
  - erlichmen 11y ago
    You don't need root to install gcloud
    - icebraining 11y ago
      I think that was just an analogy.
  - icebraining 11y ago
    And only it really does anyway is wget https://dl.google.com/dl/cloudsdk/release/install_google_cloud_sdk.bash chmod 775 install_google_cloud_sdk.bash ./install_google_cloud_sdk.bash
  - olalonde 11y ago
    In this case, * is "https://sdk.cloud.google.com" https://sdk.cloud.google.com" and I don't see how it's worse than trusting a package from PyPi. If anything, the curl command offers some guarantee that you are running code endorsed by Google.
    - lukeschlather 11y ago
      If the HTTP connection is interrupted during download (highly likely if you're doing this routinely) you'll end up with something in a broken state. Locally running a remote stream as it arrives as code is just a bad idea, unless you're talking about something like a webpage where a partial is potentially preferable to nothing at all.
      - deleted 11y ago
        [deleted]
      - cristicbz 11y ago
        Code which is meant to be piped into bash is generally written as: #!env bash f () { ...code ...code ...code } f Hence a partial stream will do nothing (syntax error, missing brace to be precise).
      - plorkyeran 11y ago
        This is a completely trivial problem to solve (wrap the logic in a function), which the script in question does solve.
  - crdoconnor 11y ago
    Only if you don't trust the source (or it's http). Otherwise you might as well give up on downloading any software from the internet. Ever.
  - brlewis 11y ago
    It's no different from steps 2 and 3. At each step you're trusting that code you just downloaded from the web is doing what you want it to do.
    - lukeschlather 11y ago
      You're also trusting that the Internet is going to stay up for the duration of the install script, which is an unreliable assumption. Imagine at some point the script does: rm -rf ~/.config/google and the connection gives out at rm -rf ~/ Suddenly your script didn't install, and you've blown away your home directory. HTTP(S) is designed for reading documents, where it's OK if you can't read the document in its entirety.
      - delroth 11y ago
        This is easily solved by putting everything in a bash function and calling the function as the last thing in the script. If you look at the Cloud SDK setup script, that is exactly what it does.
        lukeschlather 11y ago
        I agree it's a trivial problem to solve, I just think the right way is to download the code and store it separately, so that it's easy to add checksum/signature validation later.
  - Blackthorn 11y ago
    Have you ever run "./configure" on some open source code you downloaded? That's no more safe. In this case, curl | bash is probably more safe because at least it came over HTTPS.
    - discardorama 11y ago
      > Have you ever run "./configure" on some open source code you downloaded? That's no more safe. That is if you don't do a view ./configure first and go through it. You do take a look at the configure scripts, don't you?
      - Blackthorn 11y ago
        What's less readable, a ToS that you have to accept or a GNU autotools configure script? Example: expat's autoconf script is over twenty two thousand lines long.
      - cos 11y ago
        You could just as easily do 'curl someurl | less' before you run it, too.
- estefan 11y ago
  > 2. gcloud auth login (one time thing) That's a royal pain in the arse to automate with something like ansible... AWS nailed it with a token & secret (not some horrible expiring oauth2 object). And that's not to mention gsutils sleeping for a second or two every time it tells you there's an update. It's like these guys have never heard of devops. Updates are my problem; I don't expect software to sleep whenever they're available... because you never know, I might be running it in a loop under cron and it might just piss me off when I start to lose performance...
  - skj 11y ago
    'gcloud auth activate-service-account' will auth using a JWT, acquired from the Cloud Console. Also, if you are running stuff in Google Compute Engine, no auth flow is needed at all: there is a metadata service that you can connect to (and gcloud connects to by default) that provides credentials associated with the machine you're on. > It's like these guys have never heard of devops. :\ yeah...I can't imagine trying to automate a web login flow with ansible...but no one would ever suggest you do that. (disclosure: I wrote most of the client-side auth code for gcloud)
  - philip1209 11y ago
    This was also frustrating on something like CircleCI
TeeWEE 11y ago
Well, it might be a bit more difficult to get started. The cool thing is that you only have to install the google-api-python once, and you can communicate with lots of services, by creating the service object. CloudPubSub, BigQuery etc etc. Also, google comes with very good monitoring tools (they bought stackdriver and integrated it). We are running a big appengine app, with quite some compute engine instances. Together with cloud-storage and biguery its a full blown solution, without having to setup every instance yourselve. I think AWS cant compete with that, you almost need a system-engineer when you work with AWS. All in all: you are used to AWS. Making it cumbersome to get started with Google Cloud. I do agree tough that the documention is lacking often.
- piva00 11y ago
  There's also the point about vendor lock-in, in AWS you do need a systems engineer to setup and automate infrastructure, but pretty much everything (aside from a few products like Redshift, Dynamo and EMR is what I recall from my mind) you can port out of AWS pretty easily if and when the time comes to have your own data center and cut some costs. Google Cloud is a much bigger vendor lock-in, you could not setup the same environment outside of it.
  - vgt 11y ago
    I'm genuinely curious why you hold that opinion, especially in context of a thread that discusses a service that's available through the standard hbase api
    - rodionos 11y ago
      Without support for co-processors in BigTable I would hardly call it 'standard' API. Also the implementation of buffered mutation, a key attraction in HBase 1+ for fast non-blocking writes, is different enough in bigtable that you might have to change application code to work it consistently. It's nice though that bigtable API have references to HBase bugs :) > HBase 1.0.0 has a bug in it. It returns maxSize instead of the buffer size for > getWriteBufferSize. https://issues.apache.org/jira/browse/HBASE-13113 https://issues.apache.org/jira/browse/HBASE-13113
forgottenacc56 11y ago
You missed the snakes and ladders game of following googles out of date documentation. "You're doing it wrong, we deprecated that last year, I know we haven't yet updated the 4 separate places in which the same thing is documented all in a different context and different way. We really should fix that but we're busy coding. Did you know we're all PhDs at Google?" What?!?!? You're using the Google console version 3? Why? That feature isn't implemented there. Stupid you. You're MEANT to be using our new Google console version X. Why are you using our old one? Also you missed the 16 hours you'll spend trying work out why something isn't working only to find it's actually been changed or taken out of the Google developer console, without any trace or notice left in the code to say'we moved/removed the feature that you expected to be here.' Really you should ask Google for help on StackOverflow which is now the official Google channel for ignoring support questions, and where your question will within seconds be down voted, derided and deleted by the StackOverflow Community,saving Google the effort of not reading and ignoring your question about how to resolve the catastrophic failure of your software. Seriously though, why not entrust your critical systems to such capable hands?
- wslh 11y ago
  Also, it seems Google's Page Rank is sometimes optimized to show old documentation instead of the most related and current one. this is natural because there are more links pointing to the old one but needs to be fixed.
- jfoster 11y ago
  It's a shame about the tone of this post. I think it might not suit HN. As a Google Cloud customer, though, I think you do manage to be spot on about a lot of the things Google are doing wrong when it comes to the Cloud Platform. Today I came across an App Engine SDK bug that was reported almost 5 years ago, has a several years old patch in the comments, and still hasn't been remedied by Google. App Engine users who encounter it have to find the bug, apply the patch to the SDK, and repeat that upon each SDK update.
  - dlor 11y ago
    Could you send me a link to the issue? I can hopefully get that fixed.
estefan 11y ago
Let's hope the oauth token you get back doesn't expire after you think you've finished! And then lets hope it doesn't also take you a day or so to have to learn oauth2client so you know which secret method you need to call request a refresh token. Such fun!
- mataug 11y ago
  It took me more than half a day to figure out that I was doing things wrong even though I've used oauth2client before on other projects.
samkone 11y ago
Or you can use the service account to avoid dancing around gcloud and others. But I agree Google has a problem building stuff for us common humans who do not work at the big G.
- mataug 11y ago
  I this stems from a general lack of empathy for customers in the organisation. Take a look at the issue list on appengine for example https://code.google.com/p/googleappengine/issues/detail?id=10259 https://code.google.com/p/googleappengine/issues/detail?id=1... It seems like they get a bunch of engineers to come and build stuff then once they move on, the project is abandoned or poorly maintained until another team of engineers come and try to rebuild it from scratch. Here's more proof, Google Wallet for example ... https://news.ycombinator.com/item?id=9498475 https://news.ycombinator.com/item?id=9498475
thrownaway2424 11y ago
Is there something in your comment regarding bigtable?
ryanseys 11y ago
Disclaimer: I work on gcloud-node [1]. :) You're describing the OAuth 2.0 client dance which has always been cumbersome but necessary for browser-side auth. With Google Cloud Platform related services, you should never have to do this if you're using these services from the context of your server. As someone previously mentioned, Google has Service Accounts which greatly simplifies the auth process. For example, to auth with gcloud-node it's as easy as: 1. Download the key.json associated with your service account. 2. var gcloud = require('gcloud')({ projectId: 'abc-123', keyFilename: './key.json' }); 3. There is no step 3. We've also made some pretty nice docs [2] with clear examples to help you every step of the way. We have yet to integrate Cloud Bigtable but it's on our radar. For a pythonista like yourself, you may like to consider trying gcloud-python [3], and it should go without saying please feel free to file issues or reach out directly with concerns if something just doesn't feel quite right! [1]. https://github.com/GoogleCloudPlatform/gcloud-node https://github.com/GoogleCloudPlatform/gcloud-node [2]. http://googlecloudplatform.github.io/gcloud-node/#/docs/ http://googlecloudplatform.github.io/gcloud-node/#/docs/ [3]. https://github.com/GoogleCloudPlatform/gcloud-python https://github.com/GoogleCloudPlatform/gcloud-python Edit: Formatting.
- mataug 11y ago
  I'll try this out. Thanks.
- nulltype 11y ago
  I like the gcloud-node docs! I remember the oauth stuff being a pain just because oauth is kind of complicated and the terms are weird and there was no guide that just says "go here and click this" you had to just figure that out.
anthmgoogle 11y ago
(I'm a Google developer on API Platform) A few of the follow ups mentioned that Service Accounts are more appropriate for this API and closer to the AWS experience. There have been some challenges using service accounts that have been addressed with a relatively new feature set called "Application Default Credentials": https://developers.google.com/identity/protocols/application-default-credentials https://developers.google.com/identity/protocols/application... For most cloud APIs, you simply run "gcloud auth login" to enable auth on the development machine, and when you deploy to GCE or GAE it will use the more secure built-in service accounts there. And there is no token expiration in either case.
TheIronYuppie 11y ago
Interesting point in that you used a third party in step 2 of your AWS workflow. If you're going to use a 3rd party, have you tried Terraform?

by mataug 11y ago

erlichmen 11y ago

You are doing it wrong: 1. curl https://sdk.cloud.google.com | bash 2. gcloud auth login (one time thing) 3. gcloud sql instances create db-master-1

dantheman 11y ago
This is a bad practice: curl * | bash and should be discouraged. It's easy but it's like saying, hey I'm logged in as root all the time - it's easier.
- erlichmen 11y ago
  You don't need root to install gcloud
  - icebraining 11y ago
    I think that was just an analogy.
- icebraining 11y ago
  And only it really does anyway is wget https://dl.google.com/dl/cloudsdk/release/install_google_cloud_sdk.bash chmod 775 install_google_cloud_sdk.bash ./install_google_cloud_sdk.bash
- olalonde 11y ago
  In this case, * is "https://sdk.cloud.google.com" https://sdk.cloud.google.com" and I don't see how it's worse than trusting a package from PyPi. If anything, the curl command offers some guarantee that you are running code endorsed by Google.
  - lukeschlather 11y ago
    If the HTTP connection is interrupted during download (highly likely if you're doing this routinely) you'll end up with something in a broken state. Locally running a remote stream as it arrives as code is just a bad idea, unless you're talking about something like a webpage where a partial is potentially preferable to nothing at all.
    - deleted 11y ago
      [deleted]
    - cristicbz 11y ago
      Code which is meant to be piped into bash is generally written as: #!env bash f () { ...code ...code ...code } f Hence a partial stream will do nothing (syntax error, missing brace to be precise).
    - plorkyeran 11y ago
      This is a completely trivial problem to solve (wrap the logic in a function), which the script in question does solve.
- crdoconnor 11y ago
  Only if you don't trust the source (or it's http). Otherwise you might as well give up on downloading any software from the internet. Ever.
- brlewis 11y ago
  It's no different from steps 2 and 3. At each step you're trusting that code you just downloaded from the web is doing what you want it to do.
  - lukeschlather 11y ago
    You're also trusting that the Internet is going to stay up for the duration of the install script, which is an unreliable assumption. Imagine at some point the script does: rm -rf ~/.config/google and the connection gives out at rm -rf ~/ Suddenly your script didn't install, and you've blown away your home directory. HTTP(S) is designed for reading documents, where it's OK if you can't read the document in its entirety.
    - delroth 11y ago
      This is easily solved by putting everything in a bash function and calling the function as the last thing in the script. If you look at the Cloud SDK setup script, that is exactly what it does.
      - lukeschlather 11y ago
        I agree it's a trivial problem to solve, I just think the right way is to download the code and store it separately, so that it's easy to add checksum/signature validation later.
- Blackthorn 11y ago
  Have you ever run "./configure" on some open source code you downloaded? That's no more safe. In this case, curl | bash is probably more safe because at least it came over HTTPS.
  - discardorama 11y ago
    > Have you ever run "./configure" on some open source code you downloaded? That's no more safe. That is if you don't do a view ./configure first and go through it. You do take a look at the configure scripts, don't you?
    - Blackthorn 11y ago
      What's less readable, a ToS that you have to accept or a GNU autotools configure script? Example: expat's autoconf script is over twenty two thousand lines long.
    - cos 11y ago
      You could just as easily do 'curl someurl | less' before you run it, too.
estefan 11y ago
> 2. gcloud auth login (one time thing) That's a royal pain in the arse to automate with something like ansible... AWS nailed it with a token & secret (not some horrible expiring oauth2 object). And that's not to mention gsutils sleeping for a second or two every time it tells you there's an update. It's like these guys have never heard of devops. Updates are my problem; I don't expect software to sleep whenever they're available... because you never know, I might be running it in a loop under cron and it might just piss me off when I start to lose performance...
- skj 11y ago
  'gcloud auth activate-service-account' will auth using a JWT, acquired from the Cloud Console. Also, if you are running stuff in Google Compute Engine, no auth flow is needed at all: there is a metadata service that you can connect to (and gcloud connects to by default) that provides credentials associated with the machine you're on. > It's like these guys have never heard of devops. :\ yeah...I can't imagine trying to automate a web login flow with ansible...but no one would ever suggest you do that. (disclosure: I wrote most of the client-side auth code for gcloud)
- philip1209 11y ago
  This was also frustrating on something like CircleCI

TeeWEE 11y ago

Well, it might be a bit more difficult to get started. The cool thing is that you only have to install the google-api-python once, and you can communicate with lots of services, by creating the service object. CloudPubSub, BigQuery etc etc. Also, google comes with very good monitoring tools (they bought stackdriver and integrated it). We are running a big appengine app, with quite some compute engine instances. Together with cloud-storage and biguery its a full blown solution, without having to setup every instance yourselve. I think AWS cant compete with that, you almost need a system-engineer when you work with AWS. All in all: you are used to AWS. Making it cumbersome to get started with Google Cloud. I do agree tough that the documention is lacking often.

piva00 11y ago
There's also the point about vendor lock-in, in AWS you do need a systems engineer to setup and automate infrastructure, but pretty much everything (aside from a few products like Redshift, Dynamo and EMR is what I recall from my mind) you can port out of AWS pretty easily if and when the time comes to have your own data center and cut some costs. Google Cloud is a much bigger vendor lock-in, you could not setup the same environment outside of it.
- vgt 11y ago
  I'm genuinely curious why you hold that opinion, especially in context of a thread that discusses a service that's available through the standard hbase api
  - rodionos 11y ago
    Without support for co-processors in BigTable I would hardly call it 'standard' API. Also the implementation of buffered mutation, a key attraction in HBase 1+ for fast non-blocking writes, is different enough in bigtable that you might have to change application code to work it consistently. It's nice though that bigtable API have references to HBase bugs :) > HBase 1.0.0 has a bug in it. It returns maxSize instead of the buffer size for > getWriteBufferSize. https://issues.apache.org/jira/browse/HBASE-13113 https://issues.apache.org/jira/browse/HBASE-13113

forgottenacc56 11y ago

You missed the snakes and ladders game of following googles out of date documentation. "You're doing it wrong, we deprecated that last year, I know we haven't yet updated the 4 separate places in which the same thing is documented all in a different context and different way. We really should fix that but we're busy coding. Did you know we're all PhDs at Google?" What?!?!? You're using the Google console version 3? Why? That feature isn't implemented there. Stupid you. You're MEANT to be using our new Google console version X. Why are you using our old one? Also you missed the 16 hours you'll spend trying work out why something isn't working only to find it's actually been changed or taken out of the Google developer console, without any trace or notice left in the code to say'we moved/removed the feature that you expected to be here.' Really you should ask Google for help on StackOverflow which is now the official Google channel for ignoring support questions, and where your question will within seconds be down voted, derided and deleted by the StackOverflow Community,saving Google the effort of not reading and ignoring your question about how to resolve the catastrophic failure of your software. Seriously though, why not entrust your critical systems to such capable hands?

wslh 11y ago
Also, it seems Google's Page Rank is sometimes optimized to show old documentation instead of the most related and current one. this is natural because there are more links pointing to the old one but needs to be fixed.
jfoster 11y ago
It's a shame about the tone of this post. I think it might not suit HN. As a Google Cloud customer, though, I think you do manage to be spot on about a lot of the things Google are doing wrong when it comes to the Cloud Platform. Today I came across an App Engine SDK bug that was reported almost 5 years ago, has a several years old patch in the comments, and still hasn't been remedied by Google. App Engine users who encounter it have to find the bug, apply the patch to the SDK, and repeat that upon each SDK update.
- dlor 11y ago
  Could you send me a link to the issue? I can hopefully get that fixed.

estefan 11y ago

Let's hope the oauth token you get back doesn't expire after you think you've finished! And then lets hope it doesn't also take you a day or so to have to learn oauth2client so you know which secret method you need to call request a refresh token. Such fun!

mataug 11y ago
It took me more than half a day to figure out that I was doing things wrong even though I've used oauth2client before on other projects.

samkone 11y ago

Or you can use the service account to avoid dancing around gcloud and others. But I agree Google has a problem building stuff for us common humans who do not work at the big G.

mataug 11y ago
I this stems from a general lack of empathy for customers in the organisation. Take a look at the issue list on appengine for example https://code.google.com/p/googleappengine/issues/detail?id=10259 https://code.google.com/p/googleappengine/issues/detail?id=1... It seems like they get a bunch of engineers to come and build stuff then once they move on, the project is abandoned or poorly maintained until another team of engineers come and try to rebuild it from scratch. Here's more proof, Google Wallet for example ... https://news.ycombinator.com/item?id=9498475 https://news.ycombinator.com/item?id=9498475

thrownaway2424 11y ago

Is there something in your comment regarding bigtable?

ryanseys 11y ago

Disclaimer: I work on gcloud-node [1]. :) You're describing the OAuth 2.0 client dance which has always been cumbersome but necessary for browser-side auth. With Google Cloud Platform related services, you should never have to do this if you're using these services from the context of your server. As someone previously mentioned, Google has Service Accounts which greatly simplifies the auth process. For example, to auth with gcloud-node it's as easy as: 1. Download the key.json associated with your service account. 2. var gcloud = require('gcloud')({ projectId: 'abc-123', keyFilename: './key.json' }); 3. There is no step 3. We've also made some pretty nice docs [2] with clear examples to help you every step of the way. We have yet to integrate Cloud Bigtable but it's on our radar. For a pythonista like yourself, you may like to consider trying gcloud-python [3], and it should go without saying please feel free to file issues or reach out directly with concerns if something just doesn't feel quite right! [1]. https://github.com/GoogleCloudPlatform/gcloud-node https://github.com/GoogleCloudPlatform/gcloud-node [2]. http://googlecloudplatform.github.io/gcloud-node/#/docs/ http://googlecloudplatform.github.io/gcloud-node/#/docs/ [3]. https://github.com/GoogleCloudPlatform/gcloud-python https://github.com/GoogleCloudPlatform/gcloud-python Edit: Formatting.

mataug 11y ago
I'll try this out. Thanks.
nulltype 11y ago
I like the gcloud-node docs! I remember the oauth stuff being a pain just because oauth is kind of complicated and the terms are weird and there was no guide that just says "go here and click this" you had to just figure that out.

anthmgoogle 11y ago

(I'm a Google developer on API Platform) A few of the follow ups mentioned that Service Accounts are more appropriate for this API and closer to the AWS experience. There have been some challenges using service accounts that have been addressed with a relatively new feature set called "Application Default Credentials": https://developers.google.com/identity/protocols/application-default-credentials https://developers.google.com/identity/protocols/application... For most cloud APIs, you simply run "gcloud auth login" to enable auth on the development machine, and when you deploy to GCE or GAE it will use the more secure built-in service accounts there. And there is no token expiration in either case.

TheIronYuppie 11y ago

Interesting point in that you used a third party in step 2 of your AWS workflow. If you're going to use a 3rd party, have you tried Terraform?