3 ms·

I can't contribute anything (luckily), GKE has been absolutely smooth sailing for us (small-ish data eng/data science team doing most of their own cloud ops) –

I can't contribute anything (luckily), GKE has been absolutely smooth sailing for us (small-ish data eng/data science team doing most of their own cloud ops) – even BigQuery has caused more trouble. Very curious how others have fared, though.

farisjarrah 6y ago
We went from building VM's with ansible on AWS to everything being on GKE. Previously if we wanted to build out a whole new stack it could be a multiple day endeavor, now on GKE I can build out a whole new GKE cluster in a matter of hours. Its also allowed our team to sleep at night as our Ansible driven AWS architecture was fickle and fragile. I'm sure that we could have engineered our AWS infra much better, but its a lot of work, and GKE allowed us to bypass that and focus more on the business's bottom line. There are 2 major issues that we tend to run into on GKE: DNS Problems: most likely self inflicted, but they are a pain and can take a while to troubleshoot. Google Cloud Platform Quota Increases take FOREVER: as far as I am aware Google refuses to allow customers to set organization wide quotas for new projects that get spun up, and there is absolutely no way to automate quota increases requests via an API. Its absolutely infuriating and makes making disposable projects all but impossible, which I was really hoping to set up for my company. If there is anyone at GCP reading this: Please, please, please allow customers to set organization wide quotas(even for new projects that get spun up) and please give customers an API to manage these things!!!! Overall though, GKE is pretty great for us.
Jugurtha 6y ago
Would you like to exchange a bit on your work? We're building an internal (for now) machine learning platform for our team (play version in bio). We've added features to suit our needs. Examples: near real time collaboration on notebooks with user cursors for when a data scientist colleague has a problem implementing something and we need to pair-program. Automatic logging for parameters, metrics, and model to take the cognitive load off. Multiple checkpoints for our notebooks. Ability to turn a notebook into an AppBook: we automatically generate a form for parameters without the author tagging cells so you can train a model with a form or an HTTP request, without changing the notebook. Also useful when we want to allow a domain expert to train models by tweaking parameters that matter to them (no need to have Jupyter). Then click to deploy a model, or click to build its Docker image if they have a developer team that wants to send data to it. Again, not all scenarios are handled but adding features that we needed in our projects.

farisjarrah 6y ago

We went from building VM's with ansible on AWS to everything being on GKE. Previously if we wanted to build out a whole new stack it could be a multiple day endeavor, now on GKE I can build out a whole new GKE cluster in a matter of hours. Its also allowed our team to sleep at night as our Ansible driven AWS architecture was fickle and fragile. I'm sure that we could have engineered our AWS infra much better, but its a lot of work, and GKE allowed us to bypass that and focus more on the business's bottom line. There are 2 major issues that we tend to run into on GKE: DNS Problems: most likely self inflicted, but they are a pain and can take a while to troubleshoot. Google Cloud Platform Quota Increases take FOREVER: as far as I am aware Google refuses to allow customers to set organization wide quotas for new projects that get spun up, and there is absolutely no way to automate quota increases requests via an API. Its absolutely infuriating and makes making disposable projects all but impossible, which I was really hoping to set up for my company. If there is anyone at GCP reading this: Please, please, please allow customers to set organization wide quotas(even for new projects that get spun up) and please give customers an API to manage these things!!!! Overall though, GKE is pretty great for us.

Jugurtha 6y ago

Would you like to exchange a bit on your work? We're building an internal (for now) machine learning platform for our team (play version in bio). We've added features to suit our needs. Examples: near real time collaboration on notebooks with user cursors for when a data scientist colleague has a problem implementing something and we need to pair-program. Automatic logging for parameters, metrics, and model to take the cognitive load off. Multiple checkpoints for our notebooks. Ability to turn a notebook into an AppBook: we automatically generate a form for parameters without the author tagging cells so you can train a model with a form or an HTTP request, without changing the notebook. Also useful when we want to allow a domain expert to train models by tweaking parameters that matter to them (no need to have Jupyter). Then click to deploy a model, or click to build its Docker image if they have a developer team that wants to send data to it. Again, not all scenarios are handled but adding features that we needed in our projects.