16 ms·

Thank you for the feedback. The management fee is per cluster. You are not billed for replicated control planes. You can use the pricing calculator at https://c

Thank you for the feedback. The management fee is per cluster. You are not billed for replicated control planes. You can use the pricing calculator at https://cloud.google.com/products/calculator#tab=container https://cloud.google.com/products/calculator#tab=container to model pricing, but it should work out to $73/mo regardless of nodes or cluster size (again, because it's charged per-cluster). There's also one completely free zonal cluster for hobbyist projects.

rolleiflex 7y ago
Seth — I appreciate you being here to take feedback, and for the clarification as well. The very surprising email I’ve received this morning is very hazy on the details, and the docs linked from the email are not updated yet. The main issue is that not charging for the control plane and charging for the control plane leads to two very different Kubernetes architectures, and as per your docs, those decisions made at the start are very much set in stone. You cannot change your cluster from a regional cluster to a single zone cluster for example. So you have customers who built their stacks taking into account your free control plane, and you’re turning the screws in by adding a cost for it — but they cannot change the type of their cluster to optimise their spend, since, per your docs, those decisions are set in stone. That’s entrapment. You should keep existing clusters in the pricing model they’ve been built in, and apply this change for clusters created after today. That said, many of us made a bet on GCP. For us in particular, we made a bet to the point that our SQL servers are on AWS, but we still switched to GCP for ‘better’ Kubernetes and for not nickel and diming, since AWS had a charge that looked like it was designed to convey that they’d much rather have you use their own stuff than Kubernetes. It is a relatively trivial amount, but it makes a world of difference in how it feels and you guys know more than anyone how much of these GCP vs AWS decisions are made based not on data sheets but for the ‘general feel’ for the lack of a better word. AWS’ message is that they’re the staid, sort of old fashioned, but reliable business partner. GCP’s message, as of this morning, is stop using GCP.
- andrewmutz 7y ago
  If this cost bothers you a great deal, why not just deploy a new cluster?
- dward 7y ago
  GKE can't offer financial backed SLOs without charging for the service. This is something that, I assume, significant customers want and that competitors already have: https://aws.amazon.com/eks/sla/ https://aws.amazon.com/eks/sla/
  - outworlder 7y ago
    Workers are not free and never were. So they were already charging.
    - sethvargo 7y ago
      Correct, but the control plane nodes _were_ free and had no SLA. This changes that. [edit: spelling]
      - gowld 7y ago
        _were_ free. (Emphasis yours.)
- lilbobbytables 7y ago
  > and the docs linked from the email are not updated yet. That about sums up most things Google does for developers.
  - alasdair_ 7y ago
    I thought the standard advice for Google stuff was "there are always two systems - the undocumented one, and the deprecated one"
    - andybak 7y ago
      That's a wonderful quote that applies to many companies. (I think that will resonate with the Unity developer community right now.)
    - nickbarnwell 7y ago
      What I most frequently heard was "There are two solutions: the deprecated one, and the one that doesn't work yet."
- cmhnn 7y ago
  Sorry for technical tangent but curious. Your decision making on GCP appears to appeal to best of breed + cost. But you put SQL Server on AWS? If you are saying SQL Server is better on AWS than on Azure it would be interesting to learn why.
  - rolleiflex 7y ago
    We need MySQL 8 because of window functions, which GCP does not offer. That is available on AWS.
    - cmhnn 7y ago
      My bad. A clever marketing decision made me see the capital SQL as SQL Server since I am used to people saying Postgres, MySQL or SQL.
    - carterehsmith 7y ago
      I see. Curious about the latency between your GCP apps and the database on AWS - is it like 1 ms or 100ms? Does it affect the product?
      - rolleiflex 7y ago
        About 4ms for us. However, we chose our data centres on both ends very carefully. There are tables online you can find that for those pings, one such is here: https://medium.com/@sachinkagarwal/public-cloud-inter-region-network-latency-as-heat-maps-134e22a5ff19 https://medium.com/@sachinkagarwal/public-cloud-inter-region... However this means we are paying for egress on both sides. This was something we chose to eat due to GCP Kubernetes, but considering today’s changes, it probably no longer makes sense.
        vasco 7y ago
        So you decide to eat egress costs in perpetuity, which will scale as you go, but a one time increase of $70 per month is enough to make you go back? What are you even trying to optimize for?
- sethvargo 7y ago
  Thank you <3. I apologize the email was hazy on details. I can't un-send it, but I'll work with the product teams to make sure they are crystal clear in the future. I'm interested to learn more about what you mean about outdated docs? The documentation I'm seeing appears to have been updated. Can you drop me a screenshot, maybe on Twitter (same username, DMs are open). These changes won't take effect until June - customers won't start getting billed immediately. I'm sorry that you feel trapped, that's not our intention. > You should keep existing clusters in the pricing model they’ve been built in, and apply this change for clusters created after today. This is great feedback, but clusters should be treated like cattle, not pets. I'd love to learn more about why your clusters must be so static.
  - patrec 7y ago
    > clusters should be treated like cattle, not pets So my understanding that the official k8s way to upgrade your cluster is also to throw it away and start a new one (with some cloud provider proprietary alternatives). Let's say there is something actually important, stateful, single-source-of-truth in my k8s cluster, like a relational DB that must not lose data. I don't want downtime for readers or writers, and I want at least one synchronous slave at all times (since the data is important). I also don't want to eat non-trivial latency overheads from setting up layers of indirection. What's the recommended way of doing this?
    - patrec 7y ago
      It's kind of revealing that there are zero replies to this, 10 hours later.
    - zekrioca 7y ago
      In this case, one needs fault-tolerance. One way to achieve it is through replication, where an extra copy (or perhaps a reconstruction receipt) of your DB instance runs somewhere else. Usually DBs achieves this through transactions. Additionally, you can have distributed DBs, which then use distributed transactions for achieving so. I am not expert, but K8s handles task replication, and either spawn or route a request to another task instance somewhere else. However, the application logic itself must handle the fault-tolerance (by handling its states through transactions or something else) should an instance fail. K8s doesn't do that for you.
- ones_and_zeros 7y ago
  I agree the rollout is a little bumpy but I'm curious what workloads you are using k8s for where a $74/mo (or $300/mo) bill isn't a rounding error in your capex?
  - olafure 7y ago
    $3600/year is significant for a startup on a shoestring budget.
    - nimish 7y ago
      So ask for Google cloud for startups? One free cluster is enough to get started.
      - jorams 7y ago
        > Google Cloud for Startups is designed to help companies that are backed by VCs, incubators, or accelerators, so it's less applicable for small businesses, services, consultancies, and dev shops.[1] This makes it seem like Google Cloud for Startups is aimed at startups that aren't really on a shoestring budget. [1]: https://cloud.google.com/developers/startups/ https://cloud.google.com/developers/startups/
        p_l 7y ago
        Like every "special offer for startups", it's a vulture waiting for funding round to close.
    - dahfizz 7y ago
      Then manage k8s yourself. Or, better yet, don't use k8s. You don't need it, especially as a startup on a shoestring budget. You can migrate later if you decide you really need to, but just a plain LAMP gets you 99% of the way.
      - sheeshkebab 7y ago
        But then you can’t put k8s on you resume for when said startup implodes.
xur17 7y ago
We currently spin up dev clusters with a single node. $73/mo is going to basically double the cost of all of these..
- lvh 7y ago
  This highlights a sorta-weird consequence of this pricing change: suddenly pricing incentivizes you to use namespacing instead of clusters for separating environments. (As a security person: ugh.)
  - rolleiflex 7y ago
    That’s interesting - I think you’re right. We might move our staging cluster into our main production deployment. More likely though, AWS or OpenShift running on bare metal on a beefy ATX tower in the office. We want to have production and staging as close to each other as possible, so this is an additional reason and a p0 flag on reducing the dependency on Google-specific bits of Kubernetes as much as possible, hopefully also useful for our exit strategy as well.
    - lazyier 7y ago
      Kubespray works well for me for setting up a bare bones kubernetes cluster for the lab. I'll use helm to install metallb for the load balancer, which you can then tie into whatever egress controller you like to use. For persistent storage a simple NFS server is the bees knees. Works very well and a NFS provisioned is a helm install. Very nice, especially, over 10GbE. Do NOT dismiss NFSv4. It's actually very nice for this sort of thing. I just use a small separate Linux box with software raid on it for that. If you want to have the cluster self-host storage or need high availability then GlusterFS works great, but it's more overhead to manage. Then you just use normal helm install routines to install and setup logging, dashboards, and all that. Openshift is going to be a lot better for people who want to do multi-tenant stuff in a corporate enterprise environment. Like you have different teams of people, each with their own realm of responsibility. Openshift's UI and general approach is pretty good about allowing groups to self-manage without impacting one another. The additional security is a double edged. Fantastic if you need it, but annoying barrier to entry for users if you don't. As far as AWS goes... EKS recently lowered their cost from 20 cents per hour to 10 cents. So costs for the cluster is on par with what Google is charging. Azure doesn't charge for cluster management (yet), IIRC.
halbritt 7y ago
> There's also one completely free zonal cluster for hobbyist projects. Nice.
dcolkitt 7y ago
Hi Seth, What about clusters that are used for lumpy work loads? Like data science pipelines? For example, our org has a few dozen clusters being used like that. Each pipeline gets its own cluster instance as a way to enforce rough and ready isolation. Most of the times the clusters sit unused. To keep them alive we keep a small, cheap, preemptive node alive on the idle cluster. When a new batch of data comes in, we fire up kube jobs which then triggers GKE autoscaling that processes the workload. This pricing change means we're looking at thousands of dollar more in billing per month. Without any tangible improvement in service. (The keepalive node hack only costs $5 a month per cluster.) We could consolidate the segmented cluster instances into a single cluster with separate namespaces, but that would also cost thousands in valuable developer time. I don't know how common our use pattern is, but I think we would be a lot better served by a discounted management fee when the cluster is just being kept alive and not actually using any resources. At $0.01, maybe even $0.02, per hour we could justify it. But paying $0.10 to keep empty clusters alive is just egregious.
- rcarmo 7y ago
  Here you go: https://github.com/rcarmo/azure-k3s-cluster https://github.com/rcarmo/azure-k3s-cluster (this is an Azure template that I use precisely for testing that kind of workloads - spinning up one of these, master included, takes a couple of minutes at most). (full disclosure: I work at Microsoft - Azure Kubernetes Service works fine, but I built the above because I wanted full control over scaling and a very very simple shared filesystem)
- thockingoog 7y ago
  Those empty clusters that you get for free cost Google money. Perhaps it never should have been free, because that skewed incentives towards models like this.
  - twistedpair 7y ago
    Keep in mind that GKE cluster management was paid in the original GKE. GCP only stopped billing for cluster management when EKS released free cluster management.

rolleiflex 7y ago

Seth — I appreciate you being here to take feedback, and for the clarification as well. The very surprising email I’ve received this morning is very hazy on the details, and the docs linked from the email are not updated yet. The main issue is that not charging for the control plane and charging for the control plane leads to two very different Kubernetes architectures, and as per your docs, those decisions made at the start are very much set in stone. You cannot change your cluster from a regional cluster to a single zone cluster for example. So you have customers who built their stacks taking into account your free control plane, and you’re turning the screws in by adding a cost for it — but they cannot change the type of their cluster to optimise their spend, since, per your docs, those decisions are set in stone. That’s entrapment. You should keep existing clusters in the pricing model they’ve been built in, and apply this change for clusters created after today. That said, many of us made a bet on GCP. For us in particular, we made a bet to the point that our SQL servers are on AWS, but we still switched to GCP for ‘better’ Kubernetes and for not nickel and diming, since AWS had a charge that looked like it was designed to convey that they’d much rather have you use their own stuff than Kubernetes. It is a relatively trivial amount, but it makes a world of difference in how it feels and you guys know more than anyone how much of these GCP vs AWS decisions are made based not on data sheets but for the ‘general feel’ for the lack of a better word. AWS’ message is that they’re the staid, sort of old fashioned, but reliable business partner. GCP’s message, as of this morning, is stop using GCP.

andrewmutz 7y ago
If this cost bothers you a great deal, why not just deploy a new cluster?
dward 7y ago
GKE can't offer financial backed SLOs without charging for the service. This is something that, I assume, significant customers want and that competitors already have: https://aws.amazon.com/eks/sla/ https://aws.amazon.com/eks/sla/
- outworlder 7y ago
  Workers are not free and never were. So they were already charging.
  - sethvargo 7y ago
    Correct, but the control plane nodes _were_ free and had no SLA. This changes that. [edit: spelling]
    - gowld 7y ago
      _were_ free. (Emphasis yours.)
lilbobbytables 7y ago
> and the docs linked from the email are not updated yet. That about sums up most things Google does for developers.
- alasdair_ 7y ago
  I thought the standard advice for Google stuff was "there are always two systems - the undocumented one, and the deprecated one"
  - andybak 7y ago
    That's a wonderful quote that applies to many companies. (I think that will resonate with the Unity developer community right now.)
  - nickbarnwell 7y ago
    What I most frequently heard was "There are two solutions: the deprecated one, and the one that doesn't work yet."
cmhnn 7y ago
Sorry for technical tangent but curious. Your decision making on GCP appears to appeal to best of breed + cost. But you put SQL Server on AWS? If you are saying SQL Server is better on AWS than on Azure it would be interesting to learn why.
- rolleiflex 7y ago
  We need MySQL 8 because of window functions, which GCP does not offer. That is available on AWS.
  - cmhnn 7y ago
    My bad. A clever marketing decision made me see the capital SQL as SQL Server since I am used to people saying Postgres, MySQL or SQL.
  - carterehsmith 7y ago
    I see. Curious about the latency between your GCP apps and the database on AWS - is it like 1 ms or 100ms? Does it affect the product?
    - rolleiflex 7y ago
      About 4ms for us. However, we chose our data centres on both ends very carefully. There are tables online you can find that for those pings, one such is here: https://medium.com/@sachinkagarwal/public-cloud-inter-region-network-latency-as-heat-maps-134e22a5ff19 https://medium.com/@sachinkagarwal/public-cloud-inter-region... However this means we are paying for egress on both sides. This was something we chose to eat due to GCP Kubernetes, but considering today’s changes, it probably no longer makes sense.
      - vasco 7y ago
        So you decide to eat egress costs in perpetuity, which will scale as you go, but a one time increase of $70 per month is enough to make you go back? What are you even trying to optimize for?
sethvargo 7y ago
Thank you <3. I apologize the email was hazy on details. I can't un-send it, but I'll work with the product teams to make sure they are crystal clear in the future. I'm interested to learn more about what you mean about outdated docs? The documentation I'm seeing appears to have been updated. Can you drop me a screenshot, maybe on Twitter (same username, DMs are open). These changes won't take effect until June - customers won't start getting billed immediately. I'm sorry that you feel trapped, that's not our intention. > You should keep existing clusters in the pricing model they’ve been built in, and apply this change for clusters created after today. This is great feedback, but clusters should be treated like cattle, not pets. I'd love to learn more about why your clusters must be so static.
- patrec 7y ago
  > clusters should be treated like cattle, not pets So my understanding that the official k8s way to upgrade your cluster is also to throw it away and start a new one (with some cloud provider proprietary alternatives). Let's say there is something actually important, stateful, single-source-of-truth in my k8s cluster, like a relational DB that must not lose data. I don't want downtime for readers or writers, and I want at least one synchronous slave at all times (since the data is important). I also don't want to eat non-trivial latency overheads from setting up layers of indirection. What's the recommended way of doing this?
  - patrec 7y ago
    It's kind of revealing that there are zero replies to this, 10 hours later.
  - zekrioca 7y ago
    In this case, one needs fault-tolerance. One way to achieve it is through replication, where an extra copy (or perhaps a reconstruction receipt) of your DB instance runs somewhere else. Usually DBs achieves this through transactions. Additionally, you can have distributed DBs, which then use distributed transactions for achieving so. I am not expert, but K8s handles task replication, and either spawn or route a request to another task instance somewhere else. However, the application logic itself must handle the fault-tolerance (by handling its states through transactions or something else) should an instance fail. K8s doesn't do that for you.
ones_and_zeros 7y ago
I agree the rollout is a little bumpy but I'm curious what workloads you are using k8s for where a $74/mo (or $300/mo) bill isn't a rounding error in your capex?
- olafure 7y ago
  $3600/year is significant for a startup on a shoestring budget.
  - nimish 7y ago
    So ask for Google cloud for startups? One free cluster is enough to get started.
    - jorams 7y ago
      > Google Cloud for Startups is designed to help companies that are backed by VCs, incubators, or accelerators, so it's less applicable for small businesses, services, consultancies, and dev shops.[1] This makes it seem like Google Cloud for Startups is aimed at startups that aren't really on a shoestring budget. [1]: https://cloud.google.com/developers/startups/ https://cloud.google.com/developers/startups/
      - p_l 7y ago
        Like every "special offer for startups", it's a vulture waiting for funding round to close.
  - dahfizz 7y ago
    Then manage k8s yourself. Or, better yet, don't use k8s. You don't need it, especially as a startup on a shoestring budget. You can migrate later if you decide you really need to, but just a plain LAMP gets you 99% of the way.
    - sheeshkebab 7y ago
      But then you can’t put k8s on you resume for when said startup implodes.

xur17 7y ago

We currently spin up dev clusters with a single node. $73/mo is going to basically double the cost of all of these..

lvh 7y ago
This highlights a sorta-weird consequence of this pricing change: suddenly pricing incentivizes you to use namespacing instead of clusters for separating environments. (As a security person: ugh.)
- rolleiflex 7y ago
  That’s interesting - I think you’re right. We might move our staging cluster into our main production deployment. More likely though, AWS or OpenShift running on bare metal on a beefy ATX tower in the office. We want to have production and staging as close to each other as possible, so this is an additional reason and a p0 flag on reducing the dependency on Google-specific bits of Kubernetes as much as possible, hopefully also useful for our exit strategy as well.
  - lazyier 7y ago
    Kubespray works well for me for setting up a bare bones kubernetes cluster for the lab. I'll use helm to install metallb for the load balancer, which you can then tie into whatever egress controller you like to use. For persistent storage a simple NFS server is the bees knees. Works very well and a NFS provisioned is a helm install. Very nice, especially, over 10GbE. Do NOT dismiss NFSv4. It's actually very nice for this sort of thing. I just use a small separate Linux box with software raid on it for that. If you want to have the cluster self-host storage or need high availability then GlusterFS works great, but it's more overhead to manage. Then you just use normal helm install routines to install and setup logging, dashboards, and all that. Openshift is going to be a lot better for people who want to do multi-tenant stuff in a corporate enterprise environment. Like you have different teams of people, each with their own realm of responsibility. Openshift's UI and general approach is pretty good about allowing groups to self-manage without impacting one another. The additional security is a double edged. Fantastic if you need it, but annoying barrier to entry for users if you don't. As far as AWS goes... EKS recently lowered their cost from 20 cents per hour to 10 cents. So costs for the cluster is on par with what Google is charging. Azure doesn't charge for cluster management (yet), IIRC.

halbritt 7y ago

> There's also one completely free zonal cluster for hobbyist projects. Nice.

dcolkitt 7y ago

Hi Seth, What about clusters that are used for lumpy work loads? Like data science pipelines? For example, our org has a few dozen clusters being used like that. Each pipeline gets its own cluster instance as a way to enforce rough and ready isolation. Most of the times the clusters sit unused. To keep them alive we keep a small, cheap, preemptive node alive on the idle cluster. When a new batch of data comes in, we fire up kube jobs which then triggers GKE autoscaling that processes the workload. This pricing change means we're looking at thousands of dollar more in billing per month. Without any tangible improvement in service. (The keepalive node hack only costs $5 a month per cluster.) We could consolidate the segmented cluster instances into a single cluster with separate namespaces, but that would also cost thousands in valuable developer time. I don't know how common our use pattern is, but I think we would be a lot better served by a discounted management fee when the cluster is just being kept alive and not actually using any resources. At $0.01, maybe even $0.02, per hour we could justify it. But paying $0.10 to keep empty clusters alive is just egregious.

rcarmo 7y ago
Here you go: https://github.com/rcarmo/azure-k3s-cluster https://github.com/rcarmo/azure-k3s-cluster (this is an Azure template that I use precisely for testing that kind of workloads - spinning up one of these, master included, takes a couple of minutes at most). (full disclosure: I work at Microsoft - Azure Kubernetes Service works fine, but I built the above because I wanted full control over scaling and a very very simple shared filesystem)
thockingoog 7y ago
Those empty clusters that you get for free cost Google money. Perhaps it never should have been free, because that skewed incentives towards models like this.
- twistedpair 7y ago
  Keep in mind that GKE cluster management was paid in the original GKE. GCP only stopped billing for cluster management when EKS released free cluster management.