5 ms·

Just a data point: I'm the CTO at a very small company. All our stuff is running on GKE. Our monthly bill tends is a lot less than $10,000/mo. We're currently

Just a data point: I'm the CTO at a very small company. All our stuff is running on GKE. Our monthly bill tends is a lot less than $10,000/mo. We're currently in the process of splitting our stack into separate projects and clusters, because co-locating projects in a single cluster has gotten messy. We'll probably end up with 4-5. That will increase our bill by $292/mo, worst case, assuming the first cluster is free. For a company our size, it's not a huge expense. But these things add up. Since moving from DigitalOcean, our Google Cloud setup has more than doubled our monthly bill. We're paying for more compute, but certainly not twice the amount, as we've only gone from 14-15 nodes to around 20; it's just more expensive across the board, both node cost and ingress/egress. We're even cost-cutting by using self-hosted services instead of Google's; for example, we use Postgres instead of Cloud SQL. I ran the numbers earlier today; the equivalent on Cloud SQL would be 3.4 times more expensive. In short, Google Cloud is expensive, and it's not like the bill is getting smaller over time. Developments like these factor in my choice of cloud provider for future projects.

TheFiend7 7y ago
For sure, though I will say if you're trying to cut costs, it's my understanding serverless is quite cheap. So if you can turn some of your services into serverless containers/functions. I'd highly recommend it.
hn_throwaway_99 7y ago
Not sure what size a "very small company" size is, but I'm just curious as to why you chose GKE. I make tech decisions for a (probably) much smaller company, and I found things like App Engine Flexible Environment, Cloud Run and Cloud Functions let me do much of the stuff I can do with k8s but with much, much less complexity (at least on my side of things). The main factor is that I don't have a full-time infrastructure expert, and my experience in the past is that k8s essentially requires that.
- Glyptodon 7y ago
  So let's say you have a small pile of apps that are all <app_server>, <redis>, <jobqueue/workers>, <db>, <frontend>, but not all the same DB, language, etc. They're all low traffic and you want to automate/simplify and containerize them. On developer machines docker compose works great, but you need to deploy in a cloud provider. What are choice do you have other than K8s?
  - nostrebored 7y ago
    AWS Lambda with SAM local?
- atombender 7y ago
  Less than 15 employees. Several products, two teams, <= 20 nodes. We migrated our stuff from DigitalOcean around 2018. At the time, we briefly toyed with the notion of self-hosting Kubernetes on DO, but it's complex to manage, and we don't have any dedicated ops staff. GKE is significantly easier to manage. At that time we migrated, the things you mentioned weren't available/mature, I think. Even today, I'd choose Kubernetes over a complex mishmash of different systems. I like the unified, extensible ops model. In fact, I'd go so far as to say that I wish all of GCP could be managed as Kubernetes objects.
  - praveenperera 7y ago
    But DigitalOcean has managed K8s now: https://www.digitalocean.com/products/kubernetes/ https://www.digitalocean.com/products/kubernetes/
    - atombender 7y ago
      DigitalOcean did not have Kubernetes then. Are you suggesting we should spend 6-12 man months migrating back?
      - praveenperera 7y ago
        No my mistake DOKS came out late 2018. I had been using it since May 2018 but it didn't come out of early access till December.
      - tie_ 7y ago
        How about contracting an ops-oriented person for a month that would do the migration for you? Where do those cost functions intersect?
        atombender 7y ago
        Would never happen. Just the amount of time needed to dedicate to onboarding a temporary contractor would be really disruptive to the developers, not to mention the disruptive effect of the technical migration — databases to move over, persistent volumes to copy, DNS to repoint, lots of downtime, etc. There's a good reason companies don't switch clouds often.
- atombender 7y ago
  We have about 200 long-running pods right now. On Cloud Run, that would cost us more than $12,000 in CPU alone, and that's excluding memory and request costs. That also excludes stateful apps like Elasticsearch that would not be able to run on Cloud Run. Not sure what Google product is appropriate there.
- giovannibonetti 7y ago
  Indeed, for small apps that average 2 instances or less during the month it's now cheaper to run App Engine Flex than GKE. Each instance costs about US$ 50/month in US central. For those that are not aware, App Engine Flex runs services based on Docker containers with autoscaling similarly to Kubernetes. It has way less features than GKE, but if you are just running a standard app that only needs to connect to a database that is more than enough. Bonus points: you can have multiple services, like back-end and front-end and make them available in the same subdomain - to avoid CORS problems. You can also host your front-end for almost nothing with App Engine Standard. It has an awesome CDN built-in if you know how to use it.
echelon 7y ago
How much would it cost for you to provision and colocate your own hardware, run a k8s cluster, and manage upgrades? You might not be at the scale where this is feasible yet since that's probably multiple full-time engineers, but eventually the cost functions intersect.
- atombender 7y ago
  Well, we don't even have a dedicated ops person.

TheFiend7 7y ago

For sure, though I will say if you're trying to cut costs, it's my understanding serverless is quite cheap. So if you can turn some of your services into serverless containers/functions. I'd highly recommend it.

hn_throwaway_99 7y ago

Not sure what size a "very small company" size is, but I'm just curious as to why you chose GKE. I make tech decisions for a (probably) much smaller company, and I found things like App Engine Flexible Environment, Cloud Run and Cloud Functions let me do much of the stuff I can do with k8s but with much, much less complexity (at least on my side of things). The main factor is that I don't have a full-time infrastructure expert, and my experience in the past is that k8s essentially requires that.

Glyptodon 7y ago
So let's say you have a small pile of apps that are all <app_server>, <redis>, <jobqueue/workers>, <db>, <frontend>, but not all the same DB, language, etc. They're all low traffic and you want to automate/simplify and containerize them. On developer machines docker compose works great, but you need to deploy in a cloud provider. What are choice do you have other than K8s?
- nostrebored 7y ago
  AWS Lambda with SAM local?
atombender 7y ago
Less than 15 employees. Several products, two teams, <= 20 nodes. We migrated our stuff from DigitalOcean around 2018. At the time, we briefly toyed with the notion of self-hosting Kubernetes on DO, but it's complex to manage, and we don't have any dedicated ops staff. GKE is significantly easier to manage. At that time we migrated, the things you mentioned weren't available/mature, I think. Even today, I'd choose Kubernetes over a complex mishmash of different systems. I like the unified, extensible ops model. In fact, I'd go so far as to say that I wish all of GCP could be managed as Kubernetes objects.
- praveenperera 7y ago
  But DigitalOcean has managed K8s now: https://www.digitalocean.com/products/kubernetes/ https://www.digitalocean.com/products/kubernetes/
  - atombender 7y ago
    DigitalOcean did not have Kubernetes then. Are you suggesting we should spend 6-12 man months migrating back?
    - praveenperera 7y ago
      No my mistake DOKS came out late 2018. I had been using it since May 2018 but it didn't come out of early access till December.
    - tie_ 7y ago
      How about contracting an ops-oriented person for a month that would do the migration for you? Where do those cost functions intersect?
      - atombender 7y ago
        Would never happen. Just the amount of time needed to dedicate to onboarding a temporary contractor would be really disruptive to the developers, not to mention the disruptive effect of the technical migration — databases to move over, persistent volumes to copy, DNS to repoint, lots of downtime, etc. There's a good reason companies don't switch clouds often.
atombender 7y ago
We have about 200 long-running pods right now. On Cloud Run, that would cost us more than $12,000 in CPU alone, and that's excluding memory and request costs. That also excludes stateful apps like Elasticsearch that would not be able to run on Cloud Run. Not sure what Google product is appropriate there.
giovannibonetti 7y ago
Indeed, for small apps that average 2 instances or less during the month it's now cheaper to run App Engine Flex than GKE. Each instance costs about US$ 50/month in US central. For those that are not aware, App Engine Flex runs services based on Docker containers with autoscaling similarly to Kubernetes. It has way less features than GKE, but if you are just running a standard app that only needs to connect to a database that is more than enough. Bonus points: you can have multiple services, like back-end and front-end and make them available in the same subdomain - to avoid CORS problems. You can also host your front-end for almost nothing with App Engine Standard. It has an awesome CDN built-in if you know how to use it.

echelon 7y ago

How much would it cost for you to provision and colocate your own hardware, run a k8s cluster, and manage upgrades? You might not be at the scale where this is feasible yet since that's probably multiple full-time engineers, but eventually the cost functions intersect.

atombender 7y ago
Well, we don't even have a dedicated ops person.