3 ms·

We have a possible counter-example: our service is computing AB-test results (essentially reading from a database, processing totals, writing significance back)

by bertil 7y ago

We have a possible counter-example: our service is computing AB-test results (essentially reading from a database, processing totals, writing significance back). It have no non-system users, no dependencies, etc. All the test and strangeness is handled internally. We use k8s which is indeed over-engineered — it ran fine as a reminder and a local script for years. But the rest of the company has a release process that they like. We just integrate to it. Our service has a name in that space, ressources, a schedule that other engineers can read. Our description looks a little… film-school-credit-rolly because my name appears as the lead, architect, project, emergency contact, etc. I think the main oversight of those “you don’t need k8s” is that most projects are part of a system and fitting in that system gives you legibility to your peers that a nginx might not.

p_l 7y ago
I have ~62 apps (that's applications, not instances) deployed on kubernetes right now, for single client. It started out on 2 VMs with I think 2-4 cpus, already running kubernetes. The actual containers inside ran lighttpd and served static files while we fixed up the sites that we have mirrored as static to run in containers. If we had to run one, or maybe few of those sites, it would have been easy to run a single Apache + mod_php + vhost. We would have some annoying work to do on monitoring and logging side. But we have 62. Some of them are mutually incompatible to run on "standard" distro, as they have mutually exclusive dependencies (for example, PHP versions). This meant we ended up with containers to manage this in somewhat doable way (we are two people). We can't expend manpower to do a complete redo of the apps, though we have ideas on that (to go to single common CMS system for all of them). K8s saved our sanity. Because those "simple apps" altogether made for hard to manage setup, and the client doesn't like when they are not available, so we brought up HA as well. Doing this as separate VMs would be hard and expensive. Doing it on Heroku is expensive - I made a calculation, and our original setup would result in somewhere around ~1800 USD a month. Our 2016-2017 spend on GCP (GKE, Cloud SQL, a VM to host Gitlab + network traffic and DNS) was around 1000/month.
- earthboundkid 7y ago
  Do they need to be 62 though? Without having looked, my ignorant guess is that it could be reduced by an order of magnitude, it's just that doing so would take time that no one has, so the simpler choice is just to shovel the complexity into K8 instead of dealing with the complexity of reducing the number of apps.
  - p_l 7y ago
    The guess is a big miss, yes. Assuming certain pruning is done that I can see, we would reduce it to maybe 40-50. They are all separate concerns, independent from each other, the pruning would merge the most mergable elements back (those are, honestly, a tech debt and I'd welcome replacing them with one common app). BTW, those 62 apps? They map to ~260 domain names. Those domain names and what shows when you go there are what the client is paying us for.

by bertil 7y ago

p_l 7y ago

I have ~62 apps (that's applications, not instances) deployed on kubernetes right now, for single client. It started out on 2 VMs with I think 2-4 cpus, already running kubernetes. The actual containers inside ran lighttpd and served static files while we fixed up the sites that we have mirrored as static to run in containers. If we had to run one, or maybe few of those sites, it would have been easy to run a single Apache + mod_php + vhost. We would have some annoying work to do on monitoring and logging side. But we have 62. Some of them are mutually incompatible to run on "standard" distro, as they have mutually exclusive dependencies (for example, PHP versions). This meant we ended up with containers to manage this in somewhat doable way (we are two people). We can't expend manpower to do a complete redo of the apps, though we have ideas on that (to go to single common CMS system for all of them). K8s saved our sanity. Because those "simple apps" altogether made for hard to manage setup, and the client doesn't like when they are not available, so we brought up HA as well. Doing this as separate VMs would be hard and expensive. Doing it on Heroku is expensive - I made a calculation, and our original setup would result in somewhere around ~1800 USD a month. Our 2016-2017 spend on GCP (GKE, Cloud SQL, a VM to host Gitlab + network traffic and DNS) was around 1000/month.

earthboundkid 7y ago
Do they need to be 62 though? Without having looked, my ignorant guess is that it could be reduced by an order of magnitude, it's just that doing so would take time that no one has, so the simpler choice is just to shovel the complexity into K8 instead of dealing with the complexity of reducing the number of apps.
- p_l 7y ago
  The guess is a big miss, yes. Assuming certain pruning is done that I can see, we would reduce it to maybe 40-50. They are all separate concerns, independent from each other, the pruning would merge the most mergable elements back (those are, honestly, a tech debt and I'd welcome replacing them with one common app). BTW, those 62 apps? They map to ~260 domain names. Those domain names and what shows when you go there are what the client is paying us for.