5 ms·

Fun to read this comment, as I'm part of the team that deployed the exact same measure as for your wife: we now shutdown dev environments each day at 8PM. We de

by hacb 4y ago

Fun to read this comment, as I'm part of the team that deployed the exact same measure as for your wife: we now shutdown dev environments each day at 8PM. We decided to do so because over the weeks, we saw that a lot of devs "forgot" about their dev envs and they were up & running for 6+ months for absolutely nothing. On the other side, we are constantly requested to reduce infra costs, and dev environments (even unused) cost a lot. In the end, I'm glad we deployed this.

orev 4y ago
Not only would they be forgotten about, but without forced downtime/reset, they start slipping into production use. The dev asks someone to “try it on my server”, then the word gets out about this other server that works better and has more features. Before you know it 2 years have gone by and this “dev” server is a critical system the business needs to function.
- hacb 4y ago
  In our case, those dev envs are actually feature envs: they are a copy of staging with only the microservice they are working on that differs. So it is impossible for those env to end up as prod stuff for us. However, each FE is an almost exact copy of staging, and each dev has 1-2 running FE, so it requires a lot of compute and stopping them has a real impact
ericd 4y ago
Couldn't this be done by giving the devs some transparency into how it's killing their team's budget and reminders, instead, so that they'd manage this themselves? Because this sounds super frustrating to deal with.
- hacb 4y ago
  We kind of already have. We built a CLI that allows them to create/update/delete/list all their dev envs, so they have all the tooling needed. Also, we are still a small/medium sized company, so we often had the chance to talk with them to expose this issue, but in the end it never really worked.
  - wizhi 4y ago
    > we are still a small/medium sized company Sorry this is somewhat off-topic, but _why_ do you have virtual development machines?
    - hacb 4y ago
      We currently have ~40 microservices running (for the same number of devs), and having the full stack on a laptop is not possible. Especially with 80% of the devs having MacBook and Docker in a VM. Also, this allow us (the infra folks) to chose how devs will work (partially), so it's easier for us to debug when they have issues because we do not have to deal with custom Makefiles etc.
      - piva00 4y ago
        Is it really needed to have 40 microservices for 40 devs? What I mean is that you generally scale out to so many services when you need each to have variable scalability, so each part of the system can scale up/down as necessary. If you don't need this, all these services could be written as larger modules, with proper separation of concerns but easy to be deployed as a single piece. Collect some stuff that are similar in domain and create a well designed monolith. Easier for operations, easier for devs to maintain. Dependencies upgrades aren't a nightmare, or you need very good tooling to not create toil (which generally a small operation won't have). Just throwing this in the wind because it sounds pretty heavy for a 40-body shop to be carrying as many services, at least I don't see the benefits unless for the aforementioned scalability advantage.
    - aprdm 4y ago
      One example can be to target many different OSes/CPU architecture / GPUs architecture.
  - trumbitta2 4y ago
    A weekly digest over email, or a dashboard on a real screen if you all work out or the same office, indicating costs and what's contributing to them. That will do wonders.
ghostbrainalpha 4y ago
How much does it cost to have a single environment running overnight? Not shutting down my machine saves me 2-5 minutes probably. Which seems like it might be worth a few pennies, but I hate the waste and environmental costs of wasted energy. What we really need is a program that shuts off a devs environment, but then can boot it back up 5 minutes before they start working and somehow open all the applications that were running back to the exact state they were in before shutting down..
- hacb 4y ago
  Initially, we were using kube-downscaler to do this. It stops a namespace at a given time and restart it in the morning. However, a lot of namespaces (= dev env in out case) were restarted for nothing, so we created our own tool that stops them at 8PM, and restart them when the dev runs a command with our internal CLI.

by hacb 4y ago

orev 4y ago

Not only would they be forgotten about, but without forced downtime/reset, they start slipping into production use. The dev asks someone to “try it on my server”, then the word gets out about this other server that works better and has more features. Before you know it 2 years have gone by and this “dev” server is a critical system the business needs to function.

hacb 4y ago
In our case, those dev envs are actually feature envs: they are a copy of staging with only the microservice they are working on that differs. So it is impossible for those env to end up as prod stuff for us. However, each FE is an almost exact copy of staging, and each dev has 1-2 running FE, so it requires a lot of compute and stopping them has a real impact

ericd 4y ago

Couldn't this be done by giving the devs some transparency into how it's killing their team's budget and reminders, instead, so that they'd manage this themselves? Because this sounds super frustrating to deal with.

hacb 4y ago
We kind of already have. We built a CLI that allows them to create/update/delete/list all their dev envs, so they have all the tooling needed. Also, we are still a small/medium sized company, so we often had the chance to talk with them to expose this issue, but in the end it never really worked.
- wizhi 4y ago
  > we are still a small/medium sized company Sorry this is somewhat off-topic, but _why_ do you have virtual development machines?
  - hacb 4y ago
    We currently have ~40 microservices running (for the same number of devs), and having the full stack on a laptop is not possible. Especially with 80% of the devs having MacBook and Docker in a VM. Also, this allow us (the infra folks) to chose how devs will work (partially), so it's easier for us to debug when they have issues because we do not have to deal with custom Makefiles etc.
    - piva00 4y ago
      Is it really needed to have 40 microservices for 40 devs? What I mean is that you generally scale out to so many services when you need each to have variable scalability, so each part of the system can scale up/down as necessary. If you don't need this, all these services could be written as larger modules, with proper separation of concerns but easy to be deployed as a single piece. Collect some stuff that are similar in domain and create a well designed monolith. Easier for operations, easier for devs to maintain. Dependencies upgrades aren't a nightmare, or you need very good tooling to not create toil (which generally a small operation won't have). Just throwing this in the wind because it sounds pretty heavy for a 40-body shop to be carrying as many services, at least I don't see the benefits unless for the aforementioned scalability advantage.
  - aprdm 4y ago
    One example can be to target many different OSes/CPU architecture / GPUs architecture.
- trumbitta2 4y ago
  A weekly digest over email, or a dashboard on a real screen if you all work out or the same office, indicating costs and what's contributing to them. That will do wonders.

ghostbrainalpha 4y ago

How much does it cost to have a single environment running overnight? Not shutting down my machine saves me 2-5 minutes probably. Which seems like it might be worth a few pennies, but I hate the waste and environmental costs of wasted energy. What we really need is a program that shuts off a devs environment, but then can boot it back up 5 minutes before they start working and somehow open all the applications that were running back to the exact state they were in before shutting down..

hacb 4y ago
Initially, we were using kube-downscaler to do this. It stops a namespace at a given time and restart it in the morning. However, a lot of namespaces (= dev env in out case) were restarted for nothing, so we created our own tool that stops them at 8PM, and restart them when the dev runs a command with our internal CLI.