3 ms·

> You can have thundering herds, that you may not have levers to control To me, this seems almost unthinkable. You mean to tell me that you can't just coordina

> You can have thundering herds, that you may not have levers to control To me, this seems almost unthinkable. You mean to tell me that you can't just coordinate setting "replicas: 0" for all of your services, or the equivalent and to restart them one at a time? If that's really the case, then perhaps it's time to re-evaluate how that situation ever came to be - where's the technical leadership who kept systems scalable, yet manageable? Was that even a concern at any point in time? Sadly, my personal experience indicates that many enterprises don't give it any thought and just expect things to work, oftentimes not even knowing about all of their services in enough detail to be able to answer questions about their architecture, nor having a clear way of managing their service instances. > You can have cyclical dependencies sneaking in the past, that makes it impossible to startup some systems This does seem like an application problem. If an application cannot reach external services, it should either: - return an error for all services calls that require this external integration - queue up these requests internally to be retried later, if there are the necessary resources for this (RabbitMQ, or just a lot of RAM) Having an application fail fast when one of its dependencies is down makes sense, of course, but it's also a really dangerous approach, since just one cyclic dependency sneaking past could break everything. Perhaps it's better to avoid such situations entirely and not make your app crash if it cannot reach an external service, merely try again later?

KronisLV 5y ago
Clarification for the downvote(s) since the person didn't provide any: in my current workplace, i actually brought this very issue to the attention of all the stakeholders and over the past few months have been working on improving the applications that we develop: updating the frameworks and libraries to recent, maintained and more secure versions, using containers and Ansible to improve configuration management and change management, as well as to get rid of years of technical debt. A part of this was needing to understand how they truly process their configuration, when they choose to fail to start and how they attempt to address external systems - in my mind, it's imperative for the long term success of any project to have a clear understanding of all of this, instead of solving it on an "ad hoc" basis for each separate feature change request. To give you an example: - now almost everything runs in containers with Docker, Docker Compose or Docker Compose (depending on the use case), with extremely clear information about what belongs in which environment and how many replicas exist for each service. A part of testing all of this was actually taking down certain parts of the system to identify any and all dependencies that exist and creating action plans for them. - for example, if i want to build a new application version i won't be able to do that if the GitLab CI is down, or if the GitLab CI server cannot reach the Nexus registry which runs within containers, so if the server that it's on goes down, we'll need a fallback - in this case, temporarily using the public Docker image from Docker Hub, before restoring the functionality of Nexus through either Ansible or manually, and then proceeding with the rest of the servers. - it's gotten to the point where we could wipe any or all of the servers within that project's infrastructure and, as long as we have the sources for the projects within GitLab or within a backed up copy of it, we could restore everything to a working state with a few blank VMs and either GitLab CI or by launching local containers with the very same commands that the CI server executes (given the proper permissions). Of course, make no mistake - it's been an uphill battle every step of the way and frankly i've delivered way too many versions of software and configuration changes late into the evening, because for some reason we don't have an entire team of DevOps specialists for this - just me and other developers to onboard. Getting the applications not to fail-fast whenever external services are unavailable has also been a battle, especially given the old framework versions and inconsistent, CV driven development over the years, yet in my eyes it's a battle worth fighting. Not only that, but the results of this are extremely clear - no more depressive thoughts after connecting to the server for some environment through SSH and having to wonder whether it uses sysvinit, systemd, random scripts or something else for environment management. No more wondering about which ports are configured and how the httpd/Nginx instances map to Tomcat ports, since all of that is now formalized in a Compose stack. No more worrying about resource limits and old badly written services with rogue GC eating up all of the resources and making the entire server grind to a halt. No more configuration and data that's strewn across the POSIX file system, everything's now under /app for each stack. There hasn't been an outage for these services in two months, their resource usage is consistent, they have health checks and automatic restarts (if ever needed). Furthermore, suddenly it becomes extremely easy to add something like Nexus, or Zabbix, or Matomo or Skywalking to the infrastructure because they're just containers. Therefore, i'd posit that the things i mentioned in the original post are not only feasible, but also necessary - to both help in recovering from failures, but also to make the way everything runs more clear and intuitive for people. And it pains me every time when i join a new project with the goal of consulting some other enterprise and generating value with some software, but instead see something like a neglected codebase with not even a README and the expectation for the developers to pull out ideas on how to run local environments out of thin air. If you see environments like that, you're not set up for success, but rather failure. If you see environments like that, consider either fixing them or leaving.
sidlls 5y ago
Roblox has been around for 16 years. I would be shocked to learn that 16 years ago they planned the architecture of their system so meticulously as to account for dependencies and problems you describe for the long term future. Odds are some novice programmer or a few of them hacked something together and as the platform grew they just kept adding to the poor hacking. It might take a huge investment to correct it now. But perhaps an outage like this will prove to them that it's necessary. (N.B.: I did not vote up or down on your comment.)
- KronisLV 5y ago
  > I would be shocked to learn that 16 years ago they planned the architecture of their system so meticulously as to account for dependencies and problems you describe for the long term future. If their architecture hasn't changed much in that long of a time, then that is impressive, both in a positive and negative sense. In regards to the positive aspects, that indeed makes you consider the thought that must have been put in the system design and how it must have been good enough to survive this long. However, there are also the negative aspects - if there are single points of failure, or even scalability issues which may lead to a lot of downtime, then it's almost certain that rewrites in some capacity will need to be carried out. A system design that worked 16 years ago will probably run into some roadblocks today, much like C10k was a problem back in the day but has largely been solved in many situations. Either way, long term downtime doesn't reflect positively on the current state of the overall system. (No worries about the upvotes/downvotes, i merely mentioned that fact in case it was my tone that was inadequate, or perhaps there were technical inaccuracies or just inactionable advice given on my part - being told exactly why i'm wrong is helpful not just to me but to others as well.)

KronisLV 5y ago

Clarification for the downvote(s) since the person didn't provide any: in my current workplace, i actually brought this very issue to the attention of all the stakeholders and over the past few months have been working on improving the applications that we develop: updating the frameworks and libraries to recent, maintained and more secure versions, using containers and Ansible to improve configuration management and change management, as well as to get rid of years of technical debt. A part of this was needing to understand how they truly process their configuration, when they choose to fail to start and how they attempt to address external systems - in my mind, it's imperative for the long term success of any project to have a clear understanding of all of this, instead of solving it on an "ad hoc" basis for each separate feature change request. To give you an example: - now almost everything runs in containers with Docker, Docker Compose or Docker Compose (depending on the use case), with extremely clear information about what belongs in which environment and how many replicas exist for each service. A part of testing all of this was actually taking down certain parts of the system to identify any and all dependencies that exist and creating action plans for them. - for example, if i want to build a new application version i won't be able to do that if the GitLab CI is down, or if the GitLab CI server cannot reach the Nexus registry which runs within containers, so if the server that it's on goes down, we'll need a fallback - in this case, temporarily using the public Docker image from Docker Hub, before restoring the functionality of Nexus through either Ansible or manually, and then proceeding with the rest of the servers. - it's gotten to the point where we could wipe any or all of the servers within that project's infrastructure and, as long as we have the sources for the projects within GitLab or within a backed up copy of it, we could restore everything to a working state with a few blank VMs and either GitLab CI or by launching local containers with the very same commands that the CI server executes (given the proper permissions). Of course, make no mistake - it's been an uphill battle every step of the way and frankly i've delivered way too many versions of software and configuration changes late into the evening, because for some reason we don't have an entire team of DevOps specialists for this - just me and other developers to onboard. Getting the applications not to fail-fast whenever external services are unavailable has also been a battle, especially given the old framework versions and inconsistent, CV driven development over the years, yet in my eyes it's a battle worth fighting. Not only that, but the results of this are extremely clear - no more depressive thoughts after connecting to the server for some environment through SSH and having to wonder whether it uses sysvinit, systemd, random scripts or something else for environment management. No more wondering about which ports are configured and how the httpd/Nginx instances map to Tomcat ports, since all of that is now formalized in a Compose stack. No more worrying about resource limits and old badly written services with rogue GC eating up all of the resources and making the entire server grind to a halt. No more configuration and data that's strewn across the POSIX file system, everything's now under /app for each stack. There hasn't been an outage for these services in two months, their resource usage is consistent, they have health checks and automatic restarts (if ever needed). Furthermore, suddenly it becomes extremely easy to add something like Nexus, or Zabbix, or Matomo or Skywalking to the infrastructure because they're just containers. Therefore, i'd posit that the things i mentioned in the original post are not only feasible, but also necessary - to both help in recovering from failures, but also to make the way everything runs more clear and intuitive for people. And it pains me every time when i join a new project with the goal of consulting some other enterprise and generating value with some software, but instead see something like a neglected codebase with not even a README and the expectation for the developers to pull out ideas on how to run local environments out of thin air. If you see environments like that, you're not set up for success, but rather failure. If you see environments like that, consider either fixing them or leaving.

sidlls 5y ago

Roblox has been around for 16 years. I would be shocked to learn that 16 years ago they planned the architecture of their system so meticulously as to account for dependencies and problems you describe for the long term future. Odds are some novice programmer or a few of them hacked something together and as the platform grew they just kept adding to the poor hacking. It might take a huge investment to correct it now. But perhaps an outage like this will prove to them that it's necessary. (N.B.: I did not vote up or down on your comment.)

KronisLV 5y ago
> I would be shocked to learn that 16 years ago they planned the architecture of their system so meticulously as to account for dependencies and problems you describe for the long term future. If their architecture hasn't changed much in that long of a time, then that is impressive, both in a positive and negative sense. In regards to the positive aspects, that indeed makes you consider the thought that must have been put in the system design and how it must have been good enough to survive this long. However, there are also the negative aspects - if there are single points of failure, or even scalability issues which may lead to a lot of downtime, then it's almost certain that rewrites in some capacity will need to be carried out. A system design that worked 16 years ago will probably run into some roadblocks today, much like C10k was a problem back in the day but has largely been solved in many situations. Either way, long term downtime doesn't reflect positively on the current state of the overall system. (No worries about the upvotes/downvotes, i merely mentioned that fact in case it was my tone that was inadequate, or perhaps there were technical inaccuracies or just inactionable advice given on my part - being told exactly why i'm wrong is helpful not just to me but to others as well.)