5 ms·

I appreciate the contrarian point of view, and I don't want to pick on the author, but this sounds like words of wisdom from someone who doesn't have much real

I appreciate the contrarian point of view, and I don't want to pick on the author, but this sounds like words of wisdom from someone who doesn't have much real world experience. I worked in such a project who never upgraded once. And then Specter/Meltdown hit, and we had the mandate to patch all our systems (and the threat was very real). Welcome to a hellish mess of outdated deps that took weeks to sort out. But is not only me. I watched a talk from a Principal Engineer from Amazon builder tools, and one of their biggest mistakes was not to upgrade often enough, which made each upgrade more painful. So what do we do now working in multiple, large, multi-year projects? First, we set tools that tell us when deps are outdated, and alarm when vulnerabilities are found. Second, upgrading deps is part of the regular dev process. When you make a change, take the chance to upgrade the deps. Vulnerabilities are addressed proactively. The vast majority of times upgrades require no code changes. Third, we regularly upgrade deps of inactive packages. Lastly, if a dependency upgrade requires significant work, we create a ticket for it and address it as a new feature. In practice this happens very rarely, so it doesn't disrupt the development process much. As other's said, the key is to upgrade often, and then it wont be as painful.

KronisLV 5y ago
Hello there, i'm the author of the article. > I appreciate the contrarian point of view, and I don't want to pick on the author, but this sounds like words of wisdom from someone who doesn't have much real world experience. I see why you'd say this, but what i'm mostly advocating for in the article is that breaking changes often being the only option for updates (which may be forced upon you because you NEED the security updates that are included) is a problem, since no one wants to (or even feasibly can) keep a maintenance branch of their software around forever, for every single version. A pretty good example of software that doesn't cause many problems in this regard would be something like MySQL 5.7, which was released in 2015 and will be supported until 2023. Personally, 8 or more years seems like a pretty good amount of time to support a software product for, as opposed to forcing you to update to something newer after just a few years if you want security updates and bug fixes, especially if you don't have the proper amount of resources at your disposal to properly migrate over and test everything. For example, for the past few months at my dayjob i've been: - working to migrate about 7 Java services over from Java 8 to Java 11 - this also necessitated not only the migration of minor framework versions, but also major versions in some case (Java 8 --> 9 was a generational shift of sorts) - furthermore, the decision was made to also abandon Spring and migrate over to Spring Boot, both because it historically "won" and also because some of the services already ran with it, so this should increase consistency across the board - the decision to utilize containers also was made, after much deliberation and problems with the environments not being consistent otherwise - the decision to also use Ansible was made, because historically changes to the server configuration weren't entirely traceable easily and diverged otherwise - the decision to reorganize all of the servers with modern and up to date OS versions was also made, as well as the tools to manage the container clusters, as opposed to having systemd in one environment, sysvinit in another and manually run scripts in yet another environment (about 5 of those environments in total, each with all of the apps, though previously sometimes strewn across multiple servers for historical reasons, e.g. temporary environments that became permanent) - tack on a few topology changes, introducing a proper ingress, as opposed to sometimes managing SSL/TLS through Tomcat, but not always - if the scope creep doesn't sound bad enough, throw in some load tests that needed to be made due to issues in productions with performance, that had to be addressed in parallel - oh and since AngularJS is essentially a dead technology, some of those systems needed to be split up into proper separate front ends and back ends for the eventual migration to something else - besides all of that, i've also been working on introducing proper CI for all of that IaC and app builds within Docker containers, as well as any and all package repositories (Maven, npm, Docker) that we may need, both to avoid rate limits and improve performance, as well as cache dependencies in case the main source goes down It's as if "the business" looked at everything they should have been doing in the past years and decided to put it all on my plate, only very good prioritization can avoid situations like this turning into an utter failure. Is that too much for one person to do successfully? Quite possibly. Have i also been mostly successful in all of the above and have learnt a lot? Most certainly. But does something like that possibly lead to burnout? I'd say that yes, in most cases. It's not healthy. Now, i agree with you that updating often would have noticeably lessened my pain, yet when your department isn't seen as a profit center or you cannot sell your clients (assuming consulting) on the idea of things like SRE and constant updates, at most people will bump smaller versions every couple of months to tick a checkbox somewhere, because they cannot feasibly do what i'm attempting now. Not only that, but most of your suggestions simply wouldn't be seen worth the time and effort, unless you'd prefer to ask for forgiveness rather than permission, which is hard to do when you're also expected to deliver features and fixes: - you won't have much automation or automatic scanning of outdated packages with proper alerts in place; i only recently sorted out the package management with proper caching registries myself - you won't care about vulnerabilities (unless very high impact or incidence), at least not to the level of being able to react to them proactively (ITSEC teams often viewing the services as a black box, which doesn't tell them much about a whole class of issues) - you won't care about upgrading dependencies regularly, because you probably won't want to be the person who breaks something with 0 perceptible benefit to anyone - most importantly, you probably won't have an all encompassing test suite that'd do both unit tests, integration tests, performance tests and would also check everything from end to end, to make sure that everything would indeed work in a browser (or if you do, they're probably not updated regularly and don't have good enough coverage to matter) Furthermore, if you need to do a generational shift, like i had to Java 8 --> 11 and how we'll soon have to do with AngularJS to something else, you can't just go to whoever writes your checks and say: "Okay, i'll need the next 3-12 months to work on this migration to do basically a full rewrite," unless they're really on board with your past incentives and are aware of the need for keeping up with the current technologies. Any such incentive, no matter how important would generate pushback and long discussions, worst of all, you wouldn't even know if any of that is even feasible. Would you want to be to known as the person who spent 9 months migrating an enterprise monolith, just to fail in the end and deliver absolutely nothing? In most cases that's a hard sell and almost impossible to put a positive spin on it. For example, one of the systems that i haven't been able to split up and by far the largest one has the following: - i can't update from Java 8 to 11 because the version of Spring doesn't support it - if i attempt to migrate over to Java 11 alongside newer versions of Spring (Boot), the old web.xml configuration no longer works - some other configuration is randomly ignored and isn't loaded at all, whereas other needs refactoring because certain classes no longer exist - speaking of classes, there are now class path conflicts and i need to scan through the pom.xml and figure out which of the hundred dependencies are misbehaving - not only that, but about 50 of them are out of date and thus need updates, given that many of them are also incompatible with Java 11 - worst of those are the classes that just break at runtime, like class loader functionality at app startup - after all of that, i discover that many of the configuration values have been changed and need updating - some of the servlets also aren't loaded properly and thus i cannot figure out how to get the app responding to requests properly - since the back end also bundles JSP/JSF/PrimeFaces, all of those also provide further challenges, as does Java EL - there are also scheduled processes within the app that break - there are also services that deal with file uploads that break - there are also services that deal with reports and PDF export that break - there are also services that deal with database migrations that break - there are also Maven plugins that break so certain front end resources cannot be built - there are other things too, but sadly i don't keep a full list of everything that broke... I have no illusions about the fact that too much was put on my plate, but surely you can understand why in my eyes it could be pretty nice to have a framework version that's supported for 10-20 years and is so stable that it can be used with little to no changes for the entire expected lifetime of a system? Either that, or you have to do updates often, keep your individual services small so they're easier to rewrite (maybe divided by functionality within a bounded context, e.g. PDF service, file upload service, migration service) so that 5% of "dead end code" doesn't keep the rest 95% from being kept up to date. Essentially, you'd have to constantly invest time into this and not pretend that code doesn't rust. Knowing how little "the business" can care about these finer points in many industries, it doesn't surprise me that you see numerous neglected projects out there and i don't believe that it'll change - thus, we should slow down, if possible, and consider building solutions for the next decade, not just the next monthly iteration of our CVs. Maybe that's a bit of a rant, but i felt like i needed to elaborate on my point of view. Now i'll probably go write my own little tool that alerts me when a new article of mine gets posted on HN, so i can provide comments in a timely manner.
- md8z 5y ago
  Well that does sound like a lot, I hope you can negotiate for a raise at some point. Long-term I'm not sure what you mean building solutions for the next decade, it seems quite hard to design something perfectly in hindsight. And of course you have to compare the cost of it to just slapping something in a VM and putting a firewall over it and calling it a day...
  - KronisLV 5y ago
    > Well that does sound like a lot, I hope you can negotiate for a raise at some point. Oh, certainly. Though right now i'm more concerned with making the lives of my colleagues more easy and actually shipping software that works in the end. > Long-term I'm not sure what you mean building solutions for the next decade, it seems quite hard to design something perfectly in hindsight. That is a very fair point, but in my eyes trying to create stable software is a worthy pursuit in many cases regardless! Perfection is probably never going to be achievable, but the difference between creating something that will break in 3 months versus 3 years is probably pretty impactful (e.g. using bleeding edge/immature frameworks which will necessitate a rewrite, or will have breaking changes). I'd say that both the technologies used and the architecture of the solution can impact this greatly. What's also useful is thinking about limiting the fallout when something does break, e.g. making the system modular enough that it can be fixed, updated or changed bit by bit, as well as thinking about scaling, at least a little bit. I'm not saying that everything needs to be microservices, but having a tightly coupled codebase can and probably will create problems down the road. > And of course you have to compare the cost of it to just slapping something in a VM and putting a firewall over it and calling it a day... That's also a reasonable take. Of course, costs aren't always the only consideration - if my blog or personal site goes down, the impact probably isn't too bad, whereas if a governmental health care system goes down, many people won't be able to receive the services that they need in a timely manner. The latter is probably worth the investment, both monetary and in regards to consideration about all of the stuff mentioned before. I've actually experienced what it's like to see queues building up in one such institution, with the medical personnel also being frustrated, all because a system component in a data center somewhere had been neglected and had DB connection pooling issues, leading to a complete standstill. I was called in to fix that external project and somehow managed to do it by ripping out the DB pooling solution and replacing it with another one. That was problematic when the actual codebase was badly commented and there were no proper tests to speak of in place and the failure mode was both odd and hard to debug. In the end, i didn't manage to repair the old pooling solution without swapping it out entirely, because one moment everything was fine, the next threads just got stuck waiting one by one, with no errors or debug messages, regardless of the config, whereas breakpoints didn't lead to anything useful either. In short, there are systems out there that should be as stable as possible and tolerant of failure, given that not all of them will receive the constant love and care that they deserve. I'd also like to link this lovely article and talk: http://boringtechnology.club/ http://boringtechnology.club/ Especially the bit towards the bottom about the "known unknowns" and "unknown unknowns" - knowing what you can and can't do with a particular piece of technology and its characteristics is probably a good thing.
afro88 5y ago
> the key is to upgrade often * and have a very good test suite
- ink_13 5y ago
  And a good build system
Lhiw 5y ago
> but this sounds like words of wisdom from someone who doesn't have much real world experience Did you read the part in the first paragraph where the author complains about people shutting down the idea without even giving it any discussion? I mean you just dismissed the idea entirely and went off on a "have you tried jQuery" tangent. > one of their biggest mistakes was not to upgrade often enough, which made each upgrade more painful. The mistake isn't not updating frequently enough. The mistake is that the industry thinks maintaining backwards compatibility is a waste of time. Updating more frequently doesn't magically make updating less painful, it just spreads the pain out over longer timeframes. There is some benefit given that you won't be building on top of more and more stuff that may eventually be rug pulled by something that may already be released, but again, this isn't about update frequency, it's about a lack of respect for backward compatibility. If everyone maintained backwards compat indefinitely we wouldn't have so many damn problems and updating would be nearly painless at whatever frequency you like. Maintaining backwards compat isn't actually all that hard. There is two types of change. Breaking and non breaking. Most changes can be done without breakage. Want to make a breaking change to that function? Make a new function, put a 2 on the end. Problems solved. How often does leaving that old function actually cause you problems? Technically it's tech debt, but tbh it's the most benign tech debt you'll ever see. So anyway, my point isn't that we should update frequently or not at all, it shouldn't matter. My point is that semver gives inexperienced devs a pass for lazy development practices that cost everyone instead of eating the cost themselves. Want to release a major version change? Go for itz but understand it's a fork and not everyone needs your special new features you think is so important.

KronisLV 5y ago

Hello there, i'm the author of the article. > I appreciate the contrarian point of view, and I don't want to pick on the author, but this sounds like words of wisdom from someone who doesn't have much real world experience. I see why you'd say this, but what i'm mostly advocating for in the article is that breaking changes often being the only option for updates (which may be forced upon you because you NEED the security updates that are included) is a problem, since no one wants to (or even feasibly can) keep a maintenance branch of their software around forever, for every single version. A pretty good example of software that doesn't cause many problems in this regard would be something like MySQL 5.7, which was released in 2015 and will be supported until 2023. Personally, 8 or more years seems like a pretty good amount of time to support a software product for, as opposed to forcing you to update to something newer after just a few years if you want security updates and bug fixes, especially if you don't have the proper amount of resources at your disposal to properly migrate over and test everything. For example, for the past few months at my dayjob i've been: - working to migrate about 7 Java services over from Java 8 to Java 11 - this also necessitated not only the migration of minor framework versions, but also major versions in some case (Java 8 --> 9 was a generational shift of sorts) - furthermore, the decision was made to also abandon Spring and migrate over to Spring Boot, both because it historically "won" and also because some of the services already ran with it, so this should increase consistency across the board - the decision to utilize containers also was made, after much deliberation and problems with the environments not being consistent otherwise - the decision to also use Ansible was made, because historically changes to the server configuration weren't entirely traceable easily and diverged otherwise - the decision to reorganize all of the servers with modern and up to date OS versions was also made, as well as the tools to manage the container clusters, as opposed to having systemd in one environment, sysvinit in another and manually run scripts in yet another environment (about 5 of those environments in total, each with all of the apps, though previously sometimes strewn across multiple servers for historical reasons, e.g. temporary environments that became permanent) - tack on a few topology changes, introducing a proper ingress, as opposed to sometimes managing SSL/TLS through Tomcat, but not always - if the scope creep doesn't sound bad enough, throw in some load tests that needed to be made due to issues in productions with performance, that had to be addressed in parallel - oh and since AngularJS is essentially a dead technology, some of those systems needed to be split up into proper separate front ends and back ends for the eventual migration to something else - besides all of that, i've also been working on introducing proper CI for all of that IaC and app builds within Docker containers, as well as any and all package repositories (Maven, npm, Docker) that we may need, both to avoid rate limits and improve performance, as well as cache dependencies in case the main source goes down It's as if "the business" looked at everything they should have been doing in the past years and decided to put it all on my plate, only very good prioritization can avoid situations like this turning into an utter failure. Is that too much for one person to do successfully? Quite possibly. Have i also been mostly successful in all of the above and have learnt a lot? Most certainly. But does something like that possibly lead to burnout? I'd say that yes, in most cases. It's not healthy. Now, i agree with you that updating often would have noticeably lessened my pain, yet when your department isn't seen as a profit center or you cannot sell your clients (assuming consulting) on the idea of things like SRE and constant updates, at most people will bump smaller versions every couple of months to tick a checkbox somewhere, because they cannot feasibly do what i'm attempting now. Not only that, but most of your suggestions simply wouldn't be seen worth the time and effort, unless you'd prefer to ask for forgiveness rather than permission, which is hard to do when you're also expected to deliver features and fixes: - you won't have much automation or automatic scanning of outdated packages with proper alerts in place; i only recently sorted out the package management with proper caching registries myself - you won't care about vulnerabilities (unless very high impact or incidence), at least not to the level of being able to react to them proactively (ITSEC teams often viewing the services as a black box, which doesn't tell them much about a whole class of issues) - you won't care about upgrading dependencies regularly, because you probably won't want to be the person who breaks something with 0 perceptible benefit to anyone - most importantly, you probably won't have an all encompassing test suite that'd do both unit tests, integration tests, performance tests and would also check everything from end to end, to make sure that everything would indeed work in a browser (or if you do, they're probably not updated regularly and don't have good enough coverage to matter) Furthermore, if you need to do a generational shift, like i had to Java 8 --> 11 and how we'll soon have to do with AngularJS to something else, you can't just go to whoever writes your checks and say: "Okay, i'll need the next 3-12 months to work on this migration to do basically a full rewrite," unless they're really on board with your past incentives and are aware of the need for keeping up with the current technologies. Any such incentive, no matter how important would generate pushback and long discussions, worst of all, you wouldn't even know if any of that is even feasible. Would you want to be to known as the person who spent 9 months migrating an enterprise monolith, just to fail in the end and deliver absolutely nothing? In most cases that's a hard sell and almost impossible to put a positive spin on it. For example, one of the systems that i haven't been able to split up and by far the largest one has the following: - i can't update from Java 8 to 11 because the version of Spring doesn't support it - if i attempt to migrate over to Java 11 alongside newer versions of Spring (Boot), the old web.xml configuration no longer works - some other configuration is randomly ignored and isn't loaded at all, whereas other needs refactoring because certain classes no longer exist - speaking of classes, there are now class path conflicts and i need to scan through the pom.xml and figure out which of the hundred dependencies are misbehaving - not only that, but about 50 of them are out of date and thus need updates, given that many of them are also incompatible with Java 11 - worst of those are the classes that just break at runtime, like class loader functionality at app startup - after all of that, i discover that many of the configuration values have been changed and need updating - some of the servlets also aren't loaded properly and thus i cannot figure out how to get the app responding to requests properly - since the back end also bundles JSP/JSF/PrimeFaces, all of those also provide further challenges, as does Java EL - there are also scheduled processes within the app that break - there are also services that deal with file uploads that break - there are also services that deal with reports and PDF export that break - there are also services that deal with database migrations that break - there are also Maven plugins that break so certain front end resources cannot be built - there are other things too, but sadly i don't keep a full list of everything that broke... I have no illusions about the fact that too much was put on my plate, but surely you can understand why in my eyes it could be pretty nice to have a framework version that's supported for 10-20 years and is so stable that it can be used with little to no changes for the entire expected lifetime of a system? Either that, or you have to do updates often, keep your individual services small so they're easier to rewrite (maybe divided by functionality within a bounded context, e.g. PDF service, file upload service, migration service) so that 5% of "dead end code" doesn't keep the rest 95% from being kept up to date. Essentially, you'd have to constantly invest time into this and not pretend that code doesn't rust. Knowing how little "the business" can care about these finer points in many industries, it doesn't surprise me that you see numerous neglected projects out there and i don't believe that it'll change - thus, we should slow down, if possible, and consider building solutions for the next decade, not just the next monthly iteration of our CVs. Maybe that's a bit of a rant, but i felt like i needed to elaborate on my point of view. Now i'll probably go write my own little tool that alerts me when a new article of mine gets posted on HN, so i can provide comments in a timely manner.

md8z 5y ago
Well that does sound like a lot, I hope you can negotiate for a raise at some point. Long-term I'm not sure what you mean building solutions for the next decade, it seems quite hard to design something perfectly in hindsight. And of course you have to compare the cost of it to just slapping something in a VM and putting a firewall over it and calling it a day...
- KronisLV 5y ago
  > Well that does sound like a lot, I hope you can negotiate for a raise at some point. Oh, certainly. Though right now i'm more concerned with making the lives of my colleagues more easy and actually shipping software that works in the end. > Long-term I'm not sure what you mean building solutions for the next decade, it seems quite hard to design something perfectly in hindsight. That is a very fair point, but in my eyes trying to create stable software is a worthy pursuit in many cases regardless! Perfection is probably never going to be achievable, but the difference between creating something that will break in 3 months versus 3 years is probably pretty impactful (e.g. using bleeding edge/immature frameworks which will necessitate a rewrite, or will have breaking changes). I'd say that both the technologies used and the architecture of the solution can impact this greatly. What's also useful is thinking about limiting the fallout when something does break, e.g. making the system modular enough that it can be fixed, updated or changed bit by bit, as well as thinking about scaling, at least a little bit. I'm not saying that everything needs to be microservices, but having a tightly coupled codebase can and probably will create problems down the road. > And of course you have to compare the cost of it to just slapping something in a VM and putting a firewall over it and calling it a day... That's also a reasonable take. Of course, costs aren't always the only consideration - if my blog or personal site goes down, the impact probably isn't too bad, whereas if a governmental health care system goes down, many people won't be able to receive the services that they need in a timely manner. The latter is probably worth the investment, both monetary and in regards to consideration about all of the stuff mentioned before. I've actually experienced what it's like to see queues building up in one such institution, with the medical personnel also being frustrated, all because a system component in a data center somewhere had been neglected and had DB connection pooling issues, leading to a complete standstill. I was called in to fix that external project and somehow managed to do it by ripping out the DB pooling solution and replacing it with another one. That was problematic when the actual codebase was badly commented and there were no proper tests to speak of in place and the failure mode was both odd and hard to debug. In the end, i didn't manage to repair the old pooling solution without swapping it out entirely, because one moment everything was fine, the next threads just got stuck waiting one by one, with no errors or debug messages, regardless of the config, whereas breakpoints didn't lead to anything useful either. In short, there are systems out there that should be as stable as possible and tolerant of failure, given that not all of them will receive the constant love and care that they deserve. I'd also like to link this lovely article and talk: http://boringtechnology.club/ http://boringtechnology.club/ Especially the bit towards the bottom about the "known unknowns" and "unknown unknowns" - knowing what you can and can't do with a particular piece of technology and its characteristics is probably a good thing.

afro88 5y ago

> the key is to upgrade often * and have a very good test suite

ink_13 5y ago
And a good build system

Lhiw 5y ago

> but this sounds like words of wisdom from someone who doesn't have much real world experience Did you read the part in the first paragraph where the author complains about people shutting down the idea without even giving it any discussion? I mean you just dismissed the idea entirely and went off on a "have you tried jQuery" tangent. > one of their biggest mistakes was not to upgrade often enough, which made each upgrade more painful. The mistake isn't not updating frequently enough. The mistake is that the industry thinks maintaining backwards compatibility is a waste of time. Updating more frequently doesn't magically make updating less painful, it just spreads the pain out over longer timeframes. There is some benefit given that you won't be building on top of more and more stuff that may eventually be rug pulled by something that may already be released, but again, this isn't about update frequency, it's about a lack of respect for backward compatibility. If everyone maintained backwards compat indefinitely we wouldn't have so many damn problems and updating would be nearly painless at whatever frequency you like. Maintaining backwards compat isn't actually all that hard. There is two types of change. Breaking and non breaking. Most changes can be done without breakage. Want to make a breaking change to that function? Make a new function, put a 2 on the end. Problems solved. How often does leaving that old function actually cause you problems? Technically it's tech debt, but tbh it's the most benign tech debt you'll ever see. So anyway, my point isn't that we should update frequently or not at all, it shouldn't matter. My point is that semver gives inexperienced devs a pass for lazy development practices that cost everyone instead of eating the cost themselves. Want to release a major version change? Go for itz but understand it's a fork and not everyone needs your special new features you think is so important.