11 ms·

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us on the sysadmin side of the aisle capitulating and saying that developers can't be stopped from having the newest version of things, and I think that's a bad idea. 15 years ago, when a project would kick-off, as a sysadmin I'd be invited in and the developers and I would hash out what versions of each language and library involved the project would use. This worked well with Perl; once the stacks started gravitating to Ruby and Python it was a dismal failure. Why? Because those two ecosystems release like hummingbirds off of their ritalin. Take the release history for pip[1] (and I'm not calling pip out as particularly bad; I'm calling pip out as particularly average, which is the problem): in the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through 24 version bumps, introducing thirteen (documented) backwards incompatibilities. Furthermore, there were more regression fixes from previous bumps than feature additions. You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they were release candidates rather than releases. Ruby is just as bad; the famous (and I've experienced this) example is that an in-depth tutorial can be obsoleted in the two weeks it takes you to work through it. Devs are chasing a moving target, and devs who haven't been sysadmins may have trouble seeing why that's a bad idea. [1]: https://pip.pypa.io/en/stable/news/ https://pip.pypa.io/en/stable/news/

berntb 10y ago
Interesting. Is this because the Perl ecosystem is more mature or because of the philosophy of backwards compatibility? The "backwards compatibility" philosophy isn't so explicit for the ecosystem, mostly the language? Is the test-on-install-by-default making a big difference there?
- bandrami 10y ago
  That's a good question. I think it's not a coincidence that CPAN predated the widespread use of distributed source control systems whereas pypi and gems blew up just as mercurial and git were unseating svn and cvs. It's a different release philosophy (remember, in the 1990s you often didn't even get to see pre-release CVS commits of open source projects; that was an innovation of OpenBSD). I also think the widespread use of VPSs rather than accounts on shared servers (again, containerization) was a factor. In the 90s and early 2000s, you usually (even in a corporate setting) had an unprivileged account on a server with a given version of apache and perl, your own cgi-bin directory, and possibly some latitude on a personal CPAN install directory. The lack of containerization meant you had to compromise between using newer software and breaking existing use cases. So I guess I think it's not so much about Python vs. Perl per se but about the technologies available when those languages became popular among developers.
- rootusrootus 10y ago
  That would seem to be ironic. As a longtime Perl developer who switched (for pragmatic reasons -- a job) two and a half years ago, my impression is that Python [as a language] is much better suited to a business environment. What makes Perl such a wonderful language and why I enjoy it so much is exactly why it blows as a business language. Python's rigidity is very useful if you ever want someone else to read and understand code written by someone else. So the idea that Perl ends up with the more mature package management is exactly the opposite of what I would have predicted. That said, I haven't had any more problems with PyPi packages than I did in the past with CPAN. Yes, pip always wants to upgrade itself, but that sort of every-damn-day software upgrade cycle seems to have become quite prevalent, not just in the Python world.
  - berntb 10y ago
    You talk about that Python have just one possible layout standard and so on, I guess? That is a different subject. (And sure, no coding standard is bad for a project, Python removes that discussion to a degree.) I think the real problem with Python/Ruby/etc is the surprising lack of an analogue to CPAN Testers. It isn't just all of CPAN that is tested on different OS/Perl version combinations, it also stress test the Perl versions.
gsnedders 10y ago
> You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they were release candidates rather than releases. Releasing an RC often results in nobody using it and hence not finding the bug even in several weeks, but it gets caught almost instantly in a release… At least, that's my experience in shipping various RCs that have led to next-day regression-fixes once it does ship. While yes, better testing would solve such issues, but at some point the line has to be drawn as "good enough", because there's ultimately a limit to what is reasonable.
- bandrami 10y ago
  Oh, definitely, and I want to be clear I'm not wagging my finger at pip here; it's a good project. It's just that for somebody like me on the far side of 40 (and sysadmins in general are a grayer cohort than devs), that's absolutely not a release tempo that I grew up with. I still shudder when I see a Dockerfile that begins "FROM whatever/latest..." because I have no idea if whatever/latest is going to be the next leftpad.
vinceguidry 10y ago
> Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". As a Ruby developer, I can only laugh at this particular example. No Ruby project I've ever worked on ever upgraded their gems midway through a project, much less the version of Ruby. Developing procedures for this kind of ongoing maintenance is just way too much to ask. This stuff tends to get done years after the original devs have all moved on. Maybe they tried that kind of thing back in the early days, before I started working with Ruby, definitely not today.
- brianwawok 10y ago
  I upgrade Python deps all the time. Java too. How often depends on the scope of the project.
- bandrami 10y ago
  The time I'm thinking of was a team that wanted to switch from Rails 1.2 to 2.0 along with whichever interpreter bump was required to make that happen (IIRC 1.8.5 to 1.8.6, but this was a decade ago; I'm pretty sure 1.9 hadn't come out yet). Halfway through a project.
  - vinceguidry 10y ago
    Unreal. Yep, that sounds like that 'long time ago' I was talking about. Nowadays you can do that, no sysadmin to tell you not to, but nobody bothers.
    - falcolas 10y ago
      DevOps here - yes, they do bother. Pinned set of dependencies, but one of them updates? Upgrade all of the dependencies. But don't worry, it's all in a Docker container (I have a completely separate rant about Docker's compatibility ignorant hummingbird). Ironically enough, I think the current DevOps culture emerged partially because sysadmins got tired of saying no (if only so they could sleep through the night), so now they let developers tie their own nooses so they can be woken up at night. It's wonderful to give up all of those software pages back to developers. And the developers do seem motivated to fix the bugs which wake them up at 3am, so it turns into a win all around. It's still hard to watch a new team come up to speed though, knowing how little sleep they will be getting over the next month because they made their new docker program stateful...
brianwawok 10y ago
That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading? I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in setting up the infastructure and keeping it running. They subtract value when they want to tell developers what version of a library to use.
- bandrami 10y ago
  why should a sysadmin who doesn't even know Python tell me what version of a library I will use Because he maintains that installation and you don't? But, yeah, that's why virtualenv, Docker, etc. were invented, because devs kept getting sick of installations having consequences. What does a sysadmin have to approve or deny an upgrade? Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Read the changelog and look for implementation problems. Read fora and look for performance problems people are reporting. Yes, beards get stroked during this process, but time and again we see that developers refuse to do this, and wind up coming to us when they break something because of that...
  - e40 10y ago
    That's the tail wagging the dog (and I manage the sys admin department at my company).
  - btreecat 10y ago
    >Because he maintains that installation and you don't? So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK? >Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it. >Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Ok this is where I completely agree with you as we have been working on this at our company. My personal solution seems pretty logical though so hear me out. 1) Build a docker file that fully documents the install of your service as well as any OS level dependencies. Ensure that any config files are external to the container to allow sysadmin access. 2) Document in a central location (say an internal wiki) what the external services, servers, repositories, developers, and admins are responsible for the service. 3) Automate builds of containers from repo and add automated testing post containerization. 4) Sysadmins monitor repositories for changes to docker files or wiki articles for new services, databases and libraries as well as taking note of library versions. If an issue with a particular lib or service is discovered, the config files can be edited to point to a new service. Or a new container build can be triggered with zero changes to the source code, but a forced update to the OS packages for the container. In a tight situation where a developer might not be available on-call, the sysadmins have more control over a similar proprietary product but don't have a workflow for messing with source code (which they are likely not familiar with regardless). There are solutions to the issues you raise (often trivial ones at that), they just require an adjustment to workflow and an increase in communication between developers and their sysadmins.
SixSigma 10y ago
Ironically all-aboard-the-update-train was the actual reason I jumped off IIS back in the 90s when I got badly burned by updating Windows NT. Automatic database connection pooling for IIS was dropped and I started getting annoyed phone calls from clients who's websites were dying after updates. One had to read MSDN every day to keep up with what might break on sites you had no control over.
tete 10y ago
That might be a bit naive, but isn't that more of a package management Problem than everything else? I can tell you that while I know this issue I also know that on systems like Alpine Linux or FreeBSD this is less of a problem than it used to be and containers (so virtualization - even when it is OS-level) is potentially overkill and certainly not a solution. virtualenv and others seem way saner in this case. For development as a whole it is really great though in my opinion. Or you simply use something like this: https://bazel.build/ https://bazel.build/
deleted 10y ago
[deleted]
talideon 10y ago
> Why does virtualenv exist? A similar reason. The reason why virtualenv exists is because different apps may have conflicting requirements, and you have apps that need to be deployed in different environments with different versions of different libraries. I know that even if I were developing against versions of libraries in system packages, I'd still end up having to use virtualenv in development (EDIT: I wrote 'production' here by accident) because my stuff gets deployed on different versions of Debian and RHEL, necessitating virtual environments if only so that I can make my development environment as close to production as possible. > In the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through 24 version bumps, introducing thirteen (documented) backwards incompatibilities. Much of that has been down to efforts in recent years to finally fix the major issues with Python packaging. It has settled down quite a bit. Also, the 1.* to 8.* change is because the initial '1' was dropped: 8.* is essentially 1.8.* in the old versioning scheme. I'm not saying that this couldn't have been handled better, but it's not just a 'hummingbirds off of their ritalin' situation: Python spent many years with packaging stagnated, and what you're seeing is rapid development to fix the mess that years of PJE-related neglect caused.
AlexB138 10y ago
As a Sys Admin turned Automation/Tools Engineer, I think you're missing part of the point. You've got the beginning of it right in saying that Sys Admins used to be involved in pinning down versions, and even in why that was necessary, but I believe you're incorrect in saying that the containerization technologies are bad for removing that. Those technologies don't exist so Developers can get around Sys Admins and ignore your helpful advice. They exist to solve that problem that makes the Sys Admins role there necessary. It removes the underlying need for a Sys Admin to worry about the versions. Admins should see this as a good thing, but in my experience many dislike it because it takes them out of their Gatekeeper role. We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better. If something breaks due to version changes, their testing should catch it. If things are breaking in production, we ought to get involved because there's some other problem, but before that we, as a profession, need to learn to get out of the way and let people work by letting technology handle the problems. The "Gatekeeper" mentality needs to die as quickly as it possibly can.
- jstimpfle 10y ago
  > They aren't kids playing with toys that we need to nanny over In my experience (university), yes they are, and they should do that at home. Why do you need the latest bleeding versions in the first place? In my sysadmin experience, people believe software gets bad and deprecated as soon as the glory next breaking version appears. I don't think I need to argue why this is an illogical stance. With my developers hat on, bumping to the next version mid-process reliably introduces more friction than is worth it. People think the next version solves that one weird issue but ignore that it introduces two new ones and that the software must be changed to fix five new incompatibilities. But the solution reliably is to just not use that weird feature that caused the bug in the first place, and think what a clean solution would have been. And guess what, the result is a cleaner and more compatible code base. It's a tip that works for me again and again: If there is friction, think - before spending the next hours with an update that will soon lead to new problems. It's great that you can for example compile Linux without too much friction. It's great that arcane shell scripts can run on any system. Stability (in a compatibility sense) is not a nice-to-have, it's basic sanity.
awinder 10y ago
> 15 years ago, when a project would kick-off, as a sysadmin > I'd be invited in and the developers and I would hash out > what versions of each language and library involved the > project would use To wit: the new ideologies in infrastructure management are actually designed to solve the underlying problem that necessitated that kind of working setup. Why should the version of a lib in one part of the software somehow pose existential threat to the infrastructure? Engrain the dependencies into contained, independently deployable pieces, and make it so that app-level code can evolve without bringing down the world with it. Make it easy to revert back, and/or utilize phased rollouts, and you've got the ability to iterate quickly, keep pace with external dependencies, and it no longer has to be some scary thing that requires big back-and-forth meetings over mundane details. (As for software that releases often, maybe it's an over-correction, but there's a reason things don't work as they did in the glory days, and that's because they were never really that glorious.) This doesn't necessarily rule out the expertise of systems administration, because the platforms for all of this need to be built & maintained, and there's still a lot of work to be done on network boarder security, etc. It's a movement that focuses systems administration to systems administration, instead of having to be this big org arbiter of microdecisions, and all the baggage that goes along with trying to be the gatekeeper of all.
- jstimpfle 10y ago
  Counter-wit: Why can't people get their software to work with the existing libs? Hint: It's very rarely that the existing libs disqualify.
  - simplehuman 10y ago
    Because many of these old library versions go unmaintained.
  - adrianN 10y ago
    People could do great things just with punch cards, yet somehow technology kept marching on. If developers want to use newer stuff usually they have a good reason. The ability to hack around the deficiencies of old dependencies does not mean that one couldn't get a better, cheaper solution with newer technology.
paulddraper 10y ago
The problem is that sysadmins don't know everything. > in the year 2015, pip went from version 1.5.6 to 8.1.1 The only releases in 2015 were 6.x and 7.x. There were 8 documented backwards incompatibilities, 4 deprecated the previous year, and 3 documenting a couple bugs that were fixed several days after the 7.0.0 release. These are the sorts of thing an aware Python developer will know.
- bandrami 10y ago
  You're right; it was the period from December 22nd, 2014 to March 17th, 2016, so about 15 months centered around 2015. We may be counting regressions differently; I'm including both adding and removing the spinner as a regression, for instance (since both the addition and removal added unexpected behavior). Note that the undeniable regressions that occurred in releases during those 15 months included: 1. Exceptions raised in any command on Windows 2. Switching from not installing standard libraries to installing them back to not installing them 3. Blocking if the particular server pypi.python.org was down 4. An infinite loop on filesystems that do not allow hard links Note that in that time they also added yet another internal package management system (incompatible with the existing two), changed the versioning semantics twice, and dropped support for versions of python that were 3 years old at that point. And, again, there's nothing particularly wrong with or bad about pip; this is just what a younger generation of developers are used to.

berntb 10y ago

Interesting. Is this because the Perl ecosystem is more mature or because of the philosophy of backwards compatibility? The "backwards compatibility" philosophy isn't so explicit for the ecosystem, mostly the language? Is the test-on-install-by-default making a big difference there?

bandrami 10y ago
That's a good question. I think it's not a coincidence that CPAN predated the widespread use of distributed source control systems whereas pypi and gems blew up just as mercurial and git were unseating svn and cvs. It's a different release philosophy (remember, in the 1990s you often didn't even get to see pre-release CVS commits of open source projects; that was an innovation of OpenBSD). I also think the widespread use of VPSs rather than accounts on shared servers (again, containerization) was a factor. In the 90s and early 2000s, you usually (even in a corporate setting) had an unprivileged account on a server with a given version of apache and perl, your own cgi-bin directory, and possibly some latitude on a personal CPAN install directory. The lack of containerization meant you had to compromise between using newer software and breaking existing use cases. So I guess I think it's not so much about Python vs. Perl per se but about the technologies available when those languages became popular among developers.
rootusrootus 10y ago
That would seem to be ironic. As a longtime Perl developer who switched (for pragmatic reasons -- a job) two and a half years ago, my impression is that Python [as a language] is much better suited to a business environment. What makes Perl such a wonderful language and why I enjoy it so much is exactly why it blows as a business language. Python's rigidity is very useful if you ever want someone else to read and understand code written by someone else. So the idea that Perl ends up with the more mature package management is exactly the opposite of what I would have predicted. That said, I haven't had any more problems with PyPi packages than I did in the past with CPAN. Yes, pip always wants to upgrade itself, but that sort of every-damn-day software upgrade cycle seems to have become quite prevalent, not just in the Python world.
- berntb 10y ago
  You talk about that Python have just one possible layout standard and so on, I guess? That is a different subject. (And sure, no coding standard is bad for a project, Python removes that discussion to a degree.) I think the real problem with Python/Ruby/etc is the surprising lack of an analogue to CPAN Testers. It isn't just all of CPAN that is tested on different OS/Perl version combinations, it also stress test the Perl versions.

gsnedders 10y ago

> You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they were release candidates rather than releases. Releasing an RC often results in nobody using it and hence not finding the bug even in several weeks, but it gets caught almost instantly in a release… At least, that's my experience in shipping various RCs that have led to next-day regression-fixes once it does ship. While yes, better testing would solve such issues, but at some point the line has to be drawn as "good enough", because there's ultimately a limit to what is reasonable.

bandrami 10y ago
Oh, definitely, and I want to be clear I'm not wagging my finger at pip here; it's a good project. It's just that for somebody like me on the far side of 40 (and sysadmins in general are a grayer cohort than devs), that's absolutely not a release tempo that I grew up with. I still shudder when I see a Dockerfile that begins "FROM whatever/latest..." because I have no idea if whatever/latest is going to be the next leftpad.

vinceguidry 10y ago

> Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". As a Ruby developer, I can only laugh at this particular example. No Ruby project I've ever worked on ever upgraded their gems midway through a project, much less the version of Ruby. Developing procedures for this kind of ongoing maintenance is just way too much to ask. This stuff tends to get done years after the original devs have all moved on. Maybe they tried that kind of thing back in the early days, before I started working with Ruby, definitely not today.

brianwawok 10y ago
I upgrade Python deps all the time. Java too. How often depends on the scope of the project.
bandrami 10y ago
The time I'm thinking of was a team that wanted to switch from Rails 1.2 to 2.0 along with whichever interpreter bump was required to make that happen (IIRC 1.8.5 to 1.8.6, but this was a decade ago; I'm pretty sure 1.9 hadn't come out yet). Halfway through a project.
- vinceguidry 10y ago
  Unreal. Yep, that sounds like that 'long time ago' I was talking about. Nowadays you can do that, no sysadmin to tell you not to, but nobody bothers.
  - falcolas 10y ago
    DevOps here - yes, they do bother. Pinned set of dependencies, but one of them updates? Upgrade all of the dependencies. But don't worry, it's all in a Docker container (I have a completely separate rant about Docker's compatibility ignorant hummingbird). Ironically enough, I think the current DevOps culture emerged partially because sysadmins got tired of saying no (if only so they could sleep through the night), so now they let developers tie their own nooses so they can be woken up at night. It's wonderful to give up all of those software pages back to developers. And the developers do seem motivated to fix the bugs which wake them up at 3am, so it turns into a win all around. It's still hard to watch a new team come up to speed though, knowing how little sleep they will be getting over the next month because they made their new docker program stateful...

brianwawok 10y ago

That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading? I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in setting up the infastructure and keeping it running. They subtract value when they want to tell developers what version of a library to use.

bandrami 10y ago
why should a sysadmin who doesn't even know Python tell me what version of a library I will use Because he maintains that installation and you don't? But, yeah, that's why virtualenv, Docker, etc. were invented, because devs kept getting sick of installations having consequences. What does a sysadmin have to approve or deny an upgrade? Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Read the changelog and look for implementation problems. Read fora and look for performance problems people are reporting. Yes, beards get stroked during this process, but time and again we see that developers refuse to do this, and wind up coming to us when they break something because of that...
- e40 10y ago
  That's the tail wagging the dog (and I manage the sys admin department at my company).
- btreecat 10y ago
  >Because he maintains that installation and you don't? So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK? >Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it. >Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Ok this is where I completely agree with you as we have been working on this at our company. My personal solution seems pretty logical though so hear me out. 1) Build a docker file that fully documents the install of your service as well as any OS level dependencies. Ensure that any config files are external to the container to allow sysadmin access. 2) Document in a central location (say an internal wiki) what the external services, servers, repositories, developers, and admins are responsible for the service. 3) Automate builds of containers from repo and add automated testing post containerization. 4) Sysadmins monitor repositories for changes to docker files or wiki articles for new services, databases and libraries as well as taking note of library versions. If an issue with a particular lib or service is discovered, the config files can be edited to point to a new service. Or a new container build can be triggered with zero changes to the source code, but a forced update to the OS packages for the container. In a tight situation where a developer might not be available on-call, the sysadmins have more control over a similar proprietary product but don't have a workflow for messing with source code (which they are likely not familiar with regardless). There are solutions to the issues you raise (often trivial ones at that), they just require an adjustment to workflow and an increase in communication between developers and their sysadmins.

SixSigma 10y ago

Ironically all-aboard-the-update-train was the actual reason I jumped off IIS back in the 90s when I got badly burned by updating Windows NT. Automatic database connection pooling for IIS was dropped and I started getting annoyed phone calls from clients who's websites were dying after updates. One had to read MSDN every day to keep up with what might break on sites you had no control over.

tete 10y ago

That might be a bit naive, but isn't that more of a package management Problem than everything else? I can tell you that while I know this issue I also know that on systems like Alpine Linux or FreeBSD this is less of a problem than it used to be and containers (so virtualization - even when it is OS-level) is potentially overkill and certainly not a solution. virtualenv and others seem way saner in this case. For development as a whole it is really great though in my opinion. Or you simply use something like this: https://bazel.build/ https://bazel.build/

deleted 10y ago

[deleted]

talideon 10y ago

> Why does virtualenv exist? A similar reason. The reason why virtualenv exists is because different apps may have conflicting requirements, and you have apps that need to be deployed in different environments with different versions of different libraries. I know that even if I were developing against versions of libraries in system packages, I'd still end up having to use virtualenv in development (EDIT: I wrote 'production' here by accident) because my stuff gets deployed on different versions of Debian and RHEL, necessitating virtual environments if only so that I can make my development environment as close to production as possible. > In the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through 24 version bumps, introducing thirteen (documented) backwards incompatibilities. Much of that has been down to efforts in recent years to finally fix the major issues with Python packaging. It has settled down quite a bit. Also, the 1.* to 8.* change is because the initial '1' was dropped: 8.* is essentially 1.8.* in the old versioning scheme. I'm not saying that this couldn't have been handled better, but it's not just a 'hummingbirds off of their ritalin' situation: Python spent many years with packaging stagnated, and what you're seeing is rapid development to fix the mess that years of PJE-related neglect caused.

AlexB138 10y ago

As a Sys Admin turned Automation/Tools Engineer, I think you're missing part of the point. You've got the beginning of it right in saying that Sys Admins used to be involved in pinning down versions, and even in why that was necessary, but I believe you're incorrect in saying that the containerization technologies are bad for removing that. Those technologies don't exist so Developers can get around Sys Admins and ignore your helpful advice. They exist to solve that problem that makes the Sys Admins role there necessary. It removes the underlying need for a Sys Admin to worry about the versions. Admins should see this as a good thing, but in my experience many dislike it because it takes them out of their Gatekeeper role. We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better. If something breaks due to version changes, their testing should catch it. If things are breaking in production, we ought to get involved because there's some other problem, but before that we, as a profession, need to learn to get out of the way and let people work by letting technology handle the problems. The "Gatekeeper" mentality needs to die as quickly as it possibly can.

jstimpfle 10y ago
> They aren't kids playing with toys that we need to nanny over In my experience (university), yes they are, and they should do that at home. Why do you need the latest bleeding versions in the first place? In my sysadmin experience, people believe software gets bad and deprecated as soon as the glory next breaking version appears. I don't think I need to argue why this is an illogical stance. With my developers hat on, bumping to the next version mid-process reliably introduces more friction than is worth it. People think the next version solves that one weird issue but ignore that it introduces two new ones and that the software must be changed to fix five new incompatibilities. But the solution reliably is to just not use that weird feature that caused the bug in the first place, and think what a clean solution would have been. And guess what, the result is a cleaner and more compatible code base. It's a tip that works for me again and again: If there is friction, think - before spending the next hours with an update that will soon lead to new problems. It's great that you can for example compile Linux without too much friction. It's great that arcane shell scripts can run on any system. Stability (in a compatibility sense) is not a nice-to-have, it's basic sanity.

awinder 10y ago

> 15 years ago, when a project would kick-off, as a sysadmin > I'd be invited in and the developers and I would hash out > what versions of each language and library involved the > project would use To wit: the new ideologies in infrastructure management are actually designed to solve the underlying problem that necessitated that kind of working setup. Why should the version of a lib in one part of the software somehow pose existential threat to the infrastructure? Engrain the dependencies into contained, independently deployable pieces, and make it so that app-level code can evolve without bringing down the world with it. Make it easy to revert back, and/or utilize phased rollouts, and you've got the ability to iterate quickly, keep pace with external dependencies, and it no longer has to be some scary thing that requires big back-and-forth meetings over mundane details. (As for software that releases often, maybe it's an over-correction, but there's a reason things don't work as they did in the glory days, and that's because they were never really that glorious.) This doesn't necessarily rule out the expertise of systems administration, because the platforms for all of this need to be built & maintained, and there's still a lot of work to be done on network boarder security, etc. It's a movement that focuses systems administration to systems administration, instead of having to be this big org arbiter of microdecisions, and all the baggage that goes along with trying to be the gatekeeper of all.

jstimpfle 10y ago
Counter-wit: Why can't people get their software to work with the existing libs? Hint: It's very rarely that the existing libs disqualify.
- simplehuman 10y ago
  Because many of these old library versions go unmaintained.
- adrianN 10y ago
  People could do great things just with punch cards, yet somehow technology kept marching on. If developers want to use newer stuff usually they have a good reason. The ability to hack around the deficiencies of old dependencies does not mean that one couldn't get a better, cheaper solution with newer technology.

paulddraper 10y ago

The problem is that sysadmins don't know everything. > in the year 2015, pip went from version 1.5.6 to 8.1.1 The only releases in 2015 were 6.x and 7.x. There were 8 documented backwards incompatibilities, 4 deprecated the previous year, and 3 documenting a couple bugs that were fixed several days after the 7.0.0 release. These are the sorts of thing an aware Python developer will know.

bandrami 10y ago
You're right; it was the period from December 22nd, 2014 to March 17th, 2016, so about 15 months centered around 2015. We may be counting regressions differently; I'm including both adding and removing the spinner as a regression, for instance (since both the addition and removal added unexpected behavior). Note that the undeniable regressions that occurred in releases during those 15 months included: 1. Exceptions raised in any command on Windows 2. Switching from not installing standard libraries to installing them back to not installing them 3. Blocking if the particular server pypi.python.org was down 4. An infinite loop on filesystems that do not allow hard links Note that in that time they also added yet another internal package management system (incompatible with the existing two), changed the versioning semantics twice, and dropped support for versions of python that were 3 years old at that point. And, again, there's nothing particularly wrong with or bad about pip; this is just what a younger generation of developers are used to.