6 ms·

That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? I have regression tests

That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading? I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in setting up the infastructure and keeping it running. They subtract value when they want to tell developers what version of a library to use.

bandrami 10y ago
why should a sysadmin who doesn't even know Python tell me what version of a library I will use Because he maintains that installation and you don't? But, yeah, that's why virtualenv, Docker, etc. were invented, because devs kept getting sick of installations having consequences. What does a sysadmin have to approve or deny an upgrade? Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Read the changelog and look for implementation problems. Read fora and look for performance problems people are reporting. Yes, beards get stroked during this process, but time and again we see that developers refuse to do this, and wind up coming to us when they break something because of that...
- e40 10y ago
  That's the tail wagging the dog (and I manage the sys admin department at my company).
- btreecat 10y ago
  >Because he maintains that installation and you don't? So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK? >Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it. >Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Ok this is where I completely agree with you as we have been working on this at our company. My personal solution seems pretty logical though so hear me out. 1) Build a docker file that fully documents the install of your service as well as any OS level dependencies. Ensure that any config files are external to the container to allow sysadmin access. 2) Document in a central location (say an internal wiki) what the external services, servers, repositories, developers, and admins are responsible for the service. 3) Automate builds of containers from repo and add automated testing post containerization. 4) Sysadmins monitor repositories for changes to docker files or wiki articles for new services, databases and libraries as well as taking note of library versions. If an issue with a particular lib or service is discovered, the config files can be edited to point to a new service. Or a new container build can be triggered with zero changes to the source code, but a forced update to the OS packages for the container. In a tight situation where a developer might not be available on-call, the sysadmins have more control over a similar proprietary product but don't have a workflow for messing with source code (which they are likely not familiar with regardless). There are solutions to the issues you raise (often trivial ones at that), they just require an adjustment to workflow and an increase in communication between developers and their sysadmins.
  - bandrami 10y ago
    I see your point, and like I said I agree this is why Docker was invented and it's the best-in-breed at what it does (namely, being a tourniquet for a self-inflicted wound). My biggest concern really isn't "my problem" since I'm in ops: it's the leftpad worry. I still have teams starting Dockerfiles with "FROM centos:latest" because that's just the mindset they have: "Latest will fix any problems" rather than "Latest will introduce new problems". And, ultimately, Docker lets that not be my problem, because they have to deal with it when the next leftpad happens. So, yeah: they should have at it. I guess I still think there's something to be said for the cathedral pace, though.
    - btreecat 10y ago
      >I still have teams starting Dockerfiles with "FROM centos:latest" because that's just the mindset they have: "Latest will fix any problems" rather than "Latest will introduce new problems". Well there are two ways to approach this IMO. One be proactive. Create them a vetted centos or whatever OS environment for them to base off. The problem is, if you don't keep on top of it as a sysadmin, the developers will just figure out another way to wall you off. Alternatively, accept that it doesn't matter what underlying OS they use, because a patched OS >> than unpatched and that when done correctly there is minimal exposure even when the service has an exploitable lib due to the jailed nature of containers. Assuming "latest introduces new problems" too often builds an aversion to patching which can lead to worse issues down the road. >And, ultimately, Docker lets that not be my problem, because they have to deal with it when the next leftpad happens. So, yeah: they should have at it. Exactly! The only one responsible for libs are the parties directly leveraging them. Not that developers shouldn't make that info known. It has to be documented to remove the bus-factor of 1, and if it isn't the sysadmins should work with the devs to get it documented. > I guess I still think there's something to be said for the cathedral pace, though. I think it depends a lot on your resources as a department/company. You should always execute as quickly as feasible given your team size and work load. Otherwise technical debt has a way of piling up faster than you can offload it.
    - closeparen 10y ago
      At my organization, leftpad is not a reason for SRE to tell developers they can't use dependencies. Instead, leftpad is a reason for SRE to run internal package mirrors for all our supported packaging systems (debian, pip, glide, Maven, etc) and ship forks of the build tools so that when you reference a 3rd party dependency, the URL is rewritten to one at our internal mirror. The internal mirror, in turn, goes out and downloads anything it doesn't already have. They also maintain the base docker images that we're expected to use, as well as the docker build infrastructure. Facilitation with guardrails, not blockers.
- gnarbarian 10y ago
  Developers tend to fall into the trap of believing we are better than sysadmins but there is immense value in having talented admins around. I have recently experienced this first hand watching someone breeze through server archaeology and vitlrtualization tasks that I struggled through and may never have been able to accomplish in a reasonable amount of time. When admins and programmers recognize each other's strengths and play to them it is a rewarding experience. We just have to realize that we're on the same team. Also, docker (and containerization in general) is a wonderful thing for both of us. It decouples the fickle apps from systems (also moving targets) and the other apps which are constantly seeking out new and creative version incompatibilities. It makes migration and maintenance a much less frustrating endeavor with fewer surprises along the way.
- jasonlotito 10y ago
  > Because he maintains that installation and you don't? Or not. You're devs are on-call, aren't they? They are maintaining their own software, right?
xorcist 10y ago
> why should a sysadmin who doesn't even know Python Because a Python sysadmin has been through all the transitions of packaging systems, all the nasty corners of "backwards compatible" changes, and knows how underlying changes to the operating system will affect your code, what the storage behaves like under load, and why one tech is not "better" than another. If you really hired an admin (cough, "reliability engineer") for a Python codebase that doesn't know Python, well, that's a different question altogether. > I have regression tests to catch if an upgrade breaks anyyhing You don't know what you don't know. When you can reason about the multiple ways the above statement can fail, congratulations! You are now a seasoned sysadmin, the scorn of junior developers who just want to get things done (who incidentally read a great blog post the other day about a new packaging system that we should immediately transition to and by the way it's all backwards compatible).
- warbiscuit 10y ago
  I think a lot of this debate argues for the sysadmin role being part of the dev team. The only real way to get both constraints (production stability and update to date fixes/features) is to have fast feedback between the interest holders of two sides. In the python-specific case -- the requirements.in / .txt files for the virtualenv should be part of the software VCS, but the sysadmin should be able to edit & pin things just like the devs, so that they can bring their expertise to the container, rather than having to fight it. --- Mind you, my opinion might not scale - I'm part of a small enough team that I'm holding both those roles, but I try make sure to spent time wearing both "hats", so that one role doesn't get more man-hours clocked.
  - crdoconnor 10y ago
    IMO if a sysadmin wants to have visibility into the requirements.txt that's fine. If they want to enforce a policy of pinning versions, that's very welcome (though I would do that anyway). If they have specific, relevant comments about upgrades of specific packages - again, fine (though in practice they never do). If they want to be a gatekeeper for changes to that file they can fuck off.
- crdoconnor 10y ago
jstimpfle 10y ago
Another perspective: Why don't you want your software to be compatible with the system that your sysadmin provides? (Assuming that system is not completely obsolete). Minimize your dependencies. It's incidentally also what leads to clean code bases.
- brianwawok 10y ago
  > Minimize your dependencies. It's incidentally also what leads to clean code bases. You also crush velocity. Smart use of libraries lets you ship code 10x faster. Two identical businesses.. one writes all their own code, one is smart about using libraries. Which one makes it to IPO first, and which goes bankrupt?
  - noir_lord 10y ago
    That only holds true (as much as it does) if you assume the goal is always an IPO.
  - sly010 10y ago
    Are you arguing that you can keep velocity while building up technical debt? The company "smartly" using libraries might stuck maintaining a monster of dependencies that only ever was meant to be an MVP. It will require 10x more engineers and while they might move fast at the beginning, they will only slow down over time. The company minimizing their dependencies and paying attention to their stack will be able to add complexity over time without breaking a sweat. Their costs will be 10th the cost, and they will be able to run profitably. I am not anti-library or anything, but e.g. adding sci-py to your python project just because you need a gaussian function in one place in your code is just lazy. Managing dependencies wisely is one of the hardest thing in software development. It's right after cache invalidation and naming things ;)
  - deleted 10y ago
    [deleted]
  - prodigal_erik 10y ago
    If the world has already written 90% of the code I need, how likely is it that the remaining 10% is valuable enough to make a viable business?
thebeardisred 10y ago
To call out something that nobody else mentioned: because when you to use that newer version of pycurl or pyASN1 it's probably going to break the system level tools for running patches, handling license enforcement, and being able to keep the database online for other teams. I've had this problem over and over again (biggest one last being with the US Census). Folks insisted on upgrading a python library and auth to the hosts stopped working.
user5994461 10y ago
> That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? What if the sysadmin can code in the same language as you, faster and with less bugs?
- dozzie 10y ago
  Better yet: what if the sysadmin can code better and faster in the same language and also three others?
  - user5994461 10y ago
    That would be me!

bandrami 10y ago

why should a sysadmin who doesn't even know Python tell me what version of a library I will use Because he maintains that installation and you don't? But, yeah, that's why virtualenv, Docker, etc. were invented, because devs kept getting sick of installations having consequences. What does a sysadmin have to approve or deny an upgrade? Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Read the changelog and look for implementation problems. Read fora and look for performance problems people are reporting. Yes, beards get stroked during this process, but time and again we see that developers refuse to do this, and wind up coming to us when they break something because of that...

e40 10y ago
That's the tail wagging the dog (and I manage the sys admin department at my company).
btreecat 10y ago
>Because he maintains that installation and you don't? So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK? >Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it. >Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Ok this is where I completely agree with you as we have been working on this at our company. My personal solution seems pretty logical though so hear me out. 1) Build a docker file that fully documents the install of your service as well as any OS level dependencies. Ensure that any config files are external to the container to allow sysadmin access. 2) Document in a central location (say an internal wiki) what the external services, servers, repositories, developers, and admins are responsible for the service. 3) Automate builds of containers from repo and add automated testing post containerization. 4) Sysadmins monitor repositories for changes to docker files or wiki articles for new services, databases and libraries as well as taking note of library versions. If an issue with a particular lib or service is discovered, the config files can be edited to point to a new service. Or a new container build can be triggered with zero changes to the source code, but a forced update to the OS packages for the container. In a tight situation where a developer might not be available on-call, the sysadmins have more control over a similar proprietary product but don't have a workflow for messing with source code (which they are likely not familiar with regardless). There are solutions to the issues you raise (often trivial ones at that), they just require an adjustment to workflow and an increase in communication between developers and their sysadmins.
- bandrami 10y ago
  I see your point, and like I said I agree this is why Docker was invented and it's the best-in-breed at what it does (namely, being a tourniquet for a self-inflicted wound). My biggest concern really isn't "my problem" since I'm in ops: it's the leftpad worry. I still have teams starting Dockerfiles with "FROM centos:latest" because that's just the mindset they have: "Latest will fix any problems" rather than "Latest will introduce new problems". And, ultimately, Docker lets that not be my problem, because they have to deal with it when the next leftpad happens. So, yeah: they should have at it. I guess I still think there's something to be said for the cathedral pace, though.
  - btreecat 10y ago
    >I still have teams starting Dockerfiles with "FROM centos:latest" because that's just the mindset they have: "Latest will fix any problems" rather than "Latest will introduce new problems". Well there are two ways to approach this IMO. One be proactive. Create them a vetted centos or whatever OS environment for them to base off. The problem is, if you don't keep on top of it as a sysadmin, the developers will just figure out another way to wall you off. Alternatively, accept that it doesn't matter what underlying OS they use, because a patched OS >> than unpatched and that when done correctly there is minimal exposure even when the service has an exploitable lib due to the jailed nature of containers. Assuming "latest introduces new problems" too often builds an aversion to patching which can lead to worse issues down the road. >And, ultimately, Docker lets that not be my problem, because they have to deal with it when the next leftpad happens. So, yeah: they should have at it. Exactly! The only one responsible for libs are the parties directly leveraging them. Not that developers shouldn't make that info known. It has to be documented to remove the bus-factor of 1, and if it isn't the sysadmins should work with the devs to get it documented. > I guess I still think there's something to be said for the cathedral pace, though. I think it depends a lot on your resources as a department/company. You should always execute as quickly as feasible given your team size and work load. Otherwise technical debt has a way of piling up faster than you can offload it.
  - closeparen 10y ago
    At my organization, leftpad is not a reason for SRE to tell developers they can't use dependencies. Instead, leftpad is a reason for SRE to run internal package mirrors for all our supported packaging systems (debian, pip, glide, Maven, etc) and ship forks of the build tools so that when you reference a 3rd party dependency, the URL is rewritten to one at our internal mirror. The internal mirror, in turn, goes out and downloads anything it doesn't already have. They also maintain the base docker images that we're expected to use, as well as the docker build infrastructure. Facilitation with guardrails, not blockers.
gnarbarian 10y ago
Developers tend to fall into the trap of believing we are better than sysadmins but there is immense value in having talented admins around. I have recently experienced this first hand watching someone breeze through server archaeology and vitlrtualization tasks that I struggled through and may never have been able to accomplish in a reasonable amount of time. When admins and programmers recognize each other's strengths and play to them it is a rewarding experience. We just have to realize that we're on the same team. Also, docker (and containerization in general) is a wonderful thing for both of us. It decouples the fickle apps from systems (also moving targets) and the other apps which are constantly seeking out new and creative version incompatibilities. It makes migration and maintenance a much less frustrating endeavor with fewer surprises along the way.
jasonlotito 10y ago
> Because he maintains that installation and you don't? Or not. You're devs are on-call, aren't they? They are maintaining their own software, right?

xorcist 10y ago

> why should a sysadmin who doesn't even know Python Because a Python sysadmin has been through all the transitions of packaging systems, all the nasty corners of "backwards compatible" changes, and knows how underlying changes to the operating system will affect your code, what the storage behaves like under load, and why one tech is not "better" than another. If you really hired an admin (cough, "reliability engineer") for a Python codebase that doesn't know Python, well, that's a different question altogether. > I have regression tests to catch if an upgrade breaks anyyhing You don't know what you don't know. When you can reason about the multiple ways the above statement can fail, congratulations! You are now a seasoned sysadmin, the scorn of junior developers who just want to get things done (who incidentally read a great blog post the other day about a new packaging system that we should immediately transition to and by the way it's all backwards compatible).

warbiscuit 10y ago
I think a lot of this debate argues for the sysadmin role being part of the dev team. The only real way to get both constraints (production stability and update to date fixes/features) is to have fast feedback between the interest holders of two sides. In the python-specific case -- the requirements.in / .txt files for the virtualenv should be part of the software VCS, but the sysadmin should be able to edit & pin things just like the devs, so that they can bring their expertise to the container, rather than having to fight it. --- Mind you, my opinion might not scale - I'm part of a small enough team that I'm holding both those roles, but I try make sure to spent time wearing both "hats", so that one role doesn't get more man-hours clocked.
- crdoconnor 10y ago
  IMO if a sysadmin wants to have visibility into the requirements.txt that's fine. If they want to enforce a policy of pinning versions, that's very welcome (though I would do that anyway). If they have specific, relevant comments about upgrades of specific packages - again, fine (though in practice they never do). If they want to be a gatekeeper for changes to that file they can fuck off.
crdoconnor 10y ago

jstimpfle 10y ago

Another perspective: Why don't you want your software to be compatible with the system that your sysadmin provides? (Assuming that system is not completely obsolete). Minimize your dependencies. It's incidentally also what leads to clean code bases.

brianwawok 10y ago
> Minimize your dependencies. It's incidentally also what leads to clean code bases. You also crush velocity. Smart use of libraries lets you ship code 10x faster. Two identical businesses.. one writes all their own code, one is smart about using libraries. Which one makes it to IPO first, and which goes bankrupt?
- noir_lord 10y ago
  That only holds true (as much as it does) if you assume the goal is always an IPO.
- sly010 10y ago
  Are you arguing that you can keep velocity while building up technical debt? The company "smartly" using libraries might stuck maintaining a monster of dependencies that only ever was meant to be an MVP. It will require 10x more engineers and while they might move fast at the beginning, they will only slow down over time. The company minimizing their dependencies and paying attention to their stack will be able to add complexity over time without breaking a sweat. Their costs will be 10th the cost, and they will be able to run profitably. I am not anti-library or anything, but e.g. adding sci-py to your python project just because you need a gaussian function in one place in your code is just lazy. Managing dependencies wisely is one of the hardest thing in software development. It's right after cache invalidation and naming things ;)
- deleted 10y ago
  [deleted]
- prodigal_erik 10y ago
  If the world has already written 90% of the code I need, how likely is it that the remaining 10% is valuable enough to make a viable business?

thebeardisred 10y ago

To call out something that nobody else mentioned: because when you to use that newer version of pycurl or pyASN1 it's probably going to break the system level tools for running patches, handling license enforcement, and being able to keep the database online for other teams. I've had this problem over and over again (biggest one last being with the US Census). Folks insisted on upgrading a python library and auth to the hosts stopped working.

user5994461 10y ago

> That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? What if the sysadmin can code in the same language as you, faster and with less bugs?

dozzie 10y ago
Better yet: what if the sysadmin can code better and faster in the same language and also three others?
- user5994461 10y ago
  That would be me!