6 ms·

You would architect your application to not be so brittle that deploying one wrong package takes the service down. For example if you have to make breaking cha

by spotman 11y ago

You would architect your application to not be so brittle that deploying one wrong package takes the service down. For example if you have to make breaking changes try to make your backend smart enough to handle web2.2 and web2.3. Tools like configuration management (chef, puppet, ansible, etc) can help you roll out the different versions , and interrogate servers to see what is installed on them. So in the case of rpm/deb, if you need to update a package and it comes with dependencies I would let the package manager resolve them. But specifically if your packaging your own application components in this format you would not have control to tell server A to be dependent on server B's dpkg state. That would be complicated and brittle. You want to use rpm/deb/etc to resolve local dependencies only. For setting up infrastructure that needs supporting infrastructure at a certain version to work well this must coordinated at some level even if by hand. These days there is more than several handfuls of different ways to approach this. I recommend chef or ansible if your new to configuration management. Edit: oh you use puppet already I see. Just have puppet update packages as needed, and consider using it for deploys. You can group systems together conceptually and do all kinds of things even blue/green deploys, rolling deploys etc. you can use mcollective to see what packages are where.

perlgeek 11y ago
Thanks for your reply. It's clear that you've put a lot of thought into it, but I'm still struggling make it useful for me. > These days there is more than several handfuls of different ways to approach this. I recommend chef or ansible if your new to configuration management. As far as I can tell, neither ansible nor chef really help me with answering questions like "can I deploy service X in version Y to machine Z, or are dependencies missing?". Or do they? If yes, how? Or did I misunderstood what exactly they should help me with?
- spotman 11y ago
  Lets give a quasi specific example maybe.. lets say you have service X, is your ApiService 1.2, and it's in production. Lets say you have machine 1 and machine 2, both running ApiService 1.2 In puppet, you could use hiera to know what version of ApiService a machine should have. You can use mcollective to interrogate these machines to verify that puppet successfully installed ApiService 1.2 on these machines. Now, to answer the question "can i deploy ApiService 1.3 on machine 1", I think that that would be best answered by testing this in a development/staging environment with another machine that is not serving mission critical traffic. Here you would do the testing you need, and ensure it works, both in a installation capacity (say ApiService 1.3 needs a newer version of libxml2 or something), and in a general "does it pass QA" standpoint. It would be your package maintainers job to specify that ApiService 1.3 requires libxml2 version XYZ. Puppet/etc (or, specifically when dpkg runs via puppet) would fail to install the package if its not going to have libxml2 version XyZ. Being that it's your package, you would write it in a way that dpkg is able to install it on the target servers your targeting. So, you could even spell it out exactly in puppet to, "before installing ApiService-1.3.deb, install libxml2-2.x728.deb" If what your getting at, is that if ApiService 1.3 on machine 1, relies on libxml2 being updated on machine 2, then that sounds a bit brittle to me and should be refactored to be more forgiving. Maybe that helps make it more clear, if in doubt will look for more comments.
  - perlgeek 11y ago
    Let me see if I got this one straight: So puppet has some infrastructure for querying package versions (and other facts) (mcollective), and a way to store the dependency data (hiera), but the actual management still has to be done manually (potentially by trial & error in a dev/test/canary environment). Right? If yes, that's good to know, but still less than I was hoping for.
    - spotman 11y ago
      Yeah, i look at os-package dependency management as largely a problem that should be left up to the os' package manager I suppose. So, naturally just like any change, you would want to test the upgrade before rolling to prod. What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ? How is that different than it failing on dpkg? (its possible I'm not understanding..)
      - perlgeek 11y ago
        > What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ? In the past months we had several failures where new features weren't working in production, and it was usually due to neglecting to upgrade some components. So for example one developer creates a new feature in the front end, a second developer creates the backend service, notices a missing feature in a second backend service, and delegates that to a third developer. When the time comes to roll out the frontend service, the first two developers probably remember that a new version of the first backend service is required, but sometimes forget that there's also a dependency on the new version of the second backend service. So the feature goes into production, but fails because an indirectly dependent service wasn't upgraded as well. We only have one testing environment, not one per service, so we don't easily notice the problem in testing. So, if all the services where installed on machine machine, the developers would just need to declare the proper dependencies (which they can do at the time they discover them), and an "apt-get install <frontend-service>" would resolve the whole dependency chain, and either abort the upgrade if the new version of the second backend service isn't available, or upgrade it too. What I really want is the same easy of dependency management, even though the packages aren't all installed on the same machine. I simply want to declare all of a packages versioned dependencies, and then say "upgrade service X" and it'll automatically tell me what I need to deploy (or even do it for me). Currently our workaround is a wiki page in which we document the depenencies, and manually resolve them when the time comes for a deployment. But this is very frustrating, because the problem behind it is solved in any decent package manager -- but only in the single-machine case. So just because we run more than one machine, we have to reinvent the whole dependency management wheel. And it also frustrates me that I can't seem to find much discussion of this topic, even though I'm quite sure we're not the only ones who have these problems. Am I just looking in the wrong direction?

by spotman 11y ago

perlgeek 11y ago

Thanks for your reply. It's clear that you've put a lot of thought into it, but I'm still struggling make it useful for me. > These days there is more than several handfuls of different ways to approach this. I recommend chef or ansible if your new to configuration management. As far as I can tell, neither ansible nor chef really help me with answering questions like "can I deploy service X in version Y to machine Z, or are dependencies missing?". Or do they? If yes, how? Or did I misunderstood what exactly they should help me with?

spotman 11y ago
Lets give a quasi specific example maybe.. lets say you have service X, is your ApiService 1.2, and it's in production. Lets say you have machine 1 and machine 2, both running ApiService 1.2 In puppet, you could use hiera to know what version of ApiService a machine should have. You can use mcollective to interrogate these machines to verify that puppet successfully installed ApiService 1.2 on these machines. Now, to answer the question "can i deploy ApiService 1.3 on machine 1", I think that that would be best answered by testing this in a development/staging environment with another machine that is not serving mission critical traffic. Here you would do the testing you need, and ensure it works, both in a installation capacity (say ApiService 1.3 needs a newer version of libxml2 or something), and in a general "does it pass QA" standpoint. It would be your package maintainers job to specify that ApiService 1.3 requires libxml2 version XYZ. Puppet/etc (or, specifically when dpkg runs via puppet) would fail to install the package if its not going to have libxml2 version XyZ. Being that it's your package, you would write it in a way that dpkg is able to install it on the target servers your targeting. So, you could even spell it out exactly in puppet to, "before installing ApiService-1.3.deb, install libxml2-2.x728.deb" If what your getting at, is that if ApiService 1.3 on machine 1, relies on libxml2 being updated on machine 2, then that sounds a bit brittle to me and should be refactored to be more forgiving. Maybe that helps make it more clear, if in doubt will look for more comments.
- perlgeek 11y ago
  Let me see if I got this one straight: So puppet has some infrastructure for querying package versions (and other facts) (mcollective), and a way to store the dependency data (hiera), but the actual management still has to be done manually (potentially by trial & error in a dev/test/canary environment). Right? If yes, that's good to know, but still less than I was hoping for.
  - spotman 11y ago
    Yeah, i look at os-package dependency management as largely a problem that should be left up to the os' package manager I suppose. So, naturally just like any change, you would want to test the upgrade before rolling to prod. What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ? How is that different than it failing on dpkg? (its possible I'm not understanding..)
    - perlgeek 11y ago
      > What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ? In the past months we had several failures where new features weren't working in production, and it was usually due to neglecting to upgrade some components. So for example one developer creates a new feature in the front end, a second developer creates the backend service, notices a missing feature in a second backend service, and delegates that to a third developer. When the time comes to roll out the frontend service, the first two developers probably remember that a new version of the first backend service is required, but sometimes forget that there's also a dependency on the new version of the second backend service. So the feature goes into production, but fails because an indirectly dependent service wasn't upgraded as well. We only have one testing environment, not one per service, so we don't easily notice the problem in testing. So, if all the services where installed on machine machine, the developers would just need to declare the proper dependencies (which they can do at the time they discover them), and an "apt-get install <frontend-service>" would resolve the whole dependency chain, and either abort the upgrade if the new version of the second backend service isn't available, or upgrade it too. What I really want is the same easy of dependency management, even though the packages aren't all installed on the same machine. I simply want to declare all of a packages versioned dependencies, and then say "upgrade service X" and it'll automatically tell me what I need to deploy (or even do it for me). Currently our workaround is a wiki page in which we document the depenencies, and manually resolve them when the time comes for a deployment. But this is very frustrating, because the problem behind it is solved in any decent package manager -- but only in the single-machine case. So just because we run more than one machine, we have to reinvent the whole dependency management wheel. And it also frustrates me that I can't seem to find much discussion of this topic, even though I'm quite sure we're not the only ones who have these problems. Am I just looking in the wrong direction?