3 ms·

Lets give a quasi specific example maybe.. lets say you have service X, is your ApiService 1.2, and it's in production. Lets say you have machine 1 and machin

by spotman 11y ago

Lets give a quasi specific example maybe.. lets say you have service X, is your ApiService 1.2, and it's in production. Lets say you have machine 1 and machine 2, both running ApiService 1.2 In puppet, you could use hiera to know what version of ApiService a machine should have. You can use mcollective to interrogate these machines to verify that puppet successfully installed ApiService 1.2 on these machines. Now, to answer the question "can i deploy ApiService 1.3 on machine 1", I think that that would be best answered by testing this in a development/staging environment with another machine that is not serving mission critical traffic. Here you would do the testing you need, and ensure it works, both in a installation capacity (say ApiService 1.3 needs a newer version of libxml2 or something), and in a general "does it pass QA" standpoint. It would be your package maintainers job to specify that ApiService 1.3 requires libxml2 version XYZ. Puppet/etc (or, specifically when dpkg runs via puppet) would fail to install the package if its not going to have libxml2 version XyZ. Being that it's your package, you would write it in a way that dpkg is able to install it on the target servers your targeting. So, you could even spell it out exactly in puppet to, "before installing ApiService-1.3.deb, install libxml2-2.x728.deb" If what your getting at, is that if ApiService 1.3 on machine 1, relies on libxml2 being updated on machine 2, then that sounds a bit brittle to me and should be refactored to be more forgiving. Maybe that helps make it more clear, if in doubt will look for more comments.

perlgeek 11y ago
Let me see if I got this one straight: So puppet has some infrastructure for querying package versions (and other facts) (mcollective), and a way to store the dependency data (hiera), but the actual management still has to be done manually (potentially by trial & error in a dev/test/canary environment). Right? If yes, that's good to know, but still less than I was hoping for.
- spotman 11y ago
  Yeah, i look at os-package dependency management as largely a problem that should be left up to the os' package manager I suppose. So, naturally just like any change, you would want to test the upgrade before rolling to prod. What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ? How is that different than it failing on dpkg? (its possible I'm not understanding..)
  - perlgeek 11y ago
    > What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ? In the past months we had several failures where new features weren't working in production, and it was usually due to neglecting to upgrade some components. So for example one developer creates a new feature in the front end, a second developer creates the backend service, notices a missing feature in a second backend service, and delegates that to a third developer. When the time comes to roll out the frontend service, the first two developers probably remember that a new version of the first backend service is required, but sometimes forget that there's also a dependency on the new version of the second backend service. So the feature goes into production, but fails because an indirectly dependent service wasn't upgraded as well. We only have one testing environment, not one per service, so we don't easily notice the problem in testing. So, if all the services where installed on machine machine, the developers would just need to declare the proper dependencies (which they can do at the time they discover them), and an "apt-get install <frontend-service>" would resolve the whole dependency chain, and either abort the upgrade if the new version of the second backend service isn't available, or upgrade it too. What I really want is the same easy of dependency management, even though the packages aren't all installed on the same machine. I simply want to declare all of a packages versioned dependencies, and then say "upgrade service X" and it'll automatically tell me what I need to deploy (or even do it for me). Currently our workaround is a wiki page in which we document the depenencies, and manually resolve them when the time comes for a deployment. But this is very frustrating, because the problem behind it is solved in any decent package manager -- but only in the single-machine case. So just because we run more than one machine, we have to reinvent the whole dependency management wheel. And it also frustrates me that I can't seem to find much discussion of this topic, even though I'm quite sure we're not the only ones who have these problems. Am I just looking in the wrong direction?
    - spotman 11y ago
      Thanks for explaining. This is a pain point I think everyone that does service oriented architecture has to some level. Most shops I have worked in seem to mostly attack this via procedure. Ie - release is OK'd by QA, then deploy ticket is made that specifies what versions go where, and then operations is responsible for executing this procedure and ensuring its sanity. If you have a lot of hands in the pot, (too many cooks), then this is bound to have issues when everyone is busy and things get overlooked. There is a lot of automation that can help however. Dependency management in this sense I think has to be more elaborate than just "is X installed or deployed", but you might also want to ask "is X healthy?" Just because a package has been installed doesn't mean its working. So that is in my opinion why there is no one-size-fits-all solution for this. Many releases are fully compatible with each other, and in simple applications probably even more so, but once you have many teams working on many services, it can be a nightmare. A lot of places have granular feature management. So, frontend can fail gracefully if feature is not supported on backend. Often backend supports more than 1 fronted version set. You can use versioned api's (like putting a version in an http header, or namespacing features like /api/v1/x, /api/v2/x). (it's been said a huge distributed app like facebook has something like 500 different versions of components all running concurrently) You can have your deployment automation do things like check to see if backend has required version (which would be something that you would have to put in hiera/equiv of what versions are required), and even more importantly if they are responding to health checks. One project I currently work on has a very distributed architecture, and every application has to conform to a health check endpoint that returns its state and its version. So we have scripts that do either rolling deploys or green/blue deploys depending on what we are deploying, and we can define the order in which services are deployed. We can deploy them concurrently/rolling, or independent/rolling, or either of those flavors non-rolling, or green/blue. So if we say we are deploying fronted3.1, backend4.2, and we are replacing frontend3.0 and backend 4.1, this would be seen by the deploy scripts (and operations team) as a single changeset for the environment. So if frontend3.1 needed backend4.1 to function, we would specify a independent rolling deployment, so for each service thats being deployed (starting with the backend), we would roll out server by server at a specified interval, and we would wait for that server to return a healthy health check with the specified version. If this fails the deploy is rolled back and notifications are sent. ^ with this working correctly, we have to have the understanding with the application developers that we will never release a backend that is not backwards compatible with any components that rely on this. (beyond a cutoff version, which those do exist too) In practice this works quite well, but is not magic, and definitely requires some up front development of automation. So, sorry for the long post, I think this is a long winded way of saying, my opinion is that this belongs in a configuration management, as the logic encapsulating "what version goes where" gets more complex over time, and so some code, logic, decision making, and failsafes all have to be carefully thought out.

by spotman 11y ago

perlgeek 11y ago

Let me see if I got this one straight: So puppet has some infrastructure for querying package versions (and other facts) (mcollective), and a way to store the dependency data (hiera), but the actual management still has to be done manually (potentially by trial & error in a dev/test/canary environment). Right? If yes, that's good to know, but still less than I was hoping for.

spotman 11y ago
Yeah, i look at os-package dependency management as largely a problem that should be left up to the os' package manager I suppose. So, naturally just like any change, you would want to test the upgrade before rolling to prod. What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ? How is that different than it failing on dpkg? (its possible I'm not understanding..)
- perlgeek 11y ago
  > What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ? In the past months we had several failures where new features weren't working in production, and it was usually due to neglecting to upgrade some components. So for example one developer creates a new feature in the front end, a second developer creates the backend service, notices a missing feature in a second backend service, and delegates that to a third developer. When the time comes to roll out the frontend service, the first two developers probably remember that a new version of the first backend service is required, but sometimes forget that there's also a dependency on the new version of the second backend service. So the feature goes into production, but fails because an indirectly dependent service wasn't upgraded as well. We only have one testing environment, not one per service, so we don't easily notice the problem in testing. So, if all the services where installed on machine machine, the developers would just need to declare the proper dependencies (which they can do at the time they discover them), and an "apt-get install <frontend-service>" would resolve the whole dependency chain, and either abort the upgrade if the new version of the second backend service isn't available, or upgrade it too. What I really want is the same easy of dependency management, even though the packages aren't all installed on the same machine. I simply want to declare all of a packages versioned dependencies, and then say "upgrade service X" and it'll automatically tell me what I need to deploy (or even do it for me). Currently our workaround is a wiki page in which we document the depenencies, and manually resolve them when the time comes for a deployment. But this is very frustrating, because the problem behind it is solved in any decent package manager -- but only in the single-machine case. So just because we run more than one machine, we have to reinvent the whole dependency management wheel. And it also frustrates me that I can't seem to find much discussion of this topic, even though I'm quite sure we're not the only ones who have these problems. Am I just looking in the wrong direction?
  - spotman 11y ago
    Thanks for explaining. This is a pain point I think everyone that does service oriented architecture has to some level. Most shops I have worked in seem to mostly attack this via procedure. Ie - release is OK'd by QA, then deploy ticket is made that specifies what versions go where, and then operations is responsible for executing this procedure and ensuring its sanity. If you have a lot of hands in the pot, (too many cooks), then this is bound to have issues when everyone is busy and things get overlooked. There is a lot of automation that can help however. Dependency management in this sense I think has to be more elaborate than just "is X installed or deployed", but you might also want to ask "is X healthy?" Just because a package has been installed doesn't mean its working. So that is in my opinion why there is no one-size-fits-all solution for this. Many releases are fully compatible with each other, and in simple applications probably even more so, but once you have many teams working on many services, it can be a nightmare. A lot of places have granular feature management. So, frontend can fail gracefully if feature is not supported on backend. Often backend supports more than 1 fronted version set. You can use versioned api's (like putting a version in an http header, or namespacing features like /api/v1/x, /api/v2/x). (it's been said a huge distributed app like facebook has something like 500 different versions of components all running concurrently) You can have your deployment automation do things like check to see if backend has required version (which would be something that you would have to put in hiera/equiv of what versions are required), and even more importantly if they are responding to health checks. One project I currently work on has a very distributed architecture, and every application has to conform to a health check endpoint that returns its state and its version. So we have scripts that do either rolling deploys or green/blue deploys depending on what we are deploying, and we can define the order in which services are deployed. We can deploy them concurrently/rolling, or independent/rolling, or either of those flavors non-rolling, or green/blue. So if we say we are deploying fronted3.1, backend4.2, and we are replacing frontend3.0 and backend 4.1, this would be seen by the deploy scripts (and operations team) as a single changeset for the environment. So if frontend3.1 needed backend4.1 to function, we would specify a independent rolling deployment, so for each service thats being deployed (starting with the backend), we would roll out server by server at a specified interval, and we would wait for that server to return a healthy health check with the specified version. If this fails the deploy is rolled back and notifications are sent. ^ with this working correctly, we have to have the understanding with the application developers that we will never release a backend that is not backwards compatible with any components that rely on this. (beyond a cutoff version, which those do exist too) In practice this works quite well, but is not magic, and definitely requires some up front development of automation. So, sorry for the long post, I think this is a long winded way of saying, my opinion is that this belongs in a configuration management, as the logic encapsulating "what version goes where" gets more complex over time, and so some code, logic, decision making, and failsafes all have to be carefully thought out.