4 ms·

Because you can embed credentials in a Jenkins installation (and when you use Jenkins, we do.) It inevitably gets connected to resources which can cost money,

by yebyen 5y ago

Because you can embed credentials in a Jenkins installation (and when you use Jenkins, we do.) It inevitably gets connected to resources which can cost money, and since it's an internal infrastructure system, it will inevitably be connected with resources which contain replicas of private information. "Because why not, it's an internal system which means it's perfectly safe, and we should really be testing with real user data, you know, for realism." And once you cross that bridge, you have wholly and truly gone completely radioactive. This is the intersection of financial risk/attack surface and private customer data, and execution of unproven code. If we send new Groovy scripts through an approval process, we can at least narrow the risk of accident or intentional disclosure by manually vetting scripts before they are run for the first time after changes, (or effective sandboxing so they cannot escape and access for example, any secrets that were not intended for them.) But, roundly, one can argue there's not much to be done, as it is an internal system, and each control we place in the way can easily be seen as an obstacle to getting the job done straightforwardly. Then think of this also: maintaining Jenkins is known to be an operational burden, to say the least. If you have different departments with their own independent need for Jenkins or something like it, neither of these departments is going to want to own the Jenkins instance if it means being generally responsible for uptime and upkeep. You can bet that someone higher up is going to see this as a tremendous opportunity to consolidate and save money. They're going to wind up running on the same environment together, operated by someone who has no idea what either of these departments is really up to. If you're lucky it's only two tenants, and that shared-service admin department responsible for the Jenkins instance is going to actively pursue the common interest of keeping things secure on behalf of everyone. What's more likely is, shared services platform team only ever hears from their Jenkins customers when something has ceased to function and their Admin access is needed, and for those people it's basically hair-on-fire can't-work-until the admin guy can be reached. After ten or fifteen times and some political pressure, their boss says this is taking too much of our department's time and we're not adding value, so they decide to shirk the admin duties, and give admin credentials to a "designee" from each team. Now through diffusion of responsibility, nobody is in charge, and everyone is too afraid of breaking anyone else's stuff to ever upgrade even the most basic stuff. That's right, now you have potentially private customer data, from multiple departments, with access to secrets that may control resources which can be scaled up to cost money, in an environment where code runs for the first time (potentially even code from third party contributors, but not likely if it's an internal instance... right? ...) code running before it has been tested by anyone, in a multi-tenant environment that nobody wants to pay or spend time to maintain, with competing czars that don't really have much incentive to talk to each other, where nobody has enough visibility to safely upgrade anything for fear it will break something and step on somebody else's toes, and if it goes down or stops functioning for basically any possible reason because of any of those teams, the maintenance of business is completely halted until we take care of it; really it's basically even worse than having cats and dogs living in the same house. Sure, you can implement policy to solve any one of these things individually, but if you write enough policy to make it truly safe, then people will hate you for all the red tape required to work with it, "why can't we do the easy thing which is technically possible" and as you can see, there is so much wrong possible that even the best policy is going to have to concede some things, to make this tremendous expense worthwhile "otherwise why are we even using (footgun) if I can't easily shoot myself in the foot whenever I need to?"

zmmmmm 5y ago
so then somewhat reading between the lines of your answer, the core of this issue arises from the fact that the groovy code is executing "in-process" with the rest of the jenkins server whereas other CI/CD solutions execute whatever customisations you are making outside of the actual server process itself, so there isn't a possibility that the code can "see" into parts of the code it shouldn't be able to. I am guessing that this enables Jenkins plugins to do a lot more than they could from outside the server process (literally see and manipulate everything that goes on inside) but it does make me wonder if indeed the tradeoff there is worth it and whether for the more routine cusotmisations it wouldn't be better if it was running these scripts outside of Jenkins itself.
- yebyen 5y ago
  I did a technical review of our Jenkins solution with a friendly neighborhood InfoSec agent once. He was interested in a few things: 1) how is the stored credential being encrypted (is it encrypted at rest), 2) is it real encryption (like, does it use an actual encryption algorithm or is someone confusing base64 or rot13 for encryption) 3) where is the encrypted data stored, where are the keys stored, is it properly on a machine that is not situated directly on the internet (users must access through VPN) 4) can I review the code that decrypts the secret (yes you can, there are plenty of walkthroughs and gists published by many different Jenkins fans and users that show where Jenkins stores its encryption keys, and how you can borrow them with POC code you can use to decrypt individual creds or even the whole store, assuming you have admin access to Jenkins) He didn't really ask or care about sandboxing at all. There are mitigations you can apply for any threat model, but the crux of the matter is that even InfoSec assumed that any Jenkins users would probably have admin access anyway, and that they should all be allowed to decrypt those secrets. Which happens to be exactly what they tell you in articles about how to manage your Jenkins server anyway: https://www.codurance.com/publications/2019/05/30/accessing-and-dumping-jenkins-credentials https://www.codurance.com/publications/2019/05/30/accessing-... > 7. Treat all credentials stored in Jenkins as plain text > Treat everyone with access to Jenkins as an admin user, and you will be fine. Once you give someone access, even read-only, to a Jenkins it's game over. All developers on a project should know all secrets anyway. The sandbox is just a mitigation that has been built in so that people who run untrusted code can do so with some degree of safety. Jenkins servers that may be exposed to untrusted contributors are a completely different animal than internal Jenkins servers. Different use modes experience different threat models differently. The fact that you seem to believe there is a technical mitigation possible which would save this is potentially part of the problem, too. You're obviously a well-educated person. Organizations of sufficiently large size that settle on Jenkins can fight an uphill battle here, since many mitigations can only reduce risks, but cannot eliminate them. Leadership roles may not be inclined to understand the threat model issue, there is a dichotomy experienced when Jenkins is used to build software in the open vs closed developer models. Running builds for any PR from a random fork of a public repo is not the same threat model as running builds for any branch or tag in a private repo, from an IC with write access. And yet the mechanical actions are quite the same, and there are only about two checkboxes in a typical configuration separating these models from each other. I hope we are rapidly reaching a place where companies can participate in both Open and Closed style of development, but if they are supported by Jenkins, I hope it's at least two separate Jenkins servers, because the threat models and risk mitigation that are needed for closed-access teams and open-source style public benefit repos are both completely different, even if their activities are not different. Maybe I am overstating the risks, but I have been a Jenkins admin before, when it was not in my job description, and it was not supported by leadership, who did not believe in the benefits of open development, and made their posture closed by default (as companies are often wont to do.) And as a result, I'm afraid, before I left that job behind, we dug the hole so deep in the wrong direction... if I could go back and do it again, I would certainly draw some bright lines in different places than we did, and I would not allow customer data in the dev instances. (You can do that kind of testing in a staging environment, where similar controls as production can be applied and reliably maintained. But you should do this as far away from Jenkins as humanly possible!)

by yebyen 5y ago

zmmmmm 5y ago

so then somewhat reading between the lines of your answer, the core of this issue arises from the fact that the groovy code is executing "in-process" with the rest of the jenkins server whereas other CI/CD solutions execute whatever customisations you are making outside of the actual server process itself, so there isn't a possibility that the code can "see" into parts of the code it shouldn't be able to. I am guessing that this enables Jenkins plugins to do a lot more than they could from outside the server process (literally see and manipulate everything that goes on inside) but it does make me wonder if indeed the tradeoff there is worth it and whether for the more routine cusotmisations it wouldn't be better if it was running these scripts outside of Jenkins itself.

yebyen 5y ago
I did a technical review of our Jenkins solution with a friendly neighborhood InfoSec agent once. He was interested in a few things: 1) how is the stored credential being encrypted (is it encrypted at rest), 2) is it real encryption (like, does it use an actual encryption algorithm or is someone confusing base64 or rot13 for encryption) 3) where is the encrypted data stored, where are the keys stored, is it properly on a machine that is not situated directly on the internet (users must access through VPN) 4) can I review the code that decrypts the secret (yes you can, there are plenty of walkthroughs and gists published by many different Jenkins fans and users that show where Jenkins stores its encryption keys, and how you can borrow them with POC code you can use to decrypt individual creds or even the whole store, assuming you have admin access to Jenkins) He didn't really ask or care about sandboxing at all. There are mitigations you can apply for any threat model, but the crux of the matter is that even InfoSec assumed that any Jenkins users would probably have admin access anyway, and that they should all be allowed to decrypt those secrets. Which happens to be exactly what they tell you in articles about how to manage your Jenkins server anyway: https://www.codurance.com/publications/2019/05/30/accessing-and-dumping-jenkins-credentials https://www.codurance.com/publications/2019/05/30/accessing-... > 7. Treat all credentials stored in Jenkins as plain text > Treat everyone with access to Jenkins as an admin user, and you will be fine. Once you give someone access, even read-only, to a Jenkins it's game over. All developers on a project should know all secrets anyway. The sandbox is just a mitigation that has been built in so that people who run untrusted code can do so with some degree of safety. Jenkins servers that may be exposed to untrusted contributors are a completely different animal than internal Jenkins servers. Different use modes experience different threat models differently. The fact that you seem to believe there is a technical mitigation possible which would save this is potentially part of the problem, too. You're obviously a well-educated person. Organizations of sufficiently large size that settle on Jenkins can fight an uphill battle here, since many mitigations can only reduce risks, but cannot eliminate them. Leadership roles may not be inclined to understand the threat model issue, there is a dichotomy experienced when Jenkins is used to build software in the open vs closed developer models. Running builds for any PR from a random fork of a public repo is not the same threat model as running builds for any branch or tag in a private repo, from an IC with write access. And yet the mechanical actions are quite the same, and there are only about two checkboxes in a typical configuration separating these models from each other. I hope we are rapidly reaching a place where companies can participate in both Open and Closed style of development, but if they are supported by Jenkins, I hope it's at least two separate Jenkins servers, because the threat models and risk mitigation that are needed for closed-access teams and open-source style public benefit repos are both completely different, even if their activities are not different. Maybe I am overstating the risks, but I have been a Jenkins admin before, when it was not in my job description, and it was not supported by leadership, who did not believe in the benefits of open development, and made their posture closed by default (as companies are often wont to do.) And as a result, I'm afraid, before I left that job behind, we dug the hole so deep in the wrong direction... if I could go back and do it again, I would certainly draw some bright lines in different places than we did, and I would not allow customer data in the dev instances. (You can do that kind of testing in a staging environment, where similar controls as production can be applied and reliably maintained. But you should do this as far away from Jenkins as humanly possible!)