8 ms·

If you're considering using this (or any similar) tool, please keep in mind that you're adding a lot of attack surface. Your Ansible master is one of the most

by lima 9y ago

If you're considering using this (or any similar) tool, please keep in mind that you're adding a lot of attack surface. Your Ansible master is one of the most critical (if not the most critical) machine in your network. It has unrestricted access to everything, including a very detailed list of what and where everything is. If it's compromised, it's over. Something like Ansible Tower adds a LOT of attack surface. Instead of a locked-down server exposing a public key-only SSH port, you suddenly have a whole web application stack in there. Your browser and every Chrome extension with full page access now has root access to your network (and don't get me started on potential CSRF or XSS vulnerabilities...). If you don't need any of the enterprise-ey features, just might be better off with plain Ansible and Ara[1], with the latter running on a separate machine. A "sudoers" rule is much more secure than any ACL in a web application backend. If you do want to use Tower, you need to think about these risks and how you mitigate them. Of course, this also applies to any similar tool like Jenkins, Rundeck, CircleCI or whatever if you give them production credentials. [1]: https://ara.readthedocs.io/en/latest/ https://ara.readthedocs.io/en/latest/

crymer11 9y ago
> It has unrestricted access to everything It only has unrestricted access to what you give it unrestricted access to. If your ansible playbooks use a user that doesn't have root access and limited sudo powers, then it's not much different from using Ara.
- mikepurvis 9y ago
  That's fair, but Ansible culture is to do everything as sudo, and traditionally it was pretty painful to execute only portions of a playbook as the sudo. Though it looks like that's gotten better in more recent times: http://docs.ansible.com/ansible/latest/become.html http://docs.ansible.com/ansible/latest/become.html
  - emmelaich 9y ago
    The big improvement was the ability to turn sudo off/on for a particular task. Not sure when that was introduced. 1.9.something?
  - ghjm 9y ago
    Even if your playbooks run everything as sudo, that doesn't mean you have to grant AWX/Tower users the ability to create arbitrary playbooks or run anything else as sudo. You certainly can do that, but the point of the RBAC feature of Tower is that you don't have to.
- lima 9y ago
  If you're using Ansible for configuration management, it'll need root access either way.
  - user5994461 9y ago
    It doesn't need root access. It's just that most of the useful things to do on a server require root.
  - rmenr 9y ago
    To do anything useful, you'd have to give it sudoer privs - this is true. If you're squeamish about that, on principle, then Salt, Chef, and Puppet would be problematic as well. The only logical choice would be to provision and bake all of your images and push the AMI up to AWS...in which case Ansible would be an excellent utility anyway.
peterwwillis 9y ago
Given the context, there's not a big difference between a website getting owned and an admin's browser getting owned. When someone develops an exploit that breaks out of sandbox, they can do whatever they want. Also, this assumes access to Tower is unlimited. ACLs applied in Tower should be able to limit what user has what access.
- lima 9y ago
  All it takes is a compromised browser extension. Happens all the time: https://threatpost.com/seven-more-chrome-extensions-compromised/127458/ https://threatpost.com/seven-more-chrome-extensions-compromi...
  - peterwwillis 9y ago
    And a compromised node.js dependency could own a non-browser-controlled install. But you don't have to use node.js apps, and you don't need to use browser extensions.
    - lima 9y ago
      Sure, but now you'll need to make sure noone in your company who has full access to Tower is using browser extensions. It's absolutely possible to use Ansible Tower securely, it just takes some effort.
pm90 9y ago
This is a great point. Although...other CI systems already have that kind of privilege, right? e.g. Chef has a master node if I'm not mistaken. In my experience, ansible playbooks are great when run from a more general purpose task runner like jenkins, which then has permissions to access/modify one's production environment. I don't think I would personally use tower unless it provides something much much better than running ansible tasks in Jenkins... it would be just too much of a hassle to get the security/compliance aspects right.
- snuxoll 9y ago
  Tower is great for presenting playbooks to other users less familiar with ansible, besides you have the same attack surface with Jenkins anyway.
  - geerlingguy 9y ago
    And worse, in many places, due to most Jenkins instances (at least IMO) having dozens of plugins, many usually severely outdated.
- eropple 9y ago
  Chef Server has a central server. Chef Zero doesn't, especially when using tools like cfn-init/cfn-hup and S3-backed minimart to self-bootstrap. This is the approach I take, and it's the approach I see becoming more and more common. Letting nodes figure out how to deal with their own problems (better able to auto-scale, more fault tolerant) is, to me, much better than having Jenkins or whatever have to SSH into them in the first place. You can do it with Ansible, if you're going home-roll it, but I haven't seen too many people do so.
  - oblio 9y ago
    Does anyone actually use Ansible server-less? Can it even be done? (Edit: my bad, ignore this comment, I misphrased my question; the real question should have been: can you run Ansible locally on the target server, just like chef-solo/chef-zero?)
deleted 9y ago
[deleted]
rightos 9y ago
Yeah, I've largely avoided using Tower for this reason, but you could theoretically get it up to an equivalent or at least comparable security level simply by SSH tunneling it or putting it behind a good VPN. Host compromise is still a risk even with ssh.
- jdc0589 9y ago
  a tunnel does not protect against any of that. OP was already assuming it wasn't publicly accessible, or at least was ACLd to your company's public IPs.
  - rightos 9y ago
    > Instead of a locked-down server exposing a public key-only SSH port, you suddenly have a whole web application stack in there. There's no webapp stack to attack if you're only able to access it via a tunnel. If you're assuming the machine you're tunneling it to is compromised, there are bigger issues at play - ones that would compromise even a plain ssh link. I'm talking a direct tunnel from your ansible master to the host you're planning to use it on, not say, into your company's network at large.
Eridrus 9y ago
Maybe there should just be a trivial Electron wrapper around the Tower UI so that XSS/CSRF/extensions are not an issue.
chucky_z 9y ago
If you're always using `become_user: root` in your playbooks, maybe think about writing them differently! I have experience with Tower, and have never configured users to have any level of access beyond very, very basic SSH access.
- Spivak 9y ago
  Sure, but from an ops perspective you won't really be able to do much sysadmining without root privileges.
  - emmelaich 9y ago
    My method is to have a "app superuser"; a user which is a group admin* for every group that I run apps as. It can then do everything required with a few exceptions. The exceptions are creating init/systemd files (associated startup/shutdown) and creating the necessary top-level directories for new apps. Those can be allowed with a few careful sudo rules. It can be a little convoluted but much more secure. * "gpasswd -A <user>,,, <group>" Note: to be a group admin doesn't necessarily mean you're in that group; it means that you can put yourself in and out of that group as required.
emmelaich 9y ago
Good points, though unlike some other similar apps, Ansible Tower has an selinux policy.
btown 9y ago
Separate Chrome profile, no addons, proxied thru a dedicated VPN whose exit IP is the only one whitelisted by Tower or some reverse proxy in front of it. Or... just trust that of all the exploits someone could deliver to an addon, a specific attack on Tower will be pretty low on the list.
- rmenr 9y ago
  Yeah. What ^^ he said. Aside from how tightly locked down my localDev environment and development host are, I do hop into a sanitized Chrome profile (or just use an incognito window) when I'm doing things that require work through a GUI like the AWS console, Jenkins, GSuite Admin, etc... Couple in a sound architectural model, and you're good. Unless you're working on some super duper top-secret stuff...but if we're talking about compliance and security at the CIS standards level, to the best of your ability and knowledge, you should be good. Keep in mind, of course, that large, blanket vulns show their faces every so often - like the KRACK vuln - which renders all of our preparation and paranoia mostly null. Example (of a neurotically paranoid architecture): Dedicated VPN resides in a dedicated AWS account and fronts all traffic to all hosts in all of your organization accounts across AWS. The only services exposed publicly other than your VPN service are the public-facing services you run if you're hosting some SAAS, for example. Even then, your LB's better be public, and your EC2's behind them better be private. Yay port 443 and LB to instance certificate encryption. Ansible Tower lives in some other "Internal Services" AWS account, and all traffic ingressed to hosts in that account is fronted by a bastion host and traffic proxy. The bastion/proxy is governed by a set of strict VPC route tables, and security groups that are set to only ever permit or accept traffic that corresponds to the IP addresses of your VPN. If you wanna get even crazier than that, you can also strictly control egress rules for your security groups - so even if someone got in, they'd be hard pressed to get the data out of your systems without doing some acrobatics. In general, if vulnerabilities @ the Tower server level are your concern, addressing that with architectural best-practices and network-level controls for locking up access and traffic to Tower is reasonable and gets the job done. It would be the same as securing anything else that's sensitive running in your infra, like Jenkins or RunDeck. I think RedHat's done an excellent job, and a huge service to the community by finally making good on their promise of open sourcing Tower.
dmsimard 9y ago
Hi! I'm the author of ARA and just wanted to thank you for mentioning it :) Full disclosure: I'm a Software Engineer at Red Hat. Tower has a lot of great features but I hate to concede that it does come with some of the drawbacks you mentioned. I think it comes down to weighing the pros and cons, making sure you are aware of the cons and putting measures in place to avoid problems. It also heavily depends on your use case, Tower provides things like RBAC/ACL, auditing, scheduling, online editing and execution, an API, etc. If you happen to need none of that and you're perfectly happy with just using Ansible from your command line, there's probably little incentive for you to use it. If all you need is reporting, ARA is simple, easy to install and setup, doesn't get in your way and just records things transparently. I figured I might as well take the opportunity of posting to leave a video demo [1], even if a little outdated, as well as an example of live report that ara provides [2]. Let me know if you have any questions ! [1]: https://www.youtube.com/watch?v=k3i8VPCanGo https://www.youtube.com/watch?v=k3i8VPCanGo [2]: http://logs.openstack.org/21/474721/7/check/gate-openstack-ansible-openstack-ansible-ceph-ubuntu-xenial/779e047/logs/ara/ http://logs.openstack.org/21/474721/7/check/gate-openstack-a...
jerrac 9y ago
So, I've recently been evaluating AWX for my workplace. Thus, making sure it's secure is definitely important to me. That said, I'm having a hard time seeing how a compromised browser access AWX could give anyone full access to everything. Attackers could effect anything a playbook has access to, but considering how varied playbooks are, it would be really hard to guess the right values to pass into a job to get what you want. Can someone elaborate on how an attacker could do real damage using a browser based attack?
- deleted 9y ago
  [deleted]
- ibotty 9y ago
  Simply add to any playbook a play with `all` hosts that drops your remote shell.
  - AlanSE 9y ago
    You would need source control access to do that.
  - jerrac 9y ago
    I had thought of the shell module. But the attacker would still have to know your playbook is using that module, and they'd have to know the variables you are passing into that module in order to override them. So, can you explain how an attacker would figure that information out? Assuming the playbooks are stored in source control like the Tower docs recommend, and all your variable values are mostly in source control as well.
dimitar 9y ago
You can mitigate part of the risks if you use an SSH tunnel to access Tower via port 22.
ghjm 9y ago
You are absolutely correct that these factors should be carefully considered prior to any deployment of AWX or Tower. You are granting Tower a lot of authority to your networks and systems, and should not do so without good reason. If you don't need the features of AWX/Tower, then the best practice is not to use it. There's a tremendous amount you can do with Ansible itself, without AWX/Tower, and lots of people use it happily that way. That being said, I think you are overstating some of the risks. You don't need to grant every Tower user root access to everything on your network. If you're at a scale where Tower makes sense, you probably already have some sort of separation of privileges. I agree that a malicious Chrome extension could do a lot of damage - just like it could with all your other management tools like DRAC/ILO, network equipment GUIs and so forth. Yes, every web application carries the risk of CSRF/XSS or other vulnerabilities, and Tower is not immune to this, but we do spend a lot of time worrying about it, conducting audits, etc. If your operation can succeed with nothing but a sudoers rule and command-line Ansible, then by all means use that. Nobody wants to force AWX/Tower on people who don't need it. But if you do need the feature set of Tower, I think it's one of the safer options available.

by lima 9y ago

crymer11 9y ago

> It has unrestricted access to everything It only has unrestricted access to what you give it unrestricted access to. If your ansible playbooks use a user that doesn't have root access and limited sudo powers, then it's not much different from using Ara.

mikepurvis 9y ago
That's fair, but Ansible culture is to do everything as sudo, and traditionally it was pretty painful to execute only portions of a playbook as the sudo. Though it looks like that's gotten better in more recent times: http://docs.ansible.com/ansible/latest/become.html http://docs.ansible.com/ansible/latest/become.html
- emmelaich 9y ago
  The big improvement was the ability to turn sudo off/on for a particular task. Not sure when that was introduced. 1.9.something?
- ghjm 9y ago
  Even if your playbooks run everything as sudo, that doesn't mean you have to grant AWX/Tower users the ability to create arbitrary playbooks or run anything else as sudo. You certainly can do that, but the point of the RBAC feature of Tower is that you don't have to.
lima 9y ago
If you're using Ansible for configuration management, it'll need root access either way.
- user5994461 9y ago
  It doesn't need root access. It's just that most of the useful things to do on a server require root.
- rmenr 9y ago
  To do anything useful, you'd have to give it sudoer privs - this is true. If you're squeamish about that, on principle, then Salt, Chef, and Puppet would be problematic as well. The only logical choice would be to provision and bake all of your images and push the AMI up to AWS...in which case Ansible would be an excellent utility anyway.

peterwwillis 9y ago

Given the context, there's not a big difference between a website getting owned and an admin's browser getting owned. When someone develops an exploit that breaks out of sandbox, they can do whatever they want. Also, this assumes access to Tower is unlimited. ACLs applied in Tower should be able to limit what user has what access.

lima 9y ago
All it takes is a compromised browser extension. Happens all the time: https://threatpost.com/seven-more-chrome-extensions-compromised/127458/ https://threatpost.com/seven-more-chrome-extensions-compromi...
- peterwwillis 9y ago
  And a compromised node.js dependency could own a non-browser-controlled install. But you don't have to use node.js apps, and you don't need to use browser extensions.
  - lima 9y ago
    Sure, but now you'll need to make sure noone in your company who has full access to Tower is using browser extensions. It's absolutely possible to use Ansible Tower securely, it just takes some effort.

pm90 9y ago

This is a great point. Although...other CI systems already have that kind of privilege, right? e.g. Chef has a master node if I'm not mistaken. In my experience, ansible playbooks are great when run from a more general purpose task runner like jenkins, which then has permissions to access/modify one's production environment. I don't think I would personally use tower unless it provides something much much better than running ansible tasks in Jenkins... it would be just too much of a hassle to get the security/compliance aspects right.

snuxoll 9y ago
Tower is great for presenting playbooks to other users less familiar with ansible, besides you have the same attack surface with Jenkins anyway.
- geerlingguy 9y ago
  And worse, in many places, due to most Jenkins instances (at least IMO) having dozens of plugins, many usually severely outdated.
eropple 9y ago
Chef Server has a central server. Chef Zero doesn't, especially when using tools like cfn-init/cfn-hup and S3-backed minimart to self-bootstrap. This is the approach I take, and it's the approach I see becoming more and more common. Letting nodes figure out how to deal with their own problems (better able to auto-scale, more fault tolerant) is, to me, much better than having Jenkins or whatever have to SSH into them in the first place. You can do it with Ansible, if you're going home-roll it, but I haven't seen too many people do so.
- oblio 9y ago
  Does anyone actually use Ansible server-less? Can it even be done? (Edit: my bad, ignore this comment, I misphrased my question; the real question should have been: can you run Ansible locally on the target server, just like chef-solo/chef-zero?)

deleted 9y ago

[deleted]

rightos 9y ago

Yeah, I've largely avoided using Tower for this reason, but you could theoretically get it up to an equivalent or at least comparable security level simply by SSH tunneling it or putting it behind a good VPN. Host compromise is still a risk even with ssh.

jdc0589 9y ago
a tunnel does not protect against any of that. OP was already assuming it wasn't publicly accessible, or at least was ACLd to your company's public IPs.
- rightos 9y ago
  > Instead of a locked-down server exposing a public key-only SSH port, you suddenly have a whole web application stack in there. There's no webapp stack to attack if you're only able to access it via a tunnel. If you're assuming the machine you're tunneling it to is compromised, there are bigger issues at play - ones that would compromise even a plain ssh link. I'm talking a direct tunnel from your ansible master to the host you're planning to use it on, not say, into your company's network at large.

Eridrus 9y ago

Maybe there should just be a trivial Electron wrapper around the Tower UI so that XSS/CSRF/extensions are not an issue.

chucky_z 9y ago

If you're always using `become_user: root` in your playbooks, maybe think about writing them differently! I have experience with Tower, and have never configured users to have any level of access beyond very, very basic SSH access.

Spivak 9y ago
Sure, but from an ops perspective you won't really be able to do much sysadmining without root privileges.
- emmelaich 9y ago
  My method is to have a "app superuser"; a user which is a group admin* for every group that I run apps as. It can then do everything required with a few exceptions. The exceptions are creating init/systemd files (associated startup/shutdown) and creating the necessary top-level directories for new apps. Those can be allowed with a few careful sudo rules. It can be a little convoluted but much more secure. * "gpasswd -A <user>,,, <group>" Note: to be a group admin doesn't necessarily mean you're in that group; it means that you can put yourself in and out of that group as required.

emmelaich 9y ago

Good points, though unlike some other similar apps, Ansible Tower has an selinux policy.

btown 9y ago

Separate Chrome profile, no addons, proxied thru a dedicated VPN whose exit IP is the only one whitelisted by Tower or some reverse proxy in front of it. Or... just trust that of all the exploits someone could deliver to an addon, a specific attack on Tower will be pretty low on the list.

rmenr 9y ago
Yeah. What ^^ he said. Aside from how tightly locked down my localDev environment and development host are, I do hop into a sanitized Chrome profile (or just use an incognito window) when I'm doing things that require work through a GUI like the AWS console, Jenkins, GSuite Admin, etc... Couple in a sound architectural model, and you're good. Unless you're working on some super duper top-secret stuff...but if we're talking about compliance and security at the CIS standards level, to the best of your ability and knowledge, you should be good. Keep in mind, of course, that large, blanket vulns show their faces every so often - like the KRACK vuln - which renders all of our preparation and paranoia mostly null. Example (of a neurotically paranoid architecture): Dedicated VPN resides in a dedicated AWS account and fronts all traffic to all hosts in all of your organization accounts across AWS. The only services exposed publicly other than your VPN service are the public-facing services you run if you're hosting some SAAS, for example. Even then, your LB's better be public, and your EC2's behind them better be private. Yay port 443 and LB to instance certificate encryption. Ansible Tower lives in some other "Internal Services" AWS account, and all traffic ingressed to hosts in that account is fronted by a bastion host and traffic proxy. The bastion/proxy is governed by a set of strict VPC route tables, and security groups that are set to only ever permit or accept traffic that corresponds to the IP addresses of your VPN. If you wanna get even crazier than that, you can also strictly control egress rules for your security groups - so even if someone got in, they'd be hard pressed to get the data out of your systems without doing some acrobatics. In general, if vulnerabilities @ the Tower server level are your concern, addressing that with architectural best-practices and network-level controls for locking up access and traffic to Tower is reasonable and gets the job done. It would be the same as securing anything else that's sensitive running in your infra, like Jenkins or RunDeck. I think RedHat's done an excellent job, and a huge service to the community by finally making good on their promise of open sourcing Tower.

dmsimard 9y ago

Hi! I'm the author of ARA and just wanted to thank you for mentioning it :) Full disclosure: I'm a Software Engineer at Red Hat. Tower has a lot of great features but I hate to concede that it does come with some of the drawbacks you mentioned. I think it comes down to weighing the pros and cons, making sure you are aware of the cons and putting measures in place to avoid problems. It also heavily depends on your use case, Tower provides things like RBAC/ACL, auditing, scheduling, online editing and execution, an API, etc. If you happen to need none of that and you're perfectly happy with just using Ansible from your command line, there's probably little incentive for you to use it. If all you need is reporting, ARA is simple, easy to install and setup, doesn't get in your way and just records things transparently. I figured I might as well take the opportunity of posting to leave a video demo [1], even if a little outdated, as well as an example of live report that ara provides [2]. Let me know if you have any questions ! [1]: https://www.youtube.com/watch?v=k3i8VPCanGo https://www.youtube.com/watch?v=k3i8VPCanGo [2]: http://logs.openstack.org/21/474721/7/check/gate-openstack-ansible-openstack-ansible-ceph-ubuntu-xenial/779e047/logs/ara/ http://logs.openstack.org/21/474721/7/check/gate-openstack-a...

jerrac 9y ago

So, I've recently been evaluating AWX for my workplace. Thus, making sure it's secure is definitely important to me. That said, I'm having a hard time seeing how a compromised browser access AWX could give anyone full access to everything. Attackers could effect anything a playbook has access to, but considering how varied playbooks are, it would be really hard to guess the right values to pass into a job to get what you want. Can someone elaborate on how an attacker could do real damage using a browser based attack?

deleted 9y ago
[deleted]
ibotty 9y ago
Simply add to any playbook a play with `all` hosts that drops your remote shell.
- AlanSE 9y ago
  You would need source control access to do that.
- jerrac 9y ago
  I had thought of the shell module. But the attacker would still have to know your playbook is using that module, and they'd have to know the variables you are passing into that module in order to override them. So, can you explain how an attacker would figure that information out? Assuming the playbooks are stored in source control like the Tower docs recommend, and all your variable values are mostly in source control as well.

dimitar 9y ago

You can mitigate part of the risks if you use an SSH tunnel to access Tower via port 22.

ghjm 9y ago

You are absolutely correct that these factors should be carefully considered prior to any deployment of AWX or Tower. You are granting Tower a lot of authority to your networks and systems, and should not do so without good reason. If you don't need the features of AWX/Tower, then the best practice is not to use it. There's a tremendous amount you can do with Ansible itself, without AWX/Tower, and lots of people use it happily that way. That being said, I think you are overstating some of the risks. You don't need to grant every Tower user root access to everything on your network. If you're at a scale where Tower makes sense, you probably already have some sort of separation of privileges. I agree that a malicious Chrome extension could do a lot of damage - just like it could with all your other management tools like DRAC/ILO, network equipment GUIs and so forth. Yes, every web application carries the risk of CSRF/XSS or other vulnerabilities, and Tower is not immune to this, but we do spend a lot of time worrying about it, conducting audits, etc. If your operation can succeed with nothing but a sudoers rule and command-line Ansible, then by all means use that. Nobody wants to force AWX/Tower on people who don't need it. But if you do need the feature set of Tower, I think it's one of the safer options available.