18 ms·

Asking nicely for root command execution and getting it

589 points by TangerineDream 5y ago

0xbadcafebee 5y ago
Developers always ask me why their service needs authentication when it's only accessible to every network in the company (or, a little better, the network specific to their app) and doesnt have write access to a dataset. This is as good a reason as any. Assume your service is a remote exploit machine, and that you want to make it as inconvenient to exploit, and as easy to fix, as possible. (If you don't think your service is a remote exploit machine... I have news for you...)
- geofft 5y ago
  Authentication or authorization? If you let in any user from the company, what difference does it make that it required authentication beforehand? (As the article points out, it could make a difference in incident response if the service is conscientious about logging and log retention, but that's a whole different feature that "authentication" doesn't get you for free!) If you let in a restricted set of users from the company, that's now a business-logic change, not just a technical toggle. How do you determine which users are allowed to use and which ones aren't?
  - gamacodre 5y ago
    I read that as "request authentication", i.e. some way to sign or HMAC the content of the request, with keys given only to the things that need to interact with the service. If users are involved, then yeah you also need some kind of RBAC for it to make any sense.
  - outworlder 5y ago
    > If you let in a restricted set of users from the company, that's now a business-logic change No it isn't. Just deploy a proxy in front of the app. Nginx works. If you have SSO, presumably you also have some detail on the business unit, or even what corporate mailing lists they are part of. Use that. Only allow requests through if they have been authenticated and the user is allowed to access the app. A business logic change is only needed if you actually need RBAC and have different levels of access permissions.
    - geofft 5y ago
      I was inaccurate about my terminology, oops - you're right that "business logic" usually means "code inside the app," which isn't changing. What I mean is that the business context around the app have changed, not simply the technical implementation. Whereas the app may previously have been, say, "Anyone in the company can extract structured data out of certain PDF files," the app is now "Approved people in the company can extract structured data out of certain PDF files." How do you get approved? Well, because your app is asserted to be a "remote exploit machine," possibly even the team that runs the app can't just give people access for the asking. They have to demonstrate some genuine business need in a way that's sufficient to satisfy the security department, and the security department might need to be involved. This is not a change in how the app is deployed; it is a change in what the app is. It changes the project definition, because the requirement "Anyone at the company can..." not only got dropped, it got replaced with "Most people at the company cannot...", and in turn it probably changes whether upper management wanted to assign people to work on the project at all. If it's getting limited use, maybe it wasn't worth building an app for it at all and people should have just transcribed data by hand. Or maybe there was a mediocre SaaS that would do 50% of the job for you, and exploits are their problem, and you should have decided to buy the mediocre thing instead of building a better thing that not enough people can use.
      - closeparen 5y ago
        >possibly even the team that runs the app can't just give people access for the asking. They have to demonstrate some genuine business need in a way that's sufficient to satisfy the security department, and the security department might need to be involved. Is that seriously thing? I've often known a security team to demand that something go behind an ACL, and to set some general standards for the management of that ACL (periodic reviews, expiring inactive users, etc), but never to get involved about specific members.
        db48x 5y ago
        Sure, I’ve worked on systems that I had no business need to operate. Nor even the skill to know how to operate them correctly. I could run the software on my own machine, putting data into a local database, and I could deploy new versions of it to the servers, but no permission to actually use it in production.
  - MattPalmer1086 5y ago
    In most places I've worked, unauthenticated remote services are banned outright. As you say, authentication on its own isn't a particularly strong control if there's no authorization. Logging and monitoring are also required for them at a minimum. Adding authorization might involve business logic changes I guess, but who is authorised is a business decision achieved with config/administration, not hard coded into the software... I would hope!
hug 5y ago
It's completely unrelated, but it reminds me of the Metasploit UAC bypass for Windows, called "ask". Metasploit just pops up a UAC prompt asking for escalation. Yes or no. Over and over and over again. On any machine logged into by a local administrator user, if they're the standard kind of careless user, they'll just click yes without thinking and they're owned. If they're cautious, and on a box where UAC presents as a modal dialog, as per its default config, this essentially locks the user out of the OS until they click "yes"... and they're owned. On an assessment we did at a local 200-seat business where they ran with admin rights on their local boxes, we got about a 90% escalation rate. The other 10% of people got up from their desks and went to ask the IT guy, which we considered a "pass" in that scenario. Interesting stuff.
- ma2rten 5y ago
  Did people who clicked "yes" eventually report it to IT later?
- WrtCdEvrydy 5y ago
  I wonder if "annoying the user into submission" is "social engineering". We had a similar issue with a library that wouldn't install unless root so one of the developers just put his username/password into the box in a plaintext file so the script could basically assume it from the command line when needed... told noone about it.
  - hug 5y ago
    Yep. It's worse when working with vendors, and even worse when working with vendors in a Windows-centric environment. "Our application needs to run as domain admin". No, mate, it absolutely does not. "Well it definitely needs local admin". No, mate, it doesn't even need that. What it usually needs is write-access to a single folder or regkey somewhere that has default ACLs on it that only allow local admins, but trying to have that conversation is a waste of time. Of course as soon as you say something like that, all support bets are off. Depending on the business you're working for you either suck it up so you have a 'support agreement' in place, or you do it properly and never talk to the vendor again. (Or option #3, you rebuild the app in its default "holy shit this is how my business gets owned" mode, place a support call, and have them fix that instance while you watch so you can replicate their efforts elsewhere.) Most of the same set of vendors get their panties in a twist when you tell them their database is on RDS, too, but that's a whole different argument to have with these people.
mr-wendel 5y ago
That last paragraph is hard to argue with: Maybe this is our version of the "infinite monkeys" thing: given enough software people, enough computers, and enough time, someone at a company will eventually grant universal remote root access to anyone who knows how to read some source code. At one job, way back in the day, I had a dream that a shared web-hosting installer framework could give you root on whatever server hosted your stuff. Went to work the next day and indeed... it was true and trivially easy to exploit. Sadly, it was a custom wallpaper upload feature that ended up being the actual source of shenanigans. You get 3 guesses on what happened and probably need just one.
- chucky_z 5y ago
  timthumb?
  - mr-wendel 5y ago
    Nope... a small almost mom'n'pop web hosting company.
- mewse 5y ago
  > You get 3 guesses on what happened and probably need just one. You.. quickly found the issue and patched it, and then balloons and streamers fell from the ceiling as the CEO strode over to shake your hand and pat you on the back and crack open a bottle of champagne to celebrate your promotion and give you a hefty performance bonus for saving the company and Dave Matthews played on the company’s sound system while everyone cheered for you and then when you finally left the celebration and went home you were tired and happy and newly wealthy and had become best friends with literally everyone at the company. …right?
  - eecc 5y ago
    Or your boss’s secretary was pissed at you because they liked that background very much.
    - 5y ago
ibatindev 5y ago
Would You Kindly?
TomVDB 5y ago
Reminds me of a friend of mine who wanted the ability to have Unix shell access on his work machine from home. (This was early nineties, when Internet log in to work was not a thing.) He ran a service that piped all income emails to a script. If the email contained a magic cookie, the subsequent commands were executed on a shell, and the results emailed back to sender. There were no checks on who the sender was. :-) I don’t think the IT people at his very big telecom equipment company would have approved. They never found out about this stunt, but he was later fired for running a password cracker on the company server farm.
- tgsovlerkhgsel 5y ago
  I disagree with "There were no checks on who the sender was." - there was a password/token authentication. That was the check, and it's harder to spoof than a regular sender check would have been back then (nowadays with DKIM etc. the situation is a bit better).
  - TomVDB 5y ago
    I don’t think the magic cookie was anything deliberately complex, but you’re right that guessing wouldn’t have been very practical.
    - dredmorbius 5y ago
      In the early 1990s, all email was plaintext and transports were unencrypted. In the very early 1990s, PGP didn't exist, and the odds that the email content was independently encrypted were low. Any network sniffer or filesystem access would have exposed the cookie.
      - Gys 5y ago
        > the early 1990s, all email was plaintext and transports were unencrypted. I thought all email is still plaintext? The protocol does not support encryption?
deleted 5y ago
[deleted]
Johnny555 5y ago
They built a list of every pre/post command that had been in actual use (by looking through the source tree for clients of that service, naturally), and then turned it into a hard-coded check list. The program would now refuse any pre/post commands unless it was one of those already known to it. This change patched the hole It didn't really patch the hole so much as it moved the hole to every command that was in that whitelist of commands that were ok to run as root. The author doesn't say when this was, but nowadays there are almost certainly better ways to solve this without letting these pre/post setup commands run as root.
- tyingq 5y ago
  Agree, and chances are one of the whitelisted things would allow you to bootstrap something else. I've seen attempts to grant limited sudo access by whitelist, and it's usually easy to get a shell one way or the other.
- yjftsjthsd-h 5y ago
  I like to think that this was the mitigation that they applied while they more carefully whittled down the list of commands to zero. I can dream:)
  - deleted 5y ago
    [deleted]
- ericbarrett 5y ago
  These were pre- and post-deploy jobs for a service (?), so I'm guessing they were hard-coded as the whole command string.
- hutzlibu 5y ago
  "but nowadays there are almost certainly better ways to solve this without letting these pre/post setup commands run as root." How? The only clean solution I see (now or anytime in the past), would be to rewrite that "Actual Business Stuff" using the whole. But that is expensive, maybe very expensive (I think this was clients code and going around and telling them to change maybe complex code of theirs ... is not something you want or can afford to do) So they just patched the hole up as much as they could do. So I do consider it a patch(especially if the hardcoded commands as strings are verified commands and do not change or induce something else). Just maybe none that provides full security. But when you look at the state of things in IT in general ... I would say, there is indeed improvement, if at least gaping holes like this one gets fixed.
oh_sigh 5y ago
Not particularly relevant, but every time I read this blog I get the impression it is more LARPing rather than actual software engineering. Or maybe 2nd/3rd hand stories. Or maybe I just happen to have worked in various places that have their shit together a little more than the bloggers employer.
- bigiain 5y ago
  Why do I strongly suspect you wouldn’t assume that if it was “Chad by the bay” instead of “Rachel”?
  - oh_sigh 5y ago
    I guess I'd have to guess that you're sexist against women and projecting.
    - howinteresting 5y ago
      Just FYI, I have direct knowledge of around 20% of the stories she's talked about and have personally been involved in some of them, and they're all true and not exaggerated. I have no reason to believe the others are any different. Obviously some of the specifics have to be elided because they're proprietary information, though. Rachel is one of the very best SREs on the planet. Working with her has been one of the greatest privileges of my professional career.
- yjftsjthsd-h 5y ago
  > Or maybe I just happen to have worked in various places that have their shit together a little more than the bloggers employer. I hate to break it to you, but nothing in that blog is unusual in Enterprise in my experience. You appear to have been quite lucky:)
- bjenkins358 5y ago
  To make the LARP more elaborate I was in the room for the meeting described. I still have the T-shirt. This very much happened as described.
dj_mc_merlin 5y ago
The last paragraph hits the nail on the head, it's impossible to prevent developers from elevating privileges unless one spends much more time and effort than almost any business is willing to on creating a true zero trust internal environment. Developers have little RCEs throughout the entire system, it's how they do their jobs. If even system with millions of dollars worth of design and security reviews regularly get hacked by people with no internal access to it or its source code, how can you hope to stop someone who already has a foothold and can see into the system, including a bunch of its config files? The real answer is not to stop developers from doing so. That's a never ending uphill battle. You implement very good logging and RBAC systems instead. Now everybody could hack the system if they wished to, but they would be sued into oblivion if they tried to do it for nefarious reasons. IMO the main purpose of RBAC is not really to stop developers from doing malicious activity, but to stop the stupid ones from breaking the system. The smart ones will be able to get around whatever safeguards you put in, as some of them will undoubtedly be designed by those not so fortunate or experienced. Not every developer has a good understanding of infosec and its many, _many_ ways of circumventing what seems to be an impossible defence. Unless you know all the tricks in the book already, whatever you write is most definitely gonna be swiss cheese.
- eyelidlessness 5y ago
  The risk isn’t internal nefarious actors, it’s internal people who don’t know or don’t care that they’re granting privilege escalation to a wider audience. And zero trust is the only way to mitigate that. There are so many bad security takes in this discussion and I’ve barely read half of it. This isn’t an ignorant community. The implication is that every slightly or significantly bad take is compounded by culture and organizational ignorance. The only solution is systems that can’t be accessed by humans by design. And building better reporting and isolated debugging tools to accommodate that.
  - closeparen 5y ago
    Either you need to change and deploy the debugging tool to support each new investigation, or the input to the debugging tool grows powerful enough that it is effectively a remote shell, if not literally.
hnick 5y ago
I've actually used that whitelist pattern before for legacy software in a non-security context. We had some programs using old stuff, and it was a real hassle to migrate everyone off it due to the massive amount of testing involved. But it was very easy to setup new programs to use something else. The problem was that people just kept calling the old one out of habit. So I just replaced the executable with a wrapper that checked what is calling it, then called the real one if allowed. I can't remember how but we probably blocked the original executable from general use, but even if we didn't then bypassing this check shows clear intent on the user's part. They can't claim ignorance, which is enough if it shows up in source code later or dies when we finally remove this software. Meanwhile we could move the other users one by one without any new ones sprouting up. Of course, code reviews, check-in filters, and such might be a nicer solution but this place wasn't that organised. So we did what we could, and didn't really have to keep an active eye out.
sigjuice 5y ago
What does SEV mean? Googling didn’t turn up anything.
- emef 5y ago
  SEVerity level of the incident
- breput 5y ago
  A "SEV" (severity) is a production failure or incident, usually indicted with a level value. For example, a SEV 0 would be a total system unavailability incident, SEV 1 might be a major issue with parts of the system, SEV 3 is a less critical failure affecting a subset of users, etc. You'll typically have a postmortem meeting afterwards to discuss the incident, the root cause, lessons learned, and action items to prevent future occurrences, which is the meeting described in the article.
- amacneil 5y ago
  In this context, it refers to an incident. Incidents are often referred to by their severity, such as SEV1, SEV2, etc. E.g. https://www.gremlin.com/community/tutorials/how-to-establish-a-high-severity-incident-management-program/ https://www.gremlin.com/community/tutorials/how-to-establish... https://www.atlassian.com/incident-management/kpis/severity-levels https://www.atlassian.com/incident-management/kpis/severity-... https://response.pagerduty.com/before/severity_levels/ https://response.pagerduty.com/before/severity_levels/
h2odragon 5y ago
Reminded me of this classic: https://www.exploit-db.com/exploits/45789 https://www.exploit-db.com/exploits/45789 https://www.cs.unc.edu/~jeffay/courses/nidsS05/attacks/seely-RTMworm-89.html#p4.5.3 https://www.cs.unc.edu/~jeffay/courses/nidsS05/attacks/seely...
speedgoose 5y ago
It reminds me an old vulnerability on Mac OS X, where you could nicely ask a root deamon to execute some commands for you through the AppleScript API : osascript -e 'tell app "ARDAgent" to do shell script "whoami"'
atoav 5y ago
Interesting read, but as a programmer I wonder.. Who builds a feature that allows users to input arbitrary strings, turns them into unchecked commands from a process that runs as root and has no logging and after all of this doesn't realise this on the spot with the hands still on the keyboard that this smells like trouble? If any collegue of mine would do this I either suspect a backdoor or incompetence on such scale they shouldn't have the job in the first place. This is like seeing a four inch crack during the construction of a bridge and still pressing on till it collapses. No, it actually is like planning a four inch wide crack into a bridge and finding it through a random inspection a year after. It should not only never happen, it should never even have been considered. I understand that implementing a quick "do whatever you want"-feature might be practical but: - anything that could get arbitrary strings from network input (or from anything other that has network input) needs to be validated and sanitized - anything that dispatches commands on a machine from user/network input should run with the lowest possible privilege - whitelisting is always simpler than blacklisting, but never understimate what a few whitelisted unix commands with arbitrary arguments can do. Maybe writing your own simple DSL is better for such a use case. It could even turn out to be a selling factor if it has a nice GUI
- brianpan 5y ago
  Let me make this quick tool for the development environment. I'm not sure what else we'll need, let me add some pre/post flexibility. We have this useful tool let's use it for more things. Let's standardize on this. Who wrote this? It's ok, we'll do a readiness review. We have WHAT in production? This is going to make a crazy blog post.
  - atoav 5y ago
    I understand that, but the blog post specifically wrote about customers running that code. Sure I sometimes have a little high standards compared to others, but how are we ever gonna fix software if we don't stop with the "let me quick..." If you e.g. handle private data of people you should code as if every character you type had the ability to leak a persons data forever (because often it has). Programming is beautiful because it multiplies. If you do a good job, the fruits of your work can make many peoples lives better. The same is true into the other direction: With great power comes great responsibility. I also happen to be a certified electrical engineer. There you also have situations where you'd want to "just quickly" fix something. Only that you don't do it because you would be liable if you burn someone's house down or kill them. I'd love for the whole field to get a little bit more serious about the impact our work has if done badly.
isoprophlex 5y ago
"I hit up my little dataset of everything running on any machine in production" People actually have comprehensive overviews of what they're running on their landscape?!
- theblazehen 5y ago
  Could be running something like https://osquery.io/ https://osquery.io/
  - philsnow 5y ago
    If you can manage to tame it, auditd will give you a much more comprehensive look at what has happened on your machines. osquery is more like polling (on a long interval because it's heavyweight) but auditd is processing events from the kernel.
- cranekam 5y ago
  Yes, at this company there was a very neat dataset of, essentially, the output of `ps` on every host taken every N minutes. Metadata about each process also included things like binary build time, commit, open ports, UID/GID, etc. It made things like tracking software rollouts or finding hosts that (for whatever reason) were stuck running old code very easy.
  - isoprophlex 5y ago
    Brilliant in its simplicity, thanks for sharing!
MattPalmer1086 5y ago
It's not uncommon to find these "run arbitrary commands as root" systems, unfortunately. Seen it many times, and I've never seen one where it was necessary. Driven either by laziness or ignorance. In one instance, the service people told me they had to have root because they downloaded arbitrary scripts and executed them, so they couldn't possibly know what permissions would be required ahead of time... Sigh...
raesene9 5y ago
One modern version of this is containerization and container orchestration APIs. Docker/ContainerD/CRI-O etc are basically command execution as a service. Kubernetes is basically distributed command execution as a service. You can add controls to reduce/remove the risk of random people creating root processes on your cluster nodes, but out of the box it's usually possible.
citrin_ru 5y ago
> You'd think this would have touched off a new "golden age" of people being thoughtful about not running things as root unless absolutely necessary, and generally being paranoid about handling subcommands, but that's not how it panned out in general. Instead, I'm pretty sure it pissed off the wrong people, and the rest is pretty much as you would expect. I don't change jobs to often to say that this is how every largish company works, but it typical that if you point to a security hole, people are displeased (now they have to fix it). And if you don't want many enemies it is better to ignore security problems unless you can fix them yourself. That partially explains the very sad state of corporate security.
kaptain 5y ago
The ubiquity of this issue really points to a deeper, admittedly unsolvable issue: morality is not a shared value of significance. Every discussion of security assumes that the primary threat is a purposely malicious actor, not an accident prone one. I understand that people see the shared morality issue as an unsolvable one which is why we try to solve the problem at the software/hardware level. But that looks unsolvable too (as per the article).
raffraffraff 5y ago
We had a ton of them at a company I worked at. They consided themselves a start-up (or "we have a start-up culture" which means "we don't bother doing things correctly and everyone works their asses off"). The number of ways you could pwn them from inside and out absolutely astonished me when I first joined. I had come from a stuffy company that had boring stuff like change control, technical architects, project plans etc. I wasn't sure how exactly to get stuck into the workload because there was so much wrong, and so much new stuff to build on top of the wrongness. So I decided to deploy a general discovery script that ran ps/lsof stuff. The results: lots and lots of stuff running as root, listening on various ports. Next step: running `curl icanhazip.com` (yeah, EC2 classic FTW!) and then hitting the IPs with nmap from outside the network to see what could be reached directly. Maybe nobody did this before, or maybe they did, and they found a lot of scary stuff that they didn't have bandwidth to fix and simply ignored it. But right away I had a bunch of SEVs on my hands. Worst case was a provisioning system that was hooked up to AWS, served to the world over port 80, using a super-duper lame password. We also had a very junior professional services team who would implement garbage-fires all over the place based on customer requirements. That whole team was a joke. The real engineering team didn't consider them "engineers". There was little communication between them. Some of the solutions they built were simply horrendous. One implemented a web-based form that allowed the customer... any customer... actually anybody... to stop/start a particularly broken part of our product. Kinda like self-service problem resolution for customers, instead of just fixing the cause of the issue. If you discovered this webpage by accident who knows what else you could have done with it before we closed it down. There were other major issues. At one point a mysql server was exposed to the world with root/root. Did that company fail? Nope. IPO'd for billions. And most of the people who put out fires like those for years while the founders told us to our faces that we'd all be rich... got sod-all.
notacoward 5y ago
When I was in the kernel group at Encore in 1990-91, I had a habit of never becoming root legitimately. Every time I needed to be root, I cracked it a new way and then patched the hole behind me. It got a bit harder over that time, but never really enough to slow down my regular work. The moral of the story is that, even without people deliberately creating ways to run arbitrary code as root, people who are merely oblivious to security leave plenty of holes.
brian_cloutier 5y ago
Very timely post, this is roughly how the recent $611M poly network hack worked. [1] In the attack there were two contracts: the first is a proxy contract which forwards commands (allowing actors from other chains to interact with Poly. The second contract is a big pile of money which is very careful about who is allowed to interact with that money. Only "owner"s are allowed to take privileged actions. You might already see where this is going: the proxy contract is an owner. There was some validation in place to prevent the obvious from happening, which makes this attack not exactly like what Rachel describes, but there was little enough validation that it's not uncharitable to say the hacker simply asked for all the money (root) and the smart contract happily complied. [1] https://www.rekt.news/polynetwork-rekt/ https://www.rekt.news/polynetwork-rekt/

Developers always ask me why their service needs authentication when it's only accessible to every network in the company (or, a little better, the network specific to their app) and doesnt have write access to a dataset. This is as good a reason as any. Assume your service is a remote exploit machine, and that you want to make it as inconvenient to exploit, and as easy to fix, as possible. (If you don't think your service is a remote exploit machine... I have news for you...)

geofft 5y ago
Authentication or authorization? If you let in any user from the company, what difference does it make that it required authentication beforehand? (As the article points out, it could make a difference in incident response if the service is conscientious about logging and log retention, but that's a whole different feature that "authentication" doesn't get you for free!) If you let in a restricted set of users from the company, that's now a business-logic change, not just a technical toggle. How do you determine which users are allowed to use and which ones aren't?
- gamacodre 5y ago
  I read that as "request authentication", i.e. some way to sign or HMAC the content of the request, with keys given only to the things that need to interact with the service. If users are involved, then yeah you also need some kind of RBAC for it to make any sense.
- outworlder 5y ago
  > If you let in a restricted set of users from the company, that's now a business-logic change No it isn't. Just deploy a proxy in front of the app. Nginx works. If you have SSO, presumably you also have some detail on the business unit, or even what corporate mailing lists they are part of. Use that. Only allow requests through if they have been authenticated and the user is allowed to access the app. A business logic change is only needed if you actually need RBAC and have different levels of access permissions.
  - geofft 5y ago
    I was inaccurate about my terminology, oops - you're right that "business logic" usually means "code inside the app," which isn't changing. What I mean is that the business context around the app have changed, not simply the technical implementation. Whereas the app may previously have been, say, "Anyone in the company can extract structured data out of certain PDF files," the app is now "Approved people in the company can extract structured data out of certain PDF files." How do you get approved? Well, because your app is asserted to be a "remote exploit machine," possibly even the team that runs the app can't just give people access for the asking. They have to demonstrate some genuine business need in a way that's sufficient to satisfy the security department, and the security department might need to be involved. This is not a change in how the app is deployed; it is a change in what the app is. It changes the project definition, because the requirement "Anyone at the company can..." not only got dropped, it got replaced with "Most people at the company cannot...", and in turn it probably changes whether upper management wanted to assign people to work on the project at all. If it's getting limited use, maybe it wasn't worth building an app for it at all and people should have just transcribed data by hand. Or maybe there was a mediocre SaaS that would do 50% of the job for you, and exploits are their problem, and you should have decided to buy the mediocre thing instead of building a better thing that not enough people can use.
    - closeparen 5y ago
      >possibly even the team that runs the app can't just give people access for the asking. They have to demonstrate some genuine business need in a way that's sufficient to satisfy the security department, and the security department might need to be involved. Is that seriously thing? I've often known a security team to demand that something go behind an ACL, and to set some general standards for the management of that ACL (periodic reviews, expiring inactive users, etc), but never to get involved about specific members.
      - db48x 5y ago
        Sure, I’ve worked on systems that I had no business need to operate. Nor even the skill to know how to operate them correctly. I could run the software on my own machine, putting data into a local database, and I could deploy new versions of it to the servers, but no permission to actually use it in production.
- MattPalmer1086 5y ago
  In most places I've worked, unauthenticated remote services are banned outright. As you say, authentication on its own isn't a particularly strong control if there's no authorization. Logging and monitoring are also required for them at a minimum. Adding authorization might involve business logic changes I guess, but who is authorised is a business decision achieved with config/administration, not hard coded into the software... I would hope!

hug 5y ago

It's completely unrelated, but it reminds me of the Metasploit UAC bypass for Windows, called "ask". Metasploit just pops up a UAC prompt asking for escalation. Yes or no. Over and over and over again. On any machine logged into by a local administrator user, if they're the standard kind of careless user, they'll just click yes without thinking and they're owned. If they're cautious, and on a box where UAC presents as a modal dialog, as per its default config, this essentially locks the user out of the OS until they click "yes"... and they're owned. On an assessment we did at a local 200-seat business where they ran with admin rights on their local boxes, we got about a 90% escalation rate. The other 10% of people got up from their desks and went to ask the IT guy, which we considered a "pass" in that scenario. Interesting stuff.

ma2rten 5y ago
Did people who clicked "yes" eventually report it to IT later?
WrtCdEvrydy 5y ago
I wonder if "annoying the user into submission" is "social engineering". We had a similar issue with a library that wouldn't install unless root so one of the developers just put his username/password into the box in a plaintext file so the script could basically assume it from the command line when needed... told noone about it.
- hug 5y ago
  Yep. It's worse when working with vendors, and even worse when working with vendors in a Windows-centric environment. "Our application needs to run as domain admin". No, mate, it absolutely does not. "Well it definitely needs local admin". No, mate, it doesn't even need that. What it usually needs is write-access to a single folder or regkey somewhere that has default ACLs on it that only allow local admins, but trying to have that conversation is a waste of time. Of course as soon as you say something like that, all support bets are off. Depending on the business you're working for you either suck it up so you have a 'support agreement' in place, or you do it properly and never talk to the vendor again. (Or option #3, you rebuild the app in its default "holy shit this is how my business gets owned" mode, place a support call, and have them fix that instance while you watch so you can replicate their efforts elsewhere.) Most of the same set of vendors get their panties in a twist when you tell them their database is on RDS, too, but that's a whole different argument to have with these people.

mr-wendel 5y ago

That last paragraph is hard to argue with: Maybe this is our version of the "infinite monkeys" thing: given enough software people, enough computers, and enough time, someone at a company will eventually grant universal remote root access to anyone who knows how to read some source code. At one job, way back in the day, I had a dream that a shared web-hosting installer framework could give you root on whatever server hosted your stuff. Went to work the next day and indeed... it was true and trivially easy to exploit. Sadly, it was a custom wallpaper upload feature that ended up being the actual source of shenanigans. You get 3 guesses on what happened and probably need just one.

chucky_z 5y ago
timthumb?
- mr-wendel 5y ago
  Nope... a small almost mom'n'pop web hosting company.
mewse 5y ago
> You get 3 guesses on what happened and probably need just one. You.. quickly found the issue and patched it, and then balloons and streamers fell from the ceiling as the CEO strode over to shake your hand and pat you on the back and crack open a bottle of champagne to celebrate your promotion and give you a hefty performance bonus for saving the company and Dave Matthews played on the company’s sound system while everyone cheered for you and then when you finally left the celebration and went home you were tired and happy and newly wealthy and had become best friends with literally everyone at the company. …right?
- eecc 5y ago
  Or your boss’s secretary was pissed at you because they liked that background very much.
  - 5y ago

ibatindev 5y ago

Would You Kindly?

TomVDB 5y ago

Reminds me of a friend of mine who wanted the ability to have Unix shell access on his work machine from home. (This was early nineties, when Internet log in to work was not a thing.) He ran a service that piped all income emails to a script. If the email contained a magic cookie, the subsequent commands were executed on a shell, and the results emailed back to sender. There were no checks on who the sender was. :-) I don’t think the IT people at his very big telecom equipment company would have approved. They never found out about this stunt, but he was later fired for running a password cracker on the company server farm.

tgsovlerkhgsel 5y ago
I disagree with "There were no checks on who the sender was." - there was a password/token authentication. That was the check, and it's harder to spoof than a regular sender check would have been back then (nowadays with DKIM etc. the situation is a bit better).
- TomVDB 5y ago
  I don’t think the magic cookie was anything deliberately complex, but you’re right that guessing wouldn’t have been very practical.
  - dredmorbius 5y ago
    In the early 1990s, all email was plaintext and transports were unencrypted. In the very early 1990s, PGP didn't exist, and the odds that the email content was independently encrypted were low. Any network sniffer or filesystem access would have exposed the cookie.
    - Gys 5y ago
      > the early 1990s, all email was plaintext and transports were unencrypted. I thought all email is still plaintext? The protocol does not support encryption?

deleted 5y ago

[deleted]

Johnny555 5y ago

They built a list of every pre/post command that had been in actual use (by looking through the source tree for clients of that service, naturally), and then turned it into a hard-coded check list. The program would now refuse any pre/post commands unless it was one of those already known to it. This change patched the hole It didn't really patch the hole so much as it moved the hole to every command that was in that whitelist of commands that were ok to run as root. The author doesn't say when this was, but nowadays there are almost certainly better ways to solve this without letting these pre/post setup commands run as root.

tyingq 5y ago
Agree, and chances are one of the whitelisted things would allow you to bootstrap something else. I've seen attempts to grant limited sudo access by whitelist, and it's usually easy to get a shell one way or the other.
yjftsjthsd-h 5y ago
I like to think that this was the mitigation that they applied while they more carefully whittled down the list of commands to zero. I can dream:)
- deleted 5y ago
  [deleted]
ericbarrett 5y ago
These were pre- and post-deploy jobs for a service (?), so I'm guessing they were hard-coded as the whole command string.
hutzlibu 5y ago
"but nowadays there are almost certainly better ways to solve this without letting these pre/post setup commands run as root." How? The only clean solution I see (now or anytime in the past), would be to rewrite that "Actual Business Stuff" using the whole. But that is expensive, maybe very expensive (I think this was clients code and going around and telling them to change maybe complex code of theirs ... is not something you want or can afford to do) So they just patched the hole up as much as they could do. So I do consider it a patch(especially if the hardcoded commands as strings are verified commands and do not change or induce something else). Just maybe none that provides full security. But when you look at the state of things in IT in general ... I would say, there is indeed improvement, if at least gaping holes like this one gets fixed.

oh_sigh 5y ago

Not particularly relevant, but every time I read this blog I get the impression it is more LARPing rather than actual software engineering. Or maybe 2nd/3rd hand stories. Or maybe I just happen to have worked in various places that have their shit together a little more than the bloggers employer.

bigiain 5y ago
Why do I strongly suspect you wouldn’t assume that if it was “Chad by the bay” instead of “Rachel”?
- oh_sigh 5y ago
  I guess I'd have to guess that you're sexist against women and projecting.
  - howinteresting 5y ago
    Just FYI, I have direct knowledge of around 20% of the stories she's talked about and have personally been involved in some of them, and they're all true and not exaggerated. I have no reason to believe the others are any different. Obviously some of the specifics have to be elided because they're proprietary information, though. Rachel is one of the very best SREs on the planet. Working with her has been one of the greatest privileges of my professional career.
yjftsjthsd-h 5y ago
> Or maybe I just happen to have worked in various places that have their shit together a little more than the bloggers employer. I hate to break it to you, but nothing in that blog is unusual in Enterprise in my experience. You appear to have been quite lucky:)
bjenkins358 5y ago
To make the LARP more elaborate I was in the room for the meeting described. I still have the T-shirt. This very much happened as described.

dj_mc_merlin 5y ago

The last paragraph hits the nail on the head, it's impossible to prevent developers from elevating privileges unless one spends much more time and effort than almost any business is willing to on creating a true zero trust internal environment. Developers have little RCEs throughout the entire system, it's how they do their jobs. If even system with millions of dollars worth of design and security reviews regularly get hacked by people with no internal access to it or its source code, how can you hope to stop someone who already has a foothold and can see into the system, including a bunch of its config files? The real answer is not to stop developers from doing so. That's a never ending uphill battle. You implement very good logging and RBAC systems instead. Now everybody could hack the system if they wished to, but they would be sued into oblivion if they tried to do it for nefarious reasons. IMO the main purpose of RBAC is not really to stop developers from doing malicious activity, but to stop the stupid ones from breaking the system. The smart ones will be able to get around whatever safeguards you put in, as some of them will undoubtedly be designed by those not so fortunate or experienced. Not every developer has a good understanding of infosec and its many, _many_ ways of circumventing what seems to be an impossible defence. Unless you know all the tricks in the book already, whatever you write is most definitely gonna be swiss cheese.

eyelidlessness 5y ago
The risk isn’t internal nefarious actors, it’s internal people who don’t know or don’t care that they’re granting privilege escalation to a wider audience. And zero trust is the only way to mitigate that. There are so many bad security takes in this discussion and I’ve barely read half of it. This isn’t an ignorant community. The implication is that every slightly or significantly bad take is compounded by culture and organizational ignorance. The only solution is systems that can’t be accessed by humans by design. And building better reporting and isolated debugging tools to accommodate that.
- closeparen 5y ago
  Either you need to change and deploy the debugging tool to support each new investigation, or the input to the debugging tool grows powerful enough that it is effectively a remote shell, if not literally.

hnick 5y ago

I've actually used that whitelist pattern before for legacy software in a non-security context. We had some programs using old stuff, and it was a real hassle to migrate everyone off it due to the massive amount of testing involved. But it was very easy to setup new programs to use something else. The problem was that people just kept calling the old one out of habit. So I just replaced the executable with a wrapper that checked what is calling it, then called the real one if allowed. I can't remember how but we probably blocked the original executable from general use, but even if we didn't then bypassing this check shows clear intent on the user's part. They can't claim ignorance, which is enough if it shows up in source code later or dies when we finally remove this software. Meanwhile we could move the other users one by one without any new ones sprouting up. Of course, code reviews, check-in filters, and such might be a nicer solution but this place wasn't that organised. So we did what we could, and didn't really have to keep an active eye out.

sigjuice 5y ago

What does SEV mean? Googling didn’t turn up anything.

emef 5y ago
SEVerity level of the incident
breput 5y ago
A "SEV" (severity) is a production failure or incident, usually indicted with a level value. For example, a SEV 0 would be a total system unavailability incident, SEV 1 might be a major issue with parts of the system, SEV 3 is a less critical failure affecting a subset of users, etc. You'll typically have a postmortem meeting afterwards to discuss the incident, the root cause, lessons learned, and action items to prevent future occurrences, which is the meeting described in the article.
amacneil 5y ago
In this context, it refers to an incident. Incidents are often referred to by their severity, such as SEV1, SEV2, etc. E.g. https://www.gremlin.com/community/tutorials/how-to-establish-a-high-severity-incident-management-program/ https://www.gremlin.com/community/tutorials/how-to-establish... https://www.atlassian.com/incident-management/kpis/severity-levels https://www.atlassian.com/incident-management/kpis/severity-... https://response.pagerduty.com/before/severity_levels/ https://response.pagerduty.com/before/severity_levels/

h2odragon 5y ago

Reminded me of this classic: https://www.exploit-db.com/exploits/45789 https://www.exploit-db.com/exploits/45789 https://www.cs.unc.edu/~jeffay/courses/nidsS05/attacks/seely-RTMworm-89.html#p4.5.3 https://www.cs.unc.edu/~jeffay/courses/nidsS05/attacks/seely...

speedgoose 5y ago

It reminds me an old vulnerability on Mac OS X, where you could nicely ask a root deamon to execute some commands for you through the AppleScript API : osascript -e 'tell app "ARDAgent" to do shell script "whoami"'

atoav 5y ago

Interesting read, but as a programmer I wonder.. Who builds a feature that allows users to input arbitrary strings, turns them into unchecked commands from a process that runs as root and has no logging and after all of this doesn't realise this on the spot with the hands still on the keyboard that this smells like trouble? If any collegue of mine would do this I either suspect a backdoor or incompetence on such scale they shouldn't have the job in the first place. This is like seeing a four inch crack during the construction of a bridge and still pressing on till it collapses. No, it actually is like planning a four inch wide crack into a bridge and finding it through a random inspection a year after. It should not only never happen, it should never even have been considered. I understand that implementing a quick "do whatever you want"-feature might be practical but: - anything that could get arbitrary strings from network input (or from anything other that has network input) needs to be validated and sanitized - anything that dispatches commands on a machine from user/network input should run with the lowest possible privilege - whitelisting is always simpler than blacklisting, but never understimate what a few whitelisted unix commands with arbitrary arguments can do. Maybe writing your own simple DSL is better for such a use case. It could even turn out to be a selling factor if it has a nice GUI

brianpan 5y ago
Let me make this quick tool for the development environment. I'm not sure what else we'll need, let me add some pre/post flexibility. We have this useful tool let's use it for more things. Let's standardize on this. Who wrote this? It's ok, we'll do a readiness review. We have WHAT in production? This is going to make a crazy blog post.
- atoav 5y ago
  I understand that, but the blog post specifically wrote about customers running that code. Sure I sometimes have a little high standards compared to others, but how are we ever gonna fix software if we don't stop with the "let me quick..." If you e.g. handle private data of people you should code as if every character you type had the ability to leak a persons data forever (because often it has). Programming is beautiful because it multiplies. If you do a good job, the fruits of your work can make many peoples lives better. The same is true into the other direction: With great power comes great responsibility. I also happen to be a certified electrical engineer. There you also have situations where you'd want to "just quickly" fix something. Only that you don't do it because you would be liable if you burn someone's house down or kill them. I'd love for the whole field to get a little bit more serious about the impact our work has if done badly.

isoprophlex 5y ago

"I hit up my little dataset of everything running on any machine in production" People actually have comprehensive overviews of what they're running on their landscape?!

theblazehen 5y ago
Could be running something like https://osquery.io/ https://osquery.io/
- philsnow 5y ago
  If you can manage to tame it, auditd will give you a much more comprehensive look at what has happened on your machines. osquery is more like polling (on a long interval because it's heavyweight) but auditd is processing events from the kernel.
cranekam 5y ago
Yes, at this company there was a very neat dataset of, essentially, the output of `ps` on every host taken every N minutes. Metadata about each process also included things like binary build time, commit, open ports, UID/GID, etc. It made things like tracking software rollouts or finding hosts that (for whatever reason) were stuck running old code very easy.
- isoprophlex 5y ago
  Brilliant in its simplicity, thanks for sharing!

MattPalmer1086 5y ago

It's not uncommon to find these "run arbitrary commands as root" systems, unfortunately. Seen it many times, and I've never seen one where it was necessary. Driven either by laziness or ignorance. In one instance, the service people told me they had to have root because they downloaded arbitrary scripts and executed them, so they couldn't possibly know what permissions would be required ahead of time... Sigh...

raesene9 5y ago

One modern version of this is containerization and container orchestration APIs. Docker/ContainerD/CRI-O etc are basically command execution as a service. Kubernetes is basically distributed command execution as a service. You can add controls to reduce/remove the risk of random people creating root processes on your cluster nodes, but out of the box it's usually possible.

citrin_ru 5y ago

> You'd think this would have touched off a new "golden age" of people being thoughtful about not running things as root unless absolutely necessary, and generally being paranoid about handling subcommands, but that's not how it panned out in general. Instead, I'm pretty sure it pissed off the wrong people, and the rest is pretty much as you would expect. I don't change jobs to often to say that this is how every largish company works, but it typical that if you point to a security hole, people are displeased (now they have to fix it). And if you don't want many enemies it is better to ignore security problems unless you can fix them yourself. That partially explains the very sad state of corporate security.

kaptain 5y ago

The ubiquity of this issue really points to a deeper, admittedly unsolvable issue: morality is not a shared value of significance. Every discussion of security assumes that the primary threat is a purposely malicious actor, not an accident prone one. I understand that people see the shared morality issue as an unsolvable one which is why we try to solve the problem at the software/hardware level. But that looks unsolvable too (as per the article).

raffraffraff 5y ago

We had a ton of them at a company I worked at. They consided themselves a start-up (or "we have a start-up culture" which means "we don't bother doing things correctly and everyone works their asses off"). The number of ways you could pwn them from inside and out absolutely astonished me when I first joined. I had come from a stuffy company that had boring stuff like change control, technical architects, project plans etc. I wasn't sure how exactly to get stuck into the workload because there was so much wrong, and so much new stuff to build on top of the wrongness. So I decided to deploy a general discovery script that ran ps/lsof stuff. The results: lots and lots of stuff running as root, listening on various ports. Next step: running `curl icanhazip.com` (yeah, EC2 classic FTW!) and then hitting the IPs with nmap from outside the network to see what could be reached directly. Maybe nobody did this before, or maybe they did, and they found a lot of scary stuff that they didn't have bandwidth to fix and simply ignored it. But right away I had a bunch of SEVs on my hands. Worst case was a provisioning system that was hooked up to AWS, served to the world over port 80, using a super-duper lame password. We also had a very junior professional services team who would implement garbage-fires all over the place based on customer requirements. That whole team was a joke. The real engineering team didn't consider them "engineers". There was little communication between them. Some of the solutions they built were simply horrendous. One implemented a web-based form that allowed the customer... any customer... actually anybody... to stop/start a particularly broken part of our product. Kinda like self-service problem resolution for customers, instead of just fixing the cause of the issue. If you discovered this webpage by accident who knows what else you could have done with it before we closed it down. There were other major issues. At one point a mysql server was exposed to the world with root/root. Did that company fail? Nope. IPO'd for billions. And most of the people who put out fires like those for years while the founders told us to our faces that we'd all be rich... got sod-all.

notacoward 5y ago

When I was in the kernel group at Encore in 1990-91, I had a habit of never becoming root legitimately. Every time I needed to be root, I cracked it a new way and then patched the hole behind me. It got a bit harder over that time, but never really enough to slow down my regular work. The moral of the story is that, even without people deliberately creating ways to run arbitrary code as root, people who are merely oblivious to security leave plenty of holes.

brian_cloutier 5y ago

Very timely post, this is roughly how the recent $611M poly network hack worked. [1] In the attack there were two contracts: the first is a proxy contract which forwards commands (allowing actors from other chains to interact with Poly. The second contract is a big pile of money which is very careful about who is allowed to interact with that money. Only "owner"s are allowed to take privileged actions. You might already see where this is going: the proxy contract is an owner. There was some validation in place to prevent the obvious from happening, which makes this attack not exactly like what Rachel describes, but there was little enough validation that it's not uncharitable to say the hacker simply asked for all the money (root) and the smart contract happily complied. [1] https://www.rekt.news/polynetwork-rekt/ https://www.rekt.news/polynetwork-rekt/