8 ms·
So the funny thing is, when I saw that problem description, my first thought was: "oh, that's what mkdir -p does; I wonder if it would be easier to just shell o
by yjftsjthsd-h 5y ago
So the funny thing is, when I saw that problem description, my first thought was: "oh, that's what mkdir -p does; I wonder if it would be easier to just shell out to that than try and implement it yourself". I grant that that implementation is a massive SQL-injection-like vulnerability, but I'm still not 100% convinced that the basic impulse is wrong; I can easily imagine an alternate universe where the second half of this blog post is talking about how someone wrote it themselves and introduced some subtle bug that the coreutils folks fixed 20 years ago and everyone else has to painfully rediscover. I wonder if any of the major implementations are friendly to being used as a library? Obviously then you have to deal with licensing issues, which is a catch. Maybe fork+exec? I dunno, I'm just trying to feel out whether there's a middle ground between system() and rewriting functionality that someone else has written and that's been used by a billion other users for decades.
- shadowgovt 5y agoYep. Really, the meta-problem here is that someone would want to go run a subprocess because the standard library isn't powerful enough.
- bombcar 5y agoYou can do something akin to parameterizing SQL with fork and exec (directly setting the arguments) but at that point you might as well call whatever mkdir is doing.
- infotogivenm 5y agoThere’s absolutely a middle ground, and you are right: it is fork and exec with discrete arguments. It is the intermediary subshell in system() that is the root of all evil, not composing unix programs together; that is what our OS forefathers intended.
- theptip 5y agoI agree. Starting a subprocess is fine. Using an existing tool (mkdir) was probably a better approach than reimplementing it. The error: you should never pass unescaped/untrusted input into a subshell. In Python, it’s the difference between the safe: def make_dir(path): subprocess.run([“mkdir”, “-p”, path]) Instead of: def make_dir_unsafe(path): subprocess.run(“mkdir -p “ + path, shell=True) The latter is easier to type because you don’t need to use the annoying list syntax, but it’s unsafe. If you are writing a library function a subshell is not a sound approach.
- blorgle 5y agoYou shouldn't pass unescaped/untrusted input into a subshell but your suggestion doesn't solve for every case. It protects against ";" but not path traversal with "../../../../../foo/bar".
- yjftsjthsd-h 5y agoYou'd have to cover that if you wrote your own library function, though; it's attack surface, but not uniquely so.
- pabs3 5y agoI like to do this to avoid list syntax: def make_dir(path): subprocess.run('mkdir -p'.split() + [path])
- loloquwowndueo 5y agoSo you like typing more and making your program harder to read to avoid using two apostrophes and two commas.
- cowsandmilk 5y agoIf you have a command with lots of arguments, parent’s approach makes a lot of sense. 20 arguments that you can copy out of the shell into your program. And when you want to make changes and test via the shell, your copy and paste is also easy. I’ve seen bugs where some forgot to add quotes and commas to split some arguments and the program did the wrong thing. Parent’s approach avoids that. However, things like spaces in quoted strings are not correctly handled.
- div72 5y agoI am not exactly sure if parent's code execute the way you think it does. path is provided as an argument to mkdir -p, even if you set path to be a string with 20 arguments copied from your shell, it'll get passed as one. I assume you meant something like subprocess.run(['mkdir', '-p', '--', *path.split()]), which in that case using shlex.split will fix your problems with quoted strings.
- elurg 5y agoDoing fork/exec for the mkdir tool is still many times slower that calling a good library function.
- fivea 5y ago> Doing fork/exec for the mkdir tool is still many times slower that calling a good library function. Would it be noticeably slower to the point it's measurably relevant, though? Or are we talking about something well within the domain of premature optimization? I mean,if you're doing IO, how often does creating a dir end up being the performance bottleneck?
- thebackup 5y agoPremature optimization is a common disease amongst C and C++ devs. I often come across code that is written in an obscure/complex way just because someone thought execution speed triumphs code readability. In most cases (around 9 times out of 10) it is on a non-time-critical path of the code and the developer who wrote it didn't even bother to run a profiler on it. My work would be significantly easier if we could learn to prioritize code readability.
- mgaunard 5y agoThe point of those languages is to make it explicit when you're doing ridiculously costly things so that you're forced to do a reasonable implementation instead.
- ptx 5y ago> It is the intermediary subshell in system() that is the root of all evil Certainly the worst of it, but even if you pass the untrusted data as a discrete arguments you're still vulnerable to parameter injection. Let's say, for example, that you're running ["find", USER_SUPPLIED_PATH, "-type", "f"]. If the user supplies "-delete" as the path, the resulting behaviour would not be as intended.
- infotogivenm 5y agoVery true. A pro tip here is use ‘—-‘ :)
- me_again 5y agoI'd want a better reason before shelling out personally. Apart from security, other issues are: - performance. Obviously that depends on the circumstances but if you are working on a function that is used in many places you don't know the circumstances. - error handling and reporting. Are you going to capture stderr from mkdir? If this fails can you provide a clear indication of the problem to the caller? - paths. If you run mkdir without specifying an absolute path your program may fail if $path is not what you expect. That can also be a vulnerability. - dependencies. Now you won't run in a distroless container or other cut down environment - you lose type safety Not saying "never shell out" but I've been burned by most of these so I am cautious about it.
- hamburglar 5y ago“Never shell out” is pretty close to correct though. I’d say “only shell out as a last resort, and only after you’ve considered all the drawbacks.”
- ufmace 5y agoIt seems to me that hiding a risky implementation is the problem. If you were directly shelling out to `mkdir -p` yourself, it would (hopefully) at least be obvious that it's a risky thing to do, and you need to make sure that either the input couldn't possibly contain any data touched by a malicious user or that it was sufficiently well sanitized to protect against that sort of thing. But if it's hidden behind a friendly-look library call, it would be less obvious that it's a risk, and it's easier to see someone absent-mindedly calling it because it's handy and passing in data from an unauthenticated web user or something.
- catern 5y agoI think the real fix here is to investigate why you're creating possibly-already-existing directories anyway. Most files or directories should either be owned by a specific component (which then creates them and fails if they already exist) or should be assumed to already exist (as a requirement that the environment is set up properly.
- spc476 5y agoIt depends upon usage. My blogging software keeps entries in the file system, in a hierarchy as YYYY/MM/DD. I just create the directories as I add entries by doing a "mkdir(year); mkdir(year "/" month); mkdir(year "/" month "/" day)".
- lawl 5y agoImo the middle ground here is that this is a common enough operation that it should be in your language's standard library, see e.g. [0][1]. And if you have to use C, then yes, write it yourself or copy some other implementation with a compatible license. That's what you signed up for when you decided you need C. [0] https://pkg.go.dev/os#MkdirAll https://pkg.go.dev/os#MkdirAll [1] https://doc.rust-lang.org/std/fs/fn.create_dir_all.html https://doc.rust-lang.org/std/fs/fn.create_dir_all.html
- fivea 5y ago> Imo the middle ground here is that this is a common enough operation that it should be in your language's standard library, see e.g. [0][1]. Is it, though? I mean, C is around for around half a century, and C is perhaps the primary language in the FLOSS universe. Still, why is there no widely adopted idempotent directory tree creation library?
- mgaunard 5y agoBecause C stopped evolving long ago, in spite of the recent C standards that have more to do with a reanimated zombie than anything else. In C++ that functionality is indeed part of the standard library.
- fivea 5y ago> Because C stopped evolving long ago (...) That excuse isn't credible. I mean, I asked about a libraries that handled that usecase, not a standard library. If there was demand for that use case, and that demand popped up so often that it justified the adoption of a de-facto standard, why isn't there any third-party library specialized in that usecase? I mean, putting together a library takes far less work than presenting a proposal to the C standardization committee, which is open to proposals. So, why isn't there one? Might it be that there is actually no relevant demand for it?
- mgaunard 5y agoHow would such a trivial library achieve popularity? It would have to be part of a larger framework. Anyway I just checked and Glib does seem to have it for example.
- taneq 5y agoI think the real issue here is the lack of distinction in the rest of the codebase between trusted and untrusted input. There should be only a few, clearly identified functions that you do pass unsanitised user input into.
- horsawlarway 5y agoI think this, really, is the take away. Everyone is getting stuck on the subshell part, but it's the untrusted data part that's the problem. Especially because the early examples in the article (making subdirs for something like different packages/release - "project/staging", "project/production") are almost certainly safe to use a subshell for - so much so that I was genuinely wondering why the author wouldn't just call out to "mkdir -p" for that use case. The error came when someone decided to pass untrusted/user-supplied data to this function (the later example - usernames). But I think doing the later is a huge mistake either way. Don't pass untrusted & unsanitized user data ANYWHERE that isn't explicitly designed for it. Who gives a hoot whether that's to a subshell, or a pipe/stream, or to a sql query, or your DOM, or any other of hundred of places that user supplied data isn't safe.
- selfhoster11 5y agoI like the idea that Joel had: make untrusted user provided strings into a distinct data type that doesn't easily fit into the mold of a trusted, already-sanitised string type. If you ever see an unsafe string usage being passed into a function that expects a safe string, without an accompanying call to sanitise(), that's very wrong (and won't work if the language is strongly typed) and should be recognised/corrected as such immediately.
- taneq 5y agoYeah, I actually nearly put in something about separate trusted vs. untrusted types but didn't want to dilute the main point or hijack it religious wars over typed languages. :P
- selfhoster11 5y ago
- newaccount74 5y agoI feel strongly that you shouldn't go looking for external code that does simple things like this. You can reimplement the relevant parts of mkdir -p in 20 lines of C code, and I think that's what you should do if the standard library doesn't have it. Having the code inline makes it easy to audit, and prevents problems when a dependency changes 5 years later. It also trains people to write proper C code. Writing loops and checking errno should be second nature for a C programmer, and doing it often helps you learn.