5 ms·

Apple's libc shells out to Perl to implement wordexp

103 points by devbug 12y ago

Intermernet 12y ago
Sorry, I don't have a Mac to check, but does the OSX build of Perl depend on libc? Also, that code seems to be from 2008 (or at least that's the latest year in the copyright header) despite the commit being from 2012. Does anyone know if this has been updated? Inserting a NUL byte between each word, and at the end doesn't sound like it requires Perl...
- adamnemecek 12y ago
  > otool -L /usr/bin/perl /usr/bin/perl: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1151.14.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
  - Intermernet 12y ago
    Thanks, much appreciated. I thought there may be some strange cyclic dependency going on, but it seems not.
    - ksherlock 12y ago
      It would only be a problem if that specific perl command (perl -e 'print join(chr(0), @ARGV), chr(0)' -- args....) or the shell invoking it depended on wordexp(). Recursion is fine as long as it terminates.
- SrslyJosh 12y ago
  > Inserting a NUL byte between each word, and at the end doesn't sound like it requires Perl... No, it's more clever than that. The shell will actually execute something like this: /usr/bin/perl -e 'print join(chr(0), @ARGV), chr(0)' -- your input string goes here So, what happens is that the shell (in this case, bash) performs argument expansion on your input string, then calls perl -e '...' with the expanded words as its arguments. What perl does is join all of those arguments with a NUL byte and spit them back out for the calling process to read, which makes figuring out where each expanded "word" begins and ends really simple. Could you do this in shell? Probably. Would a correct implementation be as short and easy to understand? Probably not. Therefore, perl.
melling 12y ago
Well, you are warned: /* XXX this is _not_ designed to be fast */
- nandemo 12y ago
  Performance is the least of my worries here. I'd be more worried about security, and plain dependency management. I mean, let's say this libc thing is supposed to be installed in quite a lot of systems, are we really sure that perl thing is installed in all of those systems? And what if, by some crazy coincidence, perl happens to depend on libc?
  - derefr 12y ago
    In Debian, at least, there's actually a number of circular dependencies within the (small) set of packages marked as "essential"; they all sort of lean on one-another like a house of cards. Perl is in that set, if you're wondering.
  - ryanlol 12y ago
    Almost everything depends on perl. Unless you're running a very small embedded setup, you'll have extremely hard time living without perl. Circular dependencies are perfectly normal.
    - nandemo 12y ago
      I'm aware that a lot of projects use perl for the build. But on runtime? I doubt it.
      - merijnv 12y ago
        It's Apple's libc. They ship always ship Perl on OSX (and presumably iOS too?), so what do they care whether other platforms have to deal with an extra dependency? Their platform has Perl anyway, so the dependency is essentially free.
      - ryanlol 12y ago
ryan0x00 12y ago
sometimes you have to get shit done
- _pmf_ 12y ago
  > sometimes you have to get shit done And Apple have always been masters in selling polished, nicely packages shit as the most advanced technology ever conceived.
  - jes 12y ago
    With respect, is there evidence to support your claim? My direct experience is that I have received value far in excess of what I have had to pay for the Apple products and services I have purchased over the years. For example, I'm typing this reply on my PowerBook, which I use all day, every day in my work. I also have my iPhone with me constantly, and use it heavily. Finally, the Apple brand was said to be worth about $104.7B vs. $62.8B for Microsoft in 2014. [1] So, I am not saying you are wrong, only that I think some evidence would be helpful in making your case. [1] http://bgr.com/2014/03/19/apple-vs-google-vs-microsoft-brand-value/ http://bgr.com/2014/03/19/apple-vs-google-vs-microsoft-brand...
    - smcl 12y ago
      OT, but you're still using a Powerbook? That's pretty impressive. Which generation\year was it? I'm guessing you've had to swap out batteries and upgrade hard drive - have you had to modify anything else?
      - jes 12y ago
        Sorry, my mistake. It's a mid-2012 MacBook Pro. 16GB DDR and 512GB SSD. http://support.apple.com/kb/SP694 http://support.apple.com/kb/SP694
      - coldtea 12y ago
        He meant MacBook Pro. That said, I still have an iBook G4 from 2004 in perfect operation for mail/web/music (though one button is missing).
cnvogel 12y ago
As it's already written in the manual... http://linux.die.net/man/3/wordexp http://linux.die.net/man/3/wordexp wordexp, wordfree - perform word expansion like a posix-shell So, the implementer decided to take the short route, and just spawn a shell which, essentially, gets passed the input data to wordexp(), to do the work. But, of course, having something that starts with such a comment... /* XXX this is _not_ designed to be fast (...) wordexp is also rife with security "challenges", unless you pass it WRDE_NOCMD it must support subshell expansion, and even if you don't beause it has to support so much of the standard shell (all the odd little variable expansion options for example) it is hard to do without a subshell). It is probably just plan a Bad Idea to call in anything setuid, or executing remotely. */ ...in your standard C library wasn't such a smart idea to start with. Scroll down, there are many more gems in the comments! Sometimes it's better to just implement it as void wordexp() { fprintf(stderr,"wordexp() is a security nightmare. Not implemented.\n"); assert(0); } or decide to deliberately only implement a safe subset of the full functionality specified (e.g. only ~user-homedir expansion and $VARIABLES), to at least cover the common use-cases without creating a security nightmare. (EDIT: typos)
- geofft 12y ago
  "Like a POSIX shell" includes expanding command substitution. wordexp("`echo hi`") should return {"hi", NULL}. So it isn't totally ridiculous to reach for a shell; tracking multiple subprocesses, and nested command substitution (like "$(echo $(echo $(echo foo) $(echo bar)))") is a pain to reimplement correctly. In an ideal world, we'd have a libsh that exposed all of the steps of what /bin/sh does in a nice fashion, this function would call one of the libsh functions, and /bin/sh would be a 10-line while (true) { libsh_this(); libsh_that(); }. In a slightly less ideal world, the shell would have a way to separate things with null characters, without bothering Perl. We really don't live in an ideal world.
  - cnvogel 12y ago
    I understand that spawning a shell to do the expansion work probably is a good idea when your goal is to "expand a string like the shell". But my guess is that most users will call wordexp() just to expand a few variables or tilde-homedirs.
dxq 12y ago
This is source code from 2011. Checking http://opensource.apple.com http://opensource.apple.com will show that it's not longer implemented this way in 10.10. Here's the current implementation: http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/FreeBSD/wordexp.c http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/Fr...
- Jaecen 12y ago
  Can you explain why wordexp.c on the referenced site has a completely different (and older) copyright? Did they merge in another implementation?
  - dxq 12y ago
    I'm guessing that the linked implementation probably hadn't been touched since 2008. My comment that this was source code from 2011 was referring to the fact that this release of libc (Libc-763.11) is from 2011.
- mrpippy 12y ago
  The current one is a derivative of NetBSD's implementation (where it shells out to use a sh builtin)
  - dxq 12y ago
    Judging by both the source code and its placement in the repository (gen/FreeBSD/wordexp.c), I'm thinking it's a derivative of the FreeBSD implementation (which may itself be a derivative of the NetBSD implementation, or perhaps the other way around).
    - JoachimS 12y ago
      Yes, the OSX code seems to come from FreeBSD. Here is the FreeBSD version of the source: https://github.com/freebsd/freebsd/blob/master/lib/libc/gen/wordexp.c https://github.com/freebsd/freebsd/blob/master/lib/libc/gen/... It looks very similar to the current OSX file posted in an earlier comment. Calls /bin/sh too.
dunham 12y ago
They appear to be using a "wordexp-helper" in Yosemite. I found this in /usr/lib/system/libsystem_c.dylib (and no references to perl). [ $# -gt 0 ] && export IFS="$1";/usr/lib/system/wordexp-helper
- dxq 12y ago
  Source to wordexp-helper here: http://opensource.apple.com/source/system_cmds/system_cmds-643.1.1/wordexp-helper.tproj/wordexp-helper.c http://opensource.apple.com/source/system_cmds/system_cmds-6... Also the source to the Yosemite implementation is available at: http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/FreeBSD/wordexp.c http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/Fr...
comex 12y ago
Here is a vulnerability in the current version [1] for your amusement: #include <wordexp.h> #include <stdio.h> int main() { wordexp_t we; char *s = "$(($(sleep 10)))"; printf("->%d\n", wordexp(s, &we, WRDE_NOCMD)); } (Since the manual page strongly recommends not trusting wordexp with untrusted input even with WRDE_NOCMD, and based on a code search the function is rarely used in the first place, I don't think it's really sensitive.) http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/FreeBSD/wordexp.c http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/Fr...

103 points by devbug 12y ago

Sorry, I don't have a Mac to check, but does the OSX build of Perl depend on libc? Also, that code seems to be from 2008 (or at least that's the latest year in the copyright header) despite the commit being from 2012. Does anyone know if this has been updated? Inserting a NUL byte between each word, and at the end doesn't sound like it requires Perl...

adamnemecek 12y ago
> otool -L /usr/bin/perl /usr/bin/perl: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1151.14.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
- Intermernet 12y ago
  Thanks, much appreciated. I thought there may be some strange cyclic dependency going on, but it seems not.
  - ksherlock 12y ago
    It would only be a problem if that specific perl command (perl -e 'print join(chr(0), @ARGV), chr(0)' -- args....) or the shell invoking it depended on wordexp(). Recursion is fine as long as it terminates.
SrslyJosh 12y ago
> Inserting a NUL byte between each word, and at the end doesn't sound like it requires Perl... No, it's more clever than that. The shell will actually execute something like this: /usr/bin/perl -e 'print join(chr(0), @ARGV), chr(0)' -- your input string goes here So, what happens is that the shell (in this case, bash) performs argument expansion on your input string, then calls perl -e '...' with the expanded words as its arguments. What perl does is join all of those arguments with a NUL byte and spit them back out for the calling process to read, which makes figuring out where each expanded "word" begins and ends really simple. Could you do this in shell? Probably. Would a correct implementation be as short and easy to understand? Probably not. Therefore, perl.

melling 12y ago

Well, you are warned: /* XXX this is _not_ designed to be fast */

nandemo 12y ago
Performance is the least of my worries here. I'd be more worried about security, and plain dependency management. I mean, let's say this libc thing is supposed to be installed in quite a lot of systems, are we really sure that perl thing is installed in all of those systems? And what if, by some crazy coincidence, perl happens to depend on libc?
- derefr 12y ago
  In Debian, at least, there's actually a number of circular dependencies within the (small) set of packages marked as "essential"; they all sort of lean on one-another like a house of cards. Perl is in that set, if you're wondering.
- ryanlol 12y ago
  Almost everything depends on perl. Unless you're running a very small embedded setup, you'll have extremely hard time living without perl. Circular dependencies are perfectly normal.
  - nandemo 12y ago
    I'm aware that a lot of projects use perl for the build. But on runtime? I doubt it.
    - merijnv 12y ago
      It's Apple's libc. They ship always ship Perl on OSX (and presumably iOS too?), so what do they care whether other platforms have to deal with an extra dependency? Their platform has Perl anyway, so the dependency is essentially free.
    - ryanlol 12y ago

ryan0x00 12y ago

sometimes you have to get shit done

_pmf_ 12y ago
> sometimes you have to get shit done And Apple have always been masters in selling polished, nicely packages shit as the most advanced technology ever conceived.
- jes 12y ago
  With respect, is there evidence to support your claim? My direct experience is that I have received value far in excess of what I have had to pay for the Apple products and services I have purchased over the years. For example, I'm typing this reply on my PowerBook, which I use all day, every day in my work. I also have my iPhone with me constantly, and use it heavily. Finally, the Apple brand was said to be worth about $104.7B vs. $62.8B for Microsoft in 2014. [1] So, I am not saying you are wrong, only that I think some evidence would be helpful in making your case. [1] http://bgr.com/2014/03/19/apple-vs-google-vs-microsoft-brand-value/ http://bgr.com/2014/03/19/apple-vs-google-vs-microsoft-brand...
  - smcl 12y ago
    OT, but you're still using a Powerbook? That's pretty impressive. Which generation\year was it? I'm guessing you've had to swap out batteries and upgrade hard drive - have you had to modify anything else?
    - jes 12y ago
      Sorry, my mistake. It's a mid-2012 MacBook Pro. 16GB DDR and 512GB SSD. http://support.apple.com/kb/SP694 http://support.apple.com/kb/SP694
    - coldtea 12y ago
      He meant MacBook Pro. That said, I still have an iBook G4 from 2004 in perfect operation for mail/web/music (though one button is missing).

cnvogel 12y ago

As it's already written in the manual... http://linux.die.net/man/3/wordexp http://linux.die.net/man/3/wordexp wordexp, wordfree - perform word expansion like a posix-shell So, the implementer decided to take the short route, and just spawn a shell which, essentially, gets passed the input data to wordexp(), to do the work. But, of course, having something that starts with such a comment... /* XXX this is _not_ designed to be fast (...) wordexp is also rife with security "challenges", unless you pass it WRDE_NOCMD it must support subshell expansion, and even if you don't beause it has to support so much of the standard shell (all the odd little variable expansion options for example) it is hard to do without a subshell). It is probably just plan a Bad Idea to call in anything setuid, or executing remotely. */ ...in your standard C library wasn't such a smart idea to start with. Scroll down, there are many more gems in the comments! Sometimes it's better to just implement it as void wordexp() { fprintf(stderr,"wordexp() is a security nightmare. Not implemented.\n"); assert(0); } or decide to deliberately only implement a safe subset of the full functionality specified (e.g. only ~user-homedir expansion and $VARIABLES), to at least cover the common use-cases without creating a security nightmare. (EDIT: typos)

geofft 12y ago
"Like a POSIX shell" includes expanding command substitution. wordexp("`echo hi`") should return {"hi", NULL}. So it isn't totally ridiculous to reach for a shell; tracking multiple subprocesses, and nested command substitution (like "$(echo $(echo $(echo foo) $(echo bar)))") is a pain to reimplement correctly. In an ideal world, we'd have a libsh that exposed all of the steps of what /bin/sh does in a nice fashion, this function would call one of the libsh functions, and /bin/sh would be a 10-line while (true) { libsh_this(); libsh_that(); }. In a slightly less ideal world, the shell would have a way to separate things with null characters, without bothering Perl. We really don't live in an ideal world.
- cnvogel 12y ago
  I understand that spawning a shell to do the expansion work probably is a good idea when your goal is to "expand a string like the shell". But my guess is that most users will call wordexp() just to expand a few variables or tilde-homedirs.

dxq 12y ago

This is source code from 2011. Checking http://opensource.apple.com http://opensource.apple.com will show that it's not longer implemented this way in 10.10. Here's the current implementation: http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/FreeBSD/wordexp.c http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/Fr...

Jaecen 12y ago
Can you explain why wordexp.c on the referenced site has a completely different (and older) copyright? Did they merge in another implementation?
- dxq 12y ago
  I'm guessing that the linked implementation probably hadn't been touched since 2008. My comment that this was source code from 2011 was referring to the fact that this release of libc (Libc-763.11) is from 2011.
mrpippy 12y ago
The current one is a derivative of NetBSD's implementation (where it shells out to use a sh builtin)
- dxq 12y ago
  Judging by both the source code and its placement in the repository (gen/FreeBSD/wordexp.c), I'm thinking it's a derivative of the FreeBSD implementation (which may itself be a derivative of the NetBSD implementation, or perhaps the other way around).
  - JoachimS 12y ago
    Yes, the OSX code seems to come from FreeBSD. Here is the FreeBSD version of the source: https://github.com/freebsd/freebsd/blob/master/lib/libc/gen/wordexp.c https://github.com/freebsd/freebsd/blob/master/lib/libc/gen/... It looks very similar to the current OSX file posted in an earlier comment. Calls /bin/sh too.

dunham 12y ago

They appear to be using a "wordexp-helper" in Yosemite. I found this in /usr/lib/system/libsystem_c.dylib (and no references to perl). [ $# -gt 0 ] && export IFS="$1";/usr/lib/system/wordexp-helper

dxq 12y ago
Source to wordexp-helper here: http://opensource.apple.com/source/system_cmds/system_cmds-643.1.1/wordexp-helper.tproj/wordexp-helper.c http://opensource.apple.com/source/system_cmds/system_cmds-6... Also the source to the Yosemite implementation is available at: http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/FreeBSD/wordexp.c http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/Fr...

comex 12y ago

Here is a vulnerability in the current version [1] for your amusement: #include <wordexp.h> #include <stdio.h> int main() { wordexp_t we; char *s = "$(($(sleep 10)))"; printf("->%d\n", wordexp(s, &we, WRDE_NOCMD)); } (Since the manual page strongly recommends not trusting wordexp with untrusted input even with WRDE_NOCMD, and based on a code search the function is rarely used in the first place, I don't think it's really sensitive.) http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/FreeBSD/wordexp.c http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/Fr...