6 ms·

Text isn't the universal interface in Unix, byte streams are. You can quite happily send non-textual control characters and such around in Unix, or pipe data co

Text isn't the universal interface in Unix, byte streams are. You can quite happily send non-textual control characters and such around in Unix, or pipe data containing NULLs from one process to another. 'Text' is a very seductive abstraction, but it's one of the most brutal to work with once you start interacting with the real world and have to give up on ascii and deal with encodings and unicode and so on. Putting commands and data inline is a recipe for disaster and a million command injection exploits. The Unix philosophy has broken the minds of generations of programmers. It leads them to doing things like concatenating strings to build SQL queries or doing IPC with ad-hoc regex-parsed protocols or using a couple of magical characters to indicate that the contents of a variable should be parsed and executed instead of just stored. Take a read of some of the earlier threads on HN about Shellshock, and you will find numerous people blaming Apache for not "escaping" the data it was putting in a shell variable. As if it even could. Even Unix nerds have at least partially internalised the dangerousness of the paradigm -- "don't parse the output of ls" and so on. The fact that the Unix paradigm (passing everything as strings with magical characters and escape sequences) is broken for the most fundamental computing tasks like working with file names ought to be a damning inditement of the paradigm. Sadly people merely parrot the rote learned lesson "don't parse ls because file names can't be trusted", without thinking about all the other untrusted data they expose to unix shells all the time. Just this week Yahoo got exploited. At first people thought it was Shellshock, but no, it was just a routine command injection vulnerability in their log processing shell scripts. A problem blighting just about every non-trivial shell script ever written. The usual reply is "don't use shells with untrusted data". But auditing where any particular bit of data came from can be just about impossible once it has been across several systems through programmes written in a variety of languages, stored on a file system, read back and so on. The only sane solution is to never use shell scripts. Like the C memory and integer model makes writing secure C code borderline impossible, the Unix "single pipe of bytes that defaults to being commands" paradigm makes writing secure shell scripts borderline impossible. Unix needs to be taken out back and shot.

rkrzr 12y ago
Agree that text is being abused in Unix all the time. The problem with passing everything around as text is that you cannot reason about anything, because everything is of the same type. One big advantage of object-based systems is that they can catch type errors and notify you of the problem. Text pipelines will simply break because one of your implicit assumptions didn't hold.
jschwartzi 12y ago
I would not be half as capable of troubleshooting in my work as I am if I couldn't parse random byte strings on the command line. That alone is why I use Unix for embedded development. Powerful things are often dangerous. For instance, consider nuclear power, motor vehicles, rocket engines, medicine, plastics, et. al.. So how do you provide the same level of capability as C and UNIX but without mixing data and commands? Is there some alternative paradigm that is just as powerful but safe?
- BruceM 12y ago
  Watch the Mathematica-backed StrangeLoop 2014 keynote by Stephen Wolfram. That's one way to think about it. There are others like how things work on the Lisp Machine (and somewhat similarly in CLIM).
mwcampbell 12y ago
> Putting commands and data inline is a recipe for disaster and a million command injection exploits. The Unix philosophy has broken the minds of generations of programmers. It leads them to doing things like concatenating strings to build SQL queries or doing IPC with ad-hoc regex-parsed protocols or using a couple of magical characters to indicate that the contents of a variable should be parsed and executed instead of just stored. Yes! And then to compensate, they have to "sanitize" untrusted input to their systems. I had a meeting yesterday with a developer and a project manager at an organization that wants to work with my company to integrate one of our products with one of theirs. I mentioned the possibility of submitting some data in JSON format to a web API on their end, and the project manager asked about the risk of code injection attacks, by which he apparently meant SQL injection. I had to assure him, based on my knowledge of their tech stack (Node.js, CouchDB, and naturally, JSON) that code injection wouldn't be an issue. My point is that the common abuses of strings by Unix and web developers have led to well-known and widely feared security vulnerabilities which just don't exist in software that's built on a foundation of properly structured data. See also this classic by Glyph Lefkowitz: https://glyph.twistedmatrix.com/2008/06/data-in-garbage-out.html https://glyph.twistedmatrix.com/2008/06/data-in-garbage-out....
pyre 12y ago
> Take a read of some of the earlier threads on HN about Shellshock, and you will find numerous people blaming Apache for not "escaping" the data it was putting in a shell variable. It seems to me that this is the result of Cargo Cult Programming. People know that SQL strings, and user input need to be 'escaped,' so they just think "Obviously this needs to be escaped too! It's user input!" Yet they don't realize that they are trying to put a square peg in a round hole. They just know that pegs go through holes, so they keep banging away at it. Also, it's always amazed me that there was never some sort of 'standard' way to shell-escape things, even though the shell has been around for ages. Why can't I generate a shell string in the same way that I generate a SQL string? E.g.: sprintf("mv %t %t", src, dest); Where "%t" is a special token that shell-escapes the input (e.g. "My File Name.txt" => "My\ File\ Name.txt"). Instead it's something where people continue to use ad-hoc, incomplete, of 'implemented everywhere' solutions to this. Note: I don't generate SQL strings with sprintf(), but it's a close approximation of: execute('select * from table where id = ?', id);
- mw42 12y ago
  SQL injection is not avoided by escaping arguments, but by never mixing the command and user supplied arguments in the first place. The equivalent to your example would be execlp("mv", "mv", src, dest, NULL); which does not rely on the shell to try to untangle your arguments from a single string. Edit: Fixed, thanks!
  - nitrogen 12y ago
    Similarly, no amount of escaping would protect you from Shellshock. P.S. Doesn't execlp() require a NULL at the end of the parameter list?
  - pyre 12y ago
    > SQL injection is not avoided by escaping arguments, but by never mixing the command and user supplied arguments in the first place. People see: execute("select * from table where id = ?", id); as for the most part like: execute(sprintf("select * from table where id = '%s'", escape(id))); Where `escape()` is written by "smarter people" and makes sure that `id` isn't a string like this: 0'; delete all from user;' (e.g. turning it into `0''; delete all from user;''`). I realize that this isn't what actually happens, but the general idea is that you are sanitizing your inputs.
felixgallo 12y ago
so in one corner: the ability to compose small, comprehensible functions with data streams, delivering staggering riches to the world. in the other corner: the complaint that sometimes, people can put carriage returns in their filenames and screw up the output of `ls -1`. With no suggestion of what would replace composed functional streams. Nice try Lennart.
- mwcampbell 12y ago
  > in the other corner: the complaint that sometimes, people can put carriage returns in their filenames and screw up the output of `ls -1`. With no suggestion of what would replace composed functional streams. How about functions that operate on data structures such as lists and maps? These can include generic functions that can slice and dice data structures regardless of what type of data those structures contain. > Nice try Lennart. Is Mr. Poettering's name now an epithet to be thrown at anyone who opposes the Holy Unix Way? Or do you just know something I don't about the identity of the commenter to whom you were replying?
  - felixgallo 12y ago
    The unix dataset already works on list and map data structures. lists: ls -1 | wc -l maps: ps -ef | grep 'tobekilled' | grep -v grep | awk '{ print $2 }' | xargs kill would unix be better if it were cwd.files.count and processes.filter{name = 'tobekilled'}.map(kill) ? maybe for the newbie who paradoxically already understands functional and object-oriented programming, I'll grant. But the 'pipe' ("this is a bucket brigade") has always been pretty easy for people to grasp. The Lennart bit was a joke. The OP's post was so histrionically overwrought that I responded sardonically. Apologies if you were offended.
    - anon1385 12y ago
      I'm honestly not sure if your post is meant to be satire or not. >lists: ls -1 | wc -l The computer has taken a real array (probably an array in C), joined all the items together into one big string using magical characters as dividers, and then split it again on those magic characters to try and reconstruct the metadata that it threw away. I think the problem is pretty obvious and well known. >would unix be better if it were cwd.files.count Well, at least that is going to give you the correct result. Correctness seems like it should be pretty important, no? Are you really arguing for shells being easier to learn using an example of a complicated command with 4 pipes, 2 different quoted strings, several single letter arguments, and that requires implicit knowledge about the structure of the output from several commands? Compared to a much shorter, simpler, type safe, and self documenting bit of code? Also, yes my original post was hyperbolic. That was because I was responding to a histrionically overwrought post claiming that unix is perfect.
deleted 12y ago
[deleted]
rsync 12y ago
"Like the C memory and integer model makes writing secure C code borderline impossible, the Unix "single pipe of bytes that defaults to being commands" paradigm makes writing secure shell scripts borderline impossible. Unix needs to be taken out back and shot." What alternative do you advocate/propose ? Genuinely curious ...
- jcheng 12y ago
  I've hardly used it myself, but apparently Microsoft's PowerShell pipes typed objects between processes: http://technet.microsoft.com/en-us/library/dd347728.aspx http://technet.microsoft.com/en-us/library/dd347728.aspx
  - vertex-four 12y ago
    Actually, not between processes - all Powershell commands run in the same address space as the shell, and must be implemented in .Net. I don't think you can easily write an external process which takes a Powershell object directly as input.
    - jcheng 12y ago
      Thanks for the correction, that explains so much--I was picturing a lot of DCOM craziness.
- anon1385 12y ago
  For the interactive general purpose data munging and quick execution of simple commands that the shell is best at, I really don't know what a better system would look like. It seems like a really hard problem. Anything purely text based ends up being fairly cumbersome to use for simple commands if it has to use real data structures (consider having to type (["a", "b"]) instead of a b to pass arguments with json style syntax or whatever). At least that was my experience of trying to write a very simple shell. There are a hell of a lot of people a hell of a lot smarter than me though. It seems to me that a lot of shell scripts could be ported to other languages. Does DHCP on Linux need to use a shell script instead of python or something like that? The benefits of the shell grammar and semantics which are designed to make interactive use easy seem more like hindrances in a lot of those kinds of use cases. I assume it's largely done to make it easier for sysadmins to customise things. If I was a sysadmin I'd much rather learn python (and feel like I actually understood it) than the crazy byzantine grammar of bash. Maybe that's why I'm not a sysadmin. This paper by Rob Pike might also be of interest: http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf >The current UNIX® text processing tools are weakened by the built-in concept of a line.
hderms 12y ago
I agree with you to some degree. It's incredibly easy to write insecure software if it ever touches a shell.

rkrzr 12y ago

Agree that text is being abused in Unix all the time. The problem with passing everything around as text is that you cannot reason about anything, because everything is of the same type. One big advantage of object-based systems is that they can catch type errors and notify you of the problem. Text pipelines will simply break because one of your implicit assumptions didn't hold.

jschwartzi 12y ago

I would not be half as capable of troubleshooting in my work as I am if I couldn't parse random byte strings on the command line. That alone is why I use Unix for embedded development. Powerful things are often dangerous. For instance, consider nuclear power, motor vehicles, rocket engines, medicine, plastics, et. al.. So how do you provide the same level of capability as C and UNIX but without mixing data and commands? Is there some alternative paradigm that is just as powerful but safe?

BruceM 12y ago
Watch the Mathematica-backed StrangeLoop 2014 keynote by Stephen Wolfram. That's one way to think about it. There are others like how things work on the Lisp Machine (and somewhat similarly in CLIM).

mwcampbell 12y ago

> Putting commands and data inline is a recipe for disaster and a million command injection exploits. The Unix philosophy has broken the minds of generations of programmers. It leads them to doing things like concatenating strings to build SQL queries or doing IPC with ad-hoc regex-parsed protocols or using a couple of magical characters to indicate that the contents of a variable should be parsed and executed instead of just stored. Yes! And then to compensate, they have to "sanitize" untrusted input to their systems. I had a meeting yesterday with a developer and a project manager at an organization that wants to work with my company to integrate one of our products with one of theirs. I mentioned the possibility of submitting some data in JSON format to a web API on their end, and the project manager asked about the risk of code injection attacks, by which he apparently meant SQL injection. I had to assure him, based on my knowledge of their tech stack (Node.js, CouchDB, and naturally, JSON) that code injection wouldn't be an issue. My point is that the common abuses of strings by Unix and web developers have led to well-known and widely feared security vulnerabilities which just don't exist in software that's built on a foundation of properly structured data. See also this classic by Glyph Lefkowitz: https://glyph.twistedmatrix.com/2008/06/data-in-garbage-out.html https://glyph.twistedmatrix.com/2008/06/data-in-garbage-out....

pyre 12y ago

> Take a read of some of the earlier threads on HN about Shellshock, and you will find numerous people blaming Apache for not "escaping" the data it was putting in a shell variable. It seems to me that this is the result of Cargo Cult Programming. People know that SQL strings, and user input need to be 'escaped,' so they just think "Obviously this needs to be escaped too! It's user input!" Yet they don't realize that they are trying to put a square peg in a round hole. They just know that pegs go through holes, so they keep banging away at it. Also, it's always amazed me that there was never some sort of 'standard' way to shell-escape things, even though the shell has been around for ages. Why can't I generate a shell string in the same way that I generate a SQL string? E.g.: sprintf("mv %t %t", src, dest); Where "%t" is a special token that shell-escapes the input (e.g. "My File Name.txt" => "My\ File\ Name.txt"). Instead it's something where people continue to use ad-hoc, incomplete, of 'implemented everywhere' solutions to this. Note: I don't generate SQL strings with sprintf(), but it's a close approximation of: execute('select * from table where id = ?', id);

mw42 12y ago
SQL injection is not avoided by escaping arguments, but by never mixing the command and user supplied arguments in the first place. The equivalent to your example would be execlp("mv", "mv", src, dest, NULL); which does not rely on the shell to try to untangle your arguments from a single string. Edit: Fixed, thanks!
- nitrogen 12y ago
  Similarly, no amount of escaping would protect you from Shellshock. P.S. Doesn't execlp() require a NULL at the end of the parameter list?
- pyre 12y ago
  > SQL injection is not avoided by escaping arguments, but by never mixing the command and user supplied arguments in the first place. People see: execute("select * from table where id = ?", id); as for the most part like: execute(sprintf("select * from table where id = '%s'", escape(id))); Where `escape()` is written by "smarter people" and makes sure that `id` isn't a string like this: 0'; delete all from user;' (e.g. turning it into `0''; delete all from user;''`). I realize that this isn't what actually happens, but the general idea is that you are sanitizing your inputs.

felixgallo 12y ago

so in one corner: the ability to compose small, comprehensible functions with data streams, delivering staggering riches to the world. in the other corner: the complaint that sometimes, people can put carriage returns in their filenames and screw up the output of `ls -1`. With no suggestion of what would replace composed functional streams. Nice try Lennart.

mwcampbell 12y ago
> in the other corner: the complaint that sometimes, people can put carriage returns in their filenames and screw up the output of `ls -1`. With no suggestion of what would replace composed functional streams. How about functions that operate on data structures such as lists and maps? These can include generic functions that can slice and dice data structures regardless of what type of data those structures contain. > Nice try Lennart. Is Mr. Poettering's name now an epithet to be thrown at anyone who opposes the Holy Unix Way? Or do you just know something I don't about the identity of the commenter to whom you were replying?
- felixgallo 12y ago
  The unix dataset already works on list and map data structures. lists: ls -1 | wc -l maps: ps -ef | grep 'tobekilled' | grep -v grep | awk '{ print $2 }' | xargs kill would unix be better if it were cwd.files.count and processes.filter{name = 'tobekilled'}.map(kill) ? maybe for the newbie who paradoxically already understands functional and object-oriented programming, I'll grant. But the 'pipe' ("this is a bucket brigade") has always been pretty easy for people to grasp. The Lennart bit was a joke. The OP's post was so histrionically overwrought that I responded sardonically. Apologies if you were offended.
  - anon1385 12y ago
    I'm honestly not sure if your post is meant to be satire or not. >lists: ls -1 | wc -l The computer has taken a real array (probably an array in C), joined all the items together into one big string using magical characters as dividers, and then split it again on those magic characters to try and reconstruct the metadata that it threw away. I think the problem is pretty obvious and well known. >would unix be better if it were cwd.files.count Well, at least that is going to give you the correct result. Correctness seems like it should be pretty important, no? Are you really arguing for shells being easier to learn using an example of a complicated command with 4 pipes, 2 different quoted strings, several single letter arguments, and that requires implicit knowledge about the structure of the output from several commands? Compared to a much shorter, simpler, type safe, and self documenting bit of code? Also, yes my original post was hyperbolic. That was because I was responding to a histrionically overwrought post claiming that unix is perfect.

deleted 12y ago

[deleted]

rsync 12y ago

"Like the C memory and integer model makes writing secure C code borderline impossible, the Unix "single pipe of bytes that defaults to being commands" paradigm makes writing secure shell scripts borderline impossible. Unix needs to be taken out back and shot." What alternative do you advocate/propose ? Genuinely curious ...

jcheng 12y ago
I've hardly used it myself, but apparently Microsoft's PowerShell pipes typed objects between processes: http://technet.microsoft.com/en-us/library/dd347728.aspx http://technet.microsoft.com/en-us/library/dd347728.aspx
- vertex-four 12y ago
  Actually, not between processes - all Powershell commands run in the same address space as the shell, and must be implemented in .Net. I don't think you can easily write an external process which takes a Powershell object directly as input.
  - jcheng 12y ago
    Thanks for the correction, that explains so much--I was picturing a lot of DCOM craziness.
anon1385 12y ago
For the interactive general purpose data munging and quick execution of simple commands that the shell is best at, I really don't know what a better system would look like. It seems like a really hard problem. Anything purely text based ends up being fairly cumbersome to use for simple commands if it has to use real data structures (consider having to type (["a", "b"]) instead of a b to pass arguments with json style syntax or whatever). At least that was my experience of trying to write a very simple shell. There are a hell of a lot of people a hell of a lot smarter than me though. It seems to me that a lot of shell scripts could be ported to other languages. Does DHCP on Linux need to use a shell script instead of python or something like that? The benefits of the shell grammar and semantics which are designed to make interactive use easy seem more like hindrances in a lot of those kinds of use cases. I assume it's largely done to make it easier for sysadmins to customise things. If I was a sysadmin I'd much rather learn python (and feel like I actually understood it) than the crazy byzantine grammar of bash. Maybe that's why I'm not a sysadmin. This paper by Rob Pike might also be of interest: http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf >The current UNIX® text processing tools are weakened by the built-in concept of a line.

hderms 12y ago

I agree with you to some degree. It's incredibly easy to write insecure software if it ever touches a shell.