3 ms·

> They often make great toys in the sense that toys are fun but it's harder to build robust things in bash scripts. Programming is hard, regardless. You're bei

> They often make great toys in the sense that toys are fun but it's harder to build robust things in bash scripts. Programming is hard, regardless. You're being very dismissive towards shell scripters, and it comes off as elitist. > But where bash scripts fall down is the constant re-parsing of text, data has to be passed through the kernel, and extra processes (when you use xargs, which tends to happen for complicated tasks.) This adds up super quick. What do you mean by parsing? Copying, reading? The buffers sit in kernel memory, sure… but that's pretty fast if every line in your program is concurrent. > xargs Did you have a bad experience with xargs? > extra processes Did you have a bad experience with… running out of… PIDs? > > Casting bash as the insane bad language > I never said that You implied it by calling a non-bash language 'sane'; thus, bash must be insane. > I didn't recommend C Your blanket recommendation of 'anything but bash' did. I feel that sometimes you need to have more context of a situation before dictating a course of action that must be followed.

jacobparker 9y ago
> Programming is hard, regardless. You're being very dismissive towards shell scripters, and it comes off as elitist. I think you're taking this too personally. I write _a lot_ of bash scripts and like bash (as I've already indicated.) It's not always the right tool for the right job (no tool is...) > What do you mean by parsing? Copying, reading? The buffers sit in kernel memory, sure… but that's pretty fast if every line in your program is concurrent. The buffers don't just sit in the kernel, they are copied between processes (via write/read.) This involves many many context switches. By contrast (but this is just an example), libgit2 will hopefully mmap your packfiles, and once the data is mapped in you don't have to leave your process. Parsing is sometimes obvious, for example multiple passes over the data with grep and sed to shape it the way you want. It's true that components of the pipeline can run in parallel but the problem is they do a lot of unnecessary work - parallelizing unnecessary work doesn't remove it and once you hit the limit of # of cores (which is very relevant when generating HTML - something often done by webservers with many concurrent requests where you don't have cores to spare on parallelizing inefficient algorithms.) Parsing is sometime not as obvious, for example steps in the pipeline doing redundant Unicode validation. This is an artifact of squeezing things through read/write with reusable components. It's convenient (which makes it great for prototyping, or for things that "don't matter") but this adds up to poor performance. > Did you have a bad experience with xargs? > Did you have a bad experience with… running out of… PIDs? I think you're being very uncivil. xargs is useful but it (with the common options like -n and -P) can spawn many processes which has a tremendous overhead vs an alternative like a function call. It's purely a performance thing. (I probably should be using parallel but I'm used to xargs :( )
- falsedan 9y ago
  > I think you're being very uncivil. I'm sorry. I see how my questions come off as questioning your experience, and I didn't mean to do that. > I think you're taking this too personally. I think your reluctance to admit that you needlessly insulted some class of devs is exclusionary. Casual readers of these comments may think it's bad to write bash, or that it's ok to bash people who do. > The buffers don't just sit in the kernel, they are copied between processes (via write/read.) Yes, that's right. I know how pipes work, thanks. What you call 'parsing' I would call processing. Parsing has a well-defined meaning in the context of computer science: turning tokens into a data structure (like a syntax tree). > parallelizing unnecessary work doesn't remove it Is unnecessary work a problem? If you're being charged by the second, perhaps it may be cost-effective to eliminate it… but most of the time, developer productivity is a bigger cost. I see that you are mindful of the performance implication of your code. I usually have to think of developer throughput, so I'm very tolerant of inefficient use of hardware if it means someone gets their job done faster.
  - hnbroseph 9y ago
    > class of devs wait... what? is this marxist class struggle framing? there is no 'class of devs' in this sense. there is no 'social justice' here. bash is not a person, bash is not an identity, none of this applies.
    - falsedan 9y ago
      Class in the OO/taxonomy sense. I feel like you are projecting some of your existing issues onto this discussion.
      - hnbroseph 9y ago
        it's more related to the nature of your wording and framing. no group of humans is being abused or discriminated against.
        9y ago
throwawayjava 9y ago
Parent's original point is just that libraries/frameworks are a much safer way of writing programs that handle untrusted input in security-critical settings. Nothing you've said contradicts this original point (and I think it's a correct assessment). A compelling counter-argument, IMO, would be a demonstration of how to incorporate tried-and-trusted security checks into a bash script. >> I would struggle to justify the work I'd need to do to write the equivalent in C (or python, where it would be slower) for one command that's only run in the release process. The original article lists the CVEs that justify the extra work. You may say "that doesn't matter because any input to the release process must come from a trusted source anyways", but: 1) missing input validation can cause accidental catastrophic bugs just as easily as it can cause intentional catastrophic vulnerabilities, and 2) the whole point of defense in depth is that you shouldn't go around punching preventable holes in your defenses. IMO it shouldn't be possible to escalate from "request a build of the branch named 'X'" to "owning the release server".
- chubot 9y ago
  (author here) Yes, but you have to take into account the context. Every minute you spend "securing" your RELEASE NOTES is better spent securing the APPLICATION. A lot of people seem to be missing the point -- this is a quick solution that also correct (*). It's a middle ground. Yes it has drawbacks, and I analyzed those extensively in the article. Engineering is about tradeoffs. Given infinite resources, yes you want to do the "right thing". But you don't have infinite resources.
  - falsedan 9y ago
    why can't I upvote this infinite times
  - throwawayjava 9y ago
    > Every minute you spend "securing" your RELEASE NOTES is better spent securing the APPLICATION. I've never worked in a place where developer time is really that zero-sum. If your release notes are executable code running on your infrastructure, then their security does matter. I'm sure there's a team of developers at Avast who wish they had spent more time securing their "release notes".

jacobparker 9y ago

> Programming is hard, regardless. You're being very dismissive towards shell scripters, and it comes off as elitist. I think you're taking this too personally. I write _a lot_ of bash scripts and like bash (as I've already indicated.) It's not always the right tool for the right job (no tool is...) > What do you mean by parsing? Copying, reading? The buffers sit in kernel memory, sure… but that's pretty fast if every line in your program is concurrent. The buffers don't just sit in the kernel, they are copied between processes (via write/read.) This involves many many context switches. By contrast (but this is just an example), libgit2 will hopefully mmap your packfiles, and once the data is mapped in you don't have to leave your process. Parsing is sometimes obvious, for example multiple passes over the data with grep and sed to shape it the way you want. It's true that components of the pipeline can run in parallel but the problem is they do a lot of unnecessary work - parallelizing unnecessary work doesn't remove it and once you hit the limit of # of cores (which is very relevant when generating HTML - something often done by webservers with many concurrent requests where you don't have cores to spare on parallelizing inefficient algorithms.) Parsing is sometime not as obvious, for example steps in the pipeline doing redundant Unicode validation. This is an artifact of squeezing things through read/write with reusable components. It's convenient (which makes it great for prototyping, or for things that "don't matter") but this adds up to poor performance. > Did you have a bad experience with xargs? > Did you have a bad experience with… running out of… PIDs? I think you're being very uncivil. xargs is useful but it (with the common options like -n and -P) can spawn many processes which has a tremendous overhead vs an alternative like a function call. It's purely a performance thing. (I probably should be using parallel but I'm used to xargs :( )

falsedan 9y ago
> I think you're being very uncivil. I'm sorry. I see how my questions come off as questioning your experience, and I didn't mean to do that. > I think you're taking this too personally. I think your reluctance to admit that you needlessly insulted some class of devs is exclusionary. Casual readers of these comments may think it's bad to write bash, or that it's ok to bash people who do. > The buffers don't just sit in the kernel, they are copied between processes (via write/read.) Yes, that's right. I know how pipes work, thanks. What you call 'parsing' I would call processing. Parsing has a well-defined meaning in the context of computer science: turning tokens into a data structure (like a syntax tree). > parallelizing unnecessary work doesn't remove it Is unnecessary work a problem? If you're being charged by the second, perhaps it may be cost-effective to eliminate it… but most of the time, developer productivity is a bigger cost. I see that you are mindful of the performance implication of your code. I usually have to think of developer throughput, so I'm very tolerant of inefficient use of hardware if it means someone gets their job done faster.
- hnbroseph 9y ago
  > class of devs wait... what? is this marxist class struggle framing? there is no 'class of devs' in this sense. there is no 'social justice' here. bash is not a person, bash is not an identity, none of this applies.
  - falsedan 9y ago
    Class in the OO/taxonomy sense. I feel like you are projecting some of your existing issues onto this discussion.
    - hnbroseph 9y ago
      it's more related to the nature of your wording and framing. no group of humans is being abused or discriminated against.
      - 9y ago

throwawayjava 9y ago

Parent's original point is just that libraries/frameworks are a much safer way of writing programs that handle untrusted input in security-critical settings. Nothing you've said contradicts this original point (and I think it's a correct assessment). A compelling counter-argument, IMO, would be a demonstration of how to incorporate tried-and-trusted security checks into a bash script. >> I would struggle to justify the work I'd need to do to write the equivalent in C (or python, where it would be slower) for one command that's only run in the release process. The original article lists the CVEs that justify the extra work. You may say "that doesn't matter because any input to the release process must come from a trusted source anyways", but: 1) missing input validation can cause accidental catastrophic bugs just as easily as it can cause intentional catastrophic vulnerabilities, and 2) the whole point of defense in depth is that you shouldn't go around punching preventable holes in your defenses. IMO it shouldn't be possible to escalate from "request a build of the branch named 'X'" to "owning the release server".

chubot 9y ago
(author here) Yes, but you have to take into account the context. Every minute you spend "securing" your RELEASE NOTES is better spent securing the APPLICATION. A lot of people seem to be missing the point -- this is a quick solution that also correct (*). It's a middle ground. Yes it has drawbacks, and I analyzed those extensively in the article. Engineering is about tradeoffs. Given infinite resources, yes you want to do the "right thing". But you don't have infinite resources.
- falsedan 9y ago
  why can't I upvote this infinite times
- throwawayjava 9y ago
  > Every minute you spend "securing" your RELEASE NOTES is better spent securing the APPLICATION. I've never worked in a place where developer time is really that zero-sum. If your release notes are executable code running on your infrastructure, then their security does matter. I'm sure there's a team of developers at Avast who wish they had spent more time securing their "release notes".