7 ms·

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones t

by rofo1 8y ago

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong that I am surprised that even at this inefficacy, it's such a viable and well-paid profession. It's almost as if we are creating insanely complex solutions that in turn require a large amount of developers to support them, whereas we could have chosen a much more practical solution which is self-sustaining.

Aeolun 8y ago
You forget that using more developers means more headcount, and more headcount means I have more responsibility as a manager. These crazy complex solutions also look a lot more difficult on the slides than the simple 3 layer architecture that’s often shown to me. With something that simple, and requiring so few people, how am I ever going to convince my clients to pay me multiple millions of dollars for it.
- semi-extrinsic 8y ago
  I guess you should sell them the choice of either the big complex thing written by a large team for $2.3 MM, or the lean highly-optimized solution written by a handful of elite "10X" developers for $1.9 MM. (Of course the 10X developers are in high demand and very expensive.)
atoav 8y ago
As a young programmer I certainly found some love for all these unix tools and learn new things every day. I think the problem is scale. Back in the day (before I was born), very few people were programmers and the resources they could use were limited. This means they didn’t need insanely complex solutions because they already needed complex solutions just to make it work on the limited hardware. People were trying to solve problems with computers. Nowaday you take a problem that could be solved by an microcontroller with three buttons and make it a cloud app with web server, web interface and all kind of other things like containers. We donlt really tend to ask the question what a good solution would look like. Often it is the case that you just use the technology the developer wants to learn
craigsmansion 8y ago
> I feel like the art of UNIX is slowly fading into oblivion I think that by itself isn't a problem, but fading right along with it is the capacity to decompose and structure the problem domain. Even if one ends up writing a solution in a different language for whatever reasons, starting out by mapping the problem with UNIX command line tools will result in a better understanding of the problem; an understanding that is language agnostic and can be transferred to any preferred method of implementation.
agumonkey 8y ago
Things fade and shine in succession. Good bits will always come back. It's a bit like the saying about mathematics truthiness nature: doesn't matter who or when you look, they will re-emerge as is. composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words.. Societies are large and full of random fluxes and waves.. right now it might be the time for Wirth 17 pages long solutions .. but McIlroy one liner will come back.
- jstimpfle 8y ago
  > composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words.. It's not so much about "tiny" but about no fuss, low complexity, no-nonsense, etc. For example, Forth may be tiny, but it's unusable for most applications. Assembler languages are tiny, too.
- c256 8y ago
  Agreed, and well said. > right now it might be the time for Wirth 17 pages long solutions .. but McIlroy one liner will come back. In case someone is interested in &ollowing up this reference, it’s Knuth’s 17 page long solution.
- tsimionescu 8y ago
  > composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words The problem with this type of thinking is that "composing tiny bits" is obviously good to anyone. BUT there are vast differences between composing unix commands and composing lisp functions (and composing tiny JS libraries in NPM etc.) and some of these are good, while some are not. It's actually very difficult to properly create a system out of composable, re-usable components. I would actually say that Unix pipes are a particularly bad model of how to do that (as are NPM micro-libraries). No well-defined interfaces, arcane naming etc. - they are all reasons why most modern versions of these tools have grown and grown.
  - kazinator 8y ago
    Composing unix commands means endlessly adjusting the textual output of one tool to the input requirements of another. Most such coding only cares about the happy cases; it will march on with an incorrect datum because of some unexpected twist in a textual format. For instance, a field delimiting character suddenly occurs in a field. Or some code is scraping data by hard columns, but a field width overflows and the columns shift.
0815test 8y ago
> I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. ... People said the exact same thing in the 1990s, too. "Enterprise"-managed projects, using tech stacks like Win32 or Java, have generally tended to produce large, unwieldly monoliths.
lapnitnelav 8y ago
I think it's mostly a matter of chance more than anything else. If you've jumped straight into programming, you'll probably consider any of those problems as a nail to your C/JS/Java/Python Hammer. I was lucky to be initiated to the GNU / UNIX toolset by operation folks when doing tech support in a SAAS biz. We were dealing with a lot of text files and it didn't feel right to offload whatever my problem was to them, so I started learning from them and the scripts they wrote. Mostly trivial stuff, such as in a directory, only select the relevant files, iterate through them to sum up something or find exactly the bit of information you need. This allowed me to decrease time spent looking for the answer to my current case or colleague's case. Of course, I then built those functionalities into a .bashrc (or was it bash_profile?) script to help my fellow support folks finding the answer on their own.. Turns out, most people don't want to have anything to do with a command prompt, even if the hard part has been done for you. That's been a pretty good lesson BTW. Fast forward a few years later to now and I'm realising a good amount of stuff I've been writing lately could potentially be GNU'd instead of writing a Python script. So that's how it goes I guess.
- wildmindwriting 8y ago
  I have also found this to be the case too. Most people would rather have a GUI before even touching the command line. Most notably is Git; every single one of my developers use Sourcetree and if I have to help them with something, I always have to pop open the terminal. It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.
  - radiator 8y ago
    I don't see how this can be a target of their joke: they have problems with their GUI (I am assuming that's what Sourcetree is), you solve them with your CLI. If they want to laugh, they 'd better fix their own problems themselves, I guess?
    - 4thaccount 8y ago
      They probably respect his knowledge and know he is the only competent one, but they simply don't care to learn the CLI method as they are lazy, or can't justify all the time it takes to learn it if he is there to fix their problems (more efficient). I've been on both sides of that situation before. There is always an expert in something you want to know, but can't justify the time. So don't take it too hard on them. I support a Linux based app at work and thus got pretty comfortable with the command line (vim, grep, awk, head, tail, cut, sort, ls, cp, top, cat...etc). To my knowledge I'm the only one in engineering with this knowledge (not in IT support). I also notice my fellow engineers will frequently use more complicated techniques for something that is a single piped command for me. I don't fault them for it though as they have no need to use Linux.
empath75 8y ago
These complex solutions allow one operations engineer to manage thousands and thousands of servers/containers. Guys that just knew how to bang together bash and Perl scripts got laid off all over the place in favor of people that know cloud stuff.
- 4thaccount 8y ago
  I believe the author probably understands that fully and is instead referring to the situation where the optimal solution is a little bash script and somehow the developer designs this horribly complex solution that isn't any more performant, but seems more fancy. Sadly, I think developers sometime do this just to learn new things and keep their resume up to date. It's rough out there from what I read (glad I'm not a developer).
- alchemism 8y ago
  It turns out that banging together bash scripts which contain invocations of cloud SDK tools has as much relevance in the k8s era as any before. Maybe it’s Python instead of Perl but that is about the most-significant change.
deleted 8y ago
[deleted]
sigi45 8y ago
Leads are having the responsibility. Most of them are good and know there shit. So I wouldn't worry to much. All other small companies who have people who don't know better, don't have too much technology inside anyway. Or at least often enough.
forgotmypw7 8y ago
I think the issue you describe is partially a case of "when all you've got is a hammer..." When all you've got is a computer and your own hands, you use the Unix way. If you have a team (or the money to hire a team) of 20 developers, then you start looking for a problem that you can solve with that hammer.
redsavagefiero 8y ago
This is something I've noticed in the last 8-10 years. The rise of the python/js/java paradigm everywhere. Some of the associated LDIF (json) I enjoy much more than XML and flat files but the misapplication of tools is becoming an epidemic. When I can write: awk -F "," '{for (x = 1 ; x <= NF ; x++) {if ($x ~ /[0-9]+/) {a[x] = a[x] + $x}}} END { for (p in a) {printf "%d = %d\n",p,a[p]}}' to sum columns in 5 seconds and people are scrambling with libraries to do matrix operations I tend to scratch my head and walk away. The aversion to the command line is also something that bothers me but I don't run into it as much in my field.
- oarabbus_ 8y ago
  >When I can write: awk -F "," '{for (x = 1 ; x <= NF ; x++) {if ($x ~ /[0-9]+/) {a[x] = a[x] + $x}}} END { for (p in a) {printf "%d = %d\n",p,a[p]}}' To be fair, what the matrix libraries do is provide readability and clarity. Show a programmer unfamiliar with awk your statement and they're going to be spending quite a while parsing it. Show a programmer unfamiliar with numpy/pandas some matrix multiplication and they may understand it intuitively for the most part without even having to look up references. edit: I'm actually an awk noob myself but after rereading your code for a second, it makes quite a bit of sense. "For all rows in the column, take all digits 0-9 and sum them". So perhaps not much is gained by the library
- joshuamorton 8y ago
  When I can write import pandas as pd data = pd.read_csv(filename) print(data.sum()) and have the same result, I'm going to do the one that is faster to write, fewer characters, and lets me understand what's going on. And don't get me wrong, I've written some gnarly pipelined bash before, although I'm by no means an expert, but that doesn't mean its always the right tool for the job.
  - redsavagefiero 8y ago
    I was going to be the long haired *nix geek here but I have no hair and the world is moving on. I can't pick bones with python for data science/analysis and personal convenience. _However_ as a principal engineer if someone was to say, for a trivial dataset, that we need python and pandas for an operation like this where python + pandas was not already provisioned the answer would be no.
MisterOctober 8y ago
I dunno -- there's definitely a community of developers who prefer not to use the command-line / Unix toolset, but at least half of those I know [especially the macOS users and those with some ops experience] routinely reach for the core Unix utilities , especially when doing ad hoc quick fixes / file processing. For my part, I can't imagine doing my job without them [or I guess more accurately, my job wouldn't exist in the first place with them] I think git has a pretty significant amount to do with the persistence of command-line tool use among the younger set -- it's just so much more efficient / easier to use from the terminal.
l_t 8y ago
I don't think so. What is old is new again; even this blog post is an example of a developer relearning that core lesson. Also, I'd like to point out that in both this blog post and the Taco Bell one, the "UNIX way" is being compared against a straw man example, not a real example of over-engineering. Neither post actually provides any evidence of any real inefficiency. Both authors are just trying to explain and improve their own thinking about programming, not trying to cast judgement on a generation.
333c 8y ago
> I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. For what it's worth, I'm a first-year computer science major and the class I'm currently taking is very much focused on the "art of Unix." We've been doing shell scripting and regular expressions and the like. I quite enjoy it.

by rofo1 8y ago

Aeolun 8y ago

You forget that using more developers means more headcount, and more headcount means I have more responsibility as a manager. These crazy complex solutions also look a lot more difficult on the slides than the simple 3 layer architecture that’s often shown to me. With something that simple, and requiring so few people, how am I ever going to convince my clients to pay me multiple millions of dollars for it.

semi-extrinsic 8y ago
I guess you should sell them the choice of either the big complex thing written by a large team for $2.3 MM, or the lean highly-optimized solution written by a handful of elite "10X" developers for $1.9 MM. (Of course the 10X developers are in high demand and very expensive.)

atoav 8y ago

As a young programmer I certainly found some love for all these unix tools and learn new things every day. I think the problem is scale. Back in the day (before I was born), very few people were programmers and the resources they could use were limited. This means they didn’t need insanely complex solutions because they already needed complex solutions just to make it work on the limited hardware. People were trying to solve problems with computers. Nowaday you take a problem that could be solved by an microcontroller with three buttons and make it a cloud app with web server, web interface and all kind of other things like containers. We donlt really tend to ask the question what a good solution would look like. Often it is the case that you just use the technology the developer wants to learn

craigsmansion 8y ago

> I feel like the art of UNIX is slowly fading into oblivion I think that by itself isn't a problem, but fading right along with it is the capacity to decompose and structure the problem domain. Even if one ends up writing a solution in a different language for whatever reasons, starting out by mapping the problem with UNIX command line tools will result in a better understanding of the problem; an understanding that is language agnostic and can be transferred to any preferred method of implementation.

agumonkey 8y ago

Things fade and shine in succession. Good bits will always come back. It's a bit like the saying about mathematics truthiness nature: doesn't matter who or when you look, they will re-emerge as is. composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words.. Societies are large and full of random fluxes and waves.. right now it might be the time for Wirth 17 pages long solutions .. but McIlroy one liner will come back.

jstimpfle 8y ago
> composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words.. It's not so much about "tiny" but about no fuss, low complexity, no-nonsense, etc. For example, Forth may be tiny, but it's unusable for most applications. Assembler languages are tiny, too.
c256 8y ago
Agreed, and well said. > right now it might be the time for Wirth 17 pages long solutions .. but McIlroy one liner will come back. In case someone is interested in &ollowing up this reference, it’s Knuth’s 17 page long solution.
tsimionescu 8y ago
> composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words The problem with this type of thinking is that "composing tiny bits" is obviously good to anyone. BUT there are vast differences between composing unix commands and composing lisp functions (and composing tiny JS libraries in NPM etc.) and some of these are good, while some are not. It's actually very difficult to properly create a system out of composable, re-usable components. I would actually say that Unix pipes are a particularly bad model of how to do that (as are NPM micro-libraries). No well-defined interfaces, arcane naming etc. - they are all reasons why most modern versions of these tools have grown and grown.
- kazinator 8y ago
  Composing unix commands means endlessly adjusting the textual output of one tool to the input requirements of another. Most such coding only cares about the happy cases; it will march on with an incorrect datum because of some unexpected twist in a textual format. For instance, a field delimiting character suddenly occurs in a field. Or some code is scraping data by hard columns, but a field width overflows and the columns shift.

0815test 8y ago

> I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. ... People said the exact same thing in the 1990s, too. "Enterprise"-managed projects, using tech stacks like Win32 or Java, have generally tended to produce large, unwieldly monoliths.

lapnitnelav 8y ago

I think it's mostly a matter of chance more than anything else. If you've jumped straight into programming, you'll probably consider any of those problems as a nail to your C/JS/Java/Python Hammer. I was lucky to be initiated to the GNU / UNIX toolset by operation folks when doing tech support in a SAAS biz. We were dealing with a lot of text files and it didn't feel right to offload whatever my problem was to them, so I started learning from them and the scripts they wrote. Mostly trivial stuff, such as in a directory, only select the relevant files, iterate through them to sum up something or find exactly the bit of information you need. This allowed me to decrease time spent looking for the answer to my current case or colleague's case. Of course, I then built those functionalities into a .bashrc (or was it bash_profile?) script to help my fellow support folks finding the answer on their own.. Turns out, most people don't want to have anything to do with a command prompt, even if the hard part has been done for you. That's been a pretty good lesson BTW. Fast forward a few years later to now and I'm realising a good amount of stuff I've been writing lately could potentially be GNU'd instead of writing a Python script. So that's how it goes I guess.

wildmindwriting 8y ago
I have also found this to be the case too. Most people would rather have a GUI before even touching the command line. Most notably is Git; every single one of my developers use Sourcetree and if I have to help them with something, I always have to pop open the terminal. It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.
- radiator 8y ago
  I don't see how this can be a target of their joke: they have problems with their GUI (I am assuming that's what Sourcetree is), you solve them with your CLI. If they want to laugh, they 'd better fix their own problems themselves, I guess?
  - 4thaccount 8y ago
    They probably respect his knowledge and know he is the only competent one, but they simply don't care to learn the CLI method as they are lazy, or can't justify all the time it takes to learn it if he is there to fix their problems (more efficient). I've been on both sides of that situation before. There is always an expert in something you want to know, but can't justify the time. So don't take it too hard on them. I support a Linux based app at work and thus got pretty comfortable with the command line (vim, grep, awk, head, tail, cut, sort, ls, cp, top, cat...etc). To my knowledge I'm the only one in engineering with this knowledge (not in IT support). I also notice my fellow engineers will frequently use more complicated techniques for something that is a single piped command for me. I don't fault them for it though as they have no need to use Linux.

empath75 8y ago

These complex solutions allow one operations engineer to manage thousands and thousands of servers/containers. Guys that just knew how to bang together bash and Perl scripts got laid off all over the place in favor of people that know cloud stuff.

4thaccount 8y ago
I believe the author probably understands that fully and is instead referring to the situation where the optimal solution is a little bash script and somehow the developer designs this horribly complex solution that isn't any more performant, but seems more fancy. Sadly, I think developers sometime do this just to learn new things and keep their resume up to date. It's rough out there from what I read (glad I'm not a developer).
alchemism 8y ago
It turns out that banging together bash scripts which contain invocations of cloud SDK tools has as much relevance in the k8s era as any before. Maybe it’s Python instead of Perl but that is about the most-significant change.

deleted 8y ago

[deleted]

sigi45 8y ago

Leads are having the responsibility. Most of them are good and know there shit. So I wouldn't worry to much. All other small companies who have people who don't know better, don't have too much technology inside anyway. Or at least often enough.

forgotmypw7 8y ago

I think the issue you describe is partially a case of "when all you've got is a hammer..." When all you've got is a computer and your own hands, you use the Unix way. If you have a team (or the money to hire a team) of 20 developers, then you start looking for a problem that you can solve with that hammer.

redsavagefiero 8y ago

This is something I've noticed in the last 8-10 years. The rise of the python/js/java paradigm everywhere. Some of the associated LDIF (json) I enjoy much more than XML and flat files but the misapplication of tools is becoming an epidemic. When I can write: awk -F "," '{for (x = 1 ; x <= NF ; x++) {if ($x ~ /[0-9]+/) {a[x] = a[x] + $x}}} END { for (p in a) {printf "%d = %d\n",p,a[p]}}' to sum columns in 5 seconds and people are scrambling with libraries to do matrix operations I tend to scratch my head and walk away. The aversion to the command line is also something that bothers me but I don't run into it as much in my field.

oarabbus_ 8y ago
>When I can write: awk -F "," '{for (x = 1 ; x <= NF ; x++) {if ($x ~ /[0-9]+/) {a[x] = a[x] + $x}}} END { for (p in a) {printf "%d = %d\n",p,a[p]}}' To be fair, what the matrix libraries do is provide readability and clarity. Show a programmer unfamiliar with awk your statement and they're going to be spending quite a while parsing it. Show a programmer unfamiliar with numpy/pandas some matrix multiplication and they may understand it intuitively for the most part without even having to look up references. edit: I'm actually an awk noob myself but after rereading your code for a second, it makes quite a bit of sense. "For all rows in the column, take all digits 0-9 and sum them". So perhaps not much is gained by the library
joshuamorton 8y ago
When I can write import pandas as pd data = pd.read_csv(filename) print(data.sum()) and have the same result, I'm going to do the one that is faster to write, fewer characters, and lets me understand what's going on. And don't get me wrong, I've written some gnarly pipelined bash before, although I'm by no means an expert, but that doesn't mean its always the right tool for the job.
- redsavagefiero 8y ago
  I was going to be the long haired *nix geek here but I have no hair and the world is moving on. I can't pick bones with python for data science/analysis and personal convenience. _However_ as a principal engineer if someone was to say, for a trivial dataset, that we need python and pandas for an operation like this where python + pandas was not already provisioned the answer would be no.

MisterOctober 8y ago

I dunno -- there's definitely a community of developers who prefer not to use the command-line / Unix toolset, but at least half of those I know [especially the macOS users and those with some ops experience] routinely reach for the core Unix utilities , especially when doing ad hoc quick fixes / file processing. For my part, I can't imagine doing my job without them [or I guess more accurately, my job wouldn't exist in the first place with them] I think git has a pretty significant amount to do with the persistence of command-line tool use among the younger set -- it's just so much more efficient / easier to use from the terminal.

l_t 8y ago

I don't think so. What is old is new again; even this blog post is an example of a developer relearning that core lesson. Also, I'd like to point out that in both this blog post and the Taco Bell one, the "UNIX way" is being compared against a straw man example, not a real example of over-engineering. Neither post actually provides any evidence of any real inefficiency. Both authors are just trying to explain and improve their own thinking about programming, not trying to cast judgement on a generation.

333c 8y ago

> I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. For what it's worth, I'm a first-year computer science major and the class I'm currently taking is very much focused on the "art of Unix." We've been doing shell scripting and regular expressions and the like. I quite enjoy it.