4 ms·

If I do one thing this year, I need to learn more about these tried and tested command-line unix shell utilities. It's becoming increasingly obvious that so ma

If I do one thing this year, I need to learn more about these tried and tested command-line unix shell utilities. It's becoming increasingly obvious that so many complicated and overwrought things are being unleashed on the world because (statistically) nobody knows this stuff anymore.

trumped 8y ago
They try to make the cloud relevant for a lot of things that could easily be accomplished locally... Most of the time, they do that just to get your data.
- SteveNuts 8y ago
  Vendor lock-in is what they're after in the cloud case. Every time we talk to our cloud provider reps they ask us which of their SaaS offerings we're using. Their heads spin when we tell them we're just using VMs and ask us why.
beokop 8y ago
So much internal tooling at a company can be avoided if you can just get people comfortable with basic *nix programs. No, we don’t need a web interface to show that data. No, we don’t need to code a dropdown to sort it differently.
- fleitz 8y ago
  Yes, you do need a web interface for that, thankfully we have cgi-bin from the list of unix tools. Turns out browsers can render text :)
- digi_owl 8y ago
  And yet the UX people, particularly around Linux, wants to bury the _nix cli so damned deep...
  - AnIdiotOnTheNet 8y ago
    Can't say I blame them. The base concept is sound but the implementation is archaic and obtuse as hell.
  - tormeh 8y ago
    CLIs have a discoverability problem. Knowing how to use any given one is a skill that takes years to hone. Learning a single command can take up to an hour. Most webapps are more constrained, but a common operation can usually be accomplished by someone who hasn't seen it before in less than 3 minutes.
    - jonhendry18 8y ago
      The 'apropos' command is very handy on machine or OS you aren't familiar with. If you want to find out what tools are available on the CLI that operate on jpegs, you'd try 'apropos jpeg' and get a list of things that mention jpeg in their man page.
deckard1 8y ago
This applies to much of the software world. Many of the places I've worked have a bad habit of ditching perfectly fine tech in order to chase the latest thing. Part of it is driven by RDD (resume driven development) and positioning the company so they can hire the latest crop of fad chasers. You end up with a bunch of specialists that are siloed into their particular tech. You got the Docker guy. You got the Redis guy. You may even have one of those gray-haired DBAs that still talk about this "Oracle" thing all day long, and no one knows what the hell he's going on about. You got the Ruby guys, but now you guy a bunch of React or Angular guys. What to do with the Ruby guys? Can't just fire them yet. You have a dozen of 'em, and it took you years to hire them. Fine, just banish them to the backend somewhere. Give them some fidget spinners and a few pointless projects to keep them busy.
- sixdimensional 8y ago
  This is exactly right. This is real life!! And it hurts.
- tomc1985 8y ago
  Hey now, Ruby ain't dead. I am one of these "Ruby" guys and, in the 48-or-so hours since my profile was activated at a new recruiting shop this week, I've gotten several potential candidate employers (some in seed stage) who want to get in touch. The real issue is why do we silo ourselves so? It's fucking stupid. I do so much more than Ruby but conveying that to the new breed of tech people is just impossible. All they see is your biggest resume badge, and all they care about is how wide you will spread your legs for them.
- monksy 8y ago
  Let's not forget when you're interviewing: The new places expect the latest technologies. Don't have time to learn spark, flink, kafka streams, and repeat scala spec whilst having intimate knowledge of all of the REST libraries? BAD DEVELOPER.
eb0la 8y ago
You just need to master a few commands: Find (take a look at -exec option) Cut (or awk) Sed (for simple text/string substitutions) Xargs Dd (that beast can do charset transtation from ASCII to EBCDIC to use in mainframes and can also wipe disks) And bash of course :)
- sidlls 8y ago
  awk is so beautiful. A good awk one-liner can beat fancy pandas all day long in some cases.
  - flukus 8y ago
    I hate how awk gets so much praise for one liners when it's so good at building things like state machines too: https://two-wrongs.com/awk-state-machine-parser-pattern.html https://two-wrongs.com/awk-state-machine-parser-pattern.html I've got some similar ones for work that take data csv like data and output sql. A little bit of vim-foo on the csv (yy10000p) and I've got all the test data I need.
    - cosmie 8y ago
      If anything, awk should get some praise for it's flexibility to go from one liners to Doom-style 3d games[1]! [1] https://github.com/TheMozg/awk-raycaster https://github.com/TheMozg/awk-raycaster
  - cwilkes 8y ago
    An example here would be nice.
- hoffbrau99 8y ago
  Don't forget grep!
- eanzenberg 8y ago
  "master a few commands" "awk" brb 15 years
thediehl 8y ago
I started with Unix Shell Programming http://a.co/au9OUrB http://a.co/au9OUrB
nemild 8y ago
I made a list of a good set of commands for a full stack engineer: https://github.com/nemild/cli_for_full_stack https://github.com/nemild/cli_for_full_stack
- sh87 8y ago
  This is awesome! thanks.
sitkack 8y ago
You can write many of these yourself. Just reading records, and sending them across pipes in ways you can compose is a good start. I find PyPy is excellent for this kind of work, fast to write, if the pipeline runs for more than 10 seconds, it has plenty of time for the JIT to warm up and amortize the cost. If you don't need fancy libs, LuaJIT is amazing and great to doing transforms in a pipeline. I rewrote a Hadoop job to use Streaming (pipes, basic unix model) from a Java job to native. Just that switch was 10x faster. Mostly because of Hadoop overheads.
cosmie 8y ago
Companies aren't making these complicated and overwrought things because nobody knows this stuff anymore. The ones actually doing the making generally know all the tried and tested ways to do what they want, because they used them to scale. Then they eventually outgrow the existing solutions and created these complicated systems to account for whatever specific technical or business issues that necessitated hopping from existing solutions to creating a solution. The problem tends to come along with the visibility that comes when one of these get released publicly. People use these things without doing the same level of due diligence to see if something less complicated fits their needs. Either deliberately (resume driven development) or simply because it doesn't occur to them to do so and "if it's good enough for this unicorn it's good enough for me". Or in some cases they do due diligence, but incorrectly evaluate their needs and rule out simpler solutions. But the most common reason I've ran into is simply that it's more fun to play with these projects than it is to use crufty old stuff, no matter how tried and tested and potentially appropriate it is. At a previous company, I had a python script running on a cron job every 5 minutes to do some data processing needs. Once or twice a month, the batch would be so large that it took 6-7 minutes to complete and the cron job would trigger the script again before it finished, causing the second instance of the script to see the lock file, log an error, and exit. It didn't cause any problem for these periodically skipped runs, because the business need only required data to be processed within 24 hours of coming in. The 5 minute cron job was just to even out resource usage throughout the day instead of doing a larger nightly batch job. A piece of data not getting processed for 8 minutes instead of 5 did not make any material impact. Another team had noticed the errors popping up in the log and were in the process of testing out a whole bunch of real time data pipelines like Kafka, leveraging my error messages to justify the need for a "real time" system without ever even asking me about the errors. After I found out other people were noticing those superfluous errors, I moved the cron job to a 30 minute window to stop them from happening anymore. Turns out they didn't have any other justification for their data pipeline greenfield project and weren't happy to go back to their normal day to day work. I offered to let them maintain and expand my python scripts if they were interested in data processing work, but for some reason they never took me up on that offer. :(
- gaius 8y ago
  Companies aren't making these complicated and overwrought things because nobody knows this stuff anymore. The ones actually doing the making generally know all the tried and tested ways to do what they want, because they used them to scale Right, Yahoo made Hadoop because (probably) they needed it. 99% of companies... just don’t.
shawndumas 8y ago
Read The AWK Programming Language by Alfred Aho, Peter Weinberger, and Brian Kernighan "This is an absolute classic which although ancient in Computing years is an absolute gem full of relevance yet."
asicsp 8y ago
I have a repo dedicated to cli text-processing.. short-descriptions and tons of examples.. I had fun and learned a lot doing this link: https://github.com/learnbyexample/Command-line-text-processing https://github.com/learnbyexample/Command-line-text-processi...
- svde 8y ago
  This looks exactly like what I was looking for. Thank you.
  - asicsp 8y ago
    you're welcome, happy learning :)

trumped 8y ago

They try to make the cloud relevant for a lot of things that could easily be accomplished locally... Most of the time, they do that just to get your data.

SteveNuts 8y ago
Vendor lock-in is what they're after in the cloud case. Every time we talk to our cloud provider reps they ask us which of their SaaS offerings we're using. Their heads spin when we tell them we're just using VMs and ask us why.

beokop 8y ago

So much internal tooling at a company can be avoided if you can just get people comfortable with basic *nix programs. No, we don’t need a web interface to show that data. No, we don’t need to code a dropdown to sort it differently.

fleitz 8y ago
Yes, you do need a web interface for that, thankfully we have cgi-bin from the list of unix tools. Turns out browsers can render text :)
digi_owl 8y ago
And yet the UX people, particularly around Linux, wants to bury the _nix cli so damned deep...
- AnIdiotOnTheNet 8y ago
  Can't say I blame them. The base concept is sound but the implementation is archaic and obtuse as hell.
- tormeh 8y ago
  CLIs have a discoverability problem. Knowing how to use any given one is a skill that takes years to hone. Learning a single command can take up to an hour. Most webapps are more constrained, but a common operation can usually be accomplished by someone who hasn't seen it before in less than 3 minutes.
  - jonhendry18 8y ago
    The 'apropos' command is very handy on machine or OS you aren't familiar with. If you want to find out what tools are available on the CLI that operate on jpegs, you'd try 'apropos jpeg' and get a list of things that mention jpeg in their man page.

deckard1 8y ago

This applies to much of the software world. Many of the places I've worked have a bad habit of ditching perfectly fine tech in order to chase the latest thing. Part of it is driven by RDD (resume driven development) and positioning the company so they can hire the latest crop of fad chasers. You end up with a bunch of specialists that are siloed into their particular tech. You got the Docker guy. You got the Redis guy. You may even have one of those gray-haired DBAs that still talk about this "Oracle" thing all day long, and no one knows what the hell he's going on about. You got the Ruby guys, but now you guy a bunch of React or Angular guys. What to do with the Ruby guys? Can't just fire them yet. You have a dozen of 'em, and it took you years to hire them. Fine, just banish them to the backend somewhere. Give them some fidget spinners and a few pointless projects to keep them busy.

sixdimensional 8y ago
This is exactly right. This is real life!! And it hurts.
tomc1985 8y ago
Hey now, Ruby ain't dead. I am one of these "Ruby" guys and, in the 48-or-so hours since my profile was activated at a new recruiting shop this week, I've gotten several potential candidate employers (some in seed stage) who want to get in touch. The real issue is why do we silo ourselves so? It's fucking stupid. I do so much more than Ruby but conveying that to the new breed of tech people is just impossible. All they see is your biggest resume badge, and all they care about is how wide you will spread your legs for them.
monksy 8y ago
Let's not forget when you're interviewing: The new places expect the latest technologies. Don't have time to learn spark, flink, kafka streams, and repeat scala spec whilst having intimate knowledge of all of the REST libraries? BAD DEVELOPER.

eb0la 8y ago

You just need to master a few commands: Find (take a look at -exec option) Cut (or awk) Sed (for simple text/string substitutions) Xargs Dd (that beast can do charset transtation from ASCII to EBCDIC to use in mainframes and can also wipe disks) And bash of course :)

sidlls 8y ago
awk is so beautiful. A good awk one-liner can beat fancy pandas all day long in some cases.
- flukus 8y ago
  I hate how awk gets so much praise for one liners when it's so good at building things like state machines too: https://two-wrongs.com/awk-state-machine-parser-pattern.html https://two-wrongs.com/awk-state-machine-parser-pattern.html I've got some similar ones for work that take data csv like data and output sql. A little bit of vim-foo on the csv (yy10000p) and I've got all the test data I need.
  - cosmie 8y ago
    If anything, awk should get some praise for it's flexibility to go from one liners to Doom-style 3d games[1]! [1] https://github.com/TheMozg/awk-raycaster https://github.com/TheMozg/awk-raycaster
- cwilkes 8y ago
  An example here would be nice.
hoffbrau99 8y ago
Don't forget grep!
eanzenberg 8y ago
"master a few commands" "awk" brb 15 years

thediehl 8y ago

I started with Unix Shell Programming http://a.co/au9OUrB http://a.co/au9OUrB

nemild 8y ago

I made a list of a good set of commands for a full stack engineer: https://github.com/nemild/cli_for_full_stack https://github.com/nemild/cli_for_full_stack

sh87 8y ago
This is awesome! thanks.

sitkack 8y ago

You can write many of these yourself. Just reading records, and sending them across pipes in ways you can compose is a good start. I find PyPy is excellent for this kind of work, fast to write, if the pipeline runs for more than 10 seconds, it has plenty of time for the JIT to warm up and amortize the cost. If you don't need fancy libs, LuaJIT is amazing and great to doing transforms in a pipeline. I rewrote a Hadoop job to use Streaming (pipes, basic unix model) from a Java job to native. Just that switch was 10x faster. Mostly because of Hadoop overheads.

cosmie 8y ago

Companies aren't making these complicated and overwrought things because nobody knows this stuff anymore. The ones actually doing the making generally know all the tried and tested ways to do what they want, because they used them to scale. Then they eventually outgrow the existing solutions and created these complicated systems to account for whatever specific technical or business issues that necessitated hopping from existing solutions to creating a solution. The problem tends to come along with the visibility that comes when one of these get released publicly. People use these things without doing the same level of due diligence to see if something less complicated fits their needs. Either deliberately (resume driven development) or simply because it doesn't occur to them to do so and "if it's good enough for this unicorn it's good enough for me". Or in some cases they do due diligence, but incorrectly evaluate their needs and rule out simpler solutions. But the most common reason I've ran into is simply that it's more fun to play with these projects than it is to use crufty old stuff, no matter how tried and tested and potentially appropriate it is. At a previous company, I had a python script running on a cron job every 5 minutes to do some data processing needs. Once or twice a month, the batch would be so large that it took 6-7 minutes to complete and the cron job would trigger the script again before it finished, causing the second instance of the script to see the lock file, log an error, and exit. It didn't cause any problem for these periodically skipped runs, because the business need only required data to be processed within 24 hours of coming in. The 5 minute cron job was just to even out resource usage throughout the day instead of doing a larger nightly batch job. A piece of data not getting processed for 8 minutes instead of 5 did not make any material impact. Another team had noticed the errors popping up in the log and were in the process of testing out a whole bunch of real time data pipelines like Kafka, leveraging my error messages to justify the need for a "real time" system without ever even asking me about the errors. After I found out other people were noticing those superfluous errors, I moved the cron job to a 30 minute window to stop them from happening anymore. Turns out they didn't have any other justification for their data pipeline greenfield project and weren't happy to go back to their normal day to day work. I offered to let them maintain and expand my python scripts if they were interested in data processing work, but for some reason they never took me up on that offer. :(

gaius 8y ago
Companies aren't making these complicated and overwrought things because nobody knows this stuff anymore. The ones actually doing the making generally know all the tried and tested ways to do what they want, because they used them to scale Right, Yahoo made Hadoop because (probably) they needed it. 99% of companies... just don’t.

shawndumas 8y ago

Read The AWK Programming Language by Alfred Aho, Peter Weinberger, and Brian Kernighan "This is an absolute classic which although ancient in Computing years is an absolute gem full of relevance yet."

asicsp 8y ago

I have a repo dedicated to cli text-processing.. short-descriptions and tons of examples.. I had fun and learned a lot doing this link: https://github.com/learnbyexample/Command-line-text-processing https://github.com/learnbyexample/Command-line-text-processi...

svde 8y ago
This looks exactly like what I was looking for. Thank you.
- asicsp 8y ago
  you're welcome, happy learning :)