3 ms·

I feel we're witnessing a resurgence of interest in 'nix default programs such as `sed` and `awk` in part because LLMs make it so much easier to get started in

I feel we're witnessing a resurgence of interest in 'nix default programs such as `sed` and `awk` in part because LLMs make it so much easier to get started in them, and because they really do exist everywhere you might look. (The fact they were designed to be performant in bygone decades and are super-performant now as a result is also nice!) There is just something incredibly freeing about knowing you can sit down at a freshly-reinstalled box and do productive work without having to install a single thing on the box itself first. EDIT: https://hiandrewquinn.github.io/til-site/posts/what-programming-languages-come-out-of-the-box-on-debian-12/ https://hiandrewquinn.github.io/til-site/posts/what-programm... might be of interest if you want to know what you can work with right out of the box on Debian 12. Other distros might differ.

WizardClickBoy 2y ago
100% agree. I'm currently preparing several 10s of GBs of HTML in nested directories for static hosting via S3 and was floundering until Gippity recommended find + exec sed to me. I'm now batch fixing issues (think 'not enough "../" in 60000 relative hrefs in nested directories') with a single command rather than writing scripts and feel like a wizard. These tools are things I've used before but always found painful and confusing. Being able to ask Gippity for detailed explanations of what is happening, in particular being able to paste a failing command and have it explain what the problem is, has been a game changer. In general, for those of us who never had a command line wizard colleague or mentor to show what is possible, LLMs are an absolute game changer both in terms of recommending tools and showing how to use them.
- godelski 2y ago
  Primeagen detected I find him hard to listen to when he does things like this
  - WizardClickBoy 2y ago
    Primeagen is some kind of Youtuber? I am not familiar and don't understand what you are trying to convey here.
    - 000ooo000 2y ago
      Guessing 'gippity' has been used by primeagen recently, so now you're gonna be tarred with the 18-23 React bootcamp graduate brush (at least that's who I imagine find him watchable).
      - WizardClickBoy 2y ago
        It's a case of convergent evolution - I don't know where I heard it first, but I asked GPT if it minded and it said "Of course, you can call me Gippity!", so I do, because it's more fun.
    - poulpy123 2y ago
      yes, and a cringy one
- barrkel 2y ago
  If you have a lot of files, consider find piped to xargs with -P for parallelism and -n to limit the number of files per parallel invocation. Only a tiny bit more complex but often an order of magnitude faster with today's CPUs. Use -print0 on find with -0 on xargs to handle spaces in filenames correctly. GNU parallel is another step up, but xargs is generally always to hand.
  - WizardClickBoy 2y ago
    Thanks! Gippity did suggest the xargs approach as an alternative, but I found that find [...] - exec [...] {} + as opposed to find [...] - exec [...] {} \; worked fine and was performant enough for my use-case. An example command was find . -type f -name "*.html" -exec sed -i '' -e 's/\.\.\/\.\.\/\.\.\//\.\.\/\.\.\/\.\.\/source\//g' {} + which took about 20s to run
    - mdaniel 2y ago
      One can express your sed in less Leaning Toothpick Syndrome[1] via: find . -type f -name "*.html" -exec sed -i '' -e 's|\.\./\.\./\.\./|../../../source/|g' {} + Using "/" as the delineation character for "s" patterns that include "/" drives me batshit - almost as much as scripts that use the doublequote for strings that contain no variables but also contain doublequotes (looking at you, json literals in awscli examples) If your sed is GNU, or otherwise sane, one can also `sed -Ee` and then use `s|\Q../../../|` getting rid of almost every escape character. I got you half way there because one need not escape the "." in the replacement pattern because "." isn't a meta character in the replacement space - what would that even mean? 1: https://en.wikipedia.org/wiki/Leaning_toothpick_syndrome https://en.wikipedia.org/wiki/Leaning_toothpick_syndrome
  - skydhash 2y ago
    Parallel is nice when doing music conversion with ffmpeg.
deleted 2y ago
[deleted]
dools 2y ago
I needed some scripts to run a little “factory” for flashing an operating system onto some IoT devices. Lots of the work was running various shell commands but it is nonetheless something I would have traditionally written in PHP or Python but I thought “what the hell” and did the whole thing in bash with ChatGPT and it was a totally mind blowing experience. Now I use bash for all sorts of stuff. I’ve been working with *nix for 20 years but bash is so arcane and my needs always so immediate that I never did anything other than use it to run commands in sequence with maybe a $1 or a $2 in there
godelski 2y ago
I've gotten into it recently but actually not because LLMs. Actually I find them unhelpful here. The reason I've gotten into it is because I wanted to make a bunch of install scripts for programs I want on fresh boxes. Mostly it's been fun. Seeing what I can do with curl, sed, awk, regex, and bash scripting. I'm often finding that I can do a ton of things in a single line where I would have done a lot more if I wrote it in python or something else. Idk, there's just something very fun about this. Though what's been a little frustrating is that there's anti scraping measures and they break things. But they're always trivial to get around, so it's just annoying. A big reason LLMs and up failing is that I need my scripts to work on osx and nix machines. So it's always suggesting things to me that work on one but not the other. It seems to not want to listen to my constraints and grep is problematic for them in particular. Luckily man pages are great. I think they're often over looked.
- asicsp 2y ago
  If you are able to install specific implementations of the tools, go with GNU tools on all the machines. That way, you'd get more features and work the same everywhere. If that is not an option, go with Perl. It'd be a little slower, but you'll get consistent results. Plus, Perl has powerful regex, lots of standard libraries, etc.
  - godelski 2y ago
    Well the fun is, as I was trying to convey, building the tools automatically from fresh boxes. Sure, I can bootstrap my way by first installing gnu coreutils but if this was about doing things the easy way I'd just use the relevant package manager and ansible like everyone else
keybored 2y ago
I resent this combination. - We never figured out how to package programs properly (Nix needs to become easier to use) - For all kinds of smaller tasks we practically need to use those Unix tools - Those everywhere tools are for hysterical raisins hard to use in a larger context (The Unix Philosophy in practice: use these five different tools but keep in mind that they are each different from each other across six dimensions and also they have defaults from the 70’s or 80’s) - For a lot of “simple” things you need to remember the simple thing plus eight comments (on the StackOverflow answer which has 166 votes but that’s just because it was the first to answer the question) with nuance like “this won’t work for your coworker on Mac” - So you don’t: you go to SO (see previous) and use snippets (see first point: we don’t know how to package programs, this is the best we got) - This works fine until Google Search decides that you are too reliant on it for it to have to work well - Now you don’t use “random stuff from StackOverflow” which can at least have an audit trail: now you use random weights from your LLM in order to make “simple” solutions (six Unix tools in a small Bash script which you can’t read because Bash is hard) This is pretty much the opposite of what inspired me when studying computer science and programming.
- skydhash 2y ago
  > We never figured out how to package programs properly What the issue with apt, pacman, and the others? I think they’re doing their job fine. > For all kinds of smaller tasks we practically need to use those Unix tools I mean, they’re good for what they do > Those everywhere tools are for hysterical raisins hard to use in a larger context Because each does a universal task you may want to do in the unix world of files and stream of texts. > For a lot of “simple” things you need to remember the simple thing plus eight comments No, you just need the manuals. And there are books too. And yes the difference between BSD and GNU is not obvious at first glance. But they’re different software worked on by different people.
  - keybored 2y ago
    Both of the points I made are what really are tragic: 1. (the things you disagree with) 2. Using AI to compensate for (1) So if you only disagree with (1) then I don’t know if I should get into it.

WizardClickBoy 2y ago

100% agree. I'm currently preparing several 10s of GBs of HTML in nested directories for static hosting via S3 and was floundering until Gippity recommended find + exec sed to me. I'm now batch fixing issues (think 'not enough "../" in 60000 relative hrefs in nested directories') with a single command rather than writing scripts and feel like a wizard. These tools are things I've used before but always found painful and confusing. Being able to ask Gippity for detailed explanations of what is happening, in particular being able to paste a failing command and have it explain what the problem is, has been a game changer. In general, for those of us who never had a command line wizard colleague or mentor to show what is possible, LLMs are an absolute game changer both in terms of recommending tools and showing how to use them.

godelski 2y ago
Primeagen detected I find him hard to listen to when he does things like this
- WizardClickBoy 2y ago
  Primeagen is some kind of Youtuber? I am not familiar and don't understand what you are trying to convey here.
  - 000ooo000 2y ago
    Guessing 'gippity' has been used by primeagen recently, so now you're gonna be tarred with the 18-23 React bootcamp graduate brush (at least that's who I imagine find him watchable).
    - WizardClickBoy 2y ago
      It's a case of convergent evolution - I don't know where I heard it first, but I asked GPT if it minded and it said "Of course, you can call me Gippity!", so I do, because it's more fun.
  - poulpy123 2y ago
    yes, and a cringy one
barrkel 2y ago
If you have a lot of files, consider find piped to xargs with -P for parallelism and -n to limit the number of files per parallel invocation. Only a tiny bit more complex but often an order of magnitude faster with today's CPUs. Use -print0 on find with -0 on xargs to handle spaces in filenames correctly. GNU parallel is another step up, but xargs is generally always to hand.
- WizardClickBoy 2y ago
  Thanks! Gippity did suggest the xargs approach as an alternative, but I found that find [...] - exec [...] {} + as opposed to find [...] - exec [...] {} \; worked fine and was performant enough for my use-case. An example command was find . -type f -name "*.html" -exec sed -i '' -e 's/\.\.\/\.\.\/\.\.\//\.\.\/\.\.\/\.\.\/source\//g' {} + which took about 20s to run
  - mdaniel 2y ago
    One can express your sed in less Leaning Toothpick Syndrome[1] via: find . -type f -name "*.html" -exec sed -i '' -e 's|\.\./\.\./\.\./|../../../source/|g' {} + Using "/" as the delineation character for "s" patterns that include "/" drives me batshit - almost as much as scripts that use the doublequote for strings that contain no variables but also contain doublequotes (looking at you, json literals in awscli examples) If your sed is GNU, or otherwise sane, one can also `sed -Ee` and then use `s|\Q../../../|` getting rid of almost every escape character. I got you half way there because one need not escape the "." in the replacement pattern because "." isn't a meta character in the replacement space - what would that even mean? 1: https://en.wikipedia.org/wiki/Leaning_toothpick_syndrome https://en.wikipedia.org/wiki/Leaning_toothpick_syndrome
- skydhash 2y ago
  Parallel is nice when doing music conversion with ffmpeg.

deleted 2y ago

[deleted]

dools 2y ago

I needed some scripts to run a little “factory” for flashing an operating system onto some IoT devices. Lots of the work was running various shell commands but it is nonetheless something I would have traditionally written in PHP or Python but I thought “what the hell” and did the whole thing in bash with ChatGPT and it was a totally mind blowing experience. Now I use bash for all sorts of stuff. I’ve been working with *nix for 20 years but bash is so arcane and my needs always so immediate that I never did anything other than use it to run commands in sequence with maybe a $1 or a $2 in there

godelski 2y ago

I've gotten into it recently but actually not because LLMs. Actually I find them unhelpful here. The reason I've gotten into it is because I wanted to make a bunch of install scripts for programs I want on fresh boxes. Mostly it's been fun. Seeing what I can do with curl, sed, awk, regex, and bash scripting. I'm often finding that I can do a ton of things in a single line where I would have done a lot more if I wrote it in python or something else. Idk, there's just something very fun about this. Though what's been a little frustrating is that there's anti scraping measures and they break things. But they're always trivial to get around, so it's just annoying. A big reason LLMs and up failing is that I need my scripts to work on osx and nix machines. So it's always suggesting things to me that work on one but not the other. It seems to not want to listen to my constraints and grep is problematic for them in particular. Luckily man pages are great. I think they're often over looked.

asicsp 2y ago
If you are able to install specific implementations of the tools, go with GNU tools on all the machines. That way, you'd get more features and work the same everywhere. If that is not an option, go with Perl. It'd be a little slower, but you'll get consistent results. Plus, Perl has powerful regex, lots of standard libraries, etc.
- godelski 2y ago
  Well the fun is, as I was trying to convey, building the tools automatically from fresh boxes. Sure, I can bootstrap my way by first installing gnu coreutils but if this was about doing things the easy way I'd just use the relevant package manager and ansible like everyone else

keybored 2y ago

I resent this combination. - We never figured out how to package programs properly (Nix needs to become easier to use) - For all kinds of smaller tasks we practically need to use those Unix tools - Those everywhere tools are for hysterical raisins hard to use in a larger context (The Unix Philosophy in practice: use these five different tools but keep in mind that they are each different from each other across six dimensions and also they have defaults from the 70’s or 80’s) - For a lot of “simple” things you need to remember the simple thing plus eight comments (on the StackOverflow answer which has 166 votes but that’s just because it was the first to answer the question) with nuance like “this won’t work for your coworker on Mac” - So you don’t: you go to SO (see previous) and use snippets (see first point: we don’t know how to package programs, this is the best we got) - This works fine until Google Search decides that you are too reliant on it for it to have to work well - Now you don’t use “random stuff from StackOverflow” which can at least have an audit trail: now you use random weights from your LLM in order to make “simple” solutions (six Unix tools in a small Bash script which you can’t read because Bash is hard) This is pretty much the opposite of what inspired me when studying computer science and programming.

skydhash 2y ago
> We never figured out how to package programs properly What the issue with apt, pacman, and the others? I think they’re doing their job fine. > For all kinds of smaller tasks we practically need to use those Unix tools I mean, they’re good for what they do > Those everywhere tools are for hysterical raisins hard to use in a larger context Because each does a universal task you may want to do in the unix world of files and stream of texts. > For a lot of “simple” things you need to remember the simple thing plus eight comments No, you just need the manuals. And there are books too. And yes the difference between BSD and GNU is not obvious at first glance. But they’re different software worked on by different people.
- keybored 2y ago
  Both of the points I made are what really are tragic: 1. (the things you disagree with) 2. Using AI to compensate for (1) So if you only disagree with (1) then I don’t know if I should get into it.