3 ms·
"Whenever I think Sed or Awk would be correct for a problem..." is a good time to share the problem with HN. Generally, that rarely happens. We almost never s
by 1vuio0pswjnm7 4y ago
"Whenever I think Sed or Awk would be correct for a problem..." is a good time to share the problem with HN.
Generally, that rarely happens. We almost never see someone present a text processing problem (input.txt, expected_output.txt) and ask to see what some solutions using UNIX utilties, so that someone might compare with python, for example. Instead, we see endless criticisms of UNIX utilities, without presenting any specific examples (input.txt, expected_output.txt) to ilustrate the unsubstantiated arguments being made.
Without a working example to illustrate a criticism, these criticisms come across as nothing more than worthless opinions. Programmers commenting online provide a lifetime supply of such opinions.
For me, sed is a "daily driver" for text processing. There are numerous small scripts that I use every day that feature sed. I have a folder with hundreds of scripts that use sed. Some use grep and awk as well. However the best program for text processing for me is neither grep, sed nor awk.
It's flex.
If the argument is that python looks like "pseudocode" and sed does not, then I agree. No contest. No examples needed. Someone else long ago decided what "pseudocode" should look like. UNIX utility authors pursued different styles.
If the argument is that python has the best or most libraries, again I would not contest that.
However if the argument is something like "python solutions for text processing
(a) are faster to write,
(b) are smaller,
(c) run faster,
(d) are more robust/reliable,
(e) are quicker/easier to edit ("maintain"),
(f) use less memory/CPU,
(g) have fewer dependencies, or
(h) are easier to read",
then we need to look at the problem in question (input.txt, expected_output.txt). Otherwise we cannot have a meaningful discussion.
Python is just too slow for me. It may be fast enough for someone else, but not for me. Being forced to use python due to a job requirement, using python as a result of pressure from other programmers, or using python because it was "recommended", is not the same as first learning UNIX utilities and then evaluating python. I learned sed and other UNIX utilities first and so any potential "replacement" must offer the same benefits.
- xfguunbb 4y agoCan you post some of your scripts you use? I’d love to get some inspiration.
- 1vuio0pswjnm7 4y agoIf provide a text-processing problem I will try to provide a solution. Meanwhile here is one. Insert stdin or a src file at the top of a dst file. test ! -h $1||exec echo $0: error: symlink case $# in :) ;;1) x=$(sed -n '$=' $1) test $x -gt 1||exec echo usage: $0 dst src sed -i -e1r/dev/stdin -e1N $1 ;;2) printf '0r '$2'\nwq\n'|ed -s $1 esac For example, 1.sh 1.c < 1.h echo 93.184.216.34 example.com|1.sh /etc/hosts or 1.sh 1.c 1.h 1.sh /etc/hosts map-ip-host.txt How is this done in Python.
- 1vuio0pswjnm7 4y agoHere is another variation that does not used "sed -i". It uses a favourite hex editor called ired. test ! -h $1||exec echo $0: error: symlink case $# in :) ;;1) x=$(sed -n '$=' $1) test $x -gt 1||exec echo usage: $0 dst src od -tx1 -An|sed 's/^/w /'|ired -n $1 /dev/stdin ;;2) printf '0r '$2'\nwq\n'|ed -s $1 esac
- 1vuio0pswjnm7 4y agoTwo more, edited for HN of course. These are scripts/snippets that would often be used in or by other scripts. Unlike Python, these scripts will keep working for many years with zero "maintenance". They will probably still be working long after I have passed away. Task 1: Transform (a) the BIND format of stub resolver output, i.e., the format used by "dig" and many other stub resolvers, to (b) some other format, like HOSTS file, BIND/tinydns zone file, haproxy map file, etc. The input would typically be catenated stub resolver output for hundreds to thousands of domains. The epoch program is three lines of C. NB. For large input on Linux, dash is significantly faster than bash. I wrote a C program that does the job of this script, faster than dash, and much faster than python. However I still prefer the shell for testing ideas, quickly. EPOCH=$(epoch); tr -d '\12'|tr ';' '\12' \ |sed -n '/ANSWER SECTION/{s/ ANSWER SECTION://; s/\.[^-0-9a-zA-Z].*IN[^-0-9a-zA-Z]*A[^-0-9a-zA-Z]/ 1 IN A /; # list domains to exclude /www.google.com/d; /^$/d;/^.$/d;p;}' \ |{ exec 2>/dev/null 3>&3 4>&4 5>&5 6>&6 7>&7 8>&8 9>&9; while read NAME TTL CLASS TYPE IPADDR ;do echo $NAME $TTL $CLASS $TYPE $IPADDR; echo $NAME 1 $CLASS $TYPE $IPADDR >&3; echo $NAME $IPADDR >&4; echo $NAME $IPADDR \# $EPOCH >&5; echo =$NAME:$IPADDR:1 >&6; #_IP_() # { # echo $IPADDR|grepcidr -f $1-ips.txt >/dev/null; # } #if _IP_ cloudflare ... #if _IP_ aws ... #if _IP_ cloudfront ... #if _IP_ fastly ... #if _IP_ akamai ... # etc. done; } For example, drill example.com|1.sh 3>>1.zone kdig example.com|1.sh 4>>/etc/hosts drill example.com|1.sh 5>>1.map kdig example.com|1.sh 6>>data Task 2: Extract and transform CloudFront domains and their CNAMEs from BIND format stub resolver output to haproxy map format. grep -A1 ANSWER.SEC|tr '\11' '\40'|sed -n '/cloudfront.net\.$/{s/\. .* / /;s/\.$//p;}' For example, drill blogs.aws.amazon.com|2.sh >>2.map