4 ms·
Here is an Awk program (not a one-liner) that extracted select snippets of text from lots of docs and worked almost 10x faster than a grep equivalent: awk '
by j0e1 8y ago
Here is an Awk program (not a one-liner) that extracted select snippets of text from lots of docs and worked almost 10x faster than a grep equivalent:
awk 'BEGIN { ORS=" "} { print $0 }' $2 | awk --re-interval -v pat="$1" '
{
cut_content = substr($0, 1, match($0, "==== Refs"))
orig_content = substr($0, 1, match($0, "==== Refs"))
idx = 0
regex = "(^|[^a-zA-Z]{1})" pat "([^a-zA-Z]{1}|$)"
while (match(cut_content, regex)) {
if (RLENGTH > 0) {
prot = substr(cut_content, RSTART-1, RLENGTH+1)
gsub(/[:punct: ]$/, "", prot)
gsub(/^[:punct: ]/, "", prot)
print prot, "\t", substr(orig_content, idx+RSTART-400, RLENGTH+800)
idx += (RSTART + RLENGTH -1)
cut_content = substr(cut_content, RSTART+RLENGTH)
}
}
}'
- AdmiralAsshat 8y agoLooks interesting; can you explain how the hell it works? I tried writing a script that would extract entries from logs (e.g. if every entry started "====CRITICAL ERROR LOG 2018/06/15====", I wanted it to start there and grab every line below it until it hits the next log entry starting with the "=====" header), but I gave up. Your script might do something similar, if I can figure out what it's looking for.
- bcbrown 8y agoYou can use regex ranges to match every line between two regexes: https://www.gnu.org/software/gawk/manual/html_node/Ranges.html https://www.gnu.org/software/gawk/manual/html_node/Ranges.ht... Another option might be to use multi-line records (https://www.gnu.org/software/gawk/manual/html_node/Multiple-Line.html https://www.gnu.org/software/gawk/manual/html_node/Multiple-...). First pre-process the file to add an empty line before each header, then use blank lines as the record separator and newlines for the field separator.
- j0e1 8y agoSure. I'll explain the code and hopefully you'd find enough clues to adapt it for your use-case which, from what you've described, is a subset. Intent: Given a folder with multiple text files of academic papers from PubMed, remove the contents after the '====Refs' header. Then in each file find occurrences of any term in a keywords file and extract it along with 400 chars before and after it. Print this output as a tsv. input: $1) A text file containing keywords of interest that need to be searched in the files separated by `|`. $2) File containing paragraphs of lots of text with headers starting with '===='. There isn't a formatting requirement really. Just that, in my case the files happened to be so. # Note: Awk string commands have indexes that start with 1 (not 0). # The first Awk command just removes the newlines in file $2 and replaces them # with space. `pat` contains the contents of the file $1: 'keyword1|kewword2|...'. # --re-interval allows use of, well, regex intervals. awk 'BEGIN { ORS=" "} { print $0 }' $2 | awk --re-interval -v pat="$1" ' { # both variables are initialized with full-content of text file # with the part after '==== Refs' removed. cut_content = substr($0, 1, match($0, "==== Refs")) orig_content = substr($0, 1, match($0, "==== Refs")) # index (could be better named as offset, but life is too short to refactor). idx = 0 # match the keywords, even if it appears between non-alphabet chars. # e.g -keyword1. is a match. regex = "(^|[^a-zA-Z]{1})" pat "([^a-zA-Z]{1}|$)" # loop until matches are found. while (match(cut_content, regex)) { # RLENGTH is the length of the matched keyword. if (RLENGTH > 0) { # set the matched keyword to the variable `prot`. # RSTART is the character position of the matched keyword in the file. # This is the first occurrence of it in the file when read from the start. prot = substr(cut_content, RSTART-1, RLENGTH+1) # clean-up punctuation and/or space from the beginning and end of `prot`. gsub(/[:punct: ]$/, "", prot) gsub(/^[:punct: ]/, "", prot) # print the `prot` and 400 chars before and after it using idx # as an offset. Separate the results by tab char. print prot, "\t", substr(orig_content, idx+RSTART-400, RLENGTH+800) # update offset to character position right after the current matched keyword idx += (RSTART + RLENGTH -1) # cut-out all the text up until the position of the # last matched keyword (including it) and on the remaining text # repeat the above steps in the loop. This is so that the search # continues throughout the file and doesn't keep finding the same # keyword over and over again. See `while` condition. cut_content = substr(cut_content, RSTART+RLENGTH) } } }' Edit: Formatting. HN doesn't wrap text in code-block.
- bartbes 8y agoEasily solved by just keeping some state: /^=====/ { inentry = !inentry; next } inentry Possibly without 'next' if you want the marker lines too.
- deleted 8y ago[deleted]