I used dynamically generated awk scripts a few years ago to take some delimited text files that were several hundred gigabytes and pre-process them for faster l

I used dynamically generated awk scripts a few years ago to take some delimited text files that were several hundred gigabytes and pre-process them for faster loading to a DB. It sped up the whole process a huge amount - I don’t remember exactly now but I think it went from 24 hours to get it all loaded to under 2 hours. Useful tool to have in the tool box.