4 ms·

Awk and sed are generally superior for simple data transformations than python. Like cutting off specific lines, columns or replacing characters in files. They

Awk and sed are generally superior for simple data transformations than python. Like cutting off specific lines, columns or replacing characters in files. They have pretty great performance in large files. Once things get complicated though you should probably push whatever transformations you need to do to a database.

markus_zhang 7y ago
Thanks this is a good point. I did read some posts here that use sqlite for sort of middle storage in the pipelines.
manifoldgeo 7y ago
I would like to offer a counter-argument. I’ve been torrenting TV series for years, and I always have to do a mass rename of The files afterward to get Plex to process them properly and download the metadata. They’re always named something like “TV Show Name 2014.L337.H4XXX1080p-420.mkv”. The other day I set out to use BASH to mass rename, and I had such trouble backslash-escaping the dot characters and whitespaces to prevent sed from interpreting them specially. Eventually I gave up and searched for the pythonic way to rename them, and it was as simple as a string replacement followed by a call to os.rename() inside a for-loop. It was a breath of fresh air to escape “command line Kung Fu” and fearing the thrashing from shell globbing. To be fair, I got my start in using sed and awk as powertools in the BASH command pipeline, but I don’t miss them compared to a language with strong data types and simple built-in methods for handling complex manipulation. Python is built in to basically every Linux distro that has BASH, and for the sake of simple transformations, it offers a lot of succinct methods that work on either 2.7 or 3.x with no external packages.
- beojan 7y ago
  You need zmv
  - manifoldgeo 7y ago
    Thanks! I hadn't heard of zmv before, and it fits my use case perfectly.
- jumpman500 7y ago
  Your example shows a very particular shortcoming in the unix command line programs. The mv command doesn't play nice with spaces in file names. I think python is a fine solution for your use case and wouldn't recommend against your code. Especially if you had to build it out for a lot of different patterns. But if you're dealing with changing stuff in text files greater than 25 gigs I'd always recommend using the unix cmd line tool set for simple modifications if performance is a key concern. It works the same everywhere and can rip through big files. But I'm a Vim guy and I live in the shell so I'm surely quite biased lol. There are plenty of pretty easy one liners though that can rename files depending on if you can define a pattern correctly though. echo 'TV Show Name 2014.L337.H4XXX1080p-420.mkv' | awk -F . -v q="'" '{print q$0q" "q$1".mkv"q }' | xargs mv find . -type f -name '*.mkv' -printf '%f' | awk -F . -v q="'" '{print q$0q" "q$1".mkv"q}' | xargs mv

markus_zhang 7y ago

Thanks this is a good point. I did read some posts here that use sqlite for sort of middle storage in the pipelines.

manifoldgeo 7y ago

I would like to offer a counter-argument. I’ve been torrenting TV series for years, and I always have to do a mass rename of The files afterward to get Plex to process them properly and download the metadata. They’re always named something like “TV Show Name 2014.L337.H4XXX1080p-420.mkv”. The other day I set out to use BASH to mass rename, and I had such trouble backslash-escaping the dot characters and whitespaces to prevent sed from interpreting them specially. Eventually I gave up and searched for the pythonic way to rename them, and it was as simple as a string replacement followed by a call to os.rename() inside a for-loop. It was a breath of fresh air to escape “command line Kung Fu” and fearing the thrashing from shell globbing. To be fair, I got my start in using sed and awk as powertools in the BASH command pipeline, but I don’t miss them compared to a language with strong data types and simple built-in methods for handling complex manipulation. Python is built in to basically every Linux distro that has BASH, and for the sake of simple transformations, it offers a lot of succinct methods that work on either 2.7 or 3.x with no external packages.

beojan 7y ago
You need zmv
- manifoldgeo 7y ago
  Thanks! I hadn't heard of zmv before, and it fits my use case perfectly.
jumpman500 7y ago
Your example shows a very particular shortcoming in the unix command line programs. The mv command doesn't play nice with spaces in file names. I think python is a fine solution for your use case and wouldn't recommend against your code. Especially if you had to build it out for a lot of different patterns. But if you're dealing with changing stuff in text files greater than 25 gigs I'd always recommend using the unix cmd line tool set for simple modifications if performance is a key concern. It works the same everywhere and can rip through big files. But I'm a Vim guy and I live in the shell so I'm surely quite biased lol. There are plenty of pretty easy one liners though that can rename files depending on if you can define a pattern correctly though. echo 'TV Show Name 2014.L337.H4XXX1080p-420.mkv' | awk -F . -v q="'" '{print q$0q" "q$1".mkv"q }' | xargs mv find . -type f -name '*.mkv' -printf '%f' | awk -F . -v q="'" '{print q$0q" "q$1".mkv"q}' | xargs mv