3 ms·

This is super weird. The "job queue" example made me facepalm, I'd probably use a one-liner like- ls -1 *.mp3 | xargs -P4 -n1 -i ffmpeg -i "{}" "{}.ogg" (yah

by kylek 7y ago

This is super weird. The "job queue" example made me facepalm, I'd probably use a one-liner like- ls -1 *.mp3 | xargs -P4 -n1 -i ffmpeg -i "{}" "{}.ogg" (yah the filenames would end up .mp3.ogg, but so does the example...should throw basename in there somewhere)

Mic92 7y ago
It makes sense if you have multiple machines to actually distribute the load (files could be on a NFS).
- emj 7y ago
  You can send the files through the pubsub directly and just convert a stream of bytes. It is a beautiful pattern, but needs some tools to make it really easy to use (e.g. how to I metadata and bytestream to a pipe)
  - anderspitman 7y ago
    Sending the actual files would probably be really slow on the free server, due to bandwidth limiting. As for metadata, you might consider multipart/form-data, or just packing it into the query string (obviously depends on how much data you're talking about). Of course you'd probably need to write that in something other than bash. EDIT: It's pretty easy to send form-data with curl: https://ec.haxx.se/http-multipart.html https://ec.haxx.se/http-multipart.html
    - emj 7y ago
      Yes but it is the receiving that mess you up, if you want to use standard shell tools and just stream it. Everything can be solved by using other tools. # preserve filename for i in a b; do gzip -9 $i; done # does not preserve filenames while true; done curl $patchbay| mangle | gzip -9 | curl -d@- $patchbay; done
WhatIsDukkha 7y ago
You might be better served with gnu parallel - #create the directory structure first find . -type f -name '*.flac' |ionice -c3 parallel -j1 'mkdir -p $transcodes{//}' && #now the encodes find . -type f -name '*.flac' |ionice -c3 parallel -j4 "ffmpeg -n -v 0 -i {} -c:a libopus -b:a 128k $transcodes{.}.opus" Note the {.} which would actually only grab the basename.
- MachineMartin 7y ago
  The ' should be " in first example. Otherwise $transcodes is not expanded.
  - WhatIsDukkha 7y ago
    Actually no, I'm handing that line to parallel which in turn will do the expansion when it invokes.
zomglings 7y ago
ls -1 *.mp3 | awk -F. '{OFS = "."; $NF=""; print $0}' | xargs -P4 -n1 -i ffmpeg "{}mp3" "{}ogg"

by kylek 7y ago

Mic92 7y ago

It makes sense if you have multiple machines to actually distribute the load (files could be on a NFS).

emj 7y ago
You can send the files through the pubsub directly and just convert a stream of bytes. It is a beautiful pattern, but needs some tools to make it really easy to use (e.g. how to I metadata and bytestream to a pipe)
- anderspitman 7y ago
  Sending the actual files would probably be really slow on the free server, due to bandwidth limiting. As for metadata, you might consider multipart/form-data, or just packing it into the query string (obviously depends on how much data you're talking about). Of course you'd probably need to write that in something other than bash. EDIT: It's pretty easy to send form-data with curl: https://ec.haxx.se/http-multipart.html https://ec.haxx.se/http-multipart.html
  - emj 7y ago
    Yes but it is the receiving that mess you up, if you want to use standard shell tools and just stream it. Everything can be solved by using other tools. # preserve filename for i in a b; do gzip -9 $i; done # does not preserve filenames while true; done curl $patchbay| mangle | gzip -9 | curl -d@- $patchbay; done

WhatIsDukkha 7y ago

You might be better served with gnu parallel - #create the directory structure first find . -type f -name '*.flac' |ionice -c3 parallel -j1 'mkdir -p $transcodes{//}' && #now the encodes find . -type f -name '*.flac' |ionice -c3 parallel -j4 "ffmpeg -n -v 0 -i {} -c:a libopus -b:a 128k $transcodes{.}.opus" Note the {.} which would actually only grab the basename.

MachineMartin 7y ago
The ' should be " in first example. Otherwise $transcodes is not expanded.
- WhatIsDukkha 7y ago
  Actually no, I'm handing that line to parallel which in turn will do the expansion when it invokes.

zomglings 7y ago

ls -1 *.mp3 | awk -F. '{OFS = "."; $NF=""; print $0}' | xargs -P4 -n1 -i ffmpeg "{}mp3" "{}ogg"