4 ms·
Seeking a file descriptor is a basic, useful operation, and dd is the only POSIX tool that can seek an fd without actually paying the read / write costs. Suppo
by sigil 4y ago
Seeking a file descriptor is a basic, useful operation, and dd is the only POSIX tool that can seek an fd without actually paying the read / write costs.
Suppose you have 2 hours of UHD RGBA32 video, and need 5 minutes of footage from the halfway mark:
dd if=in.raw bs=$(( 3840 * 2160 * 4 * 24 )) skip=3600 count=300 | ffmpeg ...
This will be a lot faster than pointlessly catting the first 3 terabytes!
Here's a variation. Suppose the video file has a 1M header you need to skip:
{ dd bs=1M skip=1 count=0; dd bs=796262400 skip=3600 count=300; } < in.raw | ffmpeg ...
The first dd invocation does nothing more than seek stdin ahead 1M, so the second can operate on full 1 second chunks of video. Useful!
Now, are there Uncalled For Uses of dd, just like there are Useless Uses of cat? You bet.
- tomjakubowski 4y agotail can seek without reading too, but only on regular files
- sigil 4y agoCan tail seek without then writing the rest of the file to stdout?
- Self-Perfection 4y agoWell if you want to limit amount of data you are getting from a command you need `head` tail -c +STARTOFFSET $FILE | head -c MAXLENGTH There are more than one way to do stuff with UNIX tools.
- tomjakubowski 4y agodoes the head call attached to the pipe stop tail from reading the rest of the file after STARTOFFSET+MAXLENGTH?
- rosnd 4y agoYes, head exits when it's output enough. This causes tail to also call it quits as it has nothing to write to.
- Self-Perfection 4y agoIt does. tail will get SIGPIPE and exit. https://stackoverflow.com/q/8369506 https://stackoverflow.com/q/8369506 But tail might read up to pipe buffer size more data than is actually required. Also tail + head approach have an overhead of copying data between processes.
- sigil 4y agoThat’s why tail | head isn’t a reliable way to seek an fd — it will seek past the desired offset by reading and then failing to write up to PIPEBUF bytes.
- deleted 4y ago[deleted]
- mixmastamyk 4y agoffmpeg is your friend for media files, in case anyone gets ideas.
- kazinator 4y agoIf the specific job can be done with dd with your eyes closed, why would you spend 40 minutes reading the ffmpeg man page ...
- deleted 4y ago[deleted]
- mixmastamyk 4y agoBecause it takes 5 mins at stackoverflow, and the result more likely to be usable.
- verall 4y agoAlso, as in your example, it's a million times easier to make bs your frame size so you can use skip and count to slice frames as you wish. I use this all of the time working with raw video files.
- literalAardvark 4y agoWhy not use a video native tool like ffmpeg ? I'm unclear on what the advantage of dd is here. You could seek to second, keyframe, etc, and it would continue to work for formats that don't have fixed frame sizes. It's true that it is quite a lifehack if you often seek to frame in raw though.
- dfox 4y agoThe advantage of dd in this case is that dd is designed for exactly this use case of having a file (or character device, ie. magnetic tape) with some kind of fixed-size records.
- verall 4y agoBecause it's bespoke raw video, ffmpeg wouldn't know what to do with it. Sure if I'm chopping up an MP4 then dd isn't going to 'cut' it ;P
- chungy 4y agoNot to detract from the point, but if you are using ffmpeg to process raw video files, you can just use the "-ss" parameter on input and "-t" on output to get what you want. It'll seek and avoid unnecessary reads. ffmpeg -f rawvideo -r ntsc -s 1920x080 -ss 3:00 -i some_file -t 5:00 output_file
- mananaysiempre 4y agoA weird personal discovery was that `openssl x509` used as a conversion tool actually reads the first PEM certificate and leaves stdin open and positioned just past it (rather than silently discarding any trailing data the way I always thought it did), so if you pipe or redirect a bunch of certs into a shell loop (as opposed to a command inside the loop condition) you can actually decode them one by one.