5 ms·

One of my favorite party tricks is using shell redirects to do this. Using coreutils ('cp') is a nice approach I'm all for anything not 'dd' or cargo-culted li

One of my favorite party tricks is using shell redirects to do this. Using coreutils ('cp') is a nice approach I'm all for anything not 'dd' or cargo-culted like balenaEtcher This all depends on a certain format of ISO that I can't recall There's no magic. Checksum the drive and the file after

mort96 3y ago
Using shell redirects sucks though, you need to run the shell as root. Using cp seems overkill? It's really really designed to copy files between file systems and has a lot of logic to handle different cases and different optimisations which don't matter when writing to block devices. As you say, there's no magic. I want a program which just calls 'open' on a path I give it and then uses 'write' to write my data to it. cp does so, so much more related to file systems, shell redirects aren't a separate program I can run with sudo, but I can trust dd to do the job. And it has a (kinds bad) progress monitor to boot. The default 512 byte block size is unfortunate though.
GuB-42 3y ago
As far as I know, you can't copy using shell redirects. At least not with a simple POSIX shell. Redirection only open file descriptors, but you need a command to actually make the copy from one to the other. It can be "cat", which is often cargo culted in it own right. Or even "dd", which can work with stdin/out. I usually prefer not to use shell redirects if there is a command that take filenames. That's because it gives more control to the app. The app knows how the file will be used, the shell doesn't, so it can open it the most appropriate way, avoid overwriting the output file if something goes wrong, output better error messages, etc... Now, if you don't trust the app (for example if you fear it will modify your input file), then shell redirects may be the better option.
- Brian_K_White 3y ago
  You can absolutely read and write by just redirects. No cat or dd or anything needed.
  - LegionMammal978 3y ago
    Could you give an example? Like GP, I'm also not aware of any way to do such a thing without using shell-specific extensions.
    - Brian_K_White 3y ago
      I confess it's not as simple as $ <a >b Sorry about that, because it means the ansdwer while technically true in some weird case where you really need it, isn't exactly convenient like you'd actually use it. I made it sound trivial and obvious and direct and it's not. Use read in a loop, with special care with LANG and IFS to make all bytes meaningless. Except there is no way to avoid null being special, but you can handle null by making null the delimiter for read, and only reading one byte at a time. So even though you can't store an actual null in a variable, you can still detect that there was a null and print a new one back out, and since you only read one byte at a time, you do that for each individual input byte and strings of nulls are not collapsed. It looks like a lot, but, read is a builtin, and at least in bash and ksh and zsh so is printf, and although this is a loop, it's actually not even a sub-shell. If you edit variables inside the loop, they are still there after the loop, ie, you never forked a child. while LANG=C IFS= read -d '' -r -n 1 x ;do printf '%c' "$x" ;done <junk1.rnd >junk2.rnd
- bravetraveler 3y ago
  This works in ZSH but not BASH... I have a feeling you just helped me see why :D
- js2 3y ago
  You're right. You'd have to use the `read` built-in and `echo` or `printf` in a while loop combined with redirection, but the POSIX-specified `read` built-in is intended for text and isn't going work correctly with binary files. With bash you could maybe use the non-standard `read -N` option but I'm not sure about null bytes.
  - TristanBall 3y ago
    Bash not so much, Ksh93 will do this, playing with has been a recent toy project. Typeset -b to specify a binary variable, rean -n/-N and print -v. It can be really quit surprisingly fast too, 4GB/sec reading /dev/zero writing /dev/null. ( yes 40gbit/sec, binary copies, in a shell script!!! ). 2-3GB copying real data. Oh boy but the edge cases and quirks.. there are so many reasons why this is a bad way to do things.. Curiously, I double the speed bu unsetting and recreating the buffer variable for each op. There also seems to be cases where ksh will buffer reads from a pipe to allow limited seeks, but also if you do a read -N from a pipe with too large a size (>8k I think, I'd have to check) and that read can't be completed because the source finished writing less than that, then that data is gone. Less than 8k, you can still read it with a subsequent read -n. Probably the most actually useful thing I learned was that by ksh93 creates its pipes with unix sockets, not pipes, which means the buffer size is set by /proc/sys/net/core/[w|r]mem_default, rather than a fcntl call. That makes it easier to tweak from a script, and also makes ksh pipes faster by default for many streams, compared to most other shells ( depending on block sizes, and the difference goes away if you tweak up the pipe buffer size ) Don't get me wrong, not something I'd use in real life, but it was fun anyway.
- Brian_K_White 3y ago
  bash builtins only, not even a subshell, binary-complete while LANG=C IFS= read -d '' -r -n 1 x ;do printf '%c' "$x" ;done <dat1 >dat2 At least ksh and zsh have equivalent functionality though the details may differ. Not dash or ancient sh. I don't pretend it's practical, just possible. I actually do have a neck beard this month.
  - Brian_K_White 3y ago
    ksh almost identical, read -N instead of -n while LANG=C IFS= read -d'' -r -N 1 x ;do printf '%c' "$x" ;done <dat1 >dat2 With -n it handles 0x00 fine but screws up on 0x0A !

mort96 3y ago

Using shell redirects sucks though, you need to run the shell as root. Using cp seems overkill? It's really really designed to copy files between file systems and has a lot of logic to handle different cases and different optimisations which don't matter when writing to block devices. As you say, there's no magic. I want a program which just calls 'open' on a path I give it and then uses 'write' to write my data to it. cp does so, so much more related to file systems, shell redirects aren't a separate program I can run with sudo, but I can trust dd to do the job. And it has a (kinds bad) progress monitor to boot. The default 512 byte block size is unfortunate though.

GuB-42 3y ago

As far as I know, you can't copy using shell redirects. At least not with a simple POSIX shell. Redirection only open file descriptors, but you need a command to actually make the copy from one to the other. It can be "cat", which is often cargo culted in it own right. Or even "dd", which can work with stdin/out. I usually prefer not to use shell redirects if there is a command that take filenames. That's because it gives more control to the app. The app knows how the file will be used, the shell doesn't, so it can open it the most appropriate way, avoid overwriting the output file if something goes wrong, output better error messages, etc... Now, if you don't trust the app (for example if you fear it will modify your input file), then shell redirects may be the better option.

Brian_K_White 3y ago
You can absolutely read and write by just redirects. No cat or dd or anything needed.
- LegionMammal978 3y ago
  Could you give an example? Like GP, I'm also not aware of any way to do such a thing without using shell-specific extensions.
  - Brian_K_White 3y ago
    I confess it's not as simple as $ <a >b Sorry about that, because it means the ansdwer while technically true in some weird case where you really need it, isn't exactly convenient like you'd actually use it. I made it sound trivial and obvious and direct and it's not. Use read in a loop, with special care with LANG and IFS to make all bytes meaningless. Except there is no way to avoid null being special, but you can handle null by making null the delimiter for read, and only reading one byte at a time. So even though you can't store an actual null in a variable, you can still detect that there was a null and print a new one back out, and since you only read one byte at a time, you do that for each individual input byte and strings of nulls are not collapsed. It looks like a lot, but, read is a builtin, and at least in bash and ksh and zsh so is printf, and although this is a loop, it's actually not even a sub-shell. If you edit variables inside the loop, they are still there after the loop, ie, you never forked a child. while LANG=C IFS= read -d '' -r -n 1 x ;do printf '%c' "$x" ;done <junk1.rnd >junk2.rnd
bravetraveler 3y ago
This works in ZSH but not BASH... I have a feeling you just helped me see why :D
js2 3y ago
You're right. You'd have to use the `read` built-in and `echo` or `printf` in a while loop combined with redirection, but the POSIX-specified `read` built-in is intended for text and isn't going work correctly with binary files. With bash you could maybe use the non-standard `read -N` option but I'm not sure about null bytes.
- TristanBall 3y ago
  Bash not so much, Ksh93 will do this, playing with has been a recent toy project. Typeset -b to specify a binary variable, rean -n/-N and print -v. It can be really quit surprisingly fast too, 4GB/sec reading /dev/zero writing /dev/null. ( yes 40gbit/sec, binary copies, in a shell script!!! ). 2-3GB copying real data. Oh boy but the edge cases and quirks.. there are so many reasons why this is a bad way to do things.. Curiously, I double the speed bu unsetting and recreating the buffer variable for each op. There also seems to be cases where ksh will buffer reads from a pipe to allow limited seeks, but also if you do a read -N from a pipe with too large a size (>8k I think, I'd have to check) and that read can't be completed because the source finished writing less than that, then that data is gone. Less than 8k, you can still read it with a subsequent read -n. Probably the most actually useful thing I learned was that by ksh93 creates its pipes with unix sockets, not pipes, which means the buffer size is set by /proc/sys/net/core/[w|r]mem_default, rather than a fcntl call. That makes it easier to tweak from a script, and also makes ksh pipes faster by default for many streams, compared to most other shells ( depending on block sizes, and the difference goes away if you tweak up the pipe buffer size ) Don't get me wrong, not something I'd use in real life, but it was fun anyway.
Brian_K_White 3y ago
bash builtins only, not even a subshell, binary-complete while LANG=C IFS= read -d '' -r -n 1 x ;do printf '%c' "$x" ;done <dat1 >dat2 At least ksh and zsh have equivalent functionality though the details may differ. Not dash or ancient sh. I don't pretend it's practical, just possible. I actually do have a neck beard this month.
- Brian_K_White 3y ago
  ksh almost identical, read -N instead of -n while LANG=C IFS= read -d'' -r -N 1 x ;do printf '%c' "$x" ;done <dat1 >dat2 With -n it handles 0x00 fine but screws up on 0x0A !