4 ms·

>> But who cares? Why not just let the command figure out the right buffer size automatically? i think he's talking about cat being the command that figures it

>> But who cares? Why not just let the command figure out the right buffer size automatically? i think he's talking about cat being the command that figures it out automatically

GekkePrutser 4y ago
I don't read it that way. He calls out that 'weird' command specifically. But indeed he doesn't specify. I wonder what cat does in terms of buffers, I kinda doubt it has any special optimisations though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. Edit: Nope, I'm wrong there!! Also that command does more than just specify the memory buffer like he says. That's my point, it's useful for tuning which can be super helpful with huge images. It can also lead to some dangerous gotchas as well when working with files. But with full disk images these don't apply generally.
- LukeShu 4y ago
  > though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. No? The shell redirection is just int tmp_fd = open("/dev/sdb", O_WRONLY|O_TRUNC, 0666); dup2(tmp_fd, STDOUT_FILENO); close(tmp_fd); (plus error handling and whatnot) `cat` ends up with a file descriptor directly to the block device, same as `dd` does; the only difference is whether the `open()` call comes before or after the `execve()` call.
  - GekkePrutser 4y ago
    Ah ok good point! I stand corrected. I really doubt cat is smart enough to figure out a suitable block size though. At least with dd you can specify one.
    - bhaney 4y ago
      > I really doubt cat is smart enough to figure out a suitable block size though It seems to try, at least https://github.com/coreutils/coreutils/blob/master/src/cat.c#L648-L649 https://github.com/coreutils/coreutils/blob/master/src/cat.c...
      - LukeShu 4y ago
        Going through the historical versions (copying from my other comment): - >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out - >=8.23 (2014-07-18) : max(128KiB, st_blksize(infile), st_blksize(outfile)) - >=8.17 (2012-05-10) : max(64KiB, st_blksize(infile), st_blksize(outfile)) - >=7.2 (2009-03-31) : max(32KiB, st_blksize(infile), st_blksize(outfile)) - at least as far back as 1996 : max(st_blksize(infile), st_blksize(outfile)) (In my psuedo-code, `st_blksize(fd)` is the `ST_BLKSIZE(buf)` of the result of `fstat(fd, buf)`.)
LukeShu 4y ago
Right, and the parent comment is saying that `cat` doesn't figure it out very well. I recall that GNU cat used a constant value that was tuned for read performance on common hard drives, giving no consideration to write performance. Looking at the GNU cat sources today, it doesn't look at all how I remember; I'd have to study it a bit to tell you what it does. Edit: Hrmm, it seems I was mis-remembering. Perhaps I was thinking of the minimum values (32KiB, 64KiB, 128KiB below)? Or perhaps I was thinking of a BSD cat? Anyway: How GNU cat sizes its buffers, by version: - >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out - >=8.23 (2014-07-18) : max(128KiB, st_blksize(infile), st_blksize(outfile)) - >=8.17 (2012-05-10) : max(64KiB, st_blksize(infile), st_blksize(outfile)) - >=7.2 (2009-03-31) : max(32KiB, st_blksize(infile), st_blksize(outfile)) - at least as far back as 1996 : max(st_blksize(infile), st_blksize(outfile)) (In my psuedo-code, `st_blksize(fd)` is the `ST_BLKSIZE(buf)` of the result of `fstat(fd, buf)`.)
- GekkePrutser 4y ago
  No, I was talking about dd. As far as I know it does not have any smart tuning of block sizes. It definitely didn't in the past and I doubt it was added. Not sure what cat does but like I said in my other comment its not really cat itself that does the disk writing in that command but rather the shell redirect. Edit: Nope, I'm wrong there!!
  - LukeShu 4y ago
    Ah, I misunderstood you. You're correct, `dd`'s default block size was and is 512, which is specified by POSIX. I interpreted "the command" in the original article as "the command you end up running, whether it be `dd` or `cat` or something else." But you're mistaken about the shell redirect. In either scenario it's the command (`dd` or `cat`) making the write() syscall to the device. The shell passes the file descriptor to the command, then gets out of the way.
    - GekkePrutser 4y ago
      Indeed I was mistaken about the shell, sorry about that. The redirect method I forgot a really long time ago and my memory made it into something it was not. In that case I guess dd does call a sync() on every output block? Because it's definitely slower and the LED pattern on a USB stick is also much more 'flashy' when using 512 bytes.

>> But who cares? Why not just let the command figure out the right buffer size automatically? i think he's talking about cat being the command that figures it out automatically

GekkePrutser 4y ago

I don't read it that way. He calls out that 'weird' command specifically. But indeed he doesn't specify. I wonder what cat does in terms of buffers, I kinda doubt it has any special optimisations though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. Edit: Nope, I'm wrong there!! Also that command does more than just specify the memory buffer like he says. That's my point, it's useful for tuning which can be super helpful with huge images. It can also lead to some dangerous gotchas as well when working with files. But with full disk images these don't apply generally.

LukeShu 4y ago
> though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. No? The shell redirection is just int tmp_fd = open("/dev/sdb", O_WRONLY|O_TRUNC, 0666); dup2(tmp_fd, STDOUT_FILENO); close(tmp_fd); (plus error handling and whatnot) `cat` ends up with a file descriptor directly to the block device, same as `dd` does; the only difference is whether the `open()` call comes before or after the `execve()` call.
- GekkePrutser 4y ago
  Ah ok good point! I stand corrected. I really doubt cat is smart enough to figure out a suitable block size though. At least with dd you can specify one.
  - bhaney 4y ago
    > I really doubt cat is smart enough to figure out a suitable block size though It seems to try, at least https://github.com/coreutils/coreutils/blob/master/src/cat.c#L648-L649 https://github.com/coreutils/coreutils/blob/master/src/cat.c...
    - LukeShu 4y ago
      Going through the historical versions (copying from my other comment): - >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out - >=8.23 (2014-07-18) : max(128KiB, st_blksize(infile), st_blksize(outfile)) - >=8.17 (2012-05-10) : max(64KiB, st_blksize(infile), st_blksize(outfile)) - >=7.2 (2009-03-31) : max(32KiB, st_blksize(infile), st_blksize(outfile)) - at least as far back as 1996 : max(st_blksize(infile), st_blksize(outfile)) (In my psuedo-code, `st_blksize(fd)` is the `ST_BLKSIZE(buf)` of the result of `fstat(fd, buf)`.)

LukeShu 4y ago

Right, and the parent comment is saying that `cat` doesn't figure it out very well. I recall that GNU cat used a constant value that was tuned for read performance on common hard drives, giving no consideration to write performance. Looking at the GNU cat sources today, it doesn't look at all how I remember; I'd have to study it a bit to tell you what it does. Edit: Hrmm, it seems I was mis-remembering. Perhaps I was thinking of the minimum values (32KiB, 64KiB, 128KiB below)? Or perhaps I was thinking of a BSD cat? Anyway: How GNU cat sizes its buffers, by version: - >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out - >=8.23 (2014-07-18) : max(128KiB, st_blksize(infile), st_blksize(outfile)) - >=8.17 (2012-05-10) : max(64KiB, st_blksize(infile), st_blksize(outfile)) - >=7.2 (2009-03-31) : max(32KiB, st_blksize(infile), st_blksize(outfile)) - at least as far back as 1996 : max(st_blksize(infile), st_blksize(outfile)) (In my psuedo-code, `st_blksize(fd)` is the `ST_BLKSIZE(buf)` of the result of `fstat(fd, buf)`.)

GekkePrutser 4y ago
No, I was talking about dd. As far as I know it does not have any smart tuning of block sizes. It definitely didn't in the past and I doubt it was added. Not sure what cat does but like I said in my other comment its not really cat itself that does the disk writing in that command but rather the shell redirect. Edit: Nope, I'm wrong there!!
- LukeShu 4y ago
  Ah, I misunderstood you. You're correct, `dd`'s default block size was and is 512, which is specified by POSIX. I interpreted "the command" in the original article as "the command you end up running, whether it be `dd` or `cat` or something else." But you're mistaken about the shell redirect. In either scenario it's the command (`dd` or `cat`) making the write() syscall to the device. The shell passes the file descriptor to the command, then gets out of the way.
  - GekkePrutser 4y ago
    Indeed I was mistaken about the shell, sorry about that. The redirect method I forgot a really long time ago and my memory made it into something it was not. In that case I guess dd does call a sync() on every output block? Because it's definitely slower and the LED pattern on a USB stick is also much more 'flashy' when using 512 bytes.