4 ms·

For SSDs this is probably the right thing (maybe use uring tho?) For hard drives this is the opposite of what you want. For hard drives, especially when copyin

For SSDs this is probably the right thing (maybe use uring tho?) For hard drives this is the opposite of what you want. For hard drives, especially when copying large trees, you only want one thread to access the disk, and preferably sort its stat()s and open/read/close() by inode. Also this tool seems to lack most options that cp(1) has.

vthriller 5y ago
> preferably sort its stat()s and open/read/close() by inode A better approach would be to use information about location of data extents. Luckily there's already a crate that can help with that: https://github.com/the8472/platter-walk https://github.com/the8472/platter-walk
- formerly_proven 5y ago
  That's better for reading the data, but doesn't work for ordering stat()s because looking up extent information requires reading the inode, but that's what we're trying to make linear-ish at that point.
  - the8472 5y ago
    you can optimize for both. obtain directory entries, sort by inode, do a batch stat (ideally via io_uring) and fiemap and then sort again by extent order. And if you only need the `d_type` instead of the full stat struct, then you get that for free from getdents(2) on some filesystems, this is what platter_walk does.
CWuestefeld 5y ago
options that cp(1) has OK, this is something that's bugged me for a long time. I've been in the field forever, but working on Windows. So I've seen this kind of thing many times before, particularly when referencing man, and it seems completely cryptic. The way linux people occasionally put a parenthetic number after a generic command seems to me one of the ways that normal civilians, and even pros like me, are intimidated into staying away from serious linux usage. So what gives? What the heck is the parenthetic number for?
- duped 5y ago
  It refers to the manpage section. Good SO post: https://unix.stackexchange.com/questions/3586/what-do-the-numbers-in-a-man-page-mean https://unix.stackexchange.com/questions/3586/what-do-the-nu...
- amluto 5y ago
  It’s the man page section. cp(1) is documented in …/man1/…. You can type man 1 cp. Roughly, 1 is unprivileged shell commands, 2 is syscalls, 3 is library functions, and 8 is administrative commands.
- lvkv 5y ago
  The number denotes what section of the manual the command (or function, etc.) comes from. Aside from that, it also helps clear up potential confusion about what you’re talking about—a good example is stat(1) (the command) and stat(2) (the system call). Without the number, you’d have to rely on context clues to infer which stat was the relevant one. https://unix.stackexchange.com/questions/3586/what-do-the-nu https://unix.stackexchange.com/questions/3586/what-do-the-nu...
- andrewmcwatters 5y ago
  I can’t lie, if you spend most of your time in Windows, there’s probably diminished reasons to understand manpages, but for those that do any degree of meaningful work on a Linux or Unix (e.g. macOS) system, not knowing how to navigate man pages separates entire classes of developer professionalism to me. It’s night and day working with people who know how to be approachable in development, but also realize you need to understand some basic flags for everyday tools and not turn to Google or Stack Overflow for what is directly in your manpages. Admittedly those who cannot stay approachable concoct all sorts of shell hell for everyone, even those who are experienced. Excellent question, by the way. I wish we’d get some more candid questions like these on HN when relevant to the thread, like this one. It’s a learning opportunity for all readers.
  - unknown_error 5y ago
    What's wrong with Stack over man pages? Most man pages don't you give advice or context, just the raw reference, but sometimes the best way to do something is not even the tool you originally looked up (say, scp vs rsync). Stack combines the reference (which flags) with the expertise (of educated guesses by sometimes wiser users). I would never trust a man page over a good Stack answer. (edit: as a web dev who uses *nix all day, but has no desire to become a full fledged sysadmin... Just give me the tool I need to get the job done without sacrificing performance or security. I don't need to know every possible gnu tool and flag and distro dependent options...)
dekhn 5y ago
one of the things that surprised me about large cluster filesystems (although it's obvious in retrospect) is that the solution to scaling performance is many threads, not one, because the backend can handle many concurrent requests. In fact at one place I worked, the cluster filesystem wouldn't even take you seriously until you hit it with GB/s of requests.
Svetlitski 5y ago
Definitely valid points, as I mentioned in a comment below I'll be updating the README to mention this is geared towards systems with SSDs. As for lacking most of cp(1)'s options, that's a deliberate choice. My goal with fcp is to make the most common use case of cp(1) (i.e. copy some files/directories with no options) fast, not to replace cp(1) completely.

vthriller 5y ago

> preferably sort its stat()s and open/read/close() by inode A better approach would be to use information about location of data extents. Luckily there's already a crate that can help with that: https://github.com/the8472/platter-walk https://github.com/the8472/platter-walk

formerly_proven 5y ago
That's better for reading the data, but doesn't work for ordering stat()s because looking up extent information requires reading the inode, but that's what we're trying to make linear-ish at that point.
- the8472 5y ago
  you can optimize for both. obtain directory entries, sort by inode, do a batch stat (ideally via io_uring) and fiemap and then sort again by extent order. And if you only need the `d_type` instead of the full stat struct, then you get that for free from getdents(2) on some filesystems, this is what platter_walk does.

CWuestefeld 5y ago

options that cp(1) has OK, this is something that's bugged me for a long time. I've been in the field forever, but working on Windows. So I've seen this kind of thing many times before, particularly when referencing man, and it seems completely cryptic. The way linux people occasionally put a parenthetic number after a generic command seems to me one of the ways that normal civilians, and even pros like me, are intimidated into staying away from serious linux usage. So what gives? What the heck is the parenthetic number for?

duped 5y ago
It refers to the manpage section. Good SO post: https://unix.stackexchange.com/questions/3586/what-do-the-numbers-in-a-man-page-mean https://unix.stackexchange.com/questions/3586/what-do-the-nu...
amluto 5y ago
It’s the man page section. cp(1) is documented in …/man1/…. You can type man 1 cp. Roughly, 1 is unprivileged shell commands, 2 is syscalls, 3 is library functions, and 8 is administrative commands.
lvkv 5y ago
The number denotes what section of the manual the command (or function, etc.) comes from. Aside from that, it also helps clear up potential confusion about what you’re talking about—a good example is stat(1) (the command) and stat(2) (the system call). Without the number, you’d have to rely on context clues to infer which stat was the relevant one. https://unix.stackexchange.com/questions/3586/what-do-the-nu https://unix.stackexchange.com/questions/3586/what-do-the-nu...
andrewmcwatters 5y ago
I can’t lie, if you spend most of your time in Windows, there’s probably diminished reasons to understand manpages, but for those that do any degree of meaningful work on a Linux or Unix (e.g. macOS) system, not knowing how to navigate man pages separates entire classes of developer professionalism to me. It’s night and day working with people who know how to be approachable in development, but also realize you need to understand some basic flags for everyday tools and not turn to Google or Stack Overflow for what is directly in your manpages. Admittedly those who cannot stay approachable concoct all sorts of shell hell for everyone, even those who are experienced. Excellent question, by the way. I wish we’d get some more candid questions like these on HN when relevant to the thread, like this one. It’s a learning opportunity for all readers.
- unknown_error 5y ago
  What's wrong with Stack over man pages? Most man pages don't you give advice or context, just the raw reference, but sometimes the best way to do something is not even the tool you originally looked up (say, scp vs rsync). Stack combines the reference (which flags) with the expertise (of educated guesses by sometimes wiser users). I would never trust a man page over a good Stack answer. (edit: as a web dev who uses *nix all day, but has no desire to become a full fledged sysadmin... Just give me the tool I need to get the job done without sacrificing performance or security. I don't need to know every possible gnu tool and flag and distro dependent options...)

dekhn 5y ago

one of the things that surprised me about large cluster filesystems (although it's obvious in retrospect) is that the solution to scaling performance is many threads, not one, because the backend can handle many concurrent requests. In fact at one place I worked, the cluster filesystem wouldn't even take you seriously until you hit it with GB/s of requests.

Svetlitski 5y ago

Definitely valid points, as I mentioned in a comment below I'll be updating the README to mention this is geared towards systems with SSDs. As for lacking most of cp(1)'s options, that's a deliberate choice. My goal with fcp is to make the most common use case of cp(1) (i.e. copy some files/directories with no options) fast, not to replace cp(1) completely.