3 ms·

Not the OP but the complicated part to me is just that the fd has a global cursor which makes concurrent access require synchronization. The rust std::fs::File

Not the OP but the complicated part to me is just that the fd has a global cursor which makes concurrent access require synchronization. The rust std::fs::File API at least makes this clear through mutability requirements but I imagine in other languages this either can cause a lot of bugs or requires a more complicated API to surface the functionality safely.

4lDO2 2y ago
Rust does however implement the IO traits for `&File` as well (shared), and IIRC also implements `try_clone` which is the dup equivalent.
seeknotfind 2y ago
One of the reasons the FD interface is so complicated is because there are many of the same operations, but they do different things to different underlying kernel implementations. In Linux, you have no standard way to tell what the underlying FD is. It's such a wide surface, different kernel surfaces might implement some of the many different file APIs (polling especially) slightly differently. In many cases, you can reuse standard tooling, but random ioctl calls mean you can't always reuse tooling. Nonstandard implementations of standard file calls can make it dangerous if you don't know which type of FD you have. So the good thing is they are standard, you use the same set of tools (file APIs) to operate with them, so it makes the system interface smaller and simpler, but it lacks the fidelity to create orthogonal meta programming over them. It sounds like an unimportant complaint that you can't tell the underlying type, but it also means most languages refer to FDs indiscriminately. You don't typically get type safety in languages for FDs (e.g. different class for an FD backed by shared memory instead of a real exclusive file), and even if you do have these classes you can't really guarantee they are correct at runtime if you get an FD from another process (e.g. over UDS from a child process), so languages leave FDs untyped. About the global cursor, it's not an issue because you can dupe it to get a copy with a different cursor. So only references to the FD with that specific FD identifier have a location associated with itself, and that's because the OS can only look up the state on that FD. So dup gives cursor, but if an FD represents a physical seek, it might be the case that a dup'd FD actually does affect the functionality of the first FD by linking the two together, as that's a choice you can have in the kernel when implementing an FD. Dup also mean no new cursor. So that's what I mean, the complexity of the real world means these objects are fundamentally different, and having the same APIs shoe horns them into something they're not. I've worked in large systems where you have very specific type-defined capabilities and messaging, and you end with different engineers creating many types which are equivalent but have independent implementations. This is another kind of nightmare because you need to convert many types to use them, which ends up being the source of a lot of boilerplate, and many things are nice, but it takes a very influential and powerful architect to curb complexity in such a system, and in the end complexity is inevitable, even if specific bouts of it spring up and get fixed, you end up with many many many APIs. So despite FDs being a very corset type API, it's brazen simplicity has eliminated an entire layer of complexity from our software. Perhaps my complaints of FD's complexity are the misplaced pangs of an idealist, forgetting the importance of the big picture. FDs are the APIs we need, but not the ones we deserve. I think their role in the structure of our programs is far more important than their exact nature. Perhaps one day they can be replaced, but whatever does replace it will certainly have learned a lot from the humble FD.
saghm 2y ago
Ah okay, good to know. I never needed to read the same file concurrently with different cursors, so that might be why the API seemed deceptively simple to me!

4lDO2 2y ago

Rust does however implement the IO traits for `&File` as well (shared), and IIRC also implements `try_clone` which is the dup equivalent.

seeknotfind 2y ago

One of the reasons the FD interface is so complicated is because there are many of the same operations, but they do different things to different underlying kernel implementations. In Linux, you have no standard way to tell what the underlying FD is. It's such a wide surface, different kernel surfaces might implement some of the many different file APIs (polling especially) slightly differently. In many cases, you can reuse standard tooling, but random ioctl calls mean you can't always reuse tooling. Nonstandard implementations of standard file calls can make it dangerous if you don't know which type of FD you have. So the good thing is they are standard, you use the same set of tools (file APIs) to operate with them, so it makes the system interface smaller and simpler, but it lacks the fidelity to create orthogonal meta programming over them. It sounds like an unimportant complaint that you can't tell the underlying type, but it also means most languages refer to FDs indiscriminately. You don't typically get type safety in languages for FDs (e.g. different class for an FD backed by shared memory instead of a real exclusive file), and even if you do have these classes you can't really guarantee they are correct at runtime if you get an FD from another process (e.g. over UDS from a child process), so languages leave FDs untyped. About the global cursor, it's not an issue because you can dupe it to get a copy with a different cursor. So only references to the FD with that specific FD identifier have a location associated with itself, and that's because the OS can only look up the state on that FD. So dup gives cursor, but if an FD represents a physical seek, it might be the case that a dup'd FD actually does affect the functionality of the first FD by linking the two together, as that's a choice you can have in the kernel when implementing an FD. Dup also mean no new cursor. So that's what I mean, the complexity of the real world means these objects are fundamentally different, and having the same APIs shoe horns them into something they're not. I've worked in large systems where you have very specific type-defined capabilities and messaging, and you end with different engineers creating many types which are equivalent but have independent implementations. This is another kind of nightmare because you need to convert many types to use them, which ends up being the source of a lot of boilerplate, and many things are nice, but it takes a very influential and powerful architect to curb complexity in such a system, and in the end complexity is inevitable, even if specific bouts of it spring up and get fixed, you end up with many many many APIs. So despite FDs being a very corset type API, it's brazen simplicity has eliminated an entire layer of complexity from our software. Perhaps my complaints of FD's complexity are the misplaced pangs of an idealist, forgetting the importance of the big picture. FDs are the APIs we need, but not the ones we deserve. I think their role in the structure of our programs is far more important than their exact nature. Perhaps one day they can be replaced, but whatever does replace it will certainly have learned a lot from the humble FD.

saghm 2y ago

Ah okay, good to know. I never needed to read the same file concurrently with different cursors, so that might be why the API seemed deceptively simple to me!