5 ms·

"io" cannot depend on "fmt", since "fmt" depends on "io" (technically, dependency cycles in Hare can be solved, but it's messy and strongly discouraged if avoid

"io" cannot depend on "fmt", since "fmt" depends on "io" (technically, dependency cycles in Hare can be solved, but it's messy and strongly discouraged if avoidable). We did previously have a simple io::println which accepted strings but did not do formatting, but it's not super useful and fmt::* does it much better (and it was unbuffered, because bufio depends on io), so we ended up removing it. I thought about keeping it but making it use vectored writes to get around the lack of buffering, but in the end it's not really worth it.

kragen 4y ago
What do you mean by "vectored writes"? I thought you meant "invoking a function pointer" but io::stream already does that. Why is io::handle a separate type from io::stream? Is it just an efficiency hack to avoid an indirection through a function pointer for the common case where what you're writing to actually is a file? Can't io::handle eliminate the entanglement between buffered output and users of streams like the old io::println? I feel like there's something I'm not understanding here. (Is fd: int really the right way to handle a pointer to a structure representing how data is being transparently compressed into a zipfile, or a terminal emulator state that is being updated by writing bytes to it?)
- ddevault 4y ago
  By vectored writes I mean the writev(2) syscall, which can write from multiple buffers in a single syscall (offsetting the performance loss from unbuffered I/O, in theory). io::handle is separate from io::stream because it needs to store either a stream or an io::file. io::file is necessary because it's required for many operating system constructs - a TCP socket is an io::file and cannot be an io::stream, for instance. Only io::files can be passed into syscalls like poll(2). However, stream is separately useful for building userspace I/O primitives, such as io::tee, cryptographic hashes and streams, and so on. So io::handle allows you to have an I/O object which is either a file descriptor (io::file) or a userspace stream (io::stream), but your code doesn't have to care about which it is.
  - kragen 4y ago
    Ohh, I see. Yeah, on rare occasions writev() is worth the hassle, but it doesn't really get around the performance loss from unbuffered I/O; if you're calling println(f.size, " ", f.date, " ", f.filename) in a loop that overwrites f 100 times, writev() reduces that from 600 syscalls to 100, but bufio reduces it to 1. I do understand why io::file needs to be separate from io::stream. (The io documentation introduction gives an explanation of what you're explaining above; I'd additionally offer the examples of a gzip stream, a UTF-8-decoded stream on an ISO-8859-1 text file, and maybe a stream that feeds into in-process terminal emulator logic.) I was asking why io::handle does. If you're writing code that takes an io::handle, generally the code cannot rely on the fact that io::file can be used for select() or ioctl() or getpeername(); if it needed to do that, you would have written it to take an io::file, not an io::handle. So, if your code is only going to invoke io::stream-like operations on the io::handle, it would be simpler if its argument were an io::stream instead of an io::handle. But what if you want to give it an io::file? Well, it's easy enough to wrap an io::file in an io::stream that just invokes the appropriate rt operations, and in most languages the only per-call cost of doing that is that your function call is indirect (mov 12(%ebx), %ecx; call %ecx) rather than direct (call Zn#]io31337write). In fact, most code would be more* efficient that way, because right now if someone gives you an io::handle and you want to read from it, you're usually going to call io::read on it, adding an extra level of function call around the indirect call, because that's simpler than duplicating io::read's conditional call to rt::read in your own code. But dynamically most read and write calls will probably be on a bufio::bufstream (or some other io::stream), so io::read is just going to call .reader(). Does io::handle maybe exist only as an optimization for sendfile()? Maybe it's too late for such changes, given the amount of existing Hare code, and this qualifies as bikeshedding, and if so, I apologize.
    - kragen 4y ago
      I meant "indirect (mov 12(%ebx), %ecx; call *%ecx)". Oops.
    - ddevault 4y ago
      >Does io::handle maybe exist only as an optimization for sendfile()? io::file exists for where it's needed to interface with the host system. io::stream is needed for userspace streams. io::handle is needed so that code which just wants to do I/O but doesn't care which of these two paradigms is in use can be useful for both.

kragen 4y ago

What do you mean by "vectored writes"? I thought you meant "invoking a function pointer" but io::stream already does that. Why is io::handle a separate type from io::stream? Is it just an efficiency hack to avoid an indirection through a function pointer for the common case where what you're writing to actually is a file? Can't io::handle eliminate the entanglement between buffered output and users of streams like the old io::println? I feel like there's something I'm not understanding here. (Is fd: int really the right way to handle a pointer to a structure representing how data is being transparently compressed into a zipfile, or a terminal emulator state that is being updated by writing bytes to it?)

ddevault 4y ago
By vectored writes I mean the writev(2) syscall, which can write from multiple buffers in a single syscall (offsetting the performance loss from unbuffered I/O, in theory). io::handle is separate from io::stream because it needs to store either a stream or an io::file. io::file is necessary because it's required for many operating system constructs - a TCP socket is an io::file and cannot be an io::stream, for instance. Only io::files can be passed into syscalls like poll(2). However, stream is separately useful for building userspace I/O primitives, such as io::tee, cryptographic hashes and streams, and so on. So io::handle allows you to have an I/O object which is either a file descriptor (io::file) or a userspace stream (io::stream), but your code doesn't have to care about which it is.
- kragen 4y ago
  Ohh, I see. Yeah, on rare occasions writev() is worth the hassle, but it doesn't really get around the performance loss from unbuffered I/O; if you're calling println(f.size, " ", f.date, " ", f.filename) in a loop that overwrites f 100 times, writev() reduces that from 600 syscalls to 100, but bufio reduces it to 1. I do understand why io::file needs to be separate from io::stream. (The io documentation introduction gives an explanation of what you're explaining above; I'd additionally offer the examples of a gzip stream, a UTF-8-decoded stream on an ISO-8859-1 text file, and maybe a stream that feeds into in-process terminal emulator logic.) I was asking why io::handle does. If you're writing code that takes an io::handle, generally the code cannot rely on the fact that io::file can be used for select() or ioctl() or getpeername(); if it needed to do that, you would have written it to take an io::file, not an io::handle. So, if your code is only going to invoke io::stream-like operations on the io::handle, it would be simpler if its argument were an io::stream instead of an io::handle. But what if you want to give it an io::file? Well, it's easy enough to wrap an io::file in an io::stream that just invokes the appropriate rt operations, and in most languages the only per-call cost of doing that is that your function call is indirect (mov 12(%ebx), %ecx; call %ecx) rather than direct (call Zn#]io31337write). In fact, most code would be more* efficient that way, because right now if someone gives you an io::handle and you want to read from it, you're usually going to call io::read on it, adding an extra level of function call around the indirect call, because that's simpler than duplicating io::read's conditional call to rt::read in your own code. But dynamically most read and write calls will probably be on a bufio::bufstream (or some other io::stream), so io::read is just going to call .reader(). Does io::handle maybe exist only as an optimization for sendfile()? Maybe it's too late for such changes, given the amount of existing Hare code, and this qualifies as bikeshedding, and if so, I apologize.
  - kragen 4y ago
    I meant "indirect (mov 12(%ebx), %ecx; call *%ecx)". Oops.
  - ddevault 4y ago
    >Does io::handle maybe exist only as an optimization for sendfile()? io::file exists for where it's needed to interface with the host system. io::stream is needed for userspace streams. io::handle is needed so that code which just wants to do I/O but doesn't care which of these two paradigms is in use can be useful for both.