4 ms·
The problem is triviality to implement, a similar problem occurs with TCP and UDP. Lots of people implement half of TCP over UDP because they want packet based
by zaarn 7y ago
The problem is triviality to implement, a similar problem occurs with TCP and UDP.
Lots of people implement half of TCP over UDP because they want packet based transmission but also care about losing packets.
Implementing packets on top of a stream is not as trivial as implementing a stream over packets.
Implementing streaming files over record-based files is easier, you just iterate over each record and append to the stream. Implementing records in stream-based files means you have to define a framing format, internal format, bookkeeping, etc.
Record-based files would be the proper lego building blocks.
- GoblinSlayer 7y agoThe disk is a block device, i.e. it reads and writes data in blocks, the system just abstracts that into a stream interface.
- zaarn 7y agoYeah and that is IMO the wrong abstraction. Blocks on disk should be organized into records, where multiple blocks can belong to a single or multiple (if using dedup or Copy-on-Write) records. Implementing a performant stream over that can be done in libc with minimal overhead.
- GoblinSlayer 7y agoNow you want a different block structure.
- zaarn 7y agoNo? ZFS already organizes blocks on disks this way to some extend and Btrfs uses extends which include multiple blocks together and can be referenced multiple times.
- bvrmn 7y ago> Implementing streaming files over record-based files is easier What a performance-wise nightmare. Goodbye page cache and other zero copy optimizations.
- zaarn 7y agoNot really, the OS still has to abstract it onto the harddisk, this should be trivial to achieve without impacting page cache and zero copy optimizations if the on-disk metadata is seperated sufficiently. COW Filesystems already do it to a lesser degree and they're certainly fine.
- bvrmn 7y agoWhat to do with clients who want a stream of records? Separated meta + payload allows fast byte streams. Combined meta + payload allows fast record streams. Structured implementation can't provide performance for both cases. But byte-oriented can. And we see this approach in modern and fast OSes.
- zaarn 7y agoAllow clients to choose which they receive?
- bvrmn 7y agoMay be I was unclear. If we introduce record abstraction for file ops then OS low-level implementation have to decide which way to store structured data. meta + payload or meta separated from payload. In latter case there is no way to provide fast record streams. Former case does not allow fast byte streams. Client decision to receive bytes or records doesn't magically zero copy reorder stored data.
- zaarn 7y agoWhy would the later cause record streams to be slow? Snapshots in COW FS' aren't that terribly slow either, even if there is quite a few of them, which basically boils down to the same structure, really.
- imtringued 7y agoThe problem with the TCP vs UDP debate is that there exists a middle ground that satisfies most people's requirements. TCP offers a single reliable globally ordered stream. UDP offers unreliable packets in arbitrary order. What people usually want is to send whole packets(or multiple streams) in arbitrary order with configurable reliability. TCP and UDP offer neither but UDP has the least restrictions which makes it the only protocol on which you can implement your custom protocol.