As we continue to increase throughput by increasing the number of cores, high-throughput local filesystems are going to start looking more like distributed file

As we continue to increase throughput by increasing the number of cores, high-throughput local filesystems are going to start looking more like distributed filesystems. (A multicore system really is a distributed system with a low-latency interconnect and some hardware to give the illusion of flat access to main memory.) Having done a fair amount of work on long-term persistence of data from a Kafka queue to a distributed filesystem, I wish filesystems provided Lamport timestamp for the current time and a Lamport timestamp of the earliest uncommitted write. The distributed filesystem itself would internally keep track of the earliest uncommitted write using a vector clock, though only the minimum component of that vector would be externally visible. When I perform a write, I want to send an opaque 64/128-bit ID of my choosing, the write location, and the data. I want to asynchronously get back a message with either an error message and the write ID, or else [the write ID, the current Lamport timestamp, and the minimum component of the uncommitted timestamp vector] (as known by the node where I just wrote). By keeping a mapping of these Lamport timestamps to my Kafka partion read offsets, this would reduce the number of network round-trips necessary to commit my Kafka partition read offsets back to Kafka. As it stands, I periodically need some extra network round-trips to query commit status of my async writes before committing read offset changes for my consumer group back to Kafka. With the distributed filesestem keeping a vector cloock, and each async write getting back a message containing the current Lamport timestamp and the minimum uncommitted timestamp, then I could put that current time in my map of timestamps to Kafka read offsets, and then commit back to Kafka all read offsets corresponding to fs writes that are older than the earliest uncommitted write. Presumably, the Lamport timestamp and vector clock maintenance would be piggy-backed on the internal messages necessary for the distributed filesystem to persist the writes to disk on multiple nodes. That is, I write to node A, which tells me "that's a write at time 9", and node A sends messages saying "here's a write at time 9, and my earliest uncommitted write is at time 4" to nodes B and G. The next time that node G needs to forward a write to node A, then node A will update its current time and its G component of the uncommitted timestamp vector clock. You'd also want each node to guarantee a maximum amount of time between its forwarding writes to any given node, forwarding an empty write for clock update purposes if it has been too long.