4 ms·

> I don't mean to sound dismissive, but do you have any idea of how O_APPEND interacts with PIPE_BUF? Nope. I was replying to one proposed semantic and arguing

> I don't mean to sound dismissive, but do you have any idea of how O_APPEND interacts with PIPE_BUF? Nope. I was replying to one proposed semantic and arguing that the ideal semantic is slightly different. I haven't looked deeply into how such a semantic could or could not be mapped onto current APIs. > at the point where you're making >4K writes you're usually better off having some custom log system anyway and not rely on the kernel serializing things for you I disagree. The kernel has access to the I/O queue, and I don't. The kernel can fundamentally do this more efficiently and robustly than user space can. So it sounds like my options are to limit my write size, serialize writers in user space (impossible generally unless you can physically prohibit processes from running that don't follow the serialization protocol) or flock(), which "sucks." Sounds like there is room for an api that does what is desired but doesn't suck. EDIT: I just read about O_APPEND and it doesn't sound like what I'm talking about at all. It always appends to the end. It sounds like if two writers race, both chunks of new data are appended. That's not as useful as what I described. I'm basically talking about compare and swap, except instead of compare and swap its compare and append. If something changed in the meantime you don't want to just blindly append your new block that is unaware of the concurrently appended log data.

avar 11y ago
> I just read about O_APPEND and it doesn't sound like what I'm > talking about at all. It always appends to the end. It sounds like > if two writers race, both chunks of new data are appended. That's > not as useful as what I described. I'm basically talking about > compare and swap, except instead of compare and swap its compare and > append Correct. If you want to only write to the end of a file if that file is size X you'd either need your proposed syscall of "only write if size is X" or have every writer cooperate via flock() to achive that via current semantics. That's not at all what the grandparent is talking about though. They just want "log" files where you can only append and the write() is atomic. That doesn't mean "my write shouldn't work if there is a concurrent write" which is a semantic you're making up, and which I don't really see making sense for log files. I'm really finding it difficult to imagine a plausable use case for your proposed semantics. Imagine this: You have 100 concurrent web serving processes all writing to an access.log, currently they can just write() O_APPEND to an access.log with a string below PIPE_BUF and their writes will end up on disk, but may not be interleaved in "real time", but who cares? What you're proposing means that only 1 out of those 100 processes will succeed in their write() call. The rest will start a retry loop that's trying to write out some data whose order didn't even matter in the first place to try to flush it to disk because they all want to only write the data if the size of the file is N, and every single write updates N. If you have a use-case for strictly serializing writes to disk like this that's fine, but it's not the common case with log files, and the way it's done is not to try to add a new API to the kernel, you just send messages to some userspace thread that queues them up and does the I/O for you.
- haberman 11y ago
  > You have 100 concurrent web serving processes all writing to an access.log That makes no sense. Have each process write to its own log. This is making your life hard for no reason. > If you have a use-case for strictly serializing writes to disk like this that's fine, but it's not the common case with log files This isn't about access.log, it's about transaction logs for database systems. Like the commit log for Postgres or SQLite. You can think of their commit logs as like a block chain. Imagine a block chain that didn't have a pointer to the previous block, but just a timestamp. It wouldn't work. If you have two conflicting blocks you need some central arbiter to decide which wins.
  - avar 11y ago
    > That makes no sense. Have each process write to its own log. This is > making your life hard for no reason. This is literally how pretty much every daemon process works on Unix systems. You don't log in and tail /var/log/apache/access.log.{1..N} where N is the number of threads/processes. You just tail /var/log/apache/access.log which all of them write to by virtue of PIPE_BUF semantics. Having to tail N logs where N is some unknown number of processes is what's making your life harder for no reason. That's what O_APPEND and PIPE_BUF are for. > This isn't about access.log, it's about transaction logs for > database systems. [...] think of their commit logs as like a > block chain. I really don't see how any of this is relevant to the original point of having "log" files. Now you're talking about DB transaction logs and blockchains, which really don't need to have their nuances implemented in terms of POSIX file semantics. They can just have a single process that manages those expectations. Everyone's expectations don't have to be implemented on the filesystem level, particularly when those expectations inherently involve locking writes and retry loops.
    - haberman 11y ago
      > You just tail /var/log/apache/access.log which all of them write to by virtue of PIPE_BUF semantics. I can see how this would be convenient. That's fine and I have nothing against it. It's just not a problem I consider interesting, because there is no inherent need for all 100 writers to write to the same file. You could just as easily have each write to a separate file and have a separate process that merges them together if you want a single file for convenience. > I really don't see how any of this is relevant to the original point of having "log" files. Maybe you're not aware: every major database keeps a commit log, and writes it to a file called a log file (sometimes "write-ahead log file"). That is what I think of when I hear about "log" files in the context of file durability, because this is the case where file consistency and durability actually affect the consistency of the system. Here are some examples: https://www.sqlite.org/tempfiles.html#walfile https://www.sqlite.org/tempfiles.html#walfile http://www.postgresql.org/docs/9.1/static/wal-intro.html http://www.postgresql.org/docs/9.1/static/wal-intro.html https://leveldb.googlecode.com/svn/trunk/doc/impl.html https://leveldb.googlecode.com/svn/trunk/doc/impl.html This is a much more interesting problem (to me) because it is harder and relevant to the consistency of any database system. > Now you're talking about DB transaction logs and blockchains, which really don't need to have their nuances implemented in terms of POSIX file semantics. They can just have a single process that manages those expectations. You do realize that SQLite can't just corrupt your database because you ran two SQLites concurrently, right? SQLite has chosen to support concurrent SQLite processes all writing to the same database. Maybe you think they shouldn't do that, but they find it useful. And even in your world where they shouldn't do that, it's still not ok to just corrupt the database because the user didn't follow the rules. > Everyone's expectations don't have to be implemented on the filesystem level One of the kernel's key responsibilities is to arbitrate concurrent access to shared resources. It provides primitives that makes it possible to build higher-level abstractions. What I am describing is a primitive that allows for lock-free appends to consistent transaction logs. It could be useful for a lot of things, and makes at least as much sense as plenty of other things that are already in syscall interfaces.

avar 11y ago

> I just read about O_APPEND and it doesn't sound like what I'm > talking about at all. It always appends to the end. It sounds like > if two writers race, both chunks of new data are appended. That's > not as useful as what I described. I'm basically talking about > compare and swap, except instead of compare and swap its compare and > append Correct. If you want to only write to the end of a file if that file is size X you'd either need your proposed syscall of "only write if size is X" or have every writer cooperate via flock() to achive that via current semantics. That's not at all what the grandparent is talking about though. They just want "log" files where you can only append and the write() is atomic. That doesn't mean "my write shouldn't work if there is a concurrent write" which is a semantic you're making up, and which I don't really see making sense for log files. I'm really finding it difficult to imagine a plausable use case for your proposed semantics. Imagine this: You have 100 concurrent web serving processes all writing to an access.log, currently they can just write() O_APPEND to an access.log with a string below PIPE_BUF and their writes will end up on disk, but may not be interleaved in "real time", but who cares? What you're proposing means that only 1 out of those 100 processes will succeed in their write() call. The rest will start a retry loop that's trying to write out some data whose order didn't even matter in the first place to try to flush it to disk because they all want to only write the data if the size of the file is N, and every single write updates N. If you have a use-case for strictly serializing writes to disk like this that's fine, but it's not the common case with log files, and the way it's done is not to try to add a new API to the kernel, you just send messages to some userspace thread that queues them up and does the I/O for you.

haberman 11y ago
> You have 100 concurrent web serving processes all writing to an access.log That makes no sense. Have each process write to its own log. This is making your life hard for no reason. > If you have a use-case for strictly serializing writes to disk like this that's fine, but it's not the common case with log files This isn't about access.log, it's about transaction logs for database systems. Like the commit log for Postgres or SQLite. You can think of their commit logs as like a block chain. Imagine a block chain that didn't have a pointer to the previous block, but just a timestamp. It wouldn't work. If you have two conflicting blocks you need some central arbiter to decide which wins.
- avar 11y ago
  > That makes no sense. Have each process write to its own log. This is > making your life hard for no reason. This is literally how pretty much every daemon process works on Unix systems. You don't log in and tail /var/log/apache/access.log.{1..N} where N is the number of threads/processes. You just tail /var/log/apache/access.log which all of them write to by virtue of PIPE_BUF semantics. Having to tail N logs where N is some unknown number of processes is what's making your life harder for no reason. That's what O_APPEND and PIPE_BUF are for. > This isn't about access.log, it's about transaction logs for > database systems. [...] think of their commit logs as like a > block chain. I really don't see how any of this is relevant to the original point of having "log" files. Now you're talking about DB transaction logs and blockchains, which really don't need to have their nuances implemented in terms of POSIX file semantics. They can just have a single process that manages those expectations. Everyone's expectations don't have to be implemented on the filesystem level, particularly when those expectations inherently involve locking writes and retry loops.
  - haberman 11y ago
    > You just tail /var/log/apache/access.log which all of them write to by virtue of PIPE_BUF semantics. I can see how this would be convenient. That's fine and I have nothing against it. It's just not a problem I consider interesting, because there is no inherent need for all 100 writers to write to the same file. You could just as easily have each write to a separate file and have a separate process that merges them together if you want a single file for convenience. > I really don't see how any of this is relevant to the original point of having "log" files. Maybe you're not aware: every major database keeps a commit log, and writes it to a file called a log file (sometimes "write-ahead log file"). That is what I think of when I hear about "log" files in the context of file durability, because this is the case where file consistency and durability actually affect the consistency of the system. Here are some examples: https://www.sqlite.org/tempfiles.html#walfile https://www.sqlite.org/tempfiles.html#walfile http://www.postgresql.org/docs/9.1/static/wal-intro.html http://www.postgresql.org/docs/9.1/static/wal-intro.html https://leveldb.googlecode.com/svn/trunk/doc/impl.html https://leveldb.googlecode.com/svn/trunk/doc/impl.html This is a much more interesting problem (to me) because it is harder and relevant to the consistency of any database system. > Now you're talking about DB transaction logs and blockchains, which really don't need to have their nuances implemented in terms of POSIX file semantics. They can just have a single process that manages those expectations. You do realize that SQLite can't just corrupt your database because you ran two SQLites concurrently, right? SQLite has chosen to support concurrent SQLite processes all writing to the same database. Maybe you think they shouldn't do that, but they find it useful. And even in your world where they shouldn't do that, it's still not ok to just corrupt the database because the user didn't follow the rules. > Everyone's expectations don't have to be implemented on the filesystem level One of the kernel's key responsibilities is to arbitrate concurrent access to shared resources. It provides primitives that makes it possible to build higher-level abstractions. What I am describing is a primitive that allows for lock-free appends to consistent transaction logs. It could be useful for a lot of things, and makes at least as much sense as plenty of other things that are already in syscall interfaces.