3 ms·

> Regulators have never dictated where auditable logs must live. That’s true. They specify that logs cannot be lost, available for x years, not modifiable, acc

> Regulators have never dictated where auditable logs must live. That’s true. They specify that logs cannot be lost, available for x years, not modifiable, accessible only in y ways, cannot cross various boundaries/borders (depending on gov in question). Or bad things will happen to you (your company). In practice, this means that durability of that audit record “a thing happened” cannot be simply “I persisted it to disk on one machine”. You need to know that the record has been made durable (across whatever your durability mechanism is, for example a DB with HA + DR), before progressing to the next step. Depending on the stringency, RPO needs to be zero for audit, which is why I say it is a special case. I don’t know anything about linux audit, I doubt it has any relevance to regulatory compliance.

otterley 10mo ago
> In practice, this means that durability of that audit record “a thing happened” cannot be simply “I persisted it to disk on one machine” As long as the record can be located when it is sought, it does not matter how many copies there are. The regulator will not ask so long as your system is a reasonable one. Consider that technologies like RAID did not exist once upon a time, and backup copies were latent and expensive. Yet we still considered the storage (which was often just a hard copy on paper) to be sufficient to meet the applicable regulations. If a fire then happened and burned the place down, and all the records were lost, the business would not be sanctioned so long as they took reasonable precautions. Here, I’m not suggesting that “the record is on a single disk, that ought to be enough.” I am assuming that in the ordinary course of business, there is a working path to getting additional redundant copies made, but those additional copies are temporarily delayed due to overload. No reasonable regulator is going to tell you this is unacceptable. > Depending on the stringency, RPO needs to be zero for audit And it is! The record is either in local storage or in central storage.
- lll-o-lll 10mo ago
  > And it is! The record is either in local storage or in central storage. But it isn’t! Because there are many hardware failure modes that mean that you aren’t getting your log back. For the same reason that you need acks=all in Kafka for zero data loss, or synchronous_commit = remote_flush in PostgreSQL, you need to commit your audit log to more than the local disk!
  - otterley 10mo ago
    If your hardware and software can’t guarantee that writes are committed when they say they are, all bets are off. I am assuming a scenario in which your hardware and/or cloud provider doesn’t lie to you. In the world you describe, you don’t have any durability when the network is impaired. As a purchaser I would not accept such an outcome.
    - lll-o-lll 10mo ago
      It’s about avoiding single points of failure. > In the world you describe, you don’t have any durability when the network is impaired. Yes, the real world. If you want durability, a single physical machine is never enough. This is standard distributed computing, and we’ve had all (most) of the literature and understanding of this since the 70’s. It’s complicated, and painful to get right, which is why people normally default to a DB (or cloud managed service). The reason this matters for this logging scenario is that I normally don’t care if I lose a bit of logging in a catastrophic failure case. It’s not ideal, but I’m trading RPO for performance. However, when regs say “thou shalt not lose thy data”, I move the other way. Which is why the streams are separate. It does impose an architectural design constraint because audit can’t be treated as a subset of logs.
      - otterley 10mo ago
        > If you want durability, a single physical machine is never enough. It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design for similar loss probabilities. Again, auditors do not care -- a fact you admitted yourself! They care about whether you took reasonable steps to ensure correctness and availability when needed. That is all. > when regs say “thou shalt not lose thy data”, I move the other way. Which is why the streams are separate. It does impose an architectural design constraint because audit can’t be treated as a subset of logs. There's no conflict between treating audit logs as logs -- which they are -- with having separate delivery streams and treatment for different retention and durability policies. Regardless of how you manage them, it doesn't change their fundamental nature. Don't confuse the nature of logs with the level of durability you want to achieve with them. They're orthogonal matters.

otterley 10mo ago

> In practice, this means that durability of that audit record “a thing happened” cannot be simply “I persisted it to disk on one machine” As long as the record can be located when it is sought, it does not matter how many copies there are. The regulator will not ask so long as your system is a reasonable one. Consider that technologies like RAID did not exist once upon a time, and backup copies were latent and expensive. Yet we still considered the storage (which was often just a hard copy on paper) to be sufficient to meet the applicable regulations. If a fire then happened and burned the place down, and all the records were lost, the business would not be sanctioned so long as they took reasonable precautions. Here, I’m not suggesting that “the record is on a single disk, that ought to be enough.” I am assuming that in the ordinary course of business, there is a working path to getting additional redundant copies made, but those additional copies are temporarily delayed due to overload. No reasonable regulator is going to tell you this is unacceptable. > Depending on the stringency, RPO needs to be zero for audit And it is! The record is either in local storage or in central storage.

lll-o-lll 10mo ago
> And it is! The record is either in local storage or in central storage. But it isn’t! Because there are many hardware failure modes that mean that you aren’t getting your log back. For the same reason that you need acks=all in Kafka for zero data loss, or synchronous_commit = remote_flush in PostgreSQL, you need to commit your audit log to more than the local disk!
- otterley 10mo ago
  If your hardware and software can’t guarantee that writes are committed when they say they are, all bets are off. I am assuming a scenario in which your hardware and/or cloud provider doesn’t lie to you. In the world you describe, you don’t have any durability when the network is impaired. As a purchaser I would not accept such an outcome.
  - lll-o-lll 10mo ago
    It’s about avoiding single points of failure. > In the world you describe, you don’t have any durability when the network is impaired. Yes, the real world. If you want durability, a single physical machine is never enough. This is standard distributed computing, and we’ve had all (most) of the literature and understanding of this since the 70’s. It’s complicated, and painful to get right, which is why people normally default to a DB (or cloud managed service). The reason this matters for this logging scenario is that I normally don’t care if I lose a bit of logging in a catastrophic failure case. It’s not ideal, but I’m trading RPO for performance. However, when regs say “thou shalt not lose thy data”, I move the other way. Which is why the streams are separate. It does impose an architectural design constraint because audit can’t be treated as a subset of logs.
    - otterley 10mo ago
      > If you want durability, a single physical machine is never enough. It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design for similar loss probabilities. Again, auditors do not care -- a fact you admitted yourself! They care about whether you took reasonable steps to ensure correctness and availability when needed. That is all. > when regs say “thou shalt not lose thy data”, I move the other way. Which is why the streams are separate. It does impose an architectural design constraint because audit can’t be treated as a subset of logs. There's no conflict between treating audit logs as logs -- which they are -- with having separate delivery streams and treatment for different retention and durability policies. Regardless of how you manage them, it doesn't change their fundamental nature. Don't confuse the nature of logs with the level of durability you want to achieve with them. They're orthogonal matters.