3 ms·
Append workloads are common in distributed systems. Turns out nearly every time you think you need random read/write to a datastructure (eg. a hard drive/blo
by londons_explore 3y ago
Append workloads are common in distributed systems. Turns out nearly every time you think you need random read/write to a datastructure (eg. a hard drive/block device for a Windows/Linux VM), you can instead emulate that with an append-only log of changes and a set of append-only indexes.
Doing so has huge benefits: Write performance is way higher, you can do rollbacks easily (just ignore the tail of the files), you can do snapshotting easily (just make a new file and include by reference a byte range of the parent), etc.
The downside is from time to time you need to make a new file and chuck out the dead data - but such an operation can be done 'online', and can be done during times of lower system load.
- merb 3y agoyou just described wal files of a database.
- KRAKRISMOTT 3y agoThe generalized architecture is called event sourcing
- capableweb 3y agoWell, I'd argue that it's just two different names for similar concepts, but applied at different levels. WAL is a low-level implementation detail, usually for durability, while Event Source is a architecture applied to solve business problems. A WAL would usually disappear or truncate it's length after a while, and you'd only rerun things from it if you absolutely have two. Changes in business requirements shouldn't require you to do anything with a WAL. In contrast, Event sourcing log would be kept indefinitely, so when business requirements change, you could (if you want to, not required) re-run N previous events so you can apply new changes to old data in your data storage. But, if you really want to, it's basically the same, but in the end, applied differently :)
- slt2021 3y agoWAL is just one moving piece of database, that could be using SSTable or LSM tree or some other append-only immutable data structure, and the Event Sourcing is enterprise architecture pattern applied to business applications that heavily relies on immutable append-only data structures as the backbone
- tough 3y agoha I was just thinking how similar this was to postgres pg-audit way of reusing the logs to sum up the correct state
- capableweb 3y ago> Append workloads are common in distributed systems. Bringing back the topic to what the parent was saying; since S3 is a pretty common system, and a distributed system at that, are you saying that S3 does support appending data? AFAIK, S3 never supported any append operations.
- ozfive 3y agoIt does not. Consider using services like Amazon Kinesis Firehose, which can buffer and batch logs, then periodically write them to S3.
- vlovich123 3y agoI’ll add some nuance here. You can implement append yourself in a clunk way. Create a new multipart upload for the file, copy its existing contents, create a new part appending what you need, and then complete the upload. Not as elegant / fast as GCS’s and there may be other subtleties, but it’s possible to simulate.