3 ms·
There are three processes involved. Process P, the data producer, writes to file foo. Process Z, the zip-process, reads file foo and produces foo.zip Process
by scbrg 4y ago
There are three processes involved.
Process P, the data producer, writes to file foo.
Process Z, the zip-process, reads file foo and produces foo.zip
Process rm, removes file foo.
I imagine it went a bit like this:
P --output foo &
Z < foo > foo.zip
rm foo
# time passes
# process P terminates
Any data written to foo since Z was run is now lost.
User 323 assumed that either rm would fail, letting them know that foo cannot be deleted, or that P would throw an error when the file was deleted.
(I don't think this is a reasonable expectation, and it's a failure on 323's hand of not learning the UNIX file model, but there's the situation).
- trelane 4y agoIn this case, it would have been deleted anyway. I think I can see the loss: 1. Start zip. 2. In parallel, copy the intermediate zip file. 3. Delete the zip 4. Delete originals The data was deleted in step 4 and is the loss. It notably would also had occurred had the zip terminated early or not zipped _all_ the files for some reason. The process was not resilient.
- 323 4y agoI understand the UNIX file model. My mistake was that I thought that the data producer was writing to a different file (think something like log-rotation). I think it's debatable if it's reasonable to continue allowing writing to an unlinked file. There was discussion some time ago about Postgres data loss because of something similar, where the linux kernel returned successfully from fsync despite the movable media not being present anymore. > How is it possible that PostgreSQL used fsync incorrectly for 20 years, and what we'll do about it https://archive.fosdem.org/2019/schedule/event/postgresql_fsync/ https://archive.fosdem.org/2019/schedule/event/postgresql_fs...