3 ms·
What makes it atomic is running publishers and consumers on the same box (since you're sharing filesystem between those). Also listdir is a big bottleneck here
by hbrn 4y ago
What makes it atomic is running publishers and consumers on the same box (since you're sharing filesystem between those).
Also listdir is a big bottleneck here:
while True:
# get files in outbox
files_in_outbox = [f'{PREFIX}/outbox/{x}' for x in os.listdir(f'{PREFIX}/outbox')]
- andrewstuart 4y ago>> What makes it atomic is running publishers and consumers on the same box (since you're sharing filesystem between those). It's the move/rename that is atomic. https://man7.org/linux/man-pages/man2/rename.2.html https://man7.org/linux/man-pages/man2/rename.2.html
- hbrn 4y agoNot really. However, there will probably be a window in which both oldpath and newpath refer to the file being renamed. But that's not even the main point. 1. Move happens after email is sent, so there is a window where email is already being sent but file still exists. 2. Even if you do it before, there's still a window between os.listdir() and os.remove() 3. Complexity is O(N^2) due to listdir() + getctime() being called on every iteration. If you just want to ensure order, it probably works fine at a small scale. But it would be unwise to run multiple consumers on a single instance, and impossible to run them on multiple instances.
- dekhn 4y agoI worked with a network filesystem that supported atomic renames and we based an entire large-scale production system on the idea that it would work (it did). The system supported Youtube and Google Play model training, regularly processing increments of hundreds of terabytes.