This is a lovely wedge of experience. You are of course very correct that if you have a massive number of processes using files for IPC, things will fail in ve

This is a lovely wedge of experience. You are of course very correct that if you have a massive number of processes using files for IPC, things will fail in very strange ways. One of the very positive things that AWS/GCE provide is simple scalable primitives, with obvious metrics to measure limits. For example SQS is a brilliant primitive for many-many message based processing. It avoids the horror of running a HA kafka queue(or worse).