2 ms·
Ashamed to admit this happened to me once. The job output files did have unique jobid-based names, but I didn't account for the poorly-documented fact that mul
by klyrs 5y ago
Ashamed to admit this happened to me once. The job output files did have unique jobid-based names, but I didn't account for the poorly-documented fact that multiple clusters shared a filesystem (and I was running jobs on a few) so multiple jobs ended up with the same jobid. Thankfully, I had some sentinel start/stop output that allowed me to automatically detect when multiple processes had overlapping output, and I was able to re-run fractional jobs and only wasted about 5% of my compute time.
On one hand, I hated that there wasn't some fancy framework (like, a database?) to hide the footguns behind... on the other hand, the learning curve was very low and I'm happier about that on the balance.
- sarusso 5y agoThis happened to any of us in the early days... :) As a side note, a good framework does not necessarily mean to increase the learning curve, but rather to protect the user from itself.