4 ms·
I'm a sysadmin - this is how I think too. It's great for the most part but... there are some problems: - If the job isn't split up evenly or with an event que
by zdw 16y ago
I'm a sysadmin - this is how I think too. It's great for the most part but... there are some problems:
- If the job isn't split up evenly or with an event queue, you end up preallocating jobs to processes, it's possible that one may take far longer than the rest to finish.
The worst case of this I've run into is Microsoft's EXMerge, which does imports/exports from an Exchange datastore - it can be threaded, but preallocates work by splitting up alphabetically. In one case, a family business, all the heavy users got lumped in one thread because they had the same last name - that thread took 5x longer to run when all other threads had finished.
- You can run the machine out of some resource (mem/disk/CPU) by spawning a huge number of jobs that hit one subsystem hard. This is tuning dependent, of course.
Also, I'd recommend using make and similar tools for this rather than shell commands like xargs if you're going to seriously script this - those tools are made to run processes in parallel and avoid repeating work. They also tend force you to write intermediate steps to disk which can help in debugging (and can be coded around or put on a ramdisk later if it proves to be a performance issue).