4 ms·
Even if you don’t find the tool useful, the implementation under the hood is very clever. The dependency graph between the jobs is stored as a bunch of live pro
by rahimiali 6y ago
Even if you don’t find the tool useful, the implementation under the hood is very clever. The dependency graph between the jobs is stored as a bunch of live processes flock()ing on each other’s log files. Once the files are unlocked, the process execs the requested job. This makes the scheduling and error handling code much simpler. Also, right now the dependency graph is dense (a job depends on all previous jobs) but it would be straightforward to thin it out to allow parallel jobs.
Overall, a clever trick I intend to steal.
- anewhnaccount2 6y agoAgreed! I think the general principle of doing UNIXy stuff is "use the capabilities of processes and the filesystem your units of abstraction". It comes out quite nicely sometimes. Another example is SLURM, where processes are used as context managers for keeping track of allocations.
- petre 6y agoYes, but it's too bad it creates litter wherever you run it. I'd rather it stored all those logs under ~/.cache/ like the other well behaved utilities, in text files like it does or in a searchable SQLite database.
- JNRowe 6y agoFrom the man page: NQDIR Directory where lock files/job output resides. Each NQDIR can be considered a separate queue. Whether or not the default should be $XDG_CACHE_HOME/nq/<something> is a different question. For my own use cases with nq I like the current directory being used, but it would obviously be just as easy for me to set `NQDIR=.`.
- greggyb 6y agoThis is addressed directly in the first section of the README: > By default, job ids are per-directory, but you can set $NQDIR to put them elsewhere. Creating nq wrappers setting $NQDIR to provide different queues for different purposes is encouraged.