13 ms·

Nq – A simple Unix job queue system

419 points by unhammer 6y ago

gmfawcett 6y ago
> nq tries hard (but does not guarantee) to ensure the log file of the currently running job has +x bit set. Thus you can use ls -F to get a quick overview of the state of your queue. Is that a typo? Why would you need {ugo}+x to tail a file? [edit -- sorry, it's not tailing; somehow I read `tail -F` instead of `ls -F`.]
- pavpanchekha 6y ago
  Color mode (-F) in ls will highlight executables, usually in green.
  - gmfawcett 6y ago
    Ah! Thanks. I guess if they need a second color, they could mark the file `setuid` as well. :P This is clever, but I wouldn't trust any tool that marks not-intentionally-executable things as executable.
    - wnoise 6y ago
      They are in fact executable! > Due to the initial exec line in the log files, you can resubmit a job by executing it as a shell command file, i.e. running sh $jobid. (and many shells have a legacy behavior of running a file as a script if it doesn't start with a #! line.)
      - jmgao 6y ago
        > (and many shells have a legacy behavior of running a file as a script if it doesn't start with a #! line.) execlp and execvp are actually guaranteed to do this in POSIX-2017, although shells might also do this themselves if they resolve $PATH by themselves.
  - philsnow 6y ago
    -F appends a character to dentry names indicating what kind they are (directory '/' , named pipe '=', executable '*', ... symlink? dunno what else) my `man ls` has: -F, --classify append indicator (one of */=>@|) to entries
    - bmn__ 6y ago
      GNU coreutils manual explains it better. https://www.gnu.org/software/coreutils/manual/html_node/General-output-formatting.html#index-_002d_002dclassify https://www.gnu.org/software/coreutils/manual/html_node/Gene...
- icedchai 6y ago
  Sounds like they are repurposing the bit for status.
- wiml 6y ago
  "ls", not "tail". I think the idea is that "ls" will visually mark files with the execute bit set, which makes them easy to pick out in a directory listing.
amelius 6y ago
How is this different from simply serializing your commands by putting them, in the desired order, in a batch script?
- deleted 6y ago
  [deleted]
- sanxiyn 6y ago
  It is not, but REPL is useful even if you can run script from file. It's like that.
  - jolmg 6y ago
    You can also use `;` and `&` as well as redirections and job control in an interactive shell session.
- aidenn0 6y ago
  Because you can append commands at any point you like without worrying if previous commands have completed or not.
  - jolmg 6y ago
    https://unix.stackexchange.com/questions/572616/run-a-bash-command-when-a-process-is-done https://unix.stackexchange.com/questions/572616/run-a-bash-c... You can do that with the regular shell, just `^Z%; $next_command`, but I agree that nq might be more practical if that kind of workflow is done often.
    - aidenn0 6y ago
      That requires you still have the original shell open that ran the earlier commands; nq doesn't appear to have that limitation.
      - jolmg 6y ago
        Ok, that sheds light into the kind of workflow this might be useful in, like calling stuff from a launcher instead of a terminal or something like that and having them go through nq automatically. Personally, I would just keep the shell open, but options are cool.
jpalomaki 6y ago
Taskspooler provides similar service. Install and submit tasks from command-line. You can choose how many tasks run in parallel. http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
- garettmd 6y ago
  This. Taskspooler is a powerful but simple tool that seems to be fairly unknown.
  - xaduha 6y ago
    I knew about it, but years go by and I can't honestly say that I ever needed it.
- curun1r 6y ago
  Also Pueue (https://github.com/Nukesor/pueue https://github.com/Nukesor/pueue)
  - pdimitar 6y ago
    Damn that's a precious tool. Thank you for showing it to me!
  - Nukesor 6y ago
    Always happy to see my stuff getting recommended :D
- dimatura 6y ago
  I second the recommendation for task spooler, which seems to cover similar ground to nq. It's a really nice way to queue simple long-running tasks without having to write code or set up infrastructure.
- krylon 6y ago
  Heh, I use ts(1) on a fairly regular basis. It is a very nice tool, and it even supports multiple job queues via environment variables.
txdv 6y ago
awesome tool leah!
swlkr 6y ago
Yeah this is really cool, I especially like the idea of using it as a nohup alternative
wyldfire 6y ago
Maybe a public domain alternative to LSF/bsub?
- bonoboTP 6y ago
  Slurm is the go-to free software workload manager / task scheduler. It scales well from a single machine to the largest clusters in existence. Simple things are simple with it while you can also do complex things if you put in more effort.
teddyh 6y ago
No comparison with the venerable at(1) and batch(1)?
- Areading314 6y ago
  Gnu parallel is another great one
  - silasdavis 6y ago
    Apart from it being academic citation ransomware: https://news.ycombinator.com/item?id=15319715 https://news.ycombinator.com/item?id=15319715
    - SiempreViernes 6y ago
      It's nagware.
      - CameronNemo 6y ago
        Ransoming grad students' data sounds like the least profitable venture on earth.
- networked 6y ago
  1. at runs jobs at a given time. batch runs jobs "when system load levels permit" [1]. nq runs jobs in sequence with no regard to the system's load average. 2. at and batch have 52 built-in queues: a-z and A-Z. Any directory can be a queue for nq. 3. You can follow the output of an nq queue tail-style with fq. 4. The syntax is different. at and batch take whole scripts from the standard input or a file; nq takes a single command as its command line arguments. 5. nq doesn't rely on a daemon. It's an admirably simple design. Jobs are just flock()-ed output files in a directory. [1] See https://www.freebsd.org/cgi/man.cgi?query=at&sektion=1&n=1 https://www.freebsd.org/cgi/man.cgi?query=at&sektion=1&n=1.
andrey_utkin 6y ago
Having skimmed the README, I failed to grasp what it is. I would appreciate it if the README featured a clear one-sentence summary of what the tool does. Like... > cat - concatenate files and print on the standard output > nohup - run a command immune to hangups, with output to a non-tty > at — execute commands at a later time Anyway, it's good to see new conceptual tools being developed for the Unix Toolbox, keep up!
- deleted 6y ago
  [deleted]
- jolmg 6y ago
  I think the first example is pretty good: > Build targets clean, depends, all, without occupying the terminal This accommodates a workflow where one wouldn't prefer to just open up another terminal to do other stuff (maybe you're not in a graphical environment, maybe there's no tmux, etc.). With just the shell features, one could do `make ... && make ... && make ... &`, but that would cause bothersome output that would prevent you from effectively working in the same terminal. One could redirect the output of those commands, but then you lose it, or you have to think of where to collect it. This provides an alternative where you can background and still have convenient access to the output when you need it. The other use is something like: $ nq mpv $youtube_url $ #work... work... work.... $ nq mpv $next_youtube_url so that the next starts when the first finishes.
  - Zhyl 6y ago
    The mpv example was my main thought. If you want to add something to a playlist using mpv's playlist feature you have to stop and restart mpv.
    - jolmg 6y ago
      Doesn't help if you like to loop the playlist, though.
    - JNRowe 6y ago
      It probably isn't as user friendly as you'd want, but you can reload the playlist by firing commands at a socket. See the "JSON IPC" section in the mpv man page, and specifically the load* commands. jo¹ and socat² are probably the simplest way to use it if you're not looking for heavy scripting. The scriptability of mpv is really nice if you're the sort of person who likes building your perfect environment, and also a huge pit of addictive time sinks. </warning> ¹ https://github.com/jpmens/jo https://github.com/jpmens/jo ² http://www.dest-unreach.org/socat/ http://www.dest-unreach.org/socat/
watersb 6y ago
I always find great stuff when I read @leahneukirchen's blog posts.
tyingq 6y ago
Gearman is popular if your needs are more complex than this. http://gearman.org/ http://gearman.org/
- krylon 6y ago
  Hehe, my brain wanted to read "German" at first, and I was slightly confused for a moment. Thanks for the suggestion!
gigatexal 6y ago
I was just thinking there should be a tool like this. Thanks! I’ll check it out!
RickHull 6y ago
Note, this is from the creator of Ruby's Rack https://github.com/rack/rack https://github.com/rack/rack
strken 6y ago
What I really want is a way to define a graph of tasks and dependencies, and then execute it across n workers. Something like parq add --task-name wget1 wget http://example.com/foo.json parq add --task-name wget2 wget http://example.com/bar.json parq add --dependencies wget1,wget2 jq -s '.[0] * .[1]' foo.json bar.json parq run -n 2 # executes wget1 and wget2 in parallel using two workers, then merges them with jq
- shipp02 6y ago
  Check out walk[1]. It does exactly this. Lets you define a graph of dependencies in any language of your choice. [1](https://github.com/ejholmes/walk/ https://github.com/ejholmes/walk/)
- humanrebar 6y ago
  Some other fine suggestions here, but I'll also mention that luigi should do this fine as well.
- mbreese 6y ago
  What you're looking for is in the class of tools known as batch schedulers. Most commonly these are used on HPC clusters, but you can use them on any size machine. There are a number of tools in this category, and like others have mentioned, my first try would be Make, if that is an option for you. However, I normally work on HPC clusters, so submitting jobs is incredibly common for me. To keep with that workflow without needing to install SLURM or SGE on my laptop (which I've done before!?!?), my entry into this mix is here: https://github.com/compgen-io/sbs https://github.com/compgen-io/sbs. It is a single-file Python3 script. My version is setup to only run across one node, but you can have as many worker threads as you need. For what you asked for, you'd run something like this: $ sbs submit -- wget http://example.com/foo.json 1 $ sbs submit -- wget http://example.com/bar.json 2 $ sbs submit -afterok 1:2 -cmd jq -s '.[0] * .[1]' foo.json bar.json $ sbs run -maxprocs 2 This isn't heavily tested code, but works for the use-case I had (having a single-file batch scheduler for when I'm not on an HPC cluster, and testing my pipeline definition language). Normally, it assumes the parameters (CPUs, Memory, etc) are present as comments in a submitted bash script (as is the norm in HPC-land). However, I also added an option to directly submit a command. stdout/stderr are all captured and stored. The job runner also has a daemon mode, so you can keep it running in the background if you'd like to have things running on demand. Installation is as simple as copying the sbs script someplace in your $PATH (with Python3 installed). You should also set the ENV var $SBSHOME, if you'd like to have a common place for jobs. The usage is very similar to many HPC schedulers... sbs submit sbs status sbs run sbs cancel etc...
mongol 6y ago
I think this can be useful to run with git hooks. It can branch off builds etc at commit. For quick and dirty continuous integration.
soheil 6y ago
> To the extent possible under law, Leah Neukirchen leah@vuxu.org has waived all copyright and related or neighboring rights to this work. This seems a bit over the top, what is the advantage of giving up the rights to your work so slavishly?
- davidzweig 6y ago
  I'd imagine it is a statement on the author's views on software copyright. Or perhaps they wanted to offer something to the world, no strings attached. I don't think the word slavishly applies here at all.
- raggi 6y ago
  It's not too uncommon. It happens to be a convenient way to ensure that things are mostly used in a personal context. Most lawyercats will steer clear of software under these terms, and use under such terms is typically disallowed in any megacorp.
  - creata 6y ago
    Why's that? I thought megacorps would love software with no strings attached: at least, they seem to love MIT- and BSD-licensed codebases, which give up pretty much everything but attribution, if I understand correctly. Is it because it's unclear whether such a statement would really waive all copyright? Edit: that seems to be it; here are Google's views on the matter: https://opensource.google/docs/thirdparty/licenses/#unencumbered https://opensource.google/docs/thirdparty/licenses/#unencumb...
  - jeffparsons 6y ago
    I thought it was the opposite; there are some weird legal subtleties to placing things in the "public domain" in some parts of the world, so I thought people wrote statements like in the GP comment to avoid having to use that language at all, and share their code in the least encumbered way possible. I wonder whether there's a better way to express it. E.g. "if you see this code, I automatically grant to you a nonexclusive, irrevocable, royalty-free license to do whatever you want with it, including re-mixing it, re-distributing it under any license you want, or anything else you might think of." Or is this _worse_? Any lawyers in the room? :)
soheil 6y ago
So this is somehow a better version of &? Supporting only a single queue makes this no different than `nohup`, etc. Even the examples provided describe what can already be done by appending a simple & to a command: % mkdir -p /tmp/downloads % alias qget='NQDIR=/tmp/downloads nq wget' % alias qwait='NQDIR=/tmp/downloads fq -q' window1% qget http://mymirror/big1.iso window2% qget http://mymirror/big2.iso ... what about? wget http://mymirror/big1.iso & wget http://mymirror/big2.iso & It all just seems to be trying very hard to be something it's not, but I can't put my finger on it.
- CGamesPlay 6y ago
  You're aware that & doesn't actually queue, right? That would run all commands in parallel. nq runs them serially.
  - soheil 6y ago
    So this? ( wget http://mymirror/big1.iso; wget http://mymirror/big2.iso ) &
    - CGamesPlay 6y ago
      Yes, this is closer to what nq is doing :) This comparison is addressed over here: https://news.ycombinator.com/item?id=25920995 https://news.ycombinator.com/item?id=25920995
      - soheil 6y ago
        I see, what about this then?:) Run the queue: mkfifo q ( while True; do sh q; done ) & Schedule jobs: echo wget http://mymirror/big1.iso > q echo wget http://mymirror/big1.iso > q This has the added benefit of supporting multiple queues.
        CGamesPlay 6y ago
        Watch out, FIFOs don't have buffers, so that echo will block. Running the echos in the background causes a race condition on execution order.
euske 6y ago
It's offtopic, but man pages should really adopt a new format (e.g. markdown) instead of roff. roff is a terrible way to write a document. Its format is ancient and not well documented. Its behavior is not consistent across different implementations. Worst of all, no proper i18n support. There's a tool like roff2html, but again it's pretty sketchy in terms of reliability and i18n support. I wrote my own converter when I was translating OpenBSD manpages [1], but I hope more people recognize this issue. [1] https://github.com/euske/openssh-jman/blob/master/roff2html.py https://github.com/euske/openssh-jman/blob/master/roff2html....
- reidjs 6y ago
  I really like cht.sh curl cht.sh/grep
- kseistrup 6y ago
  You can use scdoc(1) to produce man-pages from something that resembles a subset of markdown: ⌘ https://git.sr.ht/~sircmpwn/scdoc https://git.sr.ht/~sircmpwn/scdoc
  - kseistrup 6y ago
    PS: You can see an example of scdoc sources here: ⌘ https://github.com/kseistrup/timestamp/tree/master/src https://github.com/kseistrup/timestamp/tree/master/src The scdocs havde the extension .md because Microsoft Github thinks .sc is SuperCollider files, whatever that is. The .1 files are “compiled” from the .1.md sources.
0xbadcafebee 6y ago
Looks pretty cool. I would love it if it weren't dependent on filesystem locking. Wondering about network locking now. If it's already forking a daemon, might as well store that lock data in memory and communicate it over a network socket. There's still the question of resolving the network service, but that's a flaw in the TCP/IP stack. I bet it could be solved by a simple wrapper command to a service resolver. Custom protocol syntax for locking per dir should be trivial. Heck, maybe move the network logic out to the wrapper, and have the locking be some other interaction between the wrapper and nq.
lboc 6y ago
The AIX print spooler can be used as a generic queueing system - I remember setting up queues with shells as backends. Jobs sent to those queues would then be executed in sequence, and could be cancelled/reprioritized/etc like any print job. Can't claim any credit for the idea though, it was actually suggested in the documentation... It might be possible on other unix systems too, but I never had to try it.
- VWWHFSfQ 6y ago
  That's awesome!
  - lboc 6y ago
    It mentions using 'cc' here: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/printer/generic_aix_spooler.html https://www.ibm.com/support/knowledgecenter/ssw_aix_72/print... An example using ksh: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/filesreference/qconfig.html https://www.ibm.com/support/knowledgecenter/ssw_aix_72/files...
- ptman 6y ago
  lpd/lpr can be used for e.g. a playlist/queue for music https://patrick.wagstrom.net/weblog/2003/05/23/lpdforfunandmp3playing/ https://patrick.wagstrom.net/weblog/2003/05/23/lpdforfunandm...
- randomswede 6y ago
  I once wrote a backend for what I think of as "the SysV print spooler" (hpux7/8, d-nix, and maybe AIX, but IIRC there were some subtle differences with AIX) to run rayshade on the files fed in and drop the resulting image back somewhere (this was ~30 years ago, I don't exactly recall the details). I recall consider doing it for the SunOS box we had, but lpd/lpr (what I think of as "the BSD print spooler") was sufficiently different that while it would have been possible, it wasn't worth the extra hassle to get one more machine to be able to render on. I don't think we ever set it up to shuffle render spool jobs from one host to another, though. And, no, "doing raytracing" was not at ALL what the machines were supposed to do, they just weren't very loaded from their normal workloads, so some of the staff played around with it for fun.
rahimiali 6y ago
Even if you don’t find the tool useful, the implementation under the hood is very clever. The dependency graph between the jobs is stored as a bunch of live processes flock()ing on each other’s log files. Once the files are unlocked, the process execs the requested job. This makes the scheduling and error handling code much simpler. Also, right now the dependency graph is dense (a job depends on all previous jobs) but it would be straightforward to thin it out to allow parallel jobs. Overall, a clever trick I intend to steal.
- anewhnaccount2 6y ago
  Agreed! I think the general principle of doing UNIXy stuff is "use the capabilities of processes and the filesystem your units of abstraction". It comes out quite nicely sometimes. Another example is SLURM, where processes are used as context managers for keeping track of allocations.
- petre 6y ago
  Yes, but it's too bad it creates litter wherever you run it. I'd rather it stored all those logs under ~/.cache/ like the other well behaved utilities, in text files like it does or in a searchable SQLite database.
  - JNRowe 6y ago
    From the man page: NQDIR Directory where lock files/job output resides. Each NQDIR can be considered a separate queue. Whether or not the default should be $XDG_CACHE_HOME/nq/<something> is a different question. For my own use cases with nq I like the current directory being used, but it would obviously be just as easy for me to set `NQDIR=.`.
  - greggyb 6y ago
    This is addressed directly in the first section of the README: > By default, job ids are per-directory, but you can set $NQDIR to put them elsewhere. Creating nq wrappers setting $NQDIR to provide different queues for different purposes is encouraged.
crb002 6y ago
GNU Make could be extended for in flight addition/removal of tasks. No reason to re-invent Make just for interactivity.
Animats 6y ago
Microsoft offers an IBM Job Control Language system for Azure.[1] It's intended for migration from mainframes, but it does the job described here. Not that you'd want to take that route by choice. [1] https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/infrastructure/mainframe-migration https://docs.microsoft.com/en-us/azure/cloud-adoption-framew...
indymike 6y ago
This is a neat little command that is kind of nice because it doesn't require a daemon like at or ts (task spooler) and does dependencies unlike a lot of informal queue techniques and job control built into the shell: $ some_command & $ wait; another_command & $ wait; yet_another_command & $ ... Some other lightweight command queue strategies: * Use a background screen or byobu session. Advantage is that you can attach to it and monitor running tasks. * bq is a bash script that lets you run a worker process and submit commands to be run in sequence. Nice because it's just a shell script.
xtf 6y ago
my simple jobqueue, if I need one waitfor() { while pgrep $1 >/dev/null; do sleep 2; done && $* }
- unhammer 6y ago
  If I may provide some feedback: Line 2: while pgrep $1 >/dev/null; do sleep 2; done && $* ^-- SC2086: Double quote to prevent globbing and word splitting. ^-- SC2048: Use "$@" (with quotes) to prevent whitespace problems. Also, how do you pick the right job to wait for if several programs match $1? (E.g. if it's a bash script, $1 will be "bash", which on my system matches lots of things.) Also, a 2s sleep will slow things down if you want to use it for a whole lot of jobs. Also, this won't run your second command if the first command you wait for has already exited! Also, shouldn't $* be `shift; "$@"` or do you only ever queue jobs of the same command?
  - xtf 6y ago
    Just a simple one when needed. cp, downloader, etc. Nothing too complicated.
kseistrup 6y ago
I'm surprised nobody seems to have mentioned run-parts(8) in this thread: ⌘ https://man.archlinux.org/man/run-parts.8 https://man.archlinux.org/man/run-parts.8 Many Linux distros use this command in the global crontab to serialise daily, weekly, monthly and annual batch jobs.
moonbug 6y ago
A month coding sure does save a morning in the library.
mac3n 6y ago
cool! see also https://gitlab.com/mac3n/qrl https://gitlab.com/mac3n/qrl "simple serverless single-system sequential scheduler" for the same trick note that many shared filesystems ae not posix-compliant

419 points by unhammer 6y ago

> nq tries hard (but does not guarantee) to ensure the log file of the currently running job has +x bit set. Thus you can use ls -F to get a quick overview of the state of your queue. Is that a typo? Why would you need {ugo}+x to tail a file? [edit -- sorry, it's not tailing; somehow I read `tail -F` instead of `ls -F`.]

pavpanchekha 6y ago
Color mode (-F) in ls will highlight executables, usually in green.
- gmfawcett 6y ago
  Ah! Thanks. I guess if they need a second color, they could mark the file `setuid` as well. :P This is clever, but I wouldn't trust any tool that marks not-intentionally-executable things as executable.
  - wnoise 6y ago
    They are in fact executable! > Due to the initial exec line in the log files, you can resubmit a job by executing it as a shell command file, i.e. running sh $jobid. (and many shells have a legacy behavior of running a file as a script if it doesn't start with a #! line.)
    - jmgao 6y ago
      > (and many shells have a legacy behavior of running a file as a script if it doesn't start with a #! line.) execlp and execvp are actually guaranteed to do this in POSIX-2017, although shells might also do this themselves if they resolve $PATH by themselves.
- philsnow 6y ago
  -F appends a character to dentry names indicating what kind they are (directory '/' , named pipe '=', executable '*', ... symlink? dunno what else) my `man ls` has: -F, --classify append indicator (one of */=>@|) to entries
  - bmn__ 6y ago
    GNU coreutils manual explains it better. https://www.gnu.org/software/coreutils/manual/html_node/General-output-formatting.html#index-_002d_002dclassify https://www.gnu.org/software/coreutils/manual/html_node/Gene...
icedchai 6y ago
Sounds like they are repurposing the bit for status.
wiml 6y ago
"ls", not "tail". I think the idea is that "ls" will visually mark files with the execute bit set, which makes them easy to pick out in a directory listing.

amelius 6y ago

How is this different from simply serializing your commands by putting them, in the desired order, in a batch script?

deleted 6y ago
[deleted]
sanxiyn 6y ago
It is not, but REPL is useful even if you can run script from file. It's like that.
- jolmg 6y ago
  You can also use `;` and `&` as well as redirections and job control in an interactive shell session.
aidenn0 6y ago
Because you can append commands at any point you like without worrying if previous commands have completed or not.
- jolmg 6y ago
  https://unix.stackexchange.com/questions/572616/run-a-bash-command-when-a-process-is-done https://unix.stackexchange.com/questions/572616/run-a-bash-c... You can do that with the regular shell, just `^Z%; $next_command`, but I agree that nq might be more practical if that kind of workflow is done often.
  - aidenn0 6y ago
    That requires you still have the original shell open that ran the earlier commands; nq doesn't appear to have that limitation.
    - jolmg 6y ago
      Ok, that sheds light into the kind of workflow this might be useful in, like calling stuff from a launcher instead of a terminal or something like that and having them go through nq automatically. Personally, I would just keep the shell open, but options are cool.

jpalomaki 6y ago

Taskspooler provides similar service. Install and submit tasks from command-line. You can choose how many tasks run in parallel. http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html

garettmd 6y ago
This. Taskspooler is a powerful but simple tool that seems to be fairly unknown.
- xaduha 6y ago
  I knew about it, but years go by and I can't honestly say that I ever needed it.
curun1r 6y ago
Also Pueue (https://github.com/Nukesor/pueue https://github.com/Nukesor/pueue)
- pdimitar 6y ago
  Damn that's a precious tool. Thank you for showing it to me!
- Nukesor 6y ago
  Always happy to see my stuff getting recommended :D
dimatura 6y ago
I second the recommendation for task spooler, which seems to cover similar ground to nq. It's a really nice way to queue simple long-running tasks without having to write code or set up infrastructure.
krylon 6y ago
Heh, I use ts(1) on a fairly regular basis. It is a very nice tool, and it even supports multiple job queues via environment variables.

txdv 6y ago

awesome tool leah!

swlkr 6y ago

Yeah this is really cool, I especially like the idea of using it as a nohup alternative

wyldfire 6y ago

Maybe a public domain alternative to LSF/bsub?

bonoboTP 6y ago
Slurm is the go-to free software workload manager / task scheduler. It scales well from a single machine to the largest clusters in existence. Simple things are simple with it while you can also do complex things if you put in more effort.

teddyh 6y ago

No comparison with the venerable at(1) and batch(1)?

Areading314 6y ago
Gnu parallel is another great one
- silasdavis 6y ago
  Apart from it being academic citation ransomware: https://news.ycombinator.com/item?id=15319715 https://news.ycombinator.com/item?id=15319715
  - SiempreViernes 6y ago
    It's nagware.
    - CameronNemo 6y ago
      Ransoming grad students' data sounds like the least profitable venture on earth.
networked 6y ago
1. at runs jobs at a given time. batch runs jobs "when system load levels permit" [1]. nq runs jobs in sequence with no regard to the system's load average. 2. at and batch have 52 built-in queues: a-z and A-Z. Any directory can be a queue for nq. 3. You can follow the output of an nq queue tail-style with fq. 4. The syntax is different. at and batch take whole scripts from the standard input or a file; nq takes a single command as its command line arguments. 5. nq doesn't rely on a daemon. It's an admirably simple design. Jobs are just flock()-ed output files in a directory. [1] See https://www.freebsd.org/cgi/man.cgi?query=at&sektion=1&n=1 https://www.freebsd.org/cgi/man.cgi?query=at&sektion=1&n=1.

andrey_utkin 6y ago

Having skimmed the README, I failed to grasp what it is. I would appreciate it if the README featured a clear one-sentence summary of what the tool does. Like... > cat - concatenate files and print on the standard output > nohup - run a command immune to hangups, with output to a non-tty > at — execute commands at a later time Anyway, it's good to see new conceptual tools being developed for the Unix Toolbox, keep up!

deleted 6y ago
[deleted]
jolmg 6y ago
I think the first example is pretty good: > Build targets clean, depends, all, without occupying the terminal This accommodates a workflow where one wouldn't prefer to just open up another terminal to do other stuff (maybe you're not in a graphical environment, maybe there's no tmux, etc.). With just the shell features, one could do `make ... && make ... && make ... &`, but that would cause bothersome output that would prevent you from effectively working in the same terminal. One could redirect the output of those commands, but then you lose it, or you have to think of where to collect it. This provides an alternative where you can background and still have convenient access to the output when you need it. The other use is something like: $ nq mpv $youtube_url $ #work... work... work.... $ nq mpv $next_youtube_url so that the next starts when the first finishes.
- Zhyl 6y ago
  The mpv example was my main thought. If you want to add something to a playlist using mpv's playlist feature you have to stop and restart mpv.
  - jolmg 6y ago
    Doesn't help if you like to loop the playlist, though.
  - JNRowe 6y ago
    It probably isn't as user friendly as you'd want, but you can reload the playlist by firing commands at a socket. See the "JSON IPC" section in the mpv man page, and specifically the load* commands. jo¹ and socat² are probably the simplest way to use it if you're not looking for heavy scripting. The scriptability of mpv is really nice if you're the sort of person who likes building your perfect environment, and also a huge pit of addictive time sinks. </warning> ¹ https://github.com/jpmens/jo https://github.com/jpmens/jo ² http://www.dest-unreach.org/socat/ http://www.dest-unreach.org/socat/

watersb 6y ago

I always find great stuff when I read @leahneukirchen's blog posts.

tyingq 6y ago

Gearman is popular if your needs are more complex than this. http://gearman.org/ http://gearman.org/

krylon 6y ago
Hehe, my brain wanted to read "German" at first, and I was slightly confused for a moment. Thanks for the suggestion!

gigatexal 6y ago

I was just thinking there should be a tool like this. Thanks! I’ll check it out!

RickHull 6y ago

Note, this is from the creator of Ruby's Rack https://github.com/rack/rack https://github.com/rack/rack

strken 6y ago

What I really want is a way to define a graph of tasks and dependencies, and then execute it across n workers. Something like parq add --task-name wget1 wget http://example.com/foo.json parq add --task-name wget2 wget http://example.com/bar.json parq add --dependencies wget1,wget2 jq -s '.[0] * .[1]' foo.json bar.json parq run -n 2 # executes wget1 and wget2 in parallel using two workers, then merges them with jq

shipp02 6y ago
Check out walk[1]. It does exactly this. Lets you define a graph of dependencies in any language of your choice. [1](https://github.com/ejholmes/walk/ https://github.com/ejholmes/walk/)
humanrebar 6y ago
Some other fine suggestions here, but I'll also mention that luigi should do this fine as well.
mbreese 6y ago
What you're looking for is in the class of tools known as batch schedulers. Most commonly these are used on HPC clusters, but you can use them on any size machine. There are a number of tools in this category, and like others have mentioned, my first try would be Make, if that is an option for you. However, I normally work on HPC clusters, so submitting jobs is incredibly common for me. To keep with that workflow without needing to install SLURM or SGE on my laptop (which I've done before!?!?), my entry into this mix is here: https://github.com/compgen-io/sbs https://github.com/compgen-io/sbs. It is a single-file Python3 script. My version is setup to only run across one node, but you can have as many worker threads as you need. For what you asked for, you'd run something like this: $ sbs submit -- wget http://example.com/foo.json 1 $ sbs submit -- wget http://example.com/bar.json 2 $ sbs submit -afterok 1:2 -cmd jq -s '.[0] * .[1]' foo.json bar.json $ sbs run -maxprocs 2 This isn't heavily tested code, but works for the use-case I had (having a single-file batch scheduler for when I'm not on an HPC cluster, and testing my pipeline definition language). Normally, it assumes the parameters (CPUs, Memory, etc) are present as comments in a submitted bash script (as is the norm in HPC-land). However, I also added an option to directly submit a command. stdout/stderr are all captured and stored. The job runner also has a daemon mode, so you can keep it running in the background if you'd like to have things running on demand. Installation is as simple as copying the sbs script someplace in your $PATH (with Python3 installed). You should also set the ENV var $SBSHOME, if you'd like to have a common place for jobs. The usage is very similar to many HPC schedulers... sbs submit sbs status sbs run sbs cancel etc...

mongol 6y ago

I think this can be useful to run with git hooks. It can branch off builds etc at commit. For quick and dirty continuous integration.

soheil 6y ago

> To the extent possible under law, Leah Neukirchen leah@vuxu.org has waived all copyright and related or neighboring rights to this work. This seems a bit over the top, what is the advantage of giving up the rights to your work so slavishly?

davidzweig 6y ago
I'd imagine it is a statement on the author's views on software copyright. Or perhaps they wanted to offer something to the world, no strings attached. I don't think the word slavishly applies here at all.
raggi 6y ago
It's not too uncommon. It happens to be a convenient way to ensure that things are mostly used in a personal context. Most lawyercats will steer clear of software under these terms, and use under such terms is typically disallowed in any megacorp.
- creata 6y ago
  Why's that? I thought megacorps would love software with no strings attached: at least, they seem to love MIT- and BSD-licensed codebases, which give up pretty much everything but attribution, if I understand correctly. Is it because it's unclear whether such a statement would really waive all copyright? Edit: that seems to be it; here are Google's views on the matter: https://opensource.google/docs/thirdparty/licenses/#unencumbered https://opensource.google/docs/thirdparty/licenses/#unencumb...
- jeffparsons 6y ago
  I thought it was the opposite; there are some weird legal subtleties to placing things in the "public domain" in some parts of the world, so I thought people wrote statements like in the GP comment to avoid having to use that language at all, and share their code in the least encumbered way possible. I wonder whether there's a better way to express it. E.g. "if you see this code, I automatically grant to you a nonexclusive, irrevocable, royalty-free license to do whatever you want with it, including re-mixing it, re-distributing it under any license you want, or anything else you might think of." Or is this _worse_? Any lawyers in the room? :)

soheil 6y ago

So this is somehow a better version of &? Supporting only a single queue makes this no different than `nohup`, etc. Even the examples provided describe what can already be done by appending a simple & to a command: % mkdir -p /tmp/downloads % alias qget='NQDIR=/tmp/downloads nq wget' % alias qwait='NQDIR=/tmp/downloads fq -q' window1% qget http://mymirror/big1.iso window2% qget http://mymirror/big2.iso ... what about? wget http://mymirror/big1.iso & wget http://mymirror/big2.iso & It all just seems to be trying very hard to be something it's not, but I can't put my finger on it.

CGamesPlay 6y ago
You're aware that & doesn't actually queue, right? That would run all commands in parallel. nq runs them serially.
- soheil 6y ago
  So this? ( wget http://mymirror/big1.iso; wget http://mymirror/big2.iso ) &
  - CGamesPlay 6y ago
    Yes, this is closer to what nq is doing :) This comparison is addressed over here: https://news.ycombinator.com/item?id=25920995 https://news.ycombinator.com/item?id=25920995
    - soheil 6y ago
      I see, what about this then?:) Run the queue: mkfifo q ( while True; do sh q; done ) & Schedule jobs: echo wget http://mymirror/big1.iso > q echo wget http://mymirror/big1.iso > q This has the added benefit of supporting multiple queues.
      - CGamesPlay 6y ago
        Watch out, FIFOs don't have buffers, so that echo will block. Running the echos in the background causes a race condition on execution order.

euske 6y ago

It's offtopic, but man pages should really adopt a new format (e.g. markdown) instead of roff. roff is a terrible way to write a document. Its format is ancient and not well documented. Its behavior is not consistent across different implementations. Worst of all, no proper i18n support. There's a tool like roff2html, but again it's pretty sketchy in terms of reliability and i18n support. I wrote my own converter when I was translating OpenBSD manpages [1], but I hope more people recognize this issue. [1] https://github.com/euske/openssh-jman/blob/master/roff2html.py https://github.com/euske/openssh-jman/blob/master/roff2html....

reidjs 6y ago
I really like cht.sh curl cht.sh/grep
kseistrup 6y ago
You can use scdoc(1) to produce man-pages from something that resembles a subset of markdown: ⌘ https://git.sr.ht/~sircmpwn/scdoc https://git.sr.ht/~sircmpwn/scdoc
- kseistrup 6y ago
  PS: You can see an example of scdoc sources here: ⌘ https://github.com/kseistrup/timestamp/tree/master/src https://github.com/kseistrup/timestamp/tree/master/src The scdocs havde the extension .md because Microsoft Github thinks .sc is SuperCollider files, whatever that is. The .1 files are “compiled” from the .1.md sources.

0xbadcafebee 6y ago

Looks pretty cool. I would love it if it weren't dependent on filesystem locking. Wondering about network locking now. If it's already forking a daemon, might as well store that lock data in memory and communicate it over a network socket. There's still the question of resolving the network service, but that's a flaw in the TCP/IP stack. I bet it could be solved by a simple wrapper command to a service resolver. Custom protocol syntax for locking per dir should be trivial. Heck, maybe move the network logic out to the wrapper, and have the locking be some other interaction between the wrapper and nq.

lboc 6y ago

The AIX print spooler can be used as a generic queueing system - I remember setting up queues with shells as backends. Jobs sent to those queues would then be executed in sequence, and could be cancelled/reprioritized/etc like any print job. Can't claim any credit for the idea though, it was actually suggested in the documentation... It might be possible on other unix systems too, but I never had to try it.

VWWHFSfQ 6y ago
That's awesome!
- lboc 6y ago
  It mentions using 'cc' here: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/printer/generic_aix_spooler.html https://www.ibm.com/support/knowledgecenter/ssw_aix_72/print... An example using ksh: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/filesreference/qconfig.html https://www.ibm.com/support/knowledgecenter/ssw_aix_72/files...
ptman 6y ago
lpd/lpr can be used for e.g. a playlist/queue for music https://patrick.wagstrom.net/weblog/2003/05/23/lpdforfunandmp3playing/ https://patrick.wagstrom.net/weblog/2003/05/23/lpdforfunandm...
randomswede 6y ago
I once wrote a backend for what I think of as "the SysV print spooler" (hpux7/8, d-nix, and maybe AIX, but IIRC there were some subtle differences with AIX) to run rayshade on the files fed in and drop the resulting image back somewhere (this was ~30 years ago, I don't exactly recall the details). I recall consider doing it for the SunOS box we had, but lpd/lpr (what I think of as "the BSD print spooler") was sufficiently different that while it would have been possible, it wasn't worth the extra hassle to get one more machine to be able to render on. I don't think we ever set it up to shuffle render spool jobs from one host to another, though. And, no, "doing raytracing" was not at ALL what the machines were supposed to do, they just weren't very loaded from their normal workloads, so some of the staff played around with it for fun.

rahimiali 6y ago

Even if you don’t find the tool useful, the implementation under the hood is very clever. The dependency graph between the jobs is stored as a bunch of live processes flock()ing on each other’s log files. Once the files are unlocked, the process execs the requested job. This makes the scheduling and error handling code much simpler. Also, right now the dependency graph is dense (a job depends on all previous jobs) but it would be straightforward to thin it out to allow parallel jobs. Overall, a clever trick I intend to steal.

anewhnaccount2 6y ago
Agreed! I think the general principle of doing UNIXy stuff is "use the capabilities of processes and the filesystem your units of abstraction". It comes out quite nicely sometimes. Another example is SLURM, where processes are used as context managers for keeping track of allocations.
petre 6y ago
Yes, but it's too bad it creates litter wherever you run it. I'd rather it stored all those logs under ~/.cache/ like the other well behaved utilities, in text files like it does or in a searchable SQLite database.
- JNRowe 6y ago
  From the man page: NQDIR Directory where lock files/job output resides. Each NQDIR can be considered a separate queue. Whether or not the default should be $XDG_CACHE_HOME/nq/<something> is a different question. For my own use cases with nq I like the current directory being used, but it would obviously be just as easy for me to set `NQDIR=.`.
- greggyb 6y ago
  This is addressed directly in the first section of the README: > By default, job ids are per-directory, but you can set $NQDIR to put them elsewhere. Creating nq wrappers setting $NQDIR to provide different queues for different purposes is encouraged.

crb002 6y ago

GNU Make could be extended for in flight addition/removal of tasks. No reason to re-invent Make just for interactivity.

Animats 6y ago

Microsoft offers an IBM Job Control Language system for Azure.[1] It's intended for migration from mainframes, but it does the job described here. Not that you'd want to take that route by choice. [1] https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/infrastructure/mainframe-migration https://docs.microsoft.com/en-us/azure/cloud-adoption-framew...

indymike 6y ago

This is a neat little command that is kind of nice because it doesn't require a daemon like at or ts (task spooler) and does dependencies unlike a lot of informal queue techniques and job control built into the shell: $ some_command & $ wait; another_command & $ wait; yet_another_command & $ ... Some other lightweight command queue strategies: * Use a background screen or byobu session. Advantage is that you can attach to it and monitor running tasks. * bq is a bash script that lets you run a worker process and submit commands to be run in sequence. Nice because it's just a shell script.

xtf 6y ago

my simple jobqueue, if I need one waitfor() { while pgrep $1 >/dev/null; do sleep 2; done && $* }

unhammer 6y ago
If I may provide some feedback: Line 2: while pgrep $1 >/dev/null; do sleep 2; done && $* ^-- SC2086: Double quote to prevent globbing and word splitting. ^-- SC2048: Use "$@" (with quotes) to prevent whitespace problems. Also, how do you pick the right job to wait for if several programs match $1? (E.g. if it's a bash script, $1 will be "bash", which on my system matches lots of things.) Also, a 2s sleep will slow things down if you want to use it for a whole lot of jobs. Also, this won't run your second command if the first command you wait for has already exited! Also, shouldn't $* be `shift; "$@"` or do you only ever queue jobs of the same command?
- xtf 6y ago
  Just a simple one when needed. cp, downloader, etc. Nothing too complicated.

kseistrup 6y ago

I'm surprised nobody seems to have mentioned run-parts(8) in this thread: ⌘ https://man.archlinux.org/man/run-parts.8 https://man.archlinux.org/man/run-parts.8 Many Linux distros use this command in the global crontab to serialise daily, weekly, monthly and annual batch jobs.

moonbug 6y ago

A month coding sure does save a morning in the library.

mac3n 6y ago

cool! see also https://gitlab.com/mac3n/qrl https://gitlab.com/mac3n/qrl "simple serverless single-system sequential scheduler" for the same trick note that many shared filesystems ae not posix-compliant