6 ms·

Sequential and parallel execution of long-running shell commands

95 points by sea-gold 3y ago

ninja3925 3y ago
This looks quite helpful actually. Thanks for sharing.
elif 3y ago
Actually pretty sweet. `ps | grep` can be tricky to navigate sometimes. I know I've certainly grabbed the wrong process a few times. I'm not exactly sure what advantage this has over managing `screen` sessions tho. Maybe it's cleaner from a process tree perspective?
- cl3misch 3y ago
  I was about to write that this has a different use case, because managing screen/tmux sessions is very manual. However the repo iself states > Pueue is not designed to be a programmable (scriptable) task scheduler/executor. The focus of pueue lies on human interaction. So I also don't really see its usecase and probably would opt for tmux instead. If you had many workers to run, but still do it manually, you would use pueue? I would be interested in such a scenario!
  - dspillett 3y ago
    The concurrency controls are not present using the tmux method. Though if I am running enough tasks for this to be an issue then something odd is happening. The queue being persisted and surviving crashes could be useful, and isn't the case with tmux unless you manually script it up. Though it could be painful if the crash leaves things remaining tasks depend upon in an odd state… The task tree could be useful: scheduling tasks to start once other tasks are completed.
    - kjkjadksj 3y ago
      You can do this in pure bash I believe. E.g. when you launch a command you can capture its pid then wait for that to exit as you expect before running further jobs.
  - wang_li 3y ago
    >The focus of pueue lies on human interaction. How is it better than me having another window open and running my long running command there?
    - Nukesor 3y ago
      Well, if you're planning on running 10 commands sequentially that might take a few hours each, you would have to keep those windows open for quite a while. See https://github.com/Nukesor/pueue/wiki/FAQ#what-can-i-use-it-for https://github.com/Nukesor/pueue/wiki/FAQ#what-can-i-use-it-...
- Nukesor 3y ago
  This question has been asked quite a lot, so I wrote a FAQ ;D : https://github.com/Nukesor/pueue/wiki/FAQ#why-should-i-use-it https://github.com/Nukesor/pueue/wiki/FAQ#why-should-i-use-i...
west0n 3y ago
If you see following error logs, $ pueue Error: Couldn't find a configuration file. Did you start the daemon yet? Run `pueued -d` first. I think this prompt should be printed on the screen the first time it runs, or automatically executed.
croemer 3y ago
The title is unclear, it should include the tool name
pdimitar 3y ago
I am planning to use this for my homemade system-wide update scripts -- that updates via the OS package manager, Golang tools, Rust tools, OCaml tools, NPM-managed tools, editor configs (NeoVim headless updating of plugins as well), and others. Seems like it has everything I need, the most crucial ones being separate queues / worker pools (f.ex. I only want the tasks that might involve source compilation to be executed one after another on a separate queue, never in parallel). Anybody knows a good alternative to `multitail`? Still haven't found any other tool that can track the output of several commands at the same time on a screen that's automatically split -- and subsequently the finished tasks get removed from view until only the last one remains, taking the entire screen, and then it says "Done, press Q to quit".
- kjkjadksj 3y ago
  You can script up something similar to your multitail example using tmux.
  - pdimitar 3y ago
    Is it going to automatically close when all tasks complete?
    - kjkjadksj 3y ago
      That can be done with ending the command with something like “; exit 0”
bionhoward 3y ago
This looks awesome and the README seems to undersell it. Would it be possible for you to put code examples all over the readme so we can see what you’re talking about without needing to dig into the codebase?
- Nukesor 3y ago
  So, there's a wiki which explains many of the usecases. I specifically didn't want to further bloat the README, as it's already super long as it is.
seabrookmx 3y ago
I've wanted to write something like this for years! For me it's always long running rsync commands I want to chain. I still might write one as it would be a fun way to play around with some low level code, but when I actually want to get things done I'll be checking this out.
- kjkjadksj 3y ago
  Could you just chain them in bash script? You can do it in a dumb way or you can even do it conditionally on the exit status of the previous rsync command.
  - seabrookmx 3y ago
    Yes or I could use the bash `&&` but the issue is I need to know all the commands at the start. I want to be able to come back an hour later and easily add a command to the end of the chain.
    - kjkjadksj 3y ago
      You can do that by abrogating your pid
  - Nukesor 3y ago
    Also, editing a command in the middle if you notice a mistake becomes tricky. Pueue also allows you to do stuff like dependencies, which get tricky in bash if a task depends on more than one tasks finishing.
    - kjkjadksj 3y ago
      If this ability is important to you then you can break your pipeline into individual files that aren’t going to be read until they are executed, giving you time to edit.
      - Nukesor 3y ago
aeblyve 3y ago
I usually just need to run one big interactive job on a server, in which case I just do tmux my-command Ctrl-b d (detach) logout, go to bed, sleep, whatever log back into the server, tmux attach Would there be an advantage to using this over that?
- eddd-ddde 3y ago
  Probably not in practice. You do save that ctrl-b keypress tho.
  - kjkjadksj 3y ago
    Ampersand also works, no tmux needed
    - renewiltord 3y ago
      Does it? I must have always configured differently but sighup. Usually have to disown preventing sighup!
      - ec109685 3y ago
        Prefix with nohup
- Nukesor 3y ago
  Nope. Pueue is designed to handle your usecase times 5+. I'm still using it for your usecase, as I'm already used to the interface by now :D.
- Alifatisk 3y ago
  Your usecase reminds me of GNU screen
  - squigz 3y ago
    They're both terminal multiplexers. In my experience tmux is a much better experience though
andsens 3y ago
> Pueue is considered feature-complete Oh that is sweet sweet music to my ears!
vram22 3y ago
Under "Features": >Pause/resume tasks, when you need some processing power right NOW! How is the pause and resume done?
- dspillett 3y ago
  > How is the pause and resume done? Perhaps by sending SIGSTOP and SIGCONT, much like hitting Ctrl+Z on the console and later running bg <pid> or fg <pid>. Note that this is not the same as Ctrl+S & Ctrl+Q on the console – that just pauses the output display not the process (though the process may subsequently pause if a buffer somewhere down the pipeline becomes full due to the terminal output pausing).
  - vram22 3y ago
    Thanks.
  - Nukesor 3y ago
    Yep, that's exactly how it's done :)
    - dspillett 3y ago
      Thought it would be, but I didn't want to state that more authoritatively as I'd not bothered checking the docs/source. And I'm lazy like that.
whatever1 3y ago
Ah, I wish this was available when I was using gnu parallel.
kjkjadksj 3y ago
What exactly is the is doing that you can’t do with pure bash? You can set up chains of commands in serial or in parallel, use job arrays, establish dependencies or other run conditions, hop into a backgrounded run, etc.
- chasil 3y ago
  Actually, the GNU xargs utility is the most efficient way to perform parallel execution on a large queue of input files. GNU parallel is also an option, but it is written in Perl and has a larger footprint. https://www.linuxjournal.com/content/parallel-shells-xargs-utilize-all-your-cpu-cores-unix-and-windows https://www.linuxjournal.com/content/parallel-shells-xargs-u...
- Nukesor 3y ago
  You can probably achieve a good subset of its functionality in bash, it's just a nicer interface with a lot of configurability and several convenience features. I'm generally a big fan of showing alternatives: https://github.com/Nukesor/pueue/?tab=readme-ov-file#similar-projects https://github.com/Nukesor/pueue/?tab=readme-ov-file#similar... Would you be willing to write a proper guide on how to do all of these things in bash? It would be great to have such a guide inside the Pueue wiki and link to it. It'll help people to make a more informed decision on whether they need this tool or not.
mshockwave 3y ago
A similar tool I highly recommend: https://github.com/justanhduc/task-spooler https://github.com/justanhduc/task-spooler At first I thought it would just be a one-off tool I used for one of my projects, not until I discovered later that it has everything I need and became my daily driver ever since.
op00to 3y ago
I do all this with systemd user services.
veyh 3y ago
Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally. [1] https://nats.io/ https://nats.io/
- Nukesor 3y ago
  Interesting. May I ask how many tasks you were managing with Pueue and what your usecase was? I also thought about using alternative formats such as CBOR, but choosing a human-readable format like JSON made debugging and such a lot easier. If there's a good usecase for it, I might consider switching to a more compact format.
  - veyh 3y ago
    I was running a lot of small tasks. With 3000 queued tasks my state file was around 25 MB. zstd compressed that down to 5% of the size. I have the code still if you want to look at it but it was just a quick experiment so I didn't add any tests. I did add it to the config, disabled by default, though. https://github.com/veyh/pueue/commit/e9dcf52227304b4b4a2ded4caf0fc1ccd2e68605 https://github.com/veyh/pueue/commit/e9dcf52227304b4b4a2ded4... Protobuf could be a pretty good alternative. It can be dumped to a human-readable format with the protoc cli.
hbarka 3y ago
“Pueue is not designed to be a programmable (scriptable) task scheduler/executor.” Any alternatives similar to Pueue with capabilities of above?
- Nukesor 3y ago
  Not that I know of. If anyone knows of stuff like that, please let me know, as this is regularly asked for :D. I would be pretty stoked to be hired to build something like that, though.
martinthought 3y ago
"pueue follow <task_id>" lets you see stdout or stderror of the specified task. If one enqueues a single chain of tasks (no parallel tasks), is there a way to monitor stdout or stderror for the chain, without having to issue the follow command for each task at the time the task starts to run? This would provide better observability of what is running, as in a shell script with the tasks sequentially listed.

95 points by sea-gold 3y ago

This looks quite helpful actually. Thanks for sharing.

elif 3y ago

Actually pretty sweet. `ps | grep` can be tricky to navigate sometimes. I know I've certainly grabbed the wrong process a few times. I'm not exactly sure what advantage this has over managing `screen` sessions tho. Maybe it's cleaner from a process tree perspective?

cl3misch 3y ago
I was about to write that this has a different use case, because managing screen/tmux sessions is very manual. However the repo iself states > Pueue is not designed to be a programmable (scriptable) task scheduler/executor. The focus of pueue lies on human interaction. So I also don't really see its usecase and probably would opt for tmux instead. If you had many workers to run, but still do it manually, you would use pueue? I would be interested in such a scenario!
- dspillett 3y ago
  The concurrency controls are not present using the tmux method. Though if I am running enough tasks for this to be an issue then something odd is happening. The queue being persisted and surviving crashes could be useful, and isn't the case with tmux unless you manually script it up. Though it could be painful if the crash leaves things remaining tasks depend upon in an odd state… The task tree could be useful: scheduling tasks to start once other tasks are completed.
  - kjkjadksj 3y ago
    You can do this in pure bash I believe. E.g. when you launch a command you can capture its pid then wait for that to exit as you expect before running further jobs.
- wang_li 3y ago
  >The focus of pueue lies on human interaction. How is it better than me having another window open and running my long running command there?
  - Nukesor 3y ago
    Well, if you're planning on running 10 commands sequentially that might take a few hours each, you would have to keep those windows open for quite a while. See https://github.com/Nukesor/pueue/wiki/FAQ#what-can-i-use-it-for https://github.com/Nukesor/pueue/wiki/FAQ#what-can-i-use-it-...
Nukesor 3y ago
This question has been asked quite a lot, so I wrote a FAQ ;D : https://github.com/Nukesor/pueue/wiki/FAQ#why-should-i-use-it https://github.com/Nukesor/pueue/wiki/FAQ#why-should-i-use-i...

west0n 3y ago

If you see following error logs, $ pueue Error: Couldn't find a configuration file. Did you start the daemon yet? Run `pueued -d` first. I think this prompt should be printed on the screen the first time it runs, or automatically executed.

croemer 3y ago

The title is unclear, it should include the tool name

pdimitar 3y ago

I am planning to use this for my homemade system-wide update scripts -- that updates via the OS package manager, Golang tools, Rust tools, OCaml tools, NPM-managed tools, editor configs (NeoVim headless updating of plugins as well), and others. Seems like it has everything I need, the most crucial ones being separate queues / worker pools (f.ex. I only want the tasks that might involve source compilation to be executed one after another on a separate queue, never in parallel). Anybody knows a good alternative to `multitail`? Still haven't found any other tool that can track the output of several commands at the same time on a screen that's automatically split -- and subsequently the finished tasks get removed from view until only the last one remains, taking the entire screen, and then it says "Done, press Q to quit".

kjkjadksj 3y ago
You can script up something similar to your multitail example using tmux.
- pdimitar 3y ago
  Is it going to automatically close when all tasks complete?
  - kjkjadksj 3y ago
    That can be done with ending the command with something like “; exit 0”

bionhoward 3y ago

This looks awesome and the README seems to undersell it. Would it be possible for you to put code examples all over the readme so we can see what you’re talking about without needing to dig into the codebase?

Nukesor 3y ago
So, there's a wiki which explains many of the usecases. I specifically didn't want to further bloat the README, as it's already super long as it is.

seabrookmx 3y ago

I've wanted to write something like this for years! For me it's always long running rsync commands I want to chain. I still might write one as it would be a fun way to play around with some low level code, but when I actually want to get things done I'll be checking this out.

kjkjadksj 3y ago
Could you just chain them in bash script? You can do it in a dumb way or you can even do it conditionally on the exit status of the previous rsync command.
- seabrookmx 3y ago
  Yes or I could use the bash `&&` but the issue is I need to know all the commands at the start. I want to be able to come back an hour later and easily add a command to the end of the chain.
  - kjkjadksj 3y ago
    You can do that by abrogating your pid
- Nukesor 3y ago
  Also, editing a command in the middle if you notice a mistake becomes tricky. Pueue also allows you to do stuff like dependencies, which get tricky in bash if a task depends on more than one tasks finishing.
  - kjkjadksj 3y ago
    If this ability is important to you then you can break your pipeline into individual files that aren’t going to be read until they are executed, giving you time to edit.
    - Nukesor 3y ago

aeblyve 3y ago

I usually just need to run one big interactive job on a server, in which case I just do tmux my-command Ctrl-b d (detach) logout, go to bed, sleep, whatever log back into the server, tmux attach Would there be an advantage to using this over that?

eddd-ddde 3y ago
Probably not in practice. You do save that ctrl-b keypress tho.
- kjkjadksj 3y ago
  Ampersand also works, no tmux needed
  - renewiltord 3y ago
    Does it? I must have always configured differently but sighup. Usually have to disown preventing sighup!
    - ec109685 3y ago
      Prefix with nohup
Nukesor 3y ago
Nope. Pueue is designed to handle your usecase times 5+. I'm still using it for your usecase, as I'm already used to the interface by now :D.
Alifatisk 3y ago
Your usecase reminds me of GNU screen
- squigz 3y ago
  They're both terminal multiplexers. In my experience tmux is a much better experience though

andsens 3y ago

> Pueue is considered feature-complete Oh that is sweet sweet music to my ears!

vram22 3y ago

Under "Features": >Pause/resume tasks, when you need some processing power right NOW! How is the pause and resume done?

dspillett 3y ago
> How is the pause and resume done? Perhaps by sending SIGSTOP and SIGCONT, much like hitting Ctrl+Z on the console and later running bg <pid> or fg <pid>. Note that this is not the same as Ctrl+S & Ctrl+Q on the console – that just pauses the output display not the process (though the process may subsequently pause if a buffer somewhere down the pipeline becomes full due to the terminal output pausing).
- vram22 3y ago
  Thanks.
- Nukesor 3y ago
  Yep, that's exactly how it's done :)
  - dspillett 3y ago
    Thought it would be, but I didn't want to state that more authoritatively as I'd not bothered checking the docs/source. And I'm lazy like that.

whatever1 3y ago

Ah, I wish this was available when I was using gnu parallel.

kjkjadksj 3y ago

What exactly is the is doing that you can’t do with pure bash? You can set up chains of commands in serial or in parallel, use job arrays, establish dependencies or other run conditions, hop into a backgrounded run, etc.

chasil 3y ago
Actually, the GNU xargs utility is the most efficient way to perform parallel execution on a large queue of input files. GNU parallel is also an option, but it is written in Perl and has a larger footprint. https://www.linuxjournal.com/content/parallel-shells-xargs-utilize-all-your-cpu-cores-unix-and-windows https://www.linuxjournal.com/content/parallel-shells-xargs-u...
Nukesor 3y ago
You can probably achieve a good subset of its functionality in bash, it's just a nicer interface with a lot of configurability and several convenience features. I'm generally a big fan of showing alternatives: https://github.com/Nukesor/pueue/?tab=readme-ov-file#similar-projects https://github.com/Nukesor/pueue/?tab=readme-ov-file#similar... Would you be willing to write a proper guide on how to do all of these things in bash? It would be great to have such a guide inside the Pueue wiki and link to it. It'll help people to make a more informed decision on whether they need this tool or not.

mshockwave 3y ago

A similar tool I highly recommend: https://github.com/justanhduc/task-spooler https://github.com/justanhduc/task-spooler At first I thought it would just be a one-off tool I used for one of my projects, not until I discovered later that it has everything I need and became my daily driver ever since.

op00to 3y ago

I do all this with systemd user services.

veyh 3y ago

Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally. [1] https://nats.io/ https://nats.io/

Nukesor 3y ago
Interesting. May I ask how many tasks you were managing with Pueue and what your usecase was? I also thought about using alternative formats such as CBOR, but choosing a human-readable format like JSON made debugging and such a lot easier. If there's a good usecase for it, I might consider switching to a more compact format.
- veyh 3y ago
  I was running a lot of small tasks. With 3000 queued tasks my state file was around 25 MB. zstd compressed that down to 5% of the size. I have the code still if you want to look at it but it was just a quick experiment so I didn't add any tests. I did add it to the config, disabled by default, though. https://github.com/veyh/pueue/commit/e9dcf52227304b4b4a2ded4caf0fc1ccd2e68605 https://github.com/veyh/pueue/commit/e9dcf52227304b4b4a2ded4... Protobuf could be a pretty good alternative. It can be dumped to a human-readable format with the protoc cli.

hbarka 3y ago

“Pueue is not designed to be a programmable (scriptable) task scheduler/executor.” Any alternatives similar to Pueue with capabilities of above?

Nukesor 3y ago
Not that I know of. If anyone knows of stuff like that, please let me know, as this is regularly asked for :D. I would be pretty stoked to be hired to build something like that, though.

martinthought 3y ago

"pueue follow <task_id>" lets you see stdout or stderror of the specified task. If one enqueues a single chain of tasks (no parallel tasks), is there a way to monitor stdout or stderror for the chain, without having to issue the follow command for each task at the time the task starts to run? This would provide better observability of what is running, as in a shell script with the tasks sequentially listed.