3 ms·

You could also just send output to separate files by re-opening stdout/stderr and use https://www.vanheusden.com/multitail/index.html https://www.vanheusden.com

by cb321 3y ago

You could also just send output to separate files by re-opening stdout/stderr and use https://www.vanheusden.com/multitail/index.html https://www.vanheusden.com/multitail/index.html (or GNU screen or tmux or whatever) to multiplex a terminal in a more organized way. This also solves an "open problem" in the article of stray prints. If you really want to share one terminal / stdout but also prevent timing-based record splitting, you could also send outputs to FIFOs / named pipes and then have a simple record boundary honoring merge program like https://github.com/c-blake/bu/blob/main/funnel.nim https://github.com/c-blake/bu/blob/main/funnel.nim with doc at https://github.com/c-blake/bu/blob/main/doc/funnel.md https://github.com/c-blake/bu/blob/main/doc/funnel.md As long as "record format" is shared (e.g. newline-terminated) this can also solve the stray print problem.

ryan-duve 3y ago
When you say "multiplex a terminal" with tmux, do you mean splitting screens so the same terminal window has multiple shells prompts in it? I'm trying to understand how that would be used to address the problem in the demo at the bottom of the post.
- cb321 3y ago
  I will try to be more concrete than I was (though I did link to some Nim/shell that "does it all" the 2nd way I mentioned). j.py: import multiprocessing as mp, contextlib as cl, time def f(x): name = mp.current_process()._name with open("o." + name, 'a+') as f: with cl.redirect_stdout(f): print(x * x) time.sleep(2) mp.Pool().map(f, range(36)) Then shell$ python3 j.py & sleep 1; multitail o.* On Unix that should create $(nproc)-panes (i.e. screen areas) of output in your terminal, each pane logging any output each worker process did. (EDIT: For fancier progress, multitail -cT may help; See the man page.) Some Py MP expert could perhaps show how to do this with only one `open` or a nicer `name`. As a bonus - the log/status outputs are not erased but left in "o.Stuff". The cost of that is having log files to clean up. Separate files means no contention at all and any stray prints from any libraries go into the log file. It's a separate exercise for the reader to extend this to stderr capture (either in e.* or also in o.*). And, really, as popular as Python and MP are, all of this is probably written up somewhere. Someone else should outline imitating `multitail o.*` in tmux, I think, as it is more involved, but a pointer to get started is here: https://github.com/tmux/tmux/wiki/Advanced-Use https://github.com/tmux/tmux/wiki/Advanced-Use (That's all I have time for today. Sorry.)

by cb321 3y ago

ryan-duve 3y ago

When you say "multiplex a terminal" with tmux, do you mean splitting screens so the same terminal window has multiple shells prompts in it? I'm trying to understand how that would be used to address the problem in the demo at the bottom of the post.

cb321 3y ago
I will try to be more concrete than I was (though I did link to some Nim/shell that "does it all" the 2nd way I mentioned). j.py: import multiprocessing as mp, contextlib as cl, time def f(x): name = mp.current_process()._name with open("o." + name, 'a+') as f: with cl.redirect_stdout(f): print(x * x) time.sleep(2) mp.Pool().map(f, range(36)) Then shell$ python3 j.py & sleep 1; multitail o.* On Unix that should create $(nproc)-panes (i.e. screen areas) of output in your terminal, each pane logging any output each worker process did. (EDIT: For fancier progress, multitail -cT may help; See the man page.) Some Py MP expert could perhaps show how to do this with only one `open` or a nicer `name`. As a bonus - the log/status outputs are not erased but left in "o.Stuff". The cost of that is having log files to clean up. Separate files means no contention at all and any stray prints from any libraries go into the log file. It's a separate exercise for the reader to extend this to stderr capture (either in e.* or also in o.*). And, really, as popular as Python and MP are, all of this is probably written up somewhere. Someone else should outline imitating `multitail o.*` in tmux, I think, as it is more involved, but a pointer to get started is here: https://github.com/tmux/tmux/wiki/Advanced-Use https://github.com/tmux/tmux/wiki/Advanced-Use (That's all I have time for today. Sorry.)