7 ms·

APT's progress bar does have issues though. When you shrink the terminal when the progress bar is visible, you're going to have gibberish in the terminal. That'

APT's progress bar does have issues though. When you shrink the terminal when the progress bar is visible, you're going to have gibberish in the terminal. That's because the terminal commands to move the cursor by lines work on visual lines, not semantic lines, so they aren't invariant under the operation of resizing the terminal window. That's a really rough area of Linux terminals: when you resize a window, the terminal is going to rewrap the lines according to semantic lines (based on line feed characters), so it's clear that all major terminals understand this distinction, but the terminal API doesn't give applications access to moving by semantic lines, so we're left with bugs like these. I think PowerShell has the concept of progress reporting specially recognized? That sounds like a good idea. It could be used to render a GUI progress bar and leave the pipeline alone, for the objects. That's just my vague idea, since I didn't spend much time with PowerShell.

agolliver 5y ago
PowerShell does have progress bars, but rendering them is (was?) on the critical path. I once had a script that was downloading a large file and took ages to finish, you could speed it up by well over 10x disabling the progress bar. https://github.com/PowerShell/PowerShell/issues/2138 https://github.com/PowerShell/PowerShell/issues/2138
- peakaboo 5y ago
  I have a hard time believing real programmers would allow something like that in production code... But I guess if its corporate, nobody cares about the quality.
  - oneweekwonder 5y ago
    npm was about 6 years old and im pretty sure a lot of corporates have adopted node by then. I don't think it has so much to do with quality but rather we like to re-invent shiny "new" wheels and then the new tooling must iron out all the corner cases. https://github.com/npm/npm/issues/11283 https://github.com/npm/npm/issues/11283
  - jrockway 5y ago
    Outputting text to a terminal is synchronous. Try connecting your favorite program's STDOUT to a program that doesn't read its STDIN. The program will stop running after the output buffer fills, because "write" blocks. Obviously you can buffer internally, and have a "UI thread" for writing stdout... but literally nobody does this. This is why people try to write the fastest terminal emulator -- a slow terminal emulator slows down apps. (And, it all makes sense. You have to apply backpressure when you are receiving data more quickly than you can process it. You can buffer to smooth over spikes, or you could drop data on the floor, but both of those would suck for output in a terminal!)
    - Sharlin 5y ago
      stdout is by default buffered in almost every language. Outputting lots of non-critical stuff to stderr, or manually flushing stdout on every write, is definitely a rookie mistake. Raw terminal output should not be a big performance sink either if you do the sane thing and only write on every 100th or 1000th or whatever iteration of your main loop. No threads needed.
      - rossy 5y ago
        Things weren't so nice in Windows before Microsoft added VT control sequences to Windows 10. Before then, if you wanted to draw fancy stuff in a console window, you had to call a synchronous out-of-band IPC API to conhost.exe. I don't know how PowerShell did it specifically, but you'd need to make the same sequence of slow IPC calls to draw the progress bar even if you did buffer them.
      - jrockway 5y ago
        It's buffered, but buffers fill up. I wrote a small program [1] that calls os.Stdout.Write to write one byte, increments a counter by the number of bytes written, and records the time of the last write. Another thread then prints the byte count and the time since the last time Write() returned every second. Running "that program | sleep infinity" yields a buffer size of 65536 and writes stop returning after the first millisecond or so of program runtime. And that makes sense; nothing is reading the output of the program, and it is written to produce an infinite stream of bytes. There is no buffer you can allocate that stores an infinite number of bytes, so it has to block or abort. [1] https://gist.github.com/jrockway/a5d96151e1c69407f491988df708733b https://gist.github.com/jrockway/a5d96151e1c69407f491988df70... Going back to the original context of the comment, we're calling some programmer at Microsoft an amateur because their progress bar blocks progress of the application. And indeed, that design could be improved (sample the progress whenever the terminal can accept however many bytes the progress bar takes to render), but it's a very common mistake. Any program that calls printf will eventually run into this problem. A fixed-length buffer smooths over quirks, but if your terminal can render 1 million characters per second and your program produces 1 million and 1 characters per second of log output, then there is no way your program can ever complete. You don't notice because programs don't output much information, terminals are fast, and 65536 is a good default. But in theory, your program suffers from the same bug as Microsoft's program. So it's pretty unfair to call them amateurs, unless the Linux kernel, GNU coreutils, etc. are all also written by amateur programmers. What the grandparent meant to say is "I've never noticed having made this mistake before" and nothing more. This is something that programmers need to think about pretty much every time they do IO. Buffer, drop, or slow down. printf picks "buffer, then slow down", writing to syslog over UDP picks "drop", but what you really want depends on your application, and it is something that you have to explicitly think about.
  - blowski 5y ago
    What does a programmer that's not a "real programmer" do? Only write code in films? Slap the keyboard on hackertyper.net?
    - aflag 5y ago
      I imagine the argument would be that the person that writes code at a corporation is not the same person who writes it for hobby. Even if they share the same body, one is just doing it for the cash and the other for fun.
- QuercusMax 5y ago
  Are they synchronizing on the UI every time they update the progress bar? That's a rookie mistake.
kccqzy 5y ago
One solution is to install a signal handler for SIGWINCH and redraw everything. In practice this works best for terminal apps that already make use of the entire screen (say vim or emacs) but not so well for mixed text and some visual gadgets.
- yakubin 5y ago
  APT already has a signal handler for SIGWINCH, in order to resize the progress bar. The problem is that after APT finds out about the resize it clears the last line on the screen, but it's a visual line, which is only a part of a semantic line which was wrapped after shrinking the terminal window. APT doesn't know how many visual lines the progress bar takes after being rewrapped by a resize, so it doesn't know how many lines to clear. It could try to compensate by remembering the old width in characters, looking at the new one and calculating what rewrapping would result in, but that would probably not be very reliable across different terminals. Redrawing everything is annoying from UX point of view, because it breaks copying lines out of the terminal. Instead of getting a bunch of lines separated by line feeds corresponding to semantic lines, the user gets a bunch of lines corresponding to the visual lines in the terminal (so a lot of lines broken, some of them midword), including loads of meaningless whitespace. I wish the modern terminal was less of an old terminal emulator and more of a GUI for displaying pipelines. So no ncurses-like applications, such as vim, more first-class support for progress bars (instead of the hacks that we have), support for jumping between the lines in scrollback where the user typed in the commands, no possibility of breaking the terminal state (such as what happens when I hit Ctrl+C in the password prompt of openvpn) forcing the user to run "reset", etc. etc.
vlunkr 5y ago
IMO this isn't a big enough issue to worry about. Most progress bars break when you resize the terminal, I just expect it at this point.
chmsky00 5y ago
The problem you’re highlighting is a simple design issue. My problem with progress bars is they are usually nonsense estimates based upon random programmer choices. I know former MSers who worked on progress bars for old Windows versions. Specifically I remember them saying the progress bar for copying files to USB was estimating how long until a USB hardware buffer was clear, without any idea how much more data there was intended to be jammed into the buffer once it cleared. Buffer would immediately refill and increase the wait time. Given how slowly Windows deprecates code, who knows what subroutines any given progress bar is relying on. They’re nonsense features meant to soothe users who seek progress. They don’t need to be interesting visually. I’d accept a simple countdown that made sense. I’m hoping manufacture of application specific chips takes off. That we embed a 3D engine into silicon and this “software engineering is life” mind virus can go away. We simply did not do it that way before because we lacked the manufacturing capabilities. If manufacturing hopes and dreams of colleagues in the chip biz come to fruition, software is on its way out as a routine part of developing new technology. But I mean they’re biased; attention on hardware versus software makes them more valuable.
- bmn__ 5y ago
  > I know former MSers who worked on progress bars for old Windows versions. https://explainxkcd.com/612 https://explainxkcd.com/612
  - chmsky00 5y ago
    Yep. That was around the time I worked with those folks. I would not be surprised if the topic came up because of that comic.
- spookthesunset 5y ago
  If we are gonna bitch about progress bars, Microsoft’s are almost always the worst. So many of them get to 99% and them stall out… dunno how they get their progress bars so bad.
  - WorldMaker 5y ago
ljm 5y ago
I hate this in homebrew and macports: they use a progress line that mostly works, until your terminal is too wide or too narrow. # ## ### #### ###### ####### ######### ############ ,,, ##################################################### ######################################################### ############################################################### ##########################################################################################################

agolliver 5y ago

PowerShell does have progress bars, but rendering them is (was?) on the critical path. I once had a script that was downloading a large file and took ages to finish, you could speed it up by well over 10x disabling the progress bar. https://github.com/PowerShell/PowerShell/issues/2138 https://github.com/PowerShell/PowerShell/issues/2138

peakaboo 5y ago
I have a hard time believing real programmers would allow something like that in production code... But I guess if its corporate, nobody cares about the quality.
- oneweekwonder 5y ago
  npm was about 6 years old and im pretty sure a lot of corporates have adopted node by then. I don't think it has so much to do with quality but rather we like to re-invent shiny "new" wheels and then the new tooling must iron out all the corner cases. https://github.com/npm/npm/issues/11283 https://github.com/npm/npm/issues/11283
- jrockway 5y ago
  Outputting text to a terminal is synchronous. Try connecting your favorite program's STDOUT to a program that doesn't read its STDIN. The program will stop running after the output buffer fills, because "write" blocks. Obviously you can buffer internally, and have a "UI thread" for writing stdout... but literally nobody does this. This is why people try to write the fastest terminal emulator -- a slow terminal emulator slows down apps. (And, it all makes sense. You have to apply backpressure when you are receiving data more quickly than you can process it. You can buffer to smooth over spikes, or you could drop data on the floor, but both of those would suck for output in a terminal!)
  - Sharlin 5y ago
    stdout is by default buffered in almost every language. Outputting lots of non-critical stuff to stderr, or manually flushing stdout on every write, is definitely a rookie mistake. Raw terminal output should not be a big performance sink either if you do the sane thing and only write on every 100th or 1000th or whatever iteration of your main loop. No threads needed.
    - rossy 5y ago
      Things weren't so nice in Windows before Microsoft added VT control sequences to Windows 10. Before then, if you wanted to draw fancy stuff in a console window, you had to call a synchronous out-of-band IPC API to conhost.exe. I don't know how PowerShell did it specifically, but you'd need to make the same sequence of slow IPC calls to draw the progress bar even if you did buffer them.
    - jrockway 5y ago
      It's buffered, but buffers fill up. I wrote a small program [1] that calls os.Stdout.Write to write one byte, increments a counter by the number of bytes written, and records the time of the last write. Another thread then prints the byte count and the time since the last time Write() returned every second. Running "that program | sleep infinity" yields a buffer size of 65536 and writes stop returning after the first millisecond or so of program runtime. And that makes sense; nothing is reading the output of the program, and it is written to produce an infinite stream of bytes. There is no buffer you can allocate that stores an infinite number of bytes, so it has to block or abort. [1] https://gist.github.com/jrockway/a5d96151e1c69407f491988df708733b https://gist.github.com/jrockway/a5d96151e1c69407f491988df70... Going back to the original context of the comment, we're calling some programmer at Microsoft an amateur because their progress bar blocks progress of the application. And indeed, that design could be improved (sample the progress whenever the terminal can accept however many bytes the progress bar takes to render), but it's a very common mistake. Any program that calls printf will eventually run into this problem. A fixed-length buffer smooths over quirks, but if your terminal can render 1 million characters per second and your program produces 1 million and 1 characters per second of log output, then there is no way your program can ever complete. You don't notice because programs don't output much information, terminals are fast, and 65536 is a good default. But in theory, your program suffers from the same bug as Microsoft's program. So it's pretty unfair to call them amateurs, unless the Linux kernel, GNU coreutils, etc. are all also written by amateur programmers. What the grandparent meant to say is "I've never noticed having made this mistake before" and nothing more. This is something that programmers need to think about pretty much every time they do IO. Buffer, drop, or slow down. printf picks "buffer, then slow down", writing to syslog over UDP picks "drop", but what you really want depends on your application, and it is something that you have to explicitly think about.
- blowski 5y ago
  What does a programmer that's not a "real programmer" do? Only write code in films? Slap the keyboard on hackertyper.net?
  - aflag 5y ago
    I imagine the argument would be that the person that writes code at a corporation is not the same person who writes it for hobby. Even if they share the same body, one is just doing it for the cash and the other for fun.
QuercusMax 5y ago
Are they synchronizing on the UI every time they update the progress bar? That's a rookie mistake.

kccqzy 5y ago

One solution is to install a signal handler for SIGWINCH and redraw everything. In practice this works best for terminal apps that already make use of the entire screen (say vim or emacs) but not so well for mixed text and some visual gadgets.

yakubin 5y ago
APT already has a signal handler for SIGWINCH, in order to resize the progress bar. The problem is that after APT finds out about the resize it clears the last line on the screen, but it's a visual line, which is only a part of a semantic line which was wrapped after shrinking the terminal window. APT doesn't know how many visual lines the progress bar takes after being rewrapped by a resize, so it doesn't know how many lines to clear. It could try to compensate by remembering the old width in characters, looking at the new one and calculating what rewrapping would result in, but that would probably not be very reliable across different terminals. Redrawing everything is annoying from UX point of view, because it breaks copying lines out of the terminal. Instead of getting a bunch of lines separated by line feeds corresponding to semantic lines, the user gets a bunch of lines corresponding to the visual lines in the terminal (so a lot of lines broken, some of them midword), including loads of meaningless whitespace. I wish the modern terminal was less of an old terminal emulator and more of a GUI for displaying pipelines. So no ncurses-like applications, such as vim, more first-class support for progress bars (instead of the hacks that we have), support for jumping between the lines in scrollback where the user typed in the commands, no possibility of breaking the terminal state (such as what happens when I hit Ctrl+C in the password prompt of openvpn) forcing the user to run "reset", etc. etc.

vlunkr 5y ago

IMO this isn't a big enough issue to worry about. Most progress bars break when you resize the terminal, I just expect it at this point.

chmsky00 5y ago

The problem you’re highlighting is a simple design issue. My problem with progress bars is they are usually nonsense estimates based upon random programmer choices. I know former MSers who worked on progress bars for old Windows versions. Specifically I remember them saying the progress bar for copying files to USB was estimating how long until a USB hardware buffer was clear, without any idea how much more data there was intended to be jammed into the buffer once it cleared. Buffer would immediately refill and increase the wait time. Given how slowly Windows deprecates code, who knows what subroutines any given progress bar is relying on. They’re nonsense features meant to soothe users who seek progress. They don’t need to be interesting visually. I’d accept a simple countdown that made sense. I’m hoping manufacture of application specific chips takes off. That we embed a 3D engine into silicon and this “software engineering is life” mind virus can go away. We simply did not do it that way before because we lacked the manufacturing capabilities. If manufacturing hopes and dreams of colleagues in the chip biz come to fruition, software is on its way out as a routine part of developing new technology. But I mean they’re biased; attention on hardware versus software makes them more valuable.

bmn__ 5y ago
> I know former MSers who worked on progress bars for old Windows versions. https://explainxkcd.com/612 https://explainxkcd.com/612
- chmsky00 5y ago
  Yep. That was around the time I worked with those folks. I would not be surprised if the topic came up because of that comic.
spookthesunset 5y ago
If we are gonna bitch about progress bars, Microsoft’s are almost always the worst. So many of them get to 99% and them stall out… dunno how they get their progress bars so bad.
- WorldMaker 5y ago

ljm 5y ago

I hate this in homebrew and macports: they use a progress line that mostly works, until your terminal is too wide or too narrow. # ## ### #### ###### ####### ######### ############ ,,, ##################################################### ######################################################### ############################################################### ##########################################################################################################