35 ms·
How does apt render its fancy progress bar?
- shmerl 5y agoA lot of terminals support Unicode, so fancy can be actually fancy in such cases using block characters that produce smooth progress animation. I.e. using those from Left one eighth block to Full block. https://en.wikipedia.org/wiki/Block_Elements https://en.wikipedia.org/wiki/Block_Elements I once added such progress bar for lgogdownloader.
- fladd 5y agoNice. I also looked into this a while ago and came up with a very simplistic (few lines) Python implementation that does this and even allows for some customization: https://gist.github.com/fladd/36c422f1c0e9bf02f41f9fad19609d06 https://gist.github.com/fladd/36c422f1c0e9bf02f41f9fad19609d...
- miohtama 5y agoFor building rich terminal UIs in Python, I recommend Rich the library https://github.com/willmcgugan/rich https://github.com/willmcgugan/rich
- julienpalard 5y agoThere's https://tqdm.github.io/ https://tqdm.github.io/ to do nice progress bars in Python. But my goal was not really the progress bars, but the placing of it below the logs.
- stavros 5y agoYeah, when I was reading the article I started out thinking "What? It's just simply rewinding the line", and then you mentioned that it's always at the bottom of the window, and I thought "Now that is a good trick, I wonder how it's done". And then I didn't wonder. Good work.
- zorr 5y agoIt's good to know the underlying low-level terminal commands that make this work like this article explains. But I guess most people will use a terminal abstraction library that has a higher level API to move the cursor, clean the screen etc. Terminals are complex beasts and they can do powerful things. If you're interested in this, I can recommend reading the source code for Tmux, JLine3 and some other libraries shell formatting or TUI libraries.
- pantulis 5y agotqdm in Python was also very nice and easy to use.
- franga2000 5y ago+1 for tqdm - not only does it take less than one line to "automatically" turn any loop into a progress bar, it also supports native progress bars in Jupyter Notebooks and even a basic GUI one
- formerly_proven 5y agoTerminal abstraction libraries feel like a relict of the past to me, because there used to be a lot of diversity in command sets and capabilities. Nowadays you can pretty much just assume that most commands work everywhere, so there is a lot less reason to drag around ncurses and friends if you want to make a progress bar that doesn't break your output or have some colored text.
- zorr 5y agoNcurses and friends could be overkill but there is still value in small wrapper libraries that provide a higher level API on top of escape codes. `setCursor(lines, 0)` is more readable than `write(f"\033[{lines};0f")`
- deleted 5y ago[deleted]
- 5y ago
- gildas 5y agoFor those who wonder how these "\033[1A..." notations are specified, look up "ANSI escape code" on the net, e.g. https://en.wikipedia.org/wiki/ANSI_escape_code https://en.wikipedia.org/wiki/ANSI_escape_code
- p-e-w 5y agoAnd if you want to play around with escape sequences from your shell, you can do so using `echo -e "\e[1A..."` in most shells (the `-e` flag enables backslash interpretation, while `\e` stands for the ESC character, alias `\033`). Note also that (traditional) escape sequences cannot execute arbitrary code, so you can't really break anything that closing and reopening the terminal won't fix.
- quotemstr 5y agoYou can be sloppy and hardcode escape sequences in your shell script, or you do it right and use the tput program to look up the right sequences at runtime.
- p-e-w 5y agoANSI escape sequences are a standard. Relying on that standard isn't sloppy, any more than relying on any other standard. Terminfo is today basically a hack to work around the problem that some uncommon sequences differ between terminals, but others (like cursor movement and SGR, which cover 99% of use cases) have been well-established for 30 years. I much prefer using such sequences directly over the complexity of shelling out to an external program. I'm not going to pretend that there is a chance my script might have to run on some obscure text terminal from the 80s some day.
- quotemstr 5y agoOpenGL is a standard too, but an OpenGL program will break on a system supporting only Vulkan. Terminfo exists for a reason and that reason is still there today. For example, I can run under TERM=dumb to disable fancy output for various reasons. No, terminfo isn't a hack, ANSI escapes aren't ubiquitous, and every single time I see someone hardcode escape sequences in some script, my opinion of that person declines.
- WayToDoor 5y agoNitpick about the website, but please don't use PGP short fingerprints like 0x46EBCD72F08E6717. They can be brute forced [0] very fast to get a different private key with the same fingerprint. You can also use them to generate keys with vanity fingerprints, which is pretty funny [1]. [0] https://github.com/lachesis/scallion https://github.com/lachesis/scallion [1] I have 0xDE4444AAAAAAAAAA, took me 5 minutes of bruteforcing on a laptop.
- deleted 5y ago[deleted]
- rhn_mk1 5y agoWhat fingerprints should be used?
- ninjin 5y agoAccording to this guide (outdated though, but this still applies) [1], `--with-fingerprint`, as even the long ID can be spoofed. [1]: https://riseup.net/en/security/message-security/openpgp/best-practices#dont-rely-on-the-key-id https://riseup.net/en/security/message-security/openpgp/best...
- jwilk 5y agoHow come? The fastest GPU in scallion's README has hash rate 11646 MH/s. To brute-force a 64-bit key ID at this rate you would need, on average, 50 GPU-years.
- arsome 5y ago50 GPU-years is fairly cheap, certainly within the realm of days for a 3 letter agency or a botnet.
- someonewhocar3s 5y agoKind of pushing it if you want to use it for regular cases though! Funny to think in GPU-years given more's law. Payloads like this (stochastic) have an expected time, not an actual time. It's pretty rare that you can buy 'enough processing power', usually have to go through figuring out and perhaps adjusting the payload to the computational platform yourself. (computing market is not per ops, it's per actual minute of computer time - rare to see it offered with 'compute this before')
- Snetry 5y agoNeat. This is one of those things I always wondered about but never cared enough to actually go and look into myself
- julienpalard 5y agoYou can't imagine how many times I wondered about it before testing it and writing the article ;)
- croes 5y agoShouldn't the title be without question mark or "How does apt render it fancy progress bar?"?
- bluedays 5y agoYeah except you dropped an s, so maybe we shouldn't be so critical.
- christophilus 5y agoYes.
- skrebbel 5y agoNot everybody is a native English speaker.
- bregma 5y agoIncluding many native English speakers.
- rob74 5y agoI'm not a native English speaker myself, but the phrasing of questions in English is one of the most basic things, English 101 if you will, so if you consider your English good enough to write a blog post, you should be capable of doing that...
- julienpalard 5y agoI'm french ;)
- deleted 5y ago[deleted]
- goohle 5y agoIt's "Newspaper Headlines English" https://www.thoughtco.com/understanding-newspaper-headlines-1212013 https://www.thoughtco.com/understanding-newspaper-headlines-...
- 5y ago
- miki123211 5y agoThis kind of fancy terminal manipulation is usually an accessibility nightmare. Real GUI frameworks usually provide semantic information to the OS, which allows screen readers to determine that a given control is a progress bar that is 50% filled, and then render that information by playing an appropriately-pitched tone. In a TUI, all they get is a line of fancy symbols changing quickly, with no context as to what those symbols mean. Same for i.e. ncurses based menus. When programs use colors to signify which menu option is selected, screen readers usually get lost. Screen readers specifically designed for text-based systems (such as DOS) usually had ways to work around this issue, but modern, GUI-focused ones don't really offer those options any more.
- darkwater 5y agoIn apt at least the fancy UI can be disabled completely, see 'Dpkg::Progress-Fancy "0";'
- d_watt 5y agoOut of curiosity, when you have a large amount of text flying by like with package installs, what is the expected accessibility behavior? I imagine a screenreader trying to keep up with the install logs in any way would be... chaotic.
- miki123211 5y agoIt tries to read everything, but you can flush it with control. If you notice it stopped speaking, you usually use some review commands to look at the last few lines. If you're looking for something specific, you'd usually use something like grep, even in situations where a sighted person could just glance at the screen.
- newswasboring 5y agomight sound a bit insensitive. But is there a video I can see of this? I cant even imagine how this would look in the hands of an expert.
- brainbag 5y agoGood explanation. It is a nice progress bar, but not particularly unique; many command line tools use progress bars like this. I've always been curious how the Heroku CLI renders its progress bar, which is unique as far as I've seen. It does not seem to use characters, it's seemingly a smooth, pixel-based fill like you'd get in a non-terminal app. Downloading a pg backup is the most common place I see it.
- julienpalard 5y agoIn Python you can use tqdm for this: python -c 'import tqdm; i=[j for j in tqdm.trange(int(1e8))]'
- kqr 5y agoThere are characters that are boxes of varying width, but maybe you've confirmed it's not even that?
- timdorr 5y agoThey use this package under the hood: https://github.com/jdxcode/smooth-progress https://github.com/jdxcode/smooth-progress It uses glyphs of boxes that have varying widths. When joined together, they appear as a solid bar.
- deleted 5y ago[deleted]
- agumonkey 5y agoterminal rendering is fun. it's often limited but the few it offers gets you going nicely.
- quotemstr 5y agoI hate it when programs hardcore control sequences like this. Just look them up at runtime with terminfo --- it's really not that hard.
- julienpalard 5y agoHow would you do the DECSTBM without hardocindg it? Honest question.
- deleted 5y ago[deleted]
- yakubin 5y agoAPT's progress bar does have issues though. When you shrink the terminal when the progress bar is visible, you're going to have gibberish in the terminal. That's because the terminal commands to move the cursor by lines work on visual lines, not semantic lines, so they aren't invariant under the operation of resizing the terminal window. That's a really rough area of Linux terminals: when you resize a window, the terminal is going to rewrap the lines according to semantic lines (based on line feed characters), so it's clear that all major terminals understand this distinction, but the terminal API doesn't give applications access to moving by semantic lines, so we're left with bugs like these. I think PowerShell has the concept of progress reporting specially recognized? That sounds like a good idea. It could be used to render a GUI progress bar and leave the pipeline alone, for the objects. That's just my vague idea, since I didn't spend much time with PowerShell.
- agolliver 5y agoPowerShell does have progress bars, but rendering them is (was?) on the critical path. I once had a script that was downloading a large file and took ages to finish, you could speed it up by well over 10x disabling the progress bar. https://github.com/PowerShell/PowerShell/issues/2138 https://github.com/PowerShell/PowerShell/issues/2138
- peakaboo 5y agoI have a hard time believing real programmers would allow something like that in production code... But I guess if its corporate, nobody cares about the quality.
- oneweekwonder 5y agonpm was about 6 years old and im pretty sure a lot of corporates have adopted node by then. I don't think it has so much to do with quality but rather we like to re-invent shiny "new" wheels and then the new tooling must iron out all the corner cases. https://github.com/npm/npm/issues/11283 https://github.com/npm/npm/issues/11283
- 5y ago
- SavantIdiot 5y agoI wouldn't call it fancy: I can't change the color so on light backgrounds I can't read it. And I've seen certain error messages slice it in two. I'd call it annoying.
- jancsika 5y agoI'd prefer a "Double Debian" distro that does the following: 1. When I run an apt command, it goes ahead and runs an automated version of the command on a "shadow" apt on the same machine. 2. It measures the time it took to do that operation on the shadow apt. 3. By the time I respond "y" to the command, it uses the duration from step 2 to display a time-based progress bar that animates smoothly over said duration. 4. If the shadow apt command hasn't finished by the time I respond "y" to the command, it brings up Ms. Pacman for me to play for a bit. I bet I'd be a much more satisfied user!
- someonewhocar3s 5y ago...You'd prefer your terminal to terribly front-run potentially huge operations... .. to get a single sample estimate .. ....because you expect the loading bars to be displayed to last on the order of milliseconds....
- someonewhocar3s 5y agoCrazy idea for apt: Concurrency Honestly tears to the fact that it doesn't multi-source and many-connection. There's something to be said about "it doesn't improve server specs" or maybe even "I don't like the idea of having too many connections" - it's just known it does speed stuff along. Basically making a lot of people sit and wait.
- buserror 5y agoI use a similar approach in libmish [0], ie I "reserve" the bottom 2 rows, and output the scrolling text in the top parts. Pretty straightforward. [0]: https://github.com/buserror/libmish https://github.com/buserror/libmish
- prajwalshetty 5y agoI implemented something like this in bash. I called this status bar sort of implementation as "message bar" to show the ongoing status of the script on a reserved lines on bottom of the screen - I had to implement progress bar one for each task inline (and not on the bottom bar) - and constantly had to switch "states" to save/restore the cursor positions on screen. This message bar would also become a prompt and take in user inputs on certain events apart from being a static status bar - super useful for the use case. Cursor positions, scrolling screens, events, progress bar updates, managing child sessions, handling screen resolution changes, taking user inputs...everything had to be catered for in this. I thoroughly enjoyed developing this and still think it was a bit over the board for what I was developing at the time but I was sent on this path of constant learning so couldn't look back - was worth it. Here's the project: https://github.com/PrajwalSD/runSAS https://github.com/PrajwalSD/runSAS
- mike_hock 5y ago> std::cout << ... << std::to_string(nr_rows - 1) > to_string What if I told you, iostream can output numbers?