32 ms·
Tqdm (Python)
- exizt88 5y agoHey, I know this is a bit off-topic, but I can't wrap my mind around one thing. I'm getting a ton of Google ads on that website. Where do they come from? Is Github adding them? Did the devs add them to their docs?
- pronoiac 5y agoI'd be stunned if Github added them, so I checked to verify, and the devs added them; I saw four matches for "adsbygoogle" in https://github.com/tqdm/tqdm.github.io/blob/gh-pages/index.html#L6 https://github.com/tqdm/tqdm.github.io/blob/gh-pages/index.h...
- nurettin 5y agoI like how tqdm can manage multiple progressbars in the same terminal. I have used it to track the progress of multi day processes. It also produces a nice animation on jupyter notebooks.
- atum47 5y agoLove tqdm, I used to use it on all my python scripts.
- perrygeo 5y agoCheck out Will Chrichton's talk "Type-Driven API Design in Rust" where he live-codes a tqdm-like progress bar for Rust iterators. Solid presentation and eye-opening how straightforward it was to extend the core language through traits.
- sgeisenh 5y agoCame here to comment the same thing. Really awesome to see how the type system can be leveraged to produce syntactically lightweight abstractions.
- djstein 5y agothis looks like a fantastic tool! would love to use this in some Python CLI's to help users understand the progress
- minimaxir 5y agotqdm is one of the very few Python packages that makes it into every script I write. It's a very high ROI for managing and tracking simple loops. My only complaint is the smoothing parameter; by default it predicts the estimated time remaining based on the most recent updates so it can fluctuate wildly; smoothing=0 predicts based on the total runtime which makes more sense given law of large numbers.
- winstonewert 5y agowell, it seems to me that its quite likely that speed varies across the process, so the more recent updates are probably a more useful default, so if your first part of the process is slower/faster this won't permanently mess up the estimates.
- bigdict 5y agoYes, assuming i.i.d. samples. If the first batch is "warmup samples", this goes out the window.
- minimaxir 5y agoEven if it's not i.i.d, a small amount of samples with much higher/lower times will just average out as noise.
- akdor1154 5y agoThe older I get, and the more i come across these issues, the more sympathy i have for the Windows file copy progress dialog...
- geenew 5y agoYet the estimate of when you'll become fully sympathetic to the Windows file copy dialog changes wildly by the day, right :)
- dwater 5y agotqdm & click make a nice CLI pairing for taking a simple idea and turning into a reusable tool that you can feel good about sharing in minutes.
- geenew 5y agoI wasn't aware of Click, it looks promising. For reference, it's a backronym for 'Command Line Interface Creation Kit': https://click.palletsprojects.com/en/8.0.x/ https://click.palletsprojects.com/en/8.0.x/
- nxpnsv 5y agoClick is really good, typer is great...
- abdusco 5y agoI love Typer, it's my go-to tool for building CLIs, but I'm worried about its future. It's developed by a single developer, there are too many issues left unattended, no development for months, lacking a good API to extend it or to interact with Click.
- deleted 5y ago[deleted]
- Spivak 5y agoI have such mixed feelings about Click; I've reached the point of using it so much that I now hate it. Which I think probably means that it's great but only as as stopgap until you learn how to bend argparse to your will. It's just too magic, too much global state, and too much spooky action at a distance.
- globular-toast 5y agoI went the opposite way. Used to do everything with argparse, then discovered click and never looked back. While argparse lets you do anything, click forces you to build a good CLI.
- curiousgal 5y agoFun fact, tqdm has an official (pre-alpha) C++ port! https://github.com/tqdm/tqdm.cpp https://github.com/tqdm/tqdm.cpp
- 1-6 5y agotqdm also works on Colab (Jupyter Notebooks) yay
- minimaxir 5y agoTo clarify, the tqdm.notebook progress bar (which is much prettier) works on Colab. If you want to make the script more agnostic, you can import tqdm.auto. A fun quirk is that tqdm.notebook didn't work with Colab's dark mode so the text was readable; this was very recently fixed.
- MapleWalnut 5y agoI wish there was something this good for Golang.
- dud3z 5y agoWhile not having all the tqdm features, i find mpb to be quite good actually. https://github.com/vbauerster/mpb https://github.com/vbauerster/mpb
- BiteCode_dev 5y agoI really love the wave of new tools that are gaining traction in the Python world: tqdm, rich (and soon textual), fastpi, typer, pydantic, shiv, toga, doit, diskcache... With the better error messages in 3.10 and 11, plus the focus on speed, it's a fantastic era for the language and it's a ton of fun. I didn't expect to find back the feeling of "too-good-to-be-true" I had when starting with 2.4, and yet. In fact, being a dev is kinda awesome right now, no matter where you look. JS and PHP are getting more ergonomics, you get a load of new low level languages to sharpen your hardware, Java is modern now, C# runs on Unix, the Rust and Go communities are busy with shipping fantastic tools (ripgrep, fdfind, docker, cue, etc), windows has a decent terminal and WSL, my mother is actually using Linux and Apple came up with the M1. IDEs and browsers are incredible, and they do eat a lot of space, but I have a 32 Go RAM + 1TO SSD laptop that's as slim as a sheet of paper. Not to mention there is a lot of money to be made. I know it's trendy right now to say everything is bad in IT, but I disagree, overall, we have it SO good.
- ropable 5y agoI'll add BeeWare (https://beeware.org/ https://beeware.org/) as a pretty nice nice to deploy Python applications cross-platform easily. It's more like a suite of tools, though.
- trutannus 5y agoTQDM isn't all that new, it's one of the first things I started to play with when learning to write code. That said, it is an amazing tool.
- dheera 5y agoYeah, I've been using it for a while, but with the barrage of memes and politics on the internet nobody finds out about the good stuff until years later.
- BiteCode_dev 5y agotqdm is indeed 8 years old, but it's been popping up everywhere for only a couple of years now. Hence the "gaining traction".
- teddyh 5y agoSee also the shell command pv(1).
- edeion 5y agoIndeed. This comes to mind right away. Question is: are we missing something? Of course, tqdm brings this functionality right into python REPL, which is new and looks great. pv and tqdm would look even better if they'd be called implicitly (with an opt-out) since I always end up regretting not using pv when my command is taking too long. Too late.
- drexlspivey 5y agoIt has no dependencies which is pretty cool
- debo_ 5y agoI just had the pleasant experience of importing tqdm into a script, checking HN, and seeing tqdm here :)
- ya3r 5y agotqdm is really great. (Also great name BTW) It has a lot of features like cli, jupter notebook widget, flexibility, etc. I have been using it for 4-5 years now.
- jasonpeacock 5y agoMight as well toss out Plumbum: https://plumbum.readthedocs.io/en/latest/ https://plumbum.readthedocs.io/en/latest/ No more bash! No more subprocess! Write shell-like code with the convenience of Python! Seriously, it's a great module. A bit of a learning curve, but then it feels natural.
- emehex 5y ago`from tqdm import tqdm; [i for i in tqdm(range(10))]` ...bit of a learning curve?
- seertaak 5y agoI think what OP is referring to is how fiddly it is to run a subprocess and capture the output. By the way, also super annoying in Java - you have to faff about with a thread (and you can't even really do that in python, which as much as I love the language, is quite pathetic) that drains stdout before it gets jettisoned by the OS, and in both cases long story short is that it's easy to write something that works when the output is short, but getting it to work on long output is an exercise in frustration. One neatly solved by plumbum, and so I'm definitely with OP it's use makes code measurably less shit.
- mrbonner 5y agowow, didn't know this existed. I've been using ipython as shell-ish replacement for not-so-serious-thing. I need to take a detail look at this from my initial glance over. thanks
- seertaak 5y agoPlumbum is great but I actually use the bang notation in Jupyter even more. Actually, for me Jupyter (I use lab but I'm sure NB is amazing too) is the tool that boosts my productivity the most. (And omg vim mode.)
- andrewshadura 5y agoAlso sh.
- mark_l_watson 5y agoI thought the name was familiar: I use it with TensorFlow. Nice project!
- mrfusion 5y agoHow would it know how long a loop is ahead of time? Is this solving the halting problem?
- nostoc 5y agoHave you ever used a for loop? You don't know how long it'll take, but you know how many iterations, and at any given point you can know how long past iterations have taken. Extrapolation is not hard, even if not always accurate.
- cowsandmilk 5y agoThere are plenty of for loops in which you don’t know how many iterations you will have. With a generator, it can even be infinite.
- berkes 5y agoIn which case the visualisation will adjust and show this fact. E.g. passing an unbounded range will jus show some info about the past iterations.
- formerly_proven 5y agoThis is not how for loops work in Python, which does not have ranged loops. The example works because range objects implement __len__ which is where tqdm will get it's total/count information from.
- wedn3sday 5y agoFor things with a fixed length, it looks it up ahead of time (i.e. things that return things when you call len() on them). For other things such as generators, you can supply a total=N value when you create the progress bar. If you dont know the total ahead of time, it doesnt give you a percent completion, but still tells you thinks like number of iterations per second and number of iterations finished.
- geofft 5y agoThe halting problem is easy to solve most of the time. For instance, you can easily tell whether the following two programs halt: while True: pass while False: pass What's impossible is solving the halting problem 100% of the time with 100% accuracy. Most people don't need to do that. Solving the halting problem most of the time and then saying "I don't know, that's too hard" the rest of the time is of immense practical value, and many practical systems (the kernel's eBPF verifier, the thing in your browser that detects stuck pages, etc.) do exactly that. In this particular case, tqdm solves the halting problem in cases where it's easy: https://tqdm.github.io/docs/tqdm/#__init__ https://tqdm.github.io/docs/tqdm/#__init__ > total: int or float, optional > The number of expected iterations. If unspecified, len(iterable) is used if possible. If float("inf") or as a last resort, only basic progress statistics are displayed (no ETA, no progressbar).
- avelis 5y agoI always wondered how a post like this a year ago didn't go anywhere but this post makes the front page. Love the Python package. https://news.ycombinator.com/item?id=23160774 https://news.ycombinator.com/item?id=23160774
- emehex 5y agochime [1] is another python package that fans of tqdm might like. [1] https://github.com/MaxHalford/chime/ https://github.com/MaxHalford/chime/
- ejdyksen 5y agoIf you like tqdm, it's worth checking out pqdm, a parallelized version. If you have embarrassingly parallel work to process in a script, it makes it dead simple to parallelize and monitor the progress of something. Highly recommend: https://github.com/niedakh/pqdm https://github.com/niedakh/pqdm
- tdekok 5y ago+1 for PQDM, I use it a ton and most of the time it just works. I did have some rare cases where PQDM was much slower than a direct joblib implemention, but that could well be a fluke on my end. Either way, amazing package!
- anonymousCar 5y agoMy current issue with tqdm is nested progress bars in multi-threading/processing causing dead locks + tqdm.write just being broken in those contexts, either deadlocking or the ui just being wrong. Does pqdm do a better job?
- pak 5y agoBig fan of tqdm in python, so I ported it to ruby 5 years ago: https://github.com/powerpak/tqdm-ruby https://github.com/powerpak/tqdm-ruby (shameless plug and an invitation for pull requests)
- QuackyTheDuck 5y agoGood stuff, thank you!
- amelius 5y agoI think the author doesn't fully appreciate how insanely difficult it is to write accurate progress meters for anything slightly more complicated than a single for-loop.
- 999900000999 5y agoPython is such an amazing language, on one hand, it's easy enough where you could probably train a non technical person on it within about two months. Yet, you can still make 200K writing it, I don't know the next time I'll create a command line application in Python, but I'll keep this little tool in mind. I hope Python eats the world.
- MichaelMoser123 5y agoit may be easy to get going with python, but it takes a non trivial amount of time to understand, what is going on. I have a advanced python3 course https://github.com/MoserMichael/python-obj-system https://github.com/MoserMichael/python-obj-system that explains some of the more advanced concepts. One of the things covered are decorators (tqdm is a decorator)
- zuj 5y agoThis is amazing. I just gave a quick glance and love your style. I just wish you would have provided ipynb instead of md. It would be so much fun to just modify and run as I am learning. Thank you!
- MichaelMoser123 5y agoThank you! I am not an expert in Jupyter Notebook, may look into it now.
- 999900000999 5y agoThis is gold, thank you !
- Nihilartikel 5y agoOne library that haa been really useful for me lately is Dataset https://dataset.readthedocs.io/en/latest/ https://dataset.readthedocs.io/en/latest/ It makes it a lot easier to casually use a database (e.g. SQLite) to persist dictionaries without explicitly building a schema. Great for CLI apps and ad hoc data crunching.
- blahgeek 5y agoWait.. so python takes several seconds to simply count from 0 to 9 million?
- thammyvienula 5y agoThẩm mỹ viện trị mụn Ula Spa giúp nàng sở hữu làn da sáng mịn chuẩn sao Hàn Thông tin liên hệ: Công ty TNHH MTV thẩm mỹ viện Ula Văn phòng đại diện: 1222 Quang Trung, Phường 8, Quận Gò Vấp, Thành phố Hồ Chí Minh, Việt Nam Website: https://thammyvienula.vn/ https://thammyvienula.vn/ Tel: 0944255226 - 0939545373 Nguồn: http://danviet.vn/tham-my-vien-tri-mun-ula-spa-giup-nang-so-huu-lan-da-sang-min-chuan-sao-han-50202171215564346.htm http://danviet.vn/tham-my-vien-tri-mun-ula-spa-giup-nang-so-...
- oars 5y agoFantastic share and subsequent discussion about in this thread. Threads like these are why I love HN. My Python skills have improved so much thanks to HN.
- k_sze 5y agotqdm is one of the (many) really useful packages that make me miss using Python (I’m stuck writing TypeScript for Node.js in my current job).
- beaugunderson 5y agodoesn't have every bell and whistle of the original but I found nqdm recently and it works: https://www.npmjs.com/package/nqdm https://www.npmjs.com/package/nqdm
- keithalewis 5y agoWhoaaaa, this tqdm thing is, like, um, toats kewl!!! It's waaay better than the <blink> tag old people used to use, like, before I was even born lol. IM so so happy idont have to think too hard to get flashing light thingys to show up on my screen.
- berkes 5y agoIf you are trying to say that youngsters are stupid for adding visualisations to their programs and terminals: sorry, but you are wrong.
- heavenlyblue 5y agoThey are stupid for using visualisations that break standard python logging.
- joelschw 5y agoIt's for interactive sessions not background ones
- heavenlyblue 5y agoYeah, exactly. The software I write is complex enough to require logging of hundreds of progress bars at the same time. The great thing that just using `logging` module is enough to log something like `Downloading file A, 55% [55Mb/100Mb]` (absolutely equivalent in terms of information to a "graphic bar") and also happens to be composable in a way that I can then reuse that package as part of anything that is also non-interactive.
- keithalewis 5y agoWo dude, not at all. But you are literally saying exactly what my gramps used to say "but you are wrong." I figured his brane was too old to tell me why I was wrong. Computers are really fast now. Why cant they just give the answer right away?
- enriquto 5y agoIs there a way to globally disable all tqdm progress bars? Something like the NO_COLOR environement variable? These progress bars are nice when you launched a single loop yourself, but when you are running an automated battery of many things they become annoying and pollute your terminal too much. I know that you can silence each particular program that uses tqdm by setting a 'disable' option. But this requires editing the python source code of the program.
- zbentley 5y agoSomething like this should work: import os import sys from tqdm import tqdm as _tqdm def tqdm(*args, **kwargs): try: disable = bool(int(os.environ['NO_PROGRESSBARS'])) except KeyError: disable = not sys.stdout.isatty() except (ValueError, TypeError): disable = False kwargs.setdefault('disable', disable) return _tqdm(*args, **kwargs) Then import that instead of tqdm.tqdm. sys.stdout.isatty() isn't a perfect answer to what people ask when they want to know "am I running in an automated environment, or is a human user looking at my output?", but it's close. More nuance is available online.
- enriquto 5y agoThanks for the answer! > import that instead of tqdm.tqdm But it's not me who is importing tqdm to begin with! I call many programs in parallel from shell scripts (out of my control) and they all call tqdm individually. I need to stop tqdm output from outside these programs. Your code should be part of tqdm itself, not written by individual programmers. > am I running in an automated environment, or is a human user looking at my output? But I want to stop tqdm output precisely because I'm a human looking at it. If you have more than one or two progress bars simultaneously, it becomes useless clutter. As much as I like to use tqdm myself for my programs, I'm sad that as tqdm becomes more and more popular, my terminal output becomes more and more cluttered, to an absurd amount. Piping the output to a file does not help and is totally the wrong idea. I'm precisely interested in seeing--in real time--the part of the output that does not come from tqdm, such as warnings and errors.
- hathym 5y agoDo you know that tqdm is pronounced like the Arabic word تقدم which mean progress?
- edanm 5y agoYes! The same as the Hebrew word תקדם for any Hebrew speakers. It makes the module so much funner to use for some reason :)
- tmabraham 5y agoTqdm is awesome and I use it all the time! Another great alternative is fastprogress: https://github.com/fastai/fastprogress https://github.com/fastai/fastprogress It often works better in Jupyter Notebooks.
- nvalis 5y agoFor notebook I am using tqdm.notebook[0]. Works quite well for me. [0] https://tqdm.github.io/docs/notebook/ https://tqdm.github.io/docs/notebook/
- davidatbu 5y agotqdm is great, except I really like my type hints, and tqdm has no type hints :(