13 ms·
CLI tools hidden in the Python standard library
- gabrielsroka 3y agoThis seems to work too https://www.google.com/search?q=site%3Adocs.python.org+%22python+-m%22 https://www.google.com/search?q=site%3Adocs.python.org+%22py...
- SushiHippie 3y ago`python -m turtledemo` may be not a hidden “tool”, but nonetheless it is absolutely mint.
- CJefferson 3y agoI use http.server all the time, particularly as modern browsers disable a bunch of functionality if you open file URLs. Had no idea there was so much other stuff here!
- icar 3y agoI use miniserve (`cargo install miniserve`). You also have available `npx serve`.
- HumblyTossed 3y agoOh and the Rust brigade have arrived... Was only a matter of time.
- madfucker69 3y agoHey, stop rust-shaming
- IshKebab 3y agoHow dare people mention tools written in Rust? It may be a fantastic, well loved language that's exploding in popularity and the source of endless very high quality CLI tools... but. The absolute cheek! We must only mention Python and Bash forevermore.
- HumblyTossed 3y agoEvery time there's a discussion about language X, Rusties jump in. Stop it. One of the biggest turn offs about Rust is the community.
- IshKebab 3y agoHe literally did not mention Rust.
- HumblyTossed 3y agoImplicitly, they did.
- pinkcan 3y agowww is aliassed on my .zshrc for that reason: www='python -m http.server'
- trotro 3y agoSame here, it's also by far the most convenient way I've found to share files between devices on my network.
- seanw444 3y agoSame for me. Although I just discovered `qrcp` which has been quite handy, and I'm sure there are many tools like it.
- bradrn 3y agoWait, how does that work? As far as I can see from the documentation, it can only serve on localhost, which to my understanding is only accessible from the single device it was launched on.
- vram22 3y agoHave you checked the full docs? Maybe it takes an optional parameter to specify the server machine's IP address or host name. Then others on the network could see it. Not near a PC now.
- ptx 3y agoI think you must have misread the documentation [1], which says: "By default, the server binds itself to all interfaces." [1] https://docs.python.org/3/library/http.server.html https://docs.python.org/3/library/http.server.html
- bradrn 3y agoNot ‘misread’, just ‘missed that bit entirely’. Testing it out, this is indeed the case.
- randlet 3y agoIf you serve on localhost you can usually access from other devices by using the "servers" ip address. So if your desktop where you're running the server has ip 192.168.1.10 then you can go to http://192.168.1.10 http://192.168.1.10 in the browser of another device on the same network.
- thrdbndndn 3y agoChrome also often takes forever to load even the simple HTMLs if it's a local file. I suspect it's related to relative path resource but never figured it out.
- lizard 3y agoIts notable, perhaps, that the if __name__ == "__main__": block allows you to do this for a _module_, i.e. a single *.py file. If you want to do this for a package, add a `__main__.py` You can also use either of these throughout your code so that you can have python -m foo python -m foo.bar python -m foo.bar.baz each doing different, but hopefully somewhat related, things.
- BiteCode_dev 3y agoIt also uses the file as a module, not a script, which means suddenly relative imports works the root dir and the cwd are the same, and it is added to sys.path. This prevents a ton of import problems, albeit for the price of more verbose typing, especially since you don't have completion on dotted path. It is my favorite way of running my projects. Unfortunalty it means you can't use "-m pdb", and that's a big loss.
- sharikous 3y agoYou can use pdb Just python -m pdb -m module
- BiteCode_dev 3y agoDamn, 15 years of python and I learn you can use -m twice. I've never even tried, didn't occur to me it would be supported. EDIT: it support other options as well, like -c. That deserves an alias: debug_module() { if python -c "import ipdb" &>/dev/null; then python -m ipdb -c c -m "$@" else python -m pdb -c c -m "$@" fi }
- abhishekjha 3y agoLike whats the general usage though? python -m http.server is the most I have done.
- BiteCode_dev 3y ago
- Noumenon72 3y agopython -m json.tool seems neat. I am always dumping JSON in single log lines and then expanding it to diff it (currently using Cmd+Opt+L in PyCharm).
- dmurray 3y agoI tend to use "jq .". An extra tool, but you don't really have to learn it, and if you're "always dumping JSON" and also use the command line a lot, you probably want to have it around anyway.
- OJFord 3y agoYou don't need the `.` fwiw, just `produce-compact-json | jq` will do.
- rich_sasha 3y agoNot always, some versions of jq get confused without the dot. I work on two Linuces at work, one requires it and one doesn't.
- OJFord 3y agoThe current version and for quite some time, then. If I said 'Windows comes with the Edge browser' would you say 'I use two at work, one does but the other only has Internet Explorer'? Surely it's generally implied we're talking about things as they are, unless specified otherwise?
- cl3misch 3y agoFYI, you can also just pipe into "jq". If anything it's faster to type :-)
- mg 3y agoThere is one problem with those: Security. Using modules (even if they are in the standard library) on the command line lets malicious code in the current dir take over your machine: https://twitter.com/marekgibney/status/1598706464583028736 https://twitter.com/marekgibney/status/1598706464583028736
- formerly_proven 3y agoiirc this is one of the things earmarked for a hypothetical Python 4, making -P the default. It's also one of the many relatively well-known (security) issues in Python that don't get addressed for a surprising amount of time. Others in the same vein would be stuff like stderr being block-buffered when not using a TTY, no randomized hashes for the longest time, loading DLLs preferably from the working directory, std* using 7-bit ASCII in a number of circumstances and many more.
- TeMPOraL 3y ago> loading DLLs preferably from the working directory That is a feature, though, isn't it?
- formerly_proven 3y agoNo, the problem was that python[.exe] would load pythonXY.dll etc. from the working directory instead of the installation. Edit: I also recall issues with wheels where .so's in unexpected locations would take preference over the .so files shipped by the wheel. I believe most of that should be fixed nowadays with auditwheel and hardcoded rpaths.
- jonnycomputer 3y ago-P?
- throwawaymobule 3y agohttps://docs.python.org/3/using/cmdline.html?highlight=q#cmdoption-P https://docs.python.org/3/using/cmdline.html?highlight=q#cmd... had to look myself. apparantly, it's like setting PYTHONSAFEPATH which prevents 'unsafe' paths from getting added to sys.path. new in 3.11
- ozymandias_kok 3y agoOooh vulnerabilities! Yum
- polyrand 3y agoPython 3.12 will include a SQLite CLI/REPL in the standard library too[0][1]. This is useful because most operating systems have sqlite3 and python3, but are missing the SQLite CLI. [0]: https://github.com/python/cpython/blob/3fb7c608e5764559a718ce8cb81350d7a3df0356/Lib/sqlite3/__main__.py#L65 https://github.com/python/cpython/blob/3fb7c608e5764559a718c... [1]: https://docs.python.org/3.12/library/sqlite3.html#command-line-interface https://docs.python.org/3.12/library/sqlite3.html#command-li...
- chmaynard 3y ago> This is useful because most operating systems have sqlite3 and python3, but are missing the SQLite CLI. Not sure what you mean -- sqlite3 is the SQLite CLI.
- TeMPOraL 3y agoThey likely mean libsqlite3, i.e. the .DLL/.so.
- polyrand 3y agoYes, I meant the DLL/.so file. The library is almost always present in the OS, the CLI is not.
- chmaynard 3y agoNot sure why my comment is being downvoted. From the sqlite.org website: The SQLite project provides a simple command-line program named sqlite3 (or sqlite3.exe on Windows) that allows the user to manually enter and execute SQL statements against an SQLite database or against a ZIP archive.
- maleldil 3y agoIt's weird that they're adding code to the stdlib without type annotations.
- RGBCube 3y agoLots of code in the stdlib does not have type annotations. Though I think most popular modules in the std either are annotated or have stubs somewhere.
- BoppreH 3y agoSpeaking of hidden Python tools, I'm a big fan of re.Scanner[0]. It's a regex-based tokenizer[1] in the `re` module, that for reasons is completely missing from any official documentation. You give it a pattern for each token type, and a function to be called on each match, and you get back a list of processed tokens. Importantly, it processes the list in one pass and ensures the matches are contiguous, where a naive `re.findall` with capture groups will ignore unmatched characters. You also get a reference to the running scanner, so you can record the location of the match for reporting errors. import re scanner = re.Scanner([ (r"[0-9]+", lambda scanner, token:("INTEGER", int(token))), (r"[a-z_]+", lambda scanner, token:("IDENTIFIER", token)), (r"[,.]+", lambda scanner, token:("PUNCTUATION", token)), (r"\s+", None), # None == skip token. ]) results, remainder = scanner.scan("45 pigeons, 23 cows, 11 spiders.") assert not remainder print(results) [('INTEGER', 45), ('IDENTIFIER', 'pigeons'), ('PUNCTUATION', ','), ('INTEGER', 23), ('IDENTIFIER', 'cows'), ('PUNCTUATION', ','), ('INTEGER', 11), ('IDENTIFIER', 'spiders'), ('PUNCTUATION', '.')] [0]: https://stackoverflow.com/a/693818/252218 https://stackoverflow.com/a/693818/252218 [1]: https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization
- Timon3 3y agoIt seems like the discussion of whether to document it died in April of 2003: https://mail.python.org/pipermail/python-dev/2003-April/035076.html https://mail.python.org/pipermail/python-dev/2003-April/0350... Bummer, it's a cool feature, but I don't feel safe relying on undocumented features.
- nbrtx 3y agoWhy would you rely on any of these? They can be deprecated at a whim like distutils and many other things. That's why Python requires so many blog posts.
- Macha 3y agoIf by on a whim, you mean after removal after 6 years of being warned against (the first "maybe use setuptools instead" note was in Python 2.7.12 in 2016), deprecation was proposed in october 2020, agreed in january 2021, and removal will happen in Python 3.12, which... hasn't been released yet.
- 9dev 3y ago> Pretty-print JSON: > echo '{"foo": "bar", "baz": [1, 2, 3]}' | python -m json.tool This is even more fun on MacOS if you combine it with the pbpaste/pbcopy utils: alias json_pretty="pbpaste | python -m json.tool | pbcopy" That command will pretty-print any JSON in your clipboard, and write it back to the clipboard, so you can paste it somewhere else formatted!
- TRiG_Ireland 3y agoalias json-format="python -m json.tool | pygmentize -l javascript"
- scruple 3y agoI have a similar mapping to use `json.tool` inside of vim buffers. Very useful tool that's gotten a ton of mileage from me over the last ~decade.
- masklinn 3y agoShame gzip has one but zlib does not, that would be a very useful addition: some software create raw zlib streams on disk (e.g. git) and there’s no standard decompressor, you need to either prepend a fake gzip header, go through openssl, qpdf‘s zlib-flate, or pigz -z.
- submeta 3y agoNice! I use this on winboxes: zipfile Decompress a zip file: python -m zipfile -e archive.zip /path/to/extract/to Compress a directory into a zip file: python -m zipfile -c new_archive.zip /path/to/directory
- deleted 3y ago[deleted]
- throwawaymobule 3y agois there an advantage to this over windows builtin zip feature? I don't know if it is usable from CLI.
- p4bl0 3y agoI often use `python -m http.server` e.g. to easily share files over local networks but I had no idea so many standard modules supported this. Thanks for sharing this link!
- tomrod 3y agoDo you have an example snippet you could share?
- jon-wood 3y ago`python3 -m http.server --help` - its really not that complex.
- tomrod 3y agoTrue. Glad to hear it's not hidden under 10 args.
- quenix 3y agoYou just start an http server with python3 -m http.server 8080 and then access the server on the receiving device through its IP address. It’ll show a basic directory listing, and you can download from there.
- tomrod 3y agoThanks! Very helpful.
- NoboruWataya 3y ago> I thought this might provide a utility for generating random numbers, but sadly it's just a benchmarking suite with no additional command-line options: I had the same experience with the tkinter ones - I thought they might be like zenity, a way to build simple UI elements from the command line. But they mostly just show simple non-configurable test widgets. The colour chooser could be helpful though.
- TelonAlex 3y agoThe cli tool [fire](https://github.com/google/python-fire/blob/master/docs/guide.md https://github.com/google/python-fire/blob/master/docs/guide...) has a nifty feature where it can generate a cli for any file for you. So random and math are somewhat usable that way $ python -m fire random uniform 0 1 0.5502786602920726 $ python -m fire math radians 180 3.141592653589793 $ python -m fire math e 2.718281828459045 Just running $ python -m fire random will give you a nice "manpage" for your module as well.
- dfc 3y agoI am not someone who complains about "stop piping cats" but using '-e' with grep is a lot quicker and easier to read. This: grep -v 'test/' | grep -v 'tests/' | grep -v idlelib | grep -v turtledemo Becomes: grep -ve 'test/' -e 'tests/' -e idlelib -e turtledemo
- sproketboy 3y ago[dead]
- jerpint 3y agoSomewhat related is the ipython repl which can replace a shell and provides many useful commands as well as execute python code
- codetrotter 3y agoMy favourite is python3 -m http.server I use it all the time
- mr_o47 3y agoThank you for putting this together and this is pretty useful as my favorite has been running http.server
- nneonneo 3y agoI answered a question on SO over a decade ago on this topic, back in the Python 2.7 era: https://stackoverflow.com/a/14545364/1204143 https://stackoverflow.com/a/14545364/1204143 One of these days it would be nice to make an unofficial Python reference book which documents these tools, hidden features (like re.Scanner!), and other corners of the stdlib or language.