15 ms·

Guide to Python Debugging

501 points by alexeiz 6y ago

cl3misch 6y ago
I use ipython a lot, and it has a great post-error debugger with `%debug` which drops you in a ipdb shell at the stackframe of the error. Concerning reloading, I find `%autoreload` much more reliable than `importlib.reload`. The latter can't handle `from lib import foo` or if a dependency inside an imported package changed. `%autoreload` just does this all automagically.
- jsmeaton 6y ago
  I’ve never had autoreload work for me - is there anything special to it? For context, I’m often working in a Django shell.
hathym 6y ago
if you are working with vscode, I highly recommand debugpy[1] or ptvsd[2] [1] https://github.com/microsoft/debugpy https://github.com/microsoft/debugpy [2] https://github.com/microsoft/ptvsd https://github.com/microsoft/ptvsd
- fireattack 6y ago
  Are they the one(s) that will be called if you use the "start debugging" in GUI with official Python extension? Or different one?
  - 0x008 6y ago
    yes. vs code python extension uses debugpy.
oli5679 6y ago
IPython breakpoints are nice from IPython import embed embed()
drpancake 6y ago
I've been a Python developer for more than ten years. Approximately 98% of the time I need to debug something I use print() statements and this: import IPython; IPython.embed()
- justmyname 6y ago
  Oh, having full ipython during debug investigations is much better than using pure pdb.set_trace(). Thank you.
  - drpancake 6y ago
    You might find this useful: https://github.com/gotcha/ipdb https://github.com/gotcha/ipdb
- quickthrower2 6y ago
  I've been a Python developer for less than 10 hours. Approximately 98% of the time debugged something was in Visual Studio. Quite a nice experience. I was lucky to have the project already set up for me.
- trymas 6y ago
  Can you step through the code when embedding ipython in your code? Does this approach have advantages versus `ipdb` or `pdb++` ?
  - drpancake 6y ago
    No, but I rarely find it helpful to step through code. That's the other 2%.
    - jsmeaton 6y ago
      import ipdb; ipdb.set_trace() IPython pdb session.
      - 6y ago
_ZeD_ 6y ago
Do yourself a favour and learn how to use use a proper IDE. debugging in pydev shows you everything
albertzeyer 6y ago
To extend on inspecting stack traces: I found that I often wanted to know the the content of (relevant) (local) variables of a stack trace. Whenever I got a stacktrace, I would often restart it in a debugger, or add print statements on variables. Until I extended the stacktrace to just include this information. The tricky part is that printing all local variables is too much. And also global variables might be involved. Or attributes (e.g. self.x or so). So now I'm parsing all variables and attributes from the source code line from the traceback (in a very simple way) and print only those. This covers about 95% of all cases - i.e. in 95% of the case, the stack trace contains all information I need to understand and fix the problem. I published that here: https://pypi.org/project/better_exchook/ https://pypi.org/project/better_exchook/ https://github.com/albertz/py_better_exchook https://github.com/albertz/py_better_exchook I also often have a SIGUSR1 handler which will print the stacktrace of all threads. This is useful on long running processes, involving multi threading, where you might run into some strange hangs or deadlocks. In addition to that, if you might get crashes (segfault or so), something like faulthandler is useful. If you also want to see the C stack trace in addition in such cases, I load libSegFault.so, like here: https://github.com/rwth-i6/returnn/blob/5b8e34ec1fd725d0e20b5b422d213dbf17d9e069/Debug.py#L226 https://github.com/rwth-i6/returnn/blob/5b8e34ec1fd725d0e20b... Profiling is another topic. Something like Py-Spy (https://github.com/benfred/py-spy https://github.com/benfred/py-spy) can be very helpful. I also found a remote background ZMQ IPython/Jupyter kernel to be useful sometimes. I published that here: https://github.com/albertz/background-zmq-ipython https://github.com/albertz/background-zmq-ipython
lmilcin 6y ago
TLDR: Guide to python debugging that doesn't mention debugger and says "logging is a must".
- nxpnsv 6y ago
  Print out debugging is the new black. Actually quite useful at times though...
- wodenokoto 6y ago
  Actually it does mention the debugger
mjbrownie 6y ago
I recommend repr in all log messages too. For a work Django project we have common BaseModel repr(obj) expression that can be easily converted (using regexes in loki and promtail) into a clickable admin link. So within loki (or any similar log tools) all logger.info(repr(object)) is a link straight to the admin. Worth the set up time.
amaajemyfren 6y ago
> Most of the time, what programming really is - is just a lot of trial and error I am not the best programmer ... but I have always felt like an imposter with how many times I googled the next expression.
gorgoiler 6y ago
I’ve always admired pdb.set_trace() as a way of inserting a debugger breakpoint. I’m surprised it isn’t mentioned in the article. You don’t need a different UI for debugging — you can just write the breakpoint into the code. Conditional breakpoints can be done with Python logic! Marvelous!
diarrhea 6y ago
The article has a couple of problems in my eyes: - "you might want to just copy-paste it and use it" ... that's maybe what people end up doing, but it's a terrible idea. - The following `__repr__` is out of alignment with the output printed below it: class Circle: def __init__(self, x, y, radius): self.x = x self.y = y self.radius = radius def __repr__(self): return f"Rectangle({self.x}, {self.y}, {self.radius})" The class name is hardcoded; maybe use: def __repr__(self): name = self.__class__.__name__ return f"{name}({self.x}, {self.y}, {self.radius})" - There's a bare `except` in the `traceback` demo. It is not a problem in this toy example and works, but bare `except` statements have no place in educational code. Some poor soul will copy-paste it and run into awful errors down the line (https://realpython.com/the-most-diabolical-python-antipattern/ https://realpython.com/the-most-diabolical-python-antipatter...)
hywel 6y ago
Right now, this site is suffering a hug of death. This looks like the same article: https://dev.to/martinheinz/ultimate-guide-to-python-debugging-1739 https://dev.to/martinheinz/ultimate-guide-to-python-debuggin...
- JimBlackwood 6y ago
  It is the same article, it’s written by the same guy. In the article you linked it mentions “This article was originally posted on [url]”, which links to the same article this post links to.
qertoip 6y ago
Nice layout Sir.
memco 6y ago
Lots of great comments and resources here! I was surprised to find out how uncommon using a debugger for Python development was when I joined my team at work. I had to muddle through lots of documentation to figure out how to use it in certain situations and also to figure out alternative things when the first approach didn't work. I looked into PyCharm but opted to use VSCode as my main editor since the debugging support seemed easier to configure both for local and especially for remote. We have a flask app and one hurdle I had to overcome was our docs all assumed you'd run the app via gunicorn, but VSCode had trouble triggering breakpoints so I had to figure out how to run the app via the flask module directly, which was a bit of work since our app isn't following most of the getting started with flask tutorials conventions. For another project we use celery, VSCode can be used to debug celery tasks but it's also work to set up and in the end I found using the rdb.set_trace() debugging provided by celery was easier. An important lesson I learned while working on that project is that you need to be sure you're setting your breakpoints in the right version of your app: if you are using a setup.py install step the code you want to debug is probably somewhere in site-packages not wherever you installed it from. For another project, we're using python 2.7 on Centos 6: VSCode debugging & remote tools don't easily support that setup so knowing pdb or doing log debugging are the best bets. Something important on log-based debugging this article doesn't mention: if you're developing a long-running app like a daemon or service of some kind: you should probably make your log config loading dynamic. The one provided in the article is nice if your script is one-off, but if you are running a service you may want to be able to dynamically adjust the logging to be more verbose when an issue occurs and then reset it when done debugging without having to start/stop the whole service. I inherited code that runs as a service and in order to change the log level I have to stop the service, change the config and restart. The start/stop is destructive: if you stop the service the action it was performing has to be redone from the beginning. These are tasks that can take 8-12 hours so restarts are painful. Debug logs can be huge so you can't just leave the service in debug log mode all the time unless you want to fill up the whole disk. Which brings up another point: rotating log files—if you're doing heavy logging you need to be sure you have set up rotation so that you don't eat through disk space indiscriminately. In my ideal world every project I work on could have breakpoints trigger my editor's tools so I can inspect and alter code on the fly, but knowing there's more than one way to go about it and how to approach it when you don't have control is important.
yig 6y ago
My favorite visual debugger for Python and many other languages is CodeRunner, because it's so lightweight. (Mac-only, not free.) https://coderunnerapp.com/ https://coderunnerapp.com/
DodgyEggplant 6y ago
But there are good tools also. I'm using wing IDE for years, it has excellent debugging: multi threaded debug, remote debug with SSH (which works almost as fast as local debugging), deep introspection to anything on the stack and call chain, conditional breakpoints, powerful debug console that can hook to any step in the call chain of any thread, and debugging a server thread easily which is useful for debugging services and scenarios that can't be fully replicated on a local machine.
tjpnz 6y ago
I'm somewhat embarrassed to say that I very rarely use debuggers. Print statements have always worked for me even in large codebases. What am I missing here?
- zwieback 6y ago
  Maybe you're not missing anything - if you're productive with print debugging and have used a debugger but found that it doesn't help that much I wouldn't worry about it. I'm the opposite - immediately run in the debugger to probe around how to use this or that API or verify that the code is really doing what I think. I also find that stepping through my code early on often gives me insight in how to implement something more efficiently.
- hansvm 6y ago
  I'm just one data point, but I also rarely use debuggers and often work with a colleague who rarely uses print statements. There are qualitative differences in the code we're usually modifying though; in particular, he's often editing code where the control flow is hard to reason about (inheritance towers, pages of conditionals, etc) and is fairly slow to build and run. In that environment I might even reach for a debugger myself -- e.g., even if you already isolated the bug to a particular portion of the code and just wanted to know which method was being dispatched you would need a ton of print statements, or at least a lot of back-and-forths editing the code and figuring out where you ended up (because of the high branch complexity which could send you far from where you started), and those restarts would be fairly slow in and of themselves. Contrast that with a debugger where you can quickly step forward and find out exactly where you end up. IMO, print statements are easier/faster to use than debuggers, so I reach for those first unless there's some feature a debugger has that I think will make my life easier overall.
dangoljames 6y ago
This is wicked af, thank you so much!
Buetol 6y ago
This is buried in the comments but `pudb` is great debugger: https://pypi.org/project/pudb/ https://pypi.org/project/pudb/ - visual interface to set breakpoint - inline interpreter - inspect on exception It's like using a modern debugger but inside your console. Can't recommend enough.
- ur-whale 6y ago
  Does it support multi-threaded python code?
  - Buetol 6y ago
    Looks like it does: https://medium.com/@auro_227/use-pudb-to-debug-python-multiprocessing-code-c0c5551d010c https://medium.com/@auro_227/use-pudb-to-debug-python-multip...
3pt14159 6y ago
Here's my list of helpful additions: 1. https://pypi.org/project/pdbpp/ https://pypi.org/project/pdbpp/ This is basically a better version of pdb. 2. ~/.pdbrc Useful for common commands that you would otherwise have to type out. For example, if you do TDD on a JSON API and often need to debug the output consider this alias: alias ppr pp response.json() 3. ~/.pdbrc.py This file loads before pdb starts, so it lets you customize it a bit and add things like history. Here is a simple version to get you started: https://gist.github.com/zachaysan/25712f3e9aa02a234437f89c60cad8d0 https://gist.github.com/zachaysan/25712f3e9aa02a234437f89c60...
petr25102018 6y ago
Shameless plug of some debugging options in Python: https://stribny.name/blog/2019/06/debugging-python-programs https://stribny.name/blog/2019/06/debugging-python-programs
kikoreis 6y ago
It misses the most important one-liner for python debugging: import pdb; pdb.set_trace() which leaves you in the debugger at a specific point in your code. When I actually did Python dev rare was the day I didn't crack that out!
- css 6y ago
  Or in 3.7 and beyond: breakpoint()
  - kikoreis 6y ago
    OH!
- mgalgs 6y ago
  Or if you have ipython installed: import ipdb; ipdb.set_trace() which gets you a slightly fancier pdb.
bitpow 6y ago
There’s a typo in the __ repr__ code section
darkteflon 6y ago
There are loads of helpful comments in here for Python devs. Thanks, everyone.
mbeex 6y ago
Since PyCharm is abundant in this discussion, here are an alternative: I've been using the Wing IDE for several years. It sometimes chokes when I try to inspect larger numpy arrays as a whole, using the application-specific array visualization (but there are other ways). Otherwise I'm quite happy with it.
alexhutcheson 6y ago
If you’re not using an IDE, then I’d highly recommend trying PuDB: https://github.com/inducer/pudb/blob/master/README.rst https://github.com/inducer/pudb/blob/master/README.rst
theshetty 6y ago
I have a complete off the topic question, how do I create a slick blog like this - simple and minimal?
- wjziv 6y ago
  The writer has his blog sourcecode on Github. https://github.com/MartinHeinz/blog-backend https://github.com/MartinHeinz/blog-backend The frontend is built on Vue.js.
fudged71 6y ago
Can anyone recommend a video/series on Python debugging? Debugging has been my greatest struggle with getting into development. I get frustrated and spend over a day on simple scripts
- chameleon_world 6y ago
  https://realpython.com/python-debugging-pdb/ https://realpython.com/python-debugging-pdb/
ur-whale 6y ago
Slightly off topic, but: how does one debug a multi-threaded python app? PDB just doesn't seem to work at all.
milin 6y ago
https://microsoft.github.io/debug-adapter-protocol/ https://microsoft.github.io/debug-adapter-protocol/ looks really promising to bring standardized debugging across most ides and languages, which is awesome.
qwertox 6y ago
Why should my browser connect to a nonstandard port just to load a webpage? https://martinheinz.dev:1234/api/v1/posts/24 https://martinheinz.dev:1234/api/v1/posts/24
luord 6y ago
In six years working with python, I was never aware of `importlib.reload` I feel we're going to have a lot of fun together.
DanFeldman 6y ago
Great article, especially nice points about __repr__ helping logging. PDB as mentioned is great, but IPDB is at a whole 'nother level. If I'm able to get an ipdb breakpoint set in a tricky part of python code, I'm set. Tab-completion, some saved history, normal debugger operations, it's a dream-come true. I often set breakpoints and code iteratively w/ a repl-like interface or explore unfamiliar functions and objects.
- Groxx 6y ago
  ipdb is one of the nicest command-line debugging experiences I've had. It's wonderful.
_coveredInBees 6y ago
This is a bit of a weird article. It spends most of the time talking about logging, which is somewhat useful for debugging but not really. pdb gets a few lines of description, and that is about it. Personally, I can't live without Pycharm when working with Python purely because of how fantastic the debugging experience is. The integration with the interactive IPython shell is simply fantastic, and the live variable viewer is also really handy. Between being able to prototype code in the interactive IPython REPL and then graduate it seamlessly to scripts/functions/classes, and then having an amazing debugging UX to fallback on for those bugs that slip through, I find myself to be orders of magnitude more efficient than when coding in something like vscode (which I absolutely love, and want to transition to for Python coding, but it is extremely limited in comparison to Pycharm, especially when it comes to the REPL experience + debugging).
- vonseel 6y ago
  Yeah I agree entirely on Pychaem debugger being awesome. But it’s no substitute for logging in production. Somethings just need to be logged, either for proof or those situations where you just can’t reasonably attach a Pycharm debugger.
  - underdeserver 6y ago
    Yes, logging is crucial for knowing what happened, but if you don't understand how it could have happened, step by step debugging is immensely useful. Assuming, of course, you can reproduce in your dev environment.
  - jerzyt 6y ago
    Production logging with accurate and precise timestamp is an absolute time saver. That means like 1ms timestamp and NNTP synchronized servers.
  - _coveredInBees 6y ago
    Oh yes, absolutely. I didn't mean to say logging isn't useful. It is vital to have good logging for production because like you said, you aren't just going to be able to attach a debugger. But for an article titled "Guide to Python Debugging", logging should be a short paragraph, not the core of the article imo. Debugging in production on the other hand... then logging makes a lot of sense to focus on.
ivan_ah 6y ago
Nice article. In particular the -i flag seems really useful. I'd like to add two packages I learned about recently that I like A LOT: pyrasite and PySnooper. Use pyrasite to "attach" to an existing running Python process (i.e. you forgot to instrument the app, but it has a bug in it, how can you debug?) (1) Install pyrasite (2) Get the PID of the running process (3) Run `pyrasite <pid> dumpstackz0.py` where dumpstackz0.pycontains `import traceback; traceback.print_stack()` (4) The stacktrace will be printed in the stdout/stderr of the running application Use PySnooper instead of print-debugging—just add a decorator to any python function to get detailed debug info like you normally would get from a full debugger: (A) code: https://github.com/cool-RR/PySnooper https://github.com/cool-RR/PySnooper (B) pypi: https://pypi.org/project/PySnooper/ https://pypi.org/project/PySnooper/ (C) talk: https://www.youtube.com/watch?v=XP5imOJc_TE https://www.youtube.com/watch?v=XP5imOJc_TE (D) podcast: https://www.youtube.com/watch?v=VUZvgJJKor0 https://www.youtube.com/watch?v=VUZvgJJKor0
- macintux 6y ago
  That -i flag trick would have saved me a lot of grief over the last year or so. Better late than never.
ed25519FUUU 6y ago
Don't forget breakpoints! Learn how to move around in the debugger, it will be a LIFESAVER. Above a troublesome line of code, drop this line: import pdb; pdb.set_trace() And run it as normally. Now you can explore. Drop into the function, step forward, go up. You just need to learn to move around. With pdb you can really explore what's going on.
- karlicoss 6y ago
  Or even just breakpoint() built-in in python 3.7+ In addition you can set PYTHONBREAKPOINT=ipdb.set_trace (or whatever you prefer) in your shell rc file
  - wodenokoto 6y ago
    It has really bothered me that `breakpoint()` didn't start ipdb when running a script using ipython. I guess this is a good workaround!
MadNinjaSkills 6y ago
Nice! This is gonna be totally useful for me. Thanks!
saagarjha 6y ago
When debugging compiled code, I often attach a debugger to the process after it’s started to introspect it. I’ve often wanted to do something similar in Python; does anyone know of a tool that might do something like this?
- jessedhillon 6y ago
  Sibling comment suggests Pyrasite for this: https://github.com/lmacken/pyrasite https://github.com/lmacken/pyrasite
- walty8 6y ago
  may be python pdb? https://docs.python.org/3/library/pdb.html https://docs.python.org/3/library/pdb.html
  - saagarjha 6y ago
    Can I somehow give that a PID so it can inject a set_trace or whatever in the current context?
gesman 6y ago
pudb / pudb3
zmmmmm 6y ago
I feel like I got in a time machine ... this like debugging from 1995. In fact, pretty sure I would have rejected this in 1995. Unless you are constrained to doing this on a server or other restricted environment where reproducing locally is not possible, just get a good IDE and set some breakpoints in there.
- zo1 6y ago
  The environment is very "VSCode" focused at the moment, hence the roundabout and inefficient ways to do actual debugging. Debugging in a proper python IDE is a breeze and on-par if not better to other traditional IDEs for staticly-typed languages.
  - 1wd 6y ago
    VSCode is not the problem here https://code.visualstudio.com/docs/python/debugging https://code.visualstudio.com/docs/python/debugging
kabacha 6y ago
I've recently started integrating PyCharm's debugger more instead of python native tools and it really has been a life changer! My favorite tiny unique feature is that it injects variable values as temp comments in the code as the program runs in debug mode. So you're seeing `foo = get_foo() # "bar"` as your code. Really helps to debug complex algorithms when you can see the values right there in the code. My only problem is that debugger shell feels very clunky with ideaVim plugin, so if someone at JetBrains reading this that would be a very welcome improvement!
- AlexPl292 6y ago
  Could you please describe some cases?
  - nwatson 6y ago
    PyCharm is a huge help when dealing with APIs that return huge or complicated data structures or where documentation is vague. Examples: interacting with AWS with boto3; interacting with Kubernetes. You can read the docs alone, but much better is to set up a live interaction with the API. I set a breakpoint and explore the returned data structure in the debugger. All types and content are clearly visible and are a guide to the next iteration of the code's form.
    - phyrex 6y ago
      You're discovering REPL-based development ;)

501 points by alexeiz 6y ago

I use ipython a lot, and it has a great post-error debugger with `%debug` which drops you in a ipdb shell at the stackframe of the error. Concerning reloading, I find `%autoreload` much more reliable than `importlib.reload`. The latter can't handle `from lib import foo` or if a dependency inside an imported package changed. `%autoreload` just does this all automagically.

jsmeaton 6y ago
I’ve never had autoreload work for me - is there anything special to it? For context, I’m often working in a Django shell.

hathym 6y ago

if you are working with vscode, I highly recommand debugpy[1] or ptvsd[2] [1] https://github.com/microsoft/debugpy https://github.com/microsoft/debugpy [2] https://github.com/microsoft/ptvsd https://github.com/microsoft/ptvsd

fireattack 6y ago
Are they the one(s) that will be called if you use the "start debugging" in GUI with official Python extension? Or different one?
- 0x008 6y ago
  yes. vs code python extension uses debugpy.

oli5679 6y ago

IPython breakpoints are nice from IPython import embed embed()

drpancake 6y ago

I've been a Python developer for more than ten years. Approximately 98% of the time I need to debug something I use print() statements and this: import IPython; IPython.embed()

justmyname 6y ago
Oh, having full ipython during debug investigations is much better than using pure pdb.set_trace(). Thank you.
- drpancake 6y ago
  You might find this useful: https://github.com/gotcha/ipdb https://github.com/gotcha/ipdb
quickthrower2 6y ago
I've been a Python developer for less than 10 hours. Approximately 98% of the time debugged something was in Visual Studio. Quite a nice experience. I was lucky to have the project already set up for me.
trymas 6y ago
Can you step through the code when embedding ipython in your code? Does this approach have advantages versus `ipdb` or `pdb++` ?
- drpancake 6y ago
  No, but I rarely find it helpful to step through code. That's the other 2%.
  - jsmeaton 6y ago
    import ipdb; ipdb.set_trace() IPython pdb session.
    - 6y ago

_ZeD_ 6y ago

Do yourself a favour and learn how to use use a proper IDE. debugging in pydev shows you everything

albertzeyer 6y ago

To extend on inspecting stack traces: I found that I often wanted to know the the content of (relevant) (local) variables of a stack trace. Whenever I got a stacktrace, I would often restart it in a debugger, or add print statements on variables. Until I extended the stacktrace to just include this information. The tricky part is that printing all local variables is too much. And also global variables might be involved. Or attributes (e.g. self.x or so). So now I'm parsing all variables and attributes from the source code line from the traceback (in a very simple way) and print only those. This covers about 95% of all cases - i.e. in 95% of the case, the stack trace contains all information I need to understand and fix the problem. I published that here: https://pypi.org/project/better_exchook/ https://pypi.org/project/better_exchook/ https://github.com/albertz/py_better_exchook https://github.com/albertz/py_better_exchook I also often have a SIGUSR1 handler which will print the stacktrace of all threads. This is useful on long running processes, involving multi threading, where you might run into some strange hangs or deadlocks. In addition to that, if you might get crashes (segfault or so), something like faulthandler is useful. If you also want to see the C stack trace in addition in such cases, I load libSegFault.so, like here: https://github.com/rwth-i6/returnn/blob/5b8e34ec1fd725d0e20b5b422d213dbf17d9e069/Debug.py#L226 https://github.com/rwth-i6/returnn/blob/5b8e34ec1fd725d0e20b... Profiling is another topic. Something like Py-Spy (https://github.com/benfred/py-spy https://github.com/benfred/py-spy) can be very helpful. I also found a remote background ZMQ IPython/Jupyter kernel to be useful sometimes. I published that here: https://github.com/albertz/background-zmq-ipython https://github.com/albertz/background-zmq-ipython

lmilcin 6y ago

TLDR: Guide to python debugging that doesn't mention debugger and says "logging is a must".

nxpnsv 6y ago
Print out debugging is the new black. Actually quite useful at times though...
wodenokoto 6y ago
Actually it does mention the debugger

mjbrownie 6y ago

I recommend repr in all log messages too. For a work Django project we have common BaseModel repr(obj) expression that can be easily converted (using regexes in loki and promtail) into a clickable admin link. So within loki (or any similar log tools) all logger.info(repr(object)) is a link straight to the admin. Worth the set up time.

amaajemyfren 6y ago

> Most of the time, what programming really is - is just a lot of trial and error I am not the best programmer ... but I have always felt like an imposter with how many times I googled the next expression.

gorgoiler 6y ago

I’ve always admired pdb.set_trace() as a way of inserting a debugger breakpoint. I’m surprised it isn’t mentioned in the article. You don’t need a different UI for debugging — you can just write the breakpoint into the code. Conditional breakpoints can be done with Python logic! Marvelous!

diarrhea 6y ago

The article has a couple of problems in my eyes: - "you might want to just copy-paste it and use it" ... that's maybe what people end up doing, but it's a terrible idea. - The following `__repr__` is out of alignment with the output printed below it: class Circle: def __init__(self, x, y, radius): self.x = x self.y = y self.radius = radius def __repr__(self): return f"Rectangle({self.x}, {self.y}, {self.radius})" The class name is hardcoded; maybe use: def __repr__(self): name = self.__class__.__name__ return f"{name}({self.x}, {self.y}, {self.radius})" - There's a bare `except` in the `traceback` demo. It is not a problem in this toy example and works, but bare `except` statements have no place in educational code. Some poor soul will copy-paste it and run into awful errors down the line (https://realpython.com/the-most-diabolical-python-antipattern/ https://realpython.com/the-most-diabolical-python-antipatter...)

hywel 6y ago

Right now, this site is suffering a hug of death. This looks like the same article: https://dev.to/martinheinz/ultimate-guide-to-python-debugging-1739 https://dev.to/martinheinz/ultimate-guide-to-python-debuggin...

JimBlackwood 6y ago
It is the same article, it’s written by the same guy. In the article you linked it mentions “This article was originally posted on [url]”, which links to the same article this post links to.

qertoip 6y ago

Nice layout Sir.

memco 6y ago

Lots of great comments and resources here! I was surprised to find out how uncommon using a debugger for Python development was when I joined my team at work. I had to muddle through lots of documentation to figure out how to use it in certain situations and also to figure out alternative things when the first approach didn't work. I looked into PyCharm but opted to use VSCode as my main editor since the debugging support seemed easier to configure both for local and especially for remote. We have a flask app and one hurdle I had to overcome was our docs all assumed you'd run the app via gunicorn, but VSCode had trouble triggering breakpoints so I had to figure out how to run the app via the flask module directly, which was a bit of work since our app isn't following most of the getting started with flask tutorials conventions. For another project we use celery, VSCode can be used to debug celery tasks but it's also work to set up and in the end I found using the rdb.set_trace() debugging provided by celery was easier. An important lesson I learned while working on that project is that you need to be sure you're setting your breakpoints in the right version of your app: if you are using a setup.py install step the code you want to debug is probably somewhere in site-packages not wherever you installed it from. For another project, we're using python 2.7 on Centos 6: VSCode debugging & remote tools don't easily support that setup so knowing pdb or doing log debugging are the best bets. Something important on log-based debugging this article doesn't mention: if you're developing a long-running app like a daemon or service of some kind: you should probably make your log config loading dynamic. The one provided in the article is nice if your script is one-off, but if you are running a service you may want to be able to dynamically adjust the logging to be more verbose when an issue occurs and then reset it when done debugging without having to start/stop the whole service. I inherited code that runs as a service and in order to change the log level I have to stop the service, change the config and restart. The start/stop is destructive: if you stop the service the action it was performing has to be redone from the beginning. These are tasks that can take 8-12 hours so restarts are painful. Debug logs can be huge so you can't just leave the service in debug log mode all the time unless you want to fill up the whole disk. Which brings up another point: rotating log files—if you're doing heavy logging you need to be sure you have set up rotation so that you don't eat through disk space indiscriminately. In my ideal world every project I work on could have breakpoints trigger my editor's tools so I can inspect and alter code on the fly, but knowing there's more than one way to go about it and how to approach it when you don't have control is important.

yig 6y ago

My favorite visual debugger for Python and many other languages is CodeRunner, because it's so lightweight. (Mac-only, not free.) https://coderunnerapp.com/ https://coderunnerapp.com/

DodgyEggplant 6y ago

But there are good tools also. I'm using wing IDE for years, it has excellent debugging: multi threaded debug, remote debug with SSH (which works almost as fast as local debugging), deep introspection to anything on the stack and call chain, conditional breakpoints, powerful debug console that can hook to any step in the call chain of any thread, and debugging a server thread easily which is useful for debugging services and scenarios that can't be fully replicated on a local machine.

tjpnz 6y ago

I'm somewhat embarrassed to say that I very rarely use debuggers. Print statements have always worked for me even in large codebases. What am I missing here?

zwieback 6y ago
Maybe you're not missing anything - if you're productive with print debugging and have used a debugger but found that it doesn't help that much I wouldn't worry about it. I'm the opposite - immediately run in the debugger to probe around how to use this or that API or verify that the code is really doing what I think. I also find that stepping through my code early on often gives me insight in how to implement something more efficiently.
hansvm 6y ago
I'm just one data point, but I also rarely use debuggers and often work with a colleague who rarely uses print statements. There are qualitative differences in the code we're usually modifying though; in particular, he's often editing code where the control flow is hard to reason about (inheritance towers, pages of conditionals, etc) and is fairly slow to build and run. In that environment I might even reach for a debugger myself -- e.g., even if you already isolated the bug to a particular portion of the code and just wanted to know which method was being dispatched you would need a ton of print statements, or at least a lot of back-and-forths editing the code and figuring out where you ended up (because of the high branch complexity which could send you far from where you started), and those restarts would be fairly slow in and of themselves. Contrast that with a debugger where you can quickly step forward and find out exactly where you end up. IMO, print statements are easier/faster to use than debuggers, so I reach for those first unless there's some feature a debugger has that I think will make my life easier overall.

dangoljames 6y ago

This is wicked af, thank you so much!

Buetol 6y ago

This is buried in the comments but `pudb` is great debugger: https://pypi.org/project/pudb/ https://pypi.org/project/pudb/ - visual interface to set breakpoint - inline interpreter - inspect on exception It's like using a modern debugger but inside your console. Can't recommend enough.

ur-whale 6y ago
Does it support multi-threaded python code?
- Buetol 6y ago
  Looks like it does: https://medium.com/@auro_227/use-pudb-to-debug-python-multiprocessing-code-c0c5551d010c https://medium.com/@auro_227/use-pudb-to-debug-python-multip...

3pt14159 6y ago

Here's my list of helpful additions: 1. https://pypi.org/project/pdbpp/ https://pypi.org/project/pdbpp/ This is basically a better version of pdb. 2. ~/.pdbrc Useful for common commands that you would otherwise have to type out. For example, if you do TDD on a JSON API and often need to debug the output consider this alias: alias ppr pp response.json() 3. ~/.pdbrc.py This file loads before pdb starts, so it lets you customize it a bit and add things like history. Here is a simple version to get you started: https://gist.github.com/zachaysan/25712f3e9aa02a234437f89c60cad8d0 https://gist.github.com/zachaysan/25712f3e9aa02a234437f89c60...

petr25102018 6y ago

Shameless plug of some debugging options in Python: https://stribny.name/blog/2019/06/debugging-python-programs https://stribny.name/blog/2019/06/debugging-python-programs

kikoreis 6y ago

It misses the most important one-liner for python debugging: import pdb; pdb.set_trace() which leaves you in the debugger at a specific point in your code. When I actually did Python dev rare was the day I didn't crack that out!

css 6y ago
Or in 3.7 and beyond: breakpoint()
- kikoreis 6y ago
  OH!
mgalgs 6y ago
Or if you have ipython installed: import ipdb; ipdb.set_trace() which gets you a slightly fancier pdb.

bitpow 6y ago

There’s a typo in the __ repr__ code section

darkteflon 6y ago

There are loads of helpful comments in here for Python devs. Thanks, everyone.

mbeex 6y ago

Since PyCharm is abundant in this discussion, here are an alternative: I've been using the Wing IDE for several years. It sometimes chokes when I try to inspect larger numpy arrays as a whole, using the application-specific array visualization (but there are other ways). Otherwise I'm quite happy with it.

alexhutcheson 6y ago

If you’re not using an IDE, then I’d highly recommend trying PuDB: https://github.com/inducer/pudb/blob/master/README.rst https://github.com/inducer/pudb/blob/master/README.rst

theshetty 6y ago

I have a complete off the topic question, how do I create a slick blog like this - simple and minimal?

wjziv 6y ago
The writer has his blog sourcecode on Github. https://github.com/MartinHeinz/blog-backend https://github.com/MartinHeinz/blog-backend The frontend is built on Vue.js.

fudged71 6y ago

Can anyone recommend a video/series on Python debugging? Debugging has been my greatest struggle with getting into development. I get frustrated and spend over a day on simple scripts

chameleon_world 6y ago
https://realpython.com/python-debugging-pdb/ https://realpython.com/python-debugging-pdb/

ur-whale 6y ago

Slightly off topic, but: how does one debug a multi-threaded python app? PDB just doesn't seem to work at all.

milin 6y ago

https://microsoft.github.io/debug-adapter-protocol/ https://microsoft.github.io/debug-adapter-protocol/ looks really promising to bring standardized debugging across most ides and languages, which is awesome.

qwertox 6y ago

Why should my browser connect to a nonstandard port just to load a webpage? https://martinheinz.dev:1234/api/v1/posts/24 https://martinheinz.dev:1234/api/v1/posts/24

luord 6y ago

In six years working with python, I was never aware of `importlib.reload` I feel we're going to have a lot of fun together.

DanFeldman 6y ago

Great article, especially nice points about __repr__ helping logging. PDB as mentioned is great, but IPDB is at a whole 'nother level. If I'm able to get an ipdb breakpoint set in a tricky part of python code, I'm set. Tab-completion, some saved history, normal debugger operations, it's a dream-come true. I often set breakpoints and code iteratively w/ a repl-like interface or explore unfamiliar functions and objects.

Groxx 6y ago
ipdb is one of the nicest command-line debugging experiences I've had. It's wonderful.

_coveredInBees 6y ago

This is a bit of a weird article. It spends most of the time talking about logging, which is somewhat useful for debugging but not really. pdb gets a few lines of description, and that is about it. Personally, I can't live without Pycharm when working with Python purely because of how fantastic the debugging experience is. The integration with the interactive IPython shell is simply fantastic, and the live variable viewer is also really handy. Between being able to prototype code in the interactive IPython REPL and then graduate it seamlessly to scripts/functions/classes, and then having an amazing debugging UX to fallback on for those bugs that slip through, I find myself to be orders of magnitude more efficient than when coding in something like vscode (which I absolutely love, and want to transition to for Python coding, but it is extremely limited in comparison to Pycharm, especially when it comes to the REPL experience + debugging).

vonseel 6y ago
Yeah I agree entirely on Pychaem debugger being awesome. But it’s no substitute for logging in production. Somethings just need to be logged, either for proof or those situations where you just can’t reasonably attach a Pycharm debugger.
- underdeserver 6y ago
  Yes, logging is crucial for knowing what happened, but if you don't understand how it could have happened, step by step debugging is immensely useful. Assuming, of course, you can reproduce in your dev environment.
- jerzyt 6y ago
  Production logging with accurate and precise timestamp is an absolute time saver. That means like 1ms timestamp and NNTP synchronized servers.
- _coveredInBees 6y ago
  Oh yes, absolutely. I didn't mean to say logging isn't useful. It is vital to have good logging for production because like you said, you aren't just going to be able to attach a debugger. But for an article titled "Guide to Python Debugging", logging should be a short paragraph, not the core of the article imo. Debugging in production on the other hand... then logging makes a lot of sense to focus on.

ivan_ah 6y ago

Nice article. In particular the -i flag seems really useful. I'd like to add two packages I learned about recently that I like A LOT: pyrasite and PySnooper. Use pyrasite to "attach" to an existing running Python process (i.e. you forgot to instrument the app, but it has a bug in it, how can you debug?) (1) Install pyrasite (2) Get the PID of the running process (3) Run `pyrasite <pid> dumpstackz0.py` where dumpstackz0.pycontains `import traceback; traceback.print_stack()` (4) The stacktrace will be printed in the stdout/stderr of the running application Use PySnooper instead of print-debugging—just add a decorator to any python function to get detailed debug info like you normally would get from a full debugger: (A) code: https://github.com/cool-RR/PySnooper https://github.com/cool-RR/PySnooper (B) pypi: https://pypi.org/project/PySnooper/ https://pypi.org/project/PySnooper/ (C) talk: https://www.youtube.com/watch?v=XP5imOJc_TE https://www.youtube.com/watch?v=XP5imOJc_TE (D) podcast: https://www.youtube.com/watch?v=VUZvgJJKor0 https://www.youtube.com/watch?v=VUZvgJJKor0

macintux 6y ago
That -i flag trick would have saved me a lot of grief over the last year or so. Better late than never.

ed25519FUUU 6y ago

Don't forget breakpoints! Learn how to move around in the debugger, it will be a LIFESAVER. Above a troublesome line of code, drop this line: import pdb; pdb.set_trace() And run it as normally. Now you can explore. Drop into the function, step forward, go up. You just need to learn to move around. With pdb you can really explore what's going on.

karlicoss 6y ago
Or even just breakpoint() built-in in python 3.7+ In addition you can set PYTHONBREAKPOINT=ipdb.set_trace (or whatever you prefer) in your shell rc file
- wodenokoto 6y ago
  It has really bothered me that `breakpoint()` didn't start ipdb when running a script using ipython. I guess this is a good workaround!

MadNinjaSkills 6y ago

Nice! This is gonna be totally useful for me. Thanks!

saagarjha 6y ago

When debugging compiled code, I often attach a debugger to the process after it’s started to introspect it. I’ve often wanted to do something similar in Python; does anyone know of a tool that might do something like this?

jessedhillon 6y ago
Sibling comment suggests Pyrasite for this: https://github.com/lmacken/pyrasite https://github.com/lmacken/pyrasite
walty8 6y ago
may be python pdb? https://docs.python.org/3/library/pdb.html https://docs.python.org/3/library/pdb.html
- saagarjha 6y ago
  Can I somehow give that a PID so it can inject a set_trace or whatever in the current context?

gesman 6y ago

pudb / pudb3

zmmmmm 6y ago

I feel like I got in a time machine ... this like debugging from 1995. In fact, pretty sure I would have rejected this in 1995. Unless you are constrained to doing this on a server or other restricted environment where reproducing locally is not possible, just get a good IDE and set some breakpoints in there.

zo1 6y ago
The environment is very "VSCode" focused at the moment, hence the roundabout and inefficient ways to do actual debugging. Debugging in a proper python IDE is a breeze and on-par if not better to other traditional IDEs for staticly-typed languages.
- 1wd 6y ago
  VSCode is not the problem here https://code.visualstudio.com/docs/python/debugging https://code.visualstudio.com/docs/python/debugging

kabacha 6y ago

I've recently started integrating PyCharm's debugger more instead of python native tools and it really has been a life changer! My favorite tiny unique feature is that it injects variable values as temp comments in the code as the program runs in debug mode. So you're seeing `foo = get_foo() # "bar"` as your code. Really helps to debug complex algorithms when you can see the values right there in the code. My only problem is that debugger shell feels very clunky with ideaVim plugin, so if someone at JetBrains reading this that would be a very welcome improvement!

AlexPl292 6y ago
Could you please describe some cases?
- nwatson 6y ago
  PyCharm is a huge help when dealing with APIs that return huge or complicated data structures or where documentation is vague. Examples: interacting with AWS with boto3; interacting with Kubernetes. You can read the docs alone, but much better is to set up a live interaction with the API. I set a breakpoint and explore the returned data structure in the debugger. All types and content are clearly visible and are a guide to the next iteration of the code's form.
  - phyrex 6y ago
    You're discovering REPL-based development ;)