6 ms·

An underrated feature in Python 3

112 points by ionelm 12y ago

erikb 12y ago
What I don't understand about Python is why it's not printing the str() of the variables in the corresponding lines of the traceback. About 95% of all debugging starts with adding prints or logging messages before the lines mentioned in the traceback. Or am I doing something wrong here (since 5 years...)?
- VMG 12y ago
  The string representation (repr() would be a better fit actually) and the amount of local variables could be enormous.
- andreasvc 12y ago
  The str or repr of a variable might result in megabytes of output (or worse, calling it might have a side effect). Still, it'd be nice to at least show the first 100 bytes or so.
- Walkman 12y ago
  If you __str__ is broken, the traceback would fail which would be awful. Once I wrote a __str__ method on a class which got into an infinite loop :) When I showed the traceback to an advanced developer, he told me immediately what was happening. How could we figure out the problem if my __str__ was broken and traceback would fail? Also recently I struggled with py.test which automatically does that (calling str() on every local variable in the test case) but it is broken, so py.test showed me strange Unicode Decode exceptions (it turned out, nothing wrong was in my code but py.test) and I had no idea what was going on. Took me hours to figure it out py.test has a issues with __future__ unicode_literals... tl;dr: not a good idea
  - raldi 12y ago
    Doesn't seem like a showstopper to me; just fall back to current behavior if str() fails or takes too long, and truncate if it's too verbose.
    - Walkman 12y ago
      How you define "too long" and "too verbose"?
      - raldi 12y ago
        Keep a running tally of time spent stringifying. If it passes 1000 msec over the entire life of the program, disable the feature. As for length, maybe 100 chars? I think that's around what repr() often uses.
        Walkman 12y ago
        > Simple is better than complex. I think Python exceptions are verbose enough, not necessary to make it more verbose, more complicated.
        pmahoney 12y ago
        Can str() call into a C extension that then loops forever?
        moe 12y ago
        Can we contrive esoteric situations where this feature would fail spectacularly? Yes. Do they matter in reality? No. Make it an optional feature to be toggled with an interpreter flag. People like me can turn it on, you can keep it turned off.
        army 12y ago
        That doesn't seem that esoteric to me: in general the problem is that repr() can result in execution of arbitrary code, and that there's no way to cleanly and reliably terminate arbitrary code. This includes Python code: it's possible to terminate it at an arbitrary point, but that leaves things in an indeterminate state. I think it's pretty important that you shouldn't add features with unpredictable behaviour into the core of the language, particularly into error handling code.
- mlader 12y ago
  I typically will throw in a debugger instead of a print statement if I'm debugging locally. import ipdb; ipdb.set_trace() Then I'll explore the context variables and figure out what the hell is going on.
  - est 12y ago
    I think 99.9999% experienced python devs do this. Why can't PSF make it default? At least some kind of commandline switch to break to last traceback context by default. e.g. python --debug my_code.py
    - the_real_bto 12y ago
      they did: python -m pdb my_code.py
  - parfe 12y ago
    ipdb just made my day. Thanks.
- aurelianito 12y ago
  For one, I think repr is more fit. And if repr (or str) has a bug, now you are chasing two issues! Besides, repr (or str) may take a lot of processing time and it is not unusual to log an exception stack trace but keep running. At last, if you are going to debug, a lot of times is better to use a debugger instead of putting print (or log) statements all around. I prefer pydev for this. You can even attach to a python process running as a different user or even on a different computer. I documented how to do it in a post on my blog (sorry, in Spanish). http://aurelianito.blogspot.com.ar/2013/05/debugueando-en-root-con-pydev.html http://aurelianito.blogspot.com.ar/2013/05/debugueando-en-ro...
  - StavrosK 12y ago
    If you prefer the console, pudb is an absolutely fantastic curses debugger. I love it.
- masklinn 12y ago
  cgitb is your friend.
  - erikb 12y ago
    That one looks nice!
- the_real_bto 12y ago
  This is a great idea, although repr() would probably be better than str(). I've spent a lot of time debugging Python over the years. This honestly never occured to me until you mentioned it. Edit: Everyone else jumped in before me. I think it would be useful as a command line switch to python to turn on the behavior (maybe call it extended exceptions). Python would need to run the extended exception generation under a try/except that fell back to regular exceptions if an exception was raised while calling repr()
- babs474 12y ago
  How about automatically giving you a repl context at the exception and letting you move up and down the call stack. http://werkzeug.pocoo.org/docs/debug/#using-the-debugger http://werkzeug.pocoo.org/docs/debug/#using-the-debugger
the_mitsuhiko 12y ago
To be honest. As cool as this feature is in the 1% of cases where I wanted it, so annoying is it in 99% of all other cases. Tracebacks on Python 3 became a lot more complex because very often exceptions are intentionally swallowed to be reraised differently and they just mess up the traceback now. For instance any custom collection that acts as a decorator will now generate very large tracebacks. I would have much preferred if the reraising with old traceback would have been a feature you can enable per site where you raise.
- Walkman 12y ago
  I don't understand what are you talking about. AFAIK, you don't have to reraise the previous exception: In [1]: try: ...: raise Exception('Something is wrong') ...: except Exception as e: ...: raise ...: --------------------------------------------------------------------------- Exception Traceback (most recent call last) <ipython-input-1-ef6cea2fccac> in <module>() 1 try: ----> 2 raise Exception('Something is wrong') 3 except Exception as e: 4 raise 5 Exception: Something is wrong In [2]: try: raise Exception('Something is wrong with RERAISE') except Exception as e: raise e ...: --------------------------------------------------------------------------- Exception Traceback (most recent call last) <ipython-input-2-c6729aaebd3d> in <module>() 2 raise Exception('Something is wrong') 3 except Exception as e: ----> 4 raise e 5 <ipython-input-2-c6729aaebd3d> in <module>() 1 try: ----> 2 raise Exception('Something is wrong with RERAISE') 3 except Exception as e: 4 raise e 5 Exception: Something is wrong with RERAISE
  - the_mitsuhiko 12y ago
    > I don't understand what are you talking about. AFAIK, you don't have to reraise the previous exception: Correct, and also you can explicitly suppress the other traceback now (3.2+) I just wish it was the default unless you opted in. If I catch down an exception and raise another one I do not want the old one. Only in very rare cases am I interested in what lead to it. I have lots of hacks in Jinja2 now to hide useless tracebacks that would only confuse users. For instance a jinja context tries different dicts through a chain in __getitem__. The naive implementation on Python 3 generates two nearly equivalent tracebacks that are very confusing.
raldi 12y ago
Can a moderator de-hype the title, please? [Edit: thanks!]
- ionelm 12y ago
  What title would you like ? What's exactly hyped ?
  - raldi 12y ago
    "The most underrated feature" is total hype. It's exactly the sort of thing the Guidelines link at the bottom of every HN page says to remove. Maybe, "Python 3 exception-handling improvement."
    - captainmuon 12y ago
      Hmm, it's just a figure of speach. "The most underrated feature" -> "I really like this feature and think it deserves more attention." Just like "the best programming language ever" -> "a programming language I am really excited about, that fits my needs very well". The hyperbole is not the problem IMO, it's the nondescriptiveness. I think the title should have something about exception handling in it. The enforcement of the Guidelines has been pretty arbitrary, and quite often good (editorialized) titles have been turned back into something nondescript. Or now the other way around, the original title is editorialized to be less interesting. I think the most important criteria for a title are: does it get across the topic of the link, and the target page author's intention?
  - jordigh 12y ago
    Yeah, I agree. The title is total linkbait. It's like "5 Python Features You Must Use Now" or something.
    - rplnt 12y ago
      Or even better "blah blah you don't know about!"
viraptor 12y ago
Actually passing the exception is not required. Even if "e" is not an argument to BarException, the behaviour is the same.
dragonwriter 12y ago
Why is the current title ("An underrated feature in Python 3") neither the original source title ("The most underrated feature in Python 3") or a title more descriptive of the content (e.g., "Exception chaining in Python 3"). If we're going to get title changes, can we at least get useful title changes?
- hueving 12y ago
  >If we're going to get title changes, can we at least get useful title changes? No. If the title changes were useful, what would we discuss on HN?
whalesalad 12y ago
Had the pleasure of working with the author (Ionel) a few years back on some Django projects. Always love reading his Python posts. Really inspiring developer.
aturek 12y ago
I don't understand Python exceptions that well. But it's great to have the pattern of maintaining the stack trace when re-raising an exception. Thanks OP! (In case anyone missed it) except foo.FooException as e: raise BarException, BarException(e), sys.exc_info()[2]

112 points by ionelm 12y ago

erikb 12y ago

What I don't understand about Python is why it's not printing the str() of the variables in the corresponding lines of the traceback. About 95% of all debugging starts with adding prints or logging messages before the lines mentioned in the traceback. Or am I doing something wrong here (since 5 years...)?

VMG 12y ago
The string representation (repr() would be a better fit actually) and the amount of local variables could be enormous.
andreasvc 12y ago
The str or repr of a variable might result in megabytes of output (or worse, calling it might have a side effect). Still, it'd be nice to at least show the first 100 bytes or so.
Walkman 12y ago
If you __str__ is broken, the traceback would fail which would be awful. Once I wrote a __str__ method on a class which got into an infinite loop :) When I showed the traceback to an advanced developer, he told me immediately what was happening. How could we figure out the problem if my __str__ was broken and traceback would fail? Also recently I struggled with py.test which automatically does that (calling str() on every local variable in the test case) but it is broken, so py.test showed me strange Unicode Decode exceptions (it turned out, nothing wrong was in my code but py.test) and I had no idea what was going on. Took me hours to figure it out py.test has a issues with __future__ unicode_literals... tl;dr: not a good idea
- raldi 12y ago
  Doesn't seem like a showstopper to me; just fall back to current behavior if str() fails or takes too long, and truncate if it's too verbose.
  - Walkman 12y ago
    How you define "too long" and "too verbose"?
    - raldi 12y ago
      Keep a running tally of time spent stringifying. If it passes 1000 msec over the entire life of the program, disable the feature. As for length, maybe 100 chars? I think that's around what repr() often uses.
      - Walkman 12y ago
        > Simple is better than complex. I think Python exceptions are verbose enough, not necessary to make it more verbose, more complicated.
      - pmahoney 12y ago
        Can str() call into a C extension that then loops forever?
        moe 12y ago
        Can we contrive esoteric situations where this feature would fail spectacularly? Yes. Do they matter in reality? No. Make it an optional feature to be toggled with an interpreter flag. People like me can turn it on, you can keep it turned off.
        army 12y ago
        That doesn't seem that esoteric to me: in general the problem is that repr() can result in execution of arbitrary code, and that there's no way to cleanly and reliably terminate arbitrary code. This includes Python code: it's possible to terminate it at an arbitrary point, but that leaves things in an indeterminate state. I think it's pretty important that you shouldn't add features with unpredictable behaviour into the core of the language, particularly into error handling code.
mlader 12y ago
I typically will throw in a debugger instead of a print statement if I'm debugging locally. import ipdb; ipdb.set_trace() Then I'll explore the context variables and figure out what the hell is going on.
- est 12y ago
  I think 99.9999% experienced python devs do this. Why can't PSF make it default? At least some kind of commandline switch to break to last traceback context by default. e.g. python --debug my_code.py
  - the_real_bto 12y ago
    they did: python -m pdb my_code.py
- parfe 12y ago
  ipdb just made my day. Thanks.
aurelianito 12y ago
For one, I think repr is more fit. And if repr (or str) has a bug, now you are chasing two issues! Besides, repr (or str) may take a lot of processing time and it is not unusual to log an exception stack trace but keep running. At last, if you are going to debug, a lot of times is better to use a debugger instead of putting print (or log) statements all around. I prefer pydev for this. You can even attach to a python process running as a different user or even on a different computer. I documented how to do it in a post on my blog (sorry, in Spanish). http://aurelianito.blogspot.com.ar/2013/05/debugueando-en-root-con-pydev.html http://aurelianito.blogspot.com.ar/2013/05/debugueando-en-ro...
- StavrosK 12y ago
  If you prefer the console, pudb is an absolutely fantastic curses debugger. I love it.
masklinn 12y ago
cgitb is your friend.
- erikb 12y ago
  That one looks nice!
the_real_bto 12y ago
This is a great idea, although repr() would probably be better than str(). I've spent a lot of time debugging Python over the years. This honestly never occured to me until you mentioned it. Edit: Everyone else jumped in before me. I think it would be useful as a command line switch to python to turn on the behavior (maybe call it extended exceptions). Python would need to run the extended exception generation under a try/except that fell back to regular exceptions if an exception was raised while calling repr()
babs474 12y ago
How about automatically giving you a repl context at the exception and letting you move up and down the call stack. http://werkzeug.pocoo.org/docs/debug/#using-the-debugger http://werkzeug.pocoo.org/docs/debug/#using-the-debugger

the_mitsuhiko 12y ago

To be honest. As cool as this feature is in the 1% of cases where I wanted it, so annoying is it in 99% of all other cases. Tracebacks on Python 3 became a lot more complex because very often exceptions are intentionally swallowed to be reraised differently and they just mess up the traceback now. For instance any custom collection that acts as a decorator will now generate very large tracebacks. I would have much preferred if the reraising with old traceback would have been a feature you can enable per site where you raise.

Walkman 12y ago
I don't understand what are you talking about. AFAIK, you don't have to reraise the previous exception: In [1]: try: ...: raise Exception('Something is wrong') ...: except Exception as e: ...: raise ...: --------------------------------------------------------------------------- Exception Traceback (most recent call last) <ipython-input-1-ef6cea2fccac> in <module>() 1 try: ----> 2 raise Exception('Something is wrong') 3 except Exception as e: 4 raise 5 Exception: Something is wrong In [2]: try: raise Exception('Something is wrong with RERAISE') except Exception as e: raise e ...: --------------------------------------------------------------------------- Exception Traceback (most recent call last) <ipython-input-2-c6729aaebd3d> in <module>() 2 raise Exception('Something is wrong') 3 except Exception as e: ----> 4 raise e 5 <ipython-input-2-c6729aaebd3d> in <module>() 1 try: ----> 2 raise Exception('Something is wrong with RERAISE') 3 except Exception as e: 4 raise e 5 Exception: Something is wrong with RERAISE
- the_mitsuhiko 12y ago
  > I don't understand what are you talking about. AFAIK, you don't have to reraise the previous exception: Correct, and also you can explicitly suppress the other traceback now (3.2+) I just wish it was the default unless you opted in. If I catch down an exception and raise another one I do not want the old one. Only in very rare cases am I interested in what lead to it. I have lots of hacks in Jinja2 now to hide useless tracebacks that would only confuse users. For instance a jinja context tries different dicts through a chain in __getitem__. The naive implementation on Python 3 generates two nearly equivalent tracebacks that are very confusing.

raldi 12y ago

Can a moderator de-hype the title, please? [Edit: thanks!]

ionelm 12y ago
What title would you like ? What's exactly hyped ?
- raldi 12y ago
  "The most underrated feature" is total hype. It's exactly the sort of thing the Guidelines link at the bottom of every HN page says to remove. Maybe, "Python 3 exception-handling improvement."
  - captainmuon 12y ago
    Hmm, it's just a figure of speach. "The most underrated feature" -> "I really like this feature and think it deserves more attention." Just like "the best programming language ever" -> "a programming language I am really excited about, that fits my needs very well". The hyperbole is not the problem IMO, it's the nondescriptiveness. I think the title should have something about exception handling in it. The enforcement of the Guidelines has been pretty arbitrary, and quite often good (editorialized) titles have been turned back into something nondescript. Or now the other way around, the original title is editorialized to be less interesting. I think the most important criteria for a title are: does it get across the topic of the link, and the target page author's intention?
- jordigh 12y ago
  Yeah, I agree. The title is total linkbait. It's like "5 Python Features You Must Use Now" or something.
  - rplnt 12y ago
    Or even better "blah blah you don't know about!"

viraptor 12y ago

Actually passing the exception is not required. Even if "e" is not an argument to BarException, the behaviour is the same.

dragonwriter 12y ago

Why is the current title ("An underrated feature in Python 3") neither the original source title ("The most underrated feature in Python 3") or a title more descriptive of the content (e.g., "Exception chaining in Python 3"). If we're going to get title changes, can we at least get useful title changes?

hueving 12y ago
>If we're going to get title changes, can we at least get useful title changes? No. If the title changes were useful, what would we discuss on HN?

whalesalad 12y ago

Had the pleasure of working with the author (Ionel) a few years back on some Django projects. Always love reading his Python posts. Really inspiring developer.

aturek 12y ago

I don't understand Python exceptions that well. But it's great to have the pattern of maintaining the stack trace when re-raising an exception. Thanks OP! (In case anyone missed it) except foo.FooException as e: raise BarException, BarException(e), sys.exc_info()[2]