4 ms·

> Python 3.11 is up to 10-60% faster than Python 3.10. Nice! > PEP 657 – Include Fine Grained Error Locations in Tracebacks Hmm, what’s this? Traceback

by teddyh 4y ago

> Python 3.11 is up to 10-60% faster than Python 3.10. Nice! > PEP 657 – Include Fine Grained Error Locations in Tracebacks Hmm, what’s this? Traceback (most recent call last): File "test.py", line 2, in <module> x['a']['b']['c']['d'] = 1 ~~~~~~~~~~~^^^^^ TypeError: 'NoneType' object is not subscriptable Yessss

deleted 4y ago
[deleted]
wodenokoto 4y ago
Does that work for chained pandas expressions too? I love writing chained expressions but debugging them is like visiting a special kind of hell.
- iruoy 4y ago
  > This highlighting will occur for every frame in the traceback. For instance, if a similar error is part of a complex function call chain, the traceback would display the code associated to the current instruction in every frame: Traceback (most recent call last): File "test.py", line 14, in <module> lel3(x) ^^^^^^^ File "test.py", line 12, in lel3 return lel2(x) / 23 ^^^^^^^ File "test.py", line 9, in lel2 return 25 + lel(x) + lel(x) ^^^^^^ File "test.py", line 6, in lel return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e) ~~~~~~~~~~~~~~~~^^^^^ TypeError: 'NoneType' object is not subscriptable
- hnews_account_1 4y ago
  All my data devs please stand up! I sometimes sacrifice readability just because I hate creating variables. But then if it affects debugging times, my boss would be furious. As such, I use a full debugger anyway so I can trace quickly.
- emehex 4y ago
  If you like writing chain-able pandas, you should check out: https://github.com/maxhumber/redframes https://github.com/maxhumber/redframes
  - IanOzsvald 4y ago
    What's your plan for redframes? It looks really new? I'm the co-author of O'Reilly's High Performance Python so I'm always on the lookout for pandas alternatives. Are you looking at speed implications too? Bigger-than-RAM use cases? "Easier than pandas" only (which, of course, is totally huge in its own right)?
    - emehex 4y ago
      "Easier than pandas" is the most important consideration for right now! If pandas is "Jira", I consider redframes to be "Trello" (appropriate for most, but not all use cases). I actually was inspired to build the library after teaching a one week intensive pandas course to a couple of Data Scientists @ a Fortune 500... (pandas is really hard for beginners!) While performance and OOM aren't priorities right now, I'd love to one day replace the pandas "backend" with Arrow (or something else) once I nail the API :)
    - bbkane 4y ago
      Ian, what are your thoughts on https://www.pola.rs/ https://www.pola.rs/ ? I've used it for a few CSV tasks and I really liked the API
      - IanOzsvald 4y ago
        I interviewed author Ritchie Vink on my newsletter (NotANumber) some months back, he's smart and the library has a nice design. I still barely know anyone trying it, it did get a write up just recently here: https://news.ycombinator.com/item?id=32998040 https://news.ycombinator.com/item?id=32998040 On one of my recent Higher Performance training courses a hedge fund person said they'd tried it with mixed success - some things faster than Pandas, others more expensive on RAM. I'm hoping to have a play soon but have only lightly tinkered so far. The parallel-by-design nature is nice, but I think the API is still evolving rapidly making it harder to develop with.
pydry 4y ago
As an encore they could bring pytest style assert errors into core.
- jononor 4y ago
  Yes, this would save me so much time. And make asserts so much more valuable, including checking of pre- and post-conditions
pknerd 4y ago
I wonder how much it is faster than other versions > 3.7?
- IanOzsvald 4y ago
  Here's a benchmark for 3.8-3.11b: https://www.phoronix.com/review/python-311-benchmarks/4 https://www.phoronix.com/review/python-311-benchmarks/4 The geometric mean of the 3.8 to 3.11b benchmarks was a 45% speedup.
rat87 4y ago
I'm worried about the speedup My understanding is that it's based on the most recent attempt to remove the GIL by Sam Gross https://github.com/colesbury/nogil https://github.com/colesbury/nogil In addition to some ways to try to not have nogil have as much overhead he added a lot of unrelated speed improvements so that python without the gil would still be faster not slower in single thread mode. They seem to have merged those performance patches first that means if they add his Gil removal patches in say python 3.12 it will still be substantially slower then 3.11 although faster then 3.10. I hope that doesn't stop them from removing the gil (at least by default)
- kzrdude 4y ago
  I hope Mark Shannon can work constructively with Sam and that they can make nogil happen

by teddyh 4y ago

deleted 4y ago

[deleted]

wodenokoto 4y ago

Does that work for chained pandas expressions too? I love writing chained expressions but debugging them is like visiting a special kind of hell.

iruoy 4y ago
> This highlighting will occur for every frame in the traceback. For instance, if a similar error is part of a complex function call chain, the traceback would display the code associated to the current instruction in every frame: Traceback (most recent call last): File "test.py", line 14, in <module> lel3(x) ^^^^^^^ File "test.py", line 12, in lel3 return lel2(x) / 23 ^^^^^^^ File "test.py", line 9, in lel2 return 25 + lel(x) + lel(x) ^^^^^^ File "test.py", line 6, in lel return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e) ~~~~~~~~~~~~~~~~^^^^^ TypeError: 'NoneType' object is not subscriptable
hnews_account_1 4y ago
All my data devs please stand up! I sometimes sacrifice readability just because I hate creating variables. But then if it affects debugging times, my boss would be furious. As such, I use a full debugger anyway so I can trace quickly.
emehex 4y ago
If you like writing chain-able pandas, you should check out: https://github.com/maxhumber/redframes https://github.com/maxhumber/redframes
- IanOzsvald 4y ago
  What's your plan for redframes? It looks really new? I'm the co-author of O'Reilly's High Performance Python so I'm always on the lookout for pandas alternatives. Are you looking at speed implications too? Bigger-than-RAM use cases? "Easier than pandas" only (which, of course, is totally huge in its own right)?
  - emehex 4y ago
    "Easier than pandas" is the most important consideration for right now! If pandas is "Jira", I consider redframes to be "Trello" (appropriate for most, but not all use cases). I actually was inspired to build the library after teaching a one week intensive pandas course to a couple of Data Scientists @ a Fortune 500... (pandas is really hard for beginners!) While performance and OOM aren't priorities right now, I'd love to one day replace the pandas "backend" with Arrow (or something else) once I nail the API :)
  - bbkane 4y ago
    Ian, what are your thoughts on https://www.pola.rs/ https://www.pola.rs/ ? I've used it for a few CSV tasks and I really liked the API
    - IanOzsvald 4y ago
      I interviewed author Ritchie Vink on my newsletter (NotANumber) some months back, he's smart and the library has a nice design. I still barely know anyone trying it, it did get a write up just recently here: https://news.ycombinator.com/item?id=32998040 https://news.ycombinator.com/item?id=32998040 On one of my recent Higher Performance training courses a hedge fund person said they'd tried it with mixed success - some things faster than Pandas, others more expensive on RAM. I'm hoping to have a play soon but have only lightly tinkered so far. The parallel-by-design nature is nice, but I think the API is still evolving rapidly making it harder to develop with.

pydry 4y ago

As an encore they could bring pytest style assert errors into core.

jononor 4y ago
Yes, this would save me so much time. And make asserts so much more valuable, including checking of pre- and post-conditions

pknerd 4y ago

I wonder how much it is faster than other versions > 3.7?

IanOzsvald 4y ago
Here's a benchmark for 3.8-3.11b: https://www.phoronix.com/review/python-311-benchmarks/4 https://www.phoronix.com/review/python-311-benchmarks/4 The geometric mean of the 3.8 to 3.11b benchmarks was a 45% speedup.

rat87 4y ago

I'm worried about the speedup My understanding is that it's based on the most recent attempt to remove the GIL by Sam Gross https://github.com/colesbury/nogil https://github.com/colesbury/nogil In addition to some ways to try to not have nogil have as much overhead he added a lot of unrelated speed improvements so that python without the gil would still be faster not slower in single thread mode. They seem to have merged those performance patches first that means if they add his Gil removal patches in say python 3.12 it will still be substantially slower then 3.11 although faster then 3.10. I hope that doesn't stop them from removing the gil (at least by default)

kzrdude 4y ago
I hope Mark Shannon can work constructively with Sam and that they can make nogil happen