53 ms·

What's New in Python 3.11?

322 points by sanketsaurav 4y ago

belval 4y ago
Does anyone actually use asyncio for processing stuff? It's been around for a while now but I still find it very awkward compared to just creating a pool and calling map or starmap. Am I missing out?
- heavyset_go 4y ago
  All the time now for IO-bound workloads. It was very awkward a few years ago, but things like asyncio.run(), asyncio.gather(), asyncio.to_thread(), and asyncio.create_task() have made things less awkward. TaskGroups should make things even less awkward, too.
  - mrfusion 4y ago
    Why not just use multi processing?
    - objektif 4y ago
      Yes curious why this package is needed at all?
      - heavyset_go 4y ago
        Single-threaded execution of IO-bound work can be faster than breaking it up between threads or processes, and it can use a lot less resources. Then there are the preemptive vs cooperative multitasking concerns and the pros and cons of processes/threads vs light-weight threads/coroutines/etc. Some IO-bound workloads are suited really well by the asyncio model, while other workloads might be better suited for processes and threads. They're three separate tools whose use cases might be similar, but they're not necessarily replacing one another. Multiple processes still have their place even while asyncio exists and vice versa.
    - nhumrich 4y ago
      If your task is io bound, aka, lots of network stuff, multi process is overkill. Also, asyncio can handle a _lot_ more tasks 100,000's as opposed to 10s. So it really shines in heavy io things. Also, multiprocessing can not share memory, and that can be a pretty busy g disadvantage depending on the task.
    - zackees 4y ago
      Probably because async io generally scales better.
    - hprotagonist 4y ago
      because pickle is an absolute shitshow.
    - quietbritishjim 4y ago
      One reason is that you often don't need to use locks. Between lines containing await (or async for or async with), you can be sure that this task won't be pre-empted to run another async task. Another reason, if you're using the Trio async library, is that managing and cancelling multiple tasks is really easy, and you can be sure that none get lost. This update to Python brings some of that to core asyncio (but I'll stick with Trio for now, thanks).
    - est 4y ago
      Why not just use Gevent?
      - d0mine 4y ago
        Explicit (await) vs. implicit (anything that uses patched I/O deep down) switch. Essentially, it makes the reasoning about the code almost as hard as with preemptive threading.
    - smitty1e 4y ago
      Asycio is really a list of tasks in the main interpreter thread. This is far easier to work with than multiprocessing. When doing, e.g. AWS work, multiprocessing is additional pain for little gain. Maybe 3.11 will make threads less painful.
      - heavyset_go 4y ago
        From what I've been following, it's going to take another release (or several) after 3.11 for Node-like worker threads to land in Python.
        smitty1e 4y ago
        I'm not deep on Node, but this sounds like a GIL thing.
        heavyset_go 4y ago
        It is, and it's a subinterpreters thing. Node has separate interpreters that run in worker threads but also share memory, which is the route Python is planning on taking.
    - heavyset_go 4y ago
      It can use less resources and can be faster than passing data between processes. There's really no reason to use multiple processes for IO-bound workloads, either, and even for some IO-bound workloads, a thread pool can be faster than processes. The second you aren't just passing raw bytes around, you have to take into consideration what can and can't be sent between processes in Python, as some objects can't be pickled and thus can't be passed between processes. You can concurrently load a lot more coroutines than processes and threads, as well.
    - hermitdev 4y ago
      Asyncio actually plays really nicely with multiprocessing, too. The concurrent.futures.ProcessPoolExecutor can handle running tasks in child processes and handles the communication seemlessly for you. I've used it quite a bit. Can easily use all 32 cores on my server this way without the GIL getting in the way.
    - dragonwriter 4y ago
      > Why not just use multi processing? Multiprocessing provides parallelism up to what the machine supports, but no additional degree of concurrency, asyncio provides a fairly high degree of concurrency, but no parallelism. OF course, you can use them together to get both. https://github.com/omnilib/aiomultiprocess https://github.com/omnilib/aiomultiprocess
    - witcher_rat 4y ago
      One reason is if you need to launch a subprocess with a timeout but don't want to use up CPU in the python script while that subprocess runs. The regular subprocess module will busy-loop in such cases, consuming CPU, while asyncio's does not. The docs even warn about this for subprocess and suggest using asyncio to avoid it, although the docs are misleading - it only busy-loops if the timeout is not None, and only when running on Mac/Linux not Windows.
- d0mine 4y ago
  Cooperative multitasking is much simpler to reason about than the general preemptive multithreading. pool+map is useful but it is not general.
- Sohcahtoa82 4y ago
  It depends...are you looking for concurrency or parallelism? The two are similar, but not the same. Using multiprocessing creates a LOT of overhead and is really intended for parallelism and is only the better solution if you're CPU-bound. If you're I/O-bound, then what you're really looking for is concurrency, and using asyncio will be more performant.
beisner 4y ago
Variadic Generics has the potential to enable all the scientific computing libraries to provide really nice type hints. One step closer to eliminating insidious broadcasting bugs… or running training for 10 hrs only to have a validation loop fail because of a shape mismatch.
- deleted 4y ago
  [deleted]
- MichaelMoser123 4y ago
  i would say it makes sense for a typed language, not sure if you can have it both in python. Why do they keep adding features upon features to the syntax of poor python? do they want to turn it into another kind of c++?
  - beisner 4y ago
    It's an optional typechecking tool. Speaking as an ML researcher who has authored my fair share of linear algebra bugs due to broadcasting, being able to optionally typecheck specific critical regions of the codebase would be a killer feature.
  - bbkane 4y ago
    I think they want Python to fit into every niche, like C++, which means a lot of knobs to turn
- davidatbu 4y ago
  THIS. OH MY GOSH. THIS. The only issue now is that pytorch has their own "shapes" solution[0], and last I checked, were kinda reserved about participating in the standardization of variadic generics because they don't expect to use it. I really really hope that the ML community comes together to use variadic generics because I believe it will save researchers and devs so many debug cycles (as well as compute resources, tbh). [0] https://pytorch.org/docs/stable/named_tensor.html https://pytorch.org/docs/stable/named_tensor.html
- aix1 4y ago
gpm 4y ago
I love that they're improving the error messages, that looks like a huge productivity improvement for beginners and experts alike.
di 4y ago
Straight from the horse's mouth, if you prefer: https://docs.python.org/3.11/whatsnew/3.11.html https://docs.python.org/3.11/whatsnew/3.11.html
- rjh29 4y ago
  I do, thank you for the link.
barosl 4y ago
The addition of a TOML parser to the standard library is really welcome. I've always wanted to use TOML in my Python scripts but having to import an external library made me use JSON instead, which was awkward due to the lack of comments.
- mixmastamyk 4y ago
  .ini configparser worked fine for decades, especially as a simple project config.
  - barosl 4y ago
    PEP 518 lists some reasons not to use configparser and I agree with them wholeheartedly.[1] Personally I don't want to use ill-defined formats, even for simple projects. [1] https://peps.python.org/pep-0518/#configparser https://peps.python.org/pep-0518/#configparser
    - mixmastamyk 4y ago
      Their reason was obsolete shortly after it was written, Python 2 was EOL years before TOML included. Toml is ill-defined as well. ini files work fine for these trivial uses. https://hitchdev.com/strictyaml/why-not/toml/ https://hitchdev.com/strictyaml/why-not/toml/ So now you have two flawed ways to do it, congrats.
    - ptx 4y ago
      The only reason offered is that there are (or might be) differences between version of Python, mainly 2 and 3, as far as I can see. Python 2 is EOL, so that's no longer a concern. As for differences between Python 3 releases, isn't there a fairly large difference in TOML support as well, since in versions before 3.11 it doesn't work at all? Wouldn't specifying the behaviour of the INI parser as whatever 3.11 is doing (and raising an exception on earlier versions) amount to essentially the same thing?
  - bulatb 4y ago
    Newer packaging-related PEPs require TOML, and many developer tools use TOML by choice, but a fresh install of Python couldn't actually read it. That required workarounds from tool and library maintainers, making it more complicated to support as many users' preferences and ways of working as possible. However you prefer to work, your tools may now be easier to test and maintain. This change is good for everyone.
throw0101a 4y ago
Crickey: > Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.25x speedup on the standard benchmark suite. See Faster CPython for details. * https://docs.python.org/3.11/whatsnew/3.11.html https://docs.python.org/3.11/whatsnew/3.11.html * https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpyth...
- jamesfinlayson 4y ago
  Among other things: "In Python 3.11, the core modules essential for Python startup are “frozen”. This means that their code objects (and bytecode) are statically allocated by the interpreter." Very nice.
  - throwaway81523 4y ago
    I'd prefer that they had done something like unexec to allow loading some arbitrary set of modules into a bootstrapping image, then dumping a runnable executable that would launch with all those modules already present. The old Emacs approaches are now mostly busted, but there are alternatives.
    - eslaught 4y ago
      Ye olde Lisp dump image. A neat approach, but results in an absolutely horrific cross compilation experience. As someone who worked briefly on a Common Lisp implementation, including trying to get it to cross-compile, I'm very glad this technique hasn't caught on in the broader community.
      - throwaway81523 4y ago
        Yeah, it is pretty unworkable now, and Emacs uses an object serialization approach that still works pretty well, but there are other methods too. There is some OpenVZ-derived way to make process snapshots in Linux whose name I've forgotten, but I've been wanting to look into it for this.
deleted 4y ago
[deleted]
bulatb 4y ago
This unblocks or simplifies so many of my projects. Variadic generics, better exceptions, and TOML in the standard library are huge, self-types are great, and extra speed is always appreciated. Thanks to everyone involved.
- daenz 4y ago
  Can you elaborate on how your projects are unblocked? My reading of it was that these improvements are nice, but nothing groundbreaking.
  - bulatb 4y ago
    I wouldn't quite say they're gamechangers, but they're exactly what I happened to need. Variadic generics will (I hope) be very nice for typing pre-2.0 SQLAlchemy code. `Query[*Ts]` or similar would be a natural type for many values from the ORM API, and hopefully allow expressing types for query transformations like adding or removing columns, joining to specific query shapes, adding arbitrary subqueries, etc. The self-type might unblock a testing tool I haven't shipped because the DX isn't where I want it. IIRC: normal TypeVar binding plus the TypeVar hack for self-types, interacting with Mypy's particular way of detecting the type of a descriptor, interacting with the way the project uses mixins, sometimes produce types that technically aren't wrong but lead to spurious type errors in user-defined data fixtures. So I'm probably the only one on earth with this particular problem. I don't know, it's been a while. Maybe it won't work. But the self-type hack is noisy anyway and getting rid of it is nice. Shipping TOML in the standard library means I can drop a bunch of Tox gymnastics in projects that support pyproject.toml or TOML-based configuration in addition to the older standards, and now I'm not stuck either forcing certain TOML libraries on people or working out a way to plug in their choice. It's not that hard, but complexity multiplies, and now it's one less thing to deal with.
salmo 4y ago
I’ll just say. Since about 3.6/3.8 the language has been growing huge. Like C++ huge. I liked f-strings. Asyncio has its place but is waaay overused. Typing was a cool idea, but so awkward in practice. All of it has really turned me off, personally. It’s getting to where reading libraries is just painful. The simplicity of the language was really beautiful and when I wanted type safety, etc. I’d use something else. I can still write simple Python, but it’s more all the other code I need to grok. I find myself going to Go more and more for stuff I used to use Python for. It’s easier to set up a dev environment for other people, easier to distribute code, and gives me type safety and concurrency as first class citizens while being a very small language. I miss dictionary comprehensions and other shortcuts from time to time. But it actually feels more ergonomic now. All personal opinion. I’m fine shifting languages. I still write C from time to time to bang bits. I play with various LISPs to exercise my brain. I never used Python for performance critical code. I guess ML is changing that need. But I know others that want Java to have functional features and Python to have this stuff. I’m not saying I’m “right”, just uncomfortable in a place I used to love to hang out in.
- Night_Thastus 4y ago
  As someone who mainly sits in the C++ and not the Python space - why? Can't you still use all those old features just as you did before? If you don't like the complexity of newer libraries and features, they're still optional aren't they? Or did something fundamental shift (aside from the obvious P2->P3 switch) that has made existing features harder to use?
  - Galanwe 4y ago
    > If you don't like the complexity of newer libraries and features, they're still optional aren't they? Chances are you're not programming in a vacuum. These features will be used by third party libraries and you will have to read/understand this code. Even in your in house codebase, there will always be co-workers that want to use these features. The optionality of language features is not real. If they exist, they will be used, you will have to understand them, they will leak from other libraries, etc.
6gvONxR4sf7o 4y ago
Better error messages is something that literally every single language/framework/library should prioritize. It improves users lives by more than almost everything else besides correctness. It’s been great to see the improvements in python.
- chrisma0 4y ago
  I agree that the exact error locations in tracebacks seem like the most impactful change here. Particularly in the light that Python is often used as an introductory language and unhelpful error messages are incredibly frustrating. Like this it feels like they can become more of a tool to improve!
- dominotw 4y ago
  best i've seen so far are in elm lang.
- drited 4y ago
  Totally agree! It's hard to fathom why so many languages leave you to be some kind of detective trying to determine exactly where the issue is rather than just telling you.
- dazzawazza 4y ago
  .. and they have the highest impact when you are starting to learn a language. Many a student turns away from a language because the developers underestimate the importance of error messages.
asicsp 4y ago
The `re` module in 3.11 will support "Atomic grouping and possessive quantifiers". I recently wrote a blog post about these features: https://learnbyexample.github.io/python-regex-possessive-quantifier/ https://learnbyexample.github.io/python-regex-possessive-qua...
TeeMassive 4y ago
Glad to see they ported Anyio's TaskGroups (or was it from trio?) to the stdlib. Starting tasks so they run concurrently was a huge pain in vanilla Python.
- tusharsadhwani 4y ago
  The idea came from trio's nurseries I believe.
- erdewit 4y ago
  Is the TaskGroup really in the 3.11 release though? It's not to be found anywhere in the Python documentation or release notes: https://docs.python.org/3.11/whatsnew/3.11.html https://docs.python.org/3.11/whatsnew/3.11.html It has been talked about for a long time, probably 4 years now.
  - Mehdi2277 4y ago
    It's implemented in 3.11. Documentation still hasn't been written for it yet though and that's why the issue is still open, https://github.com/python/cpython/issues/90908 https://github.com/python/cpython/issues/90908.
j16sdiz 4y ago
The note in exception design is interesting. This allow more contextual information to be embedded.
- stinos 4y ago
  Embedding more information isn't really the problem: alternatives have always existed, it's just that none of them were as brief and good as this one. E.g. normally you'd catch then re-raise another exception with a new message with extra information and the original one as inner exception. The issue with that is that the resulting message/stacktrace isn't as readable as these notes.
assignmenthelp0 4y ago
davidatbu 4y ago
I'm really excited for where Python's static typing is heading! I doubt that typing Python code will ever become as ergonomic as typescript, simply because typescript doesn't have the constraint of modifying the syntax of it's "target language" (JS) when it wants to add a feature or whatever, whereas typed python must still be Python, and syntactic changes therefore need to be added much more conservatively. But still, the typing story is improving before my very eyes with every release, and given that Python types are actually available at runtime (allowing for things like pydantic and FastAPI), and that Python has massive adoption in the ML space, all this is making Python a very enticing language to get back to using!
- cdrini 4y ago
  I love that Python has types available at runtime! The ability to have those type-generated CLIs or APIs is fantastic. I learned from this article about functools.singledispatch, and can't wait to use it. I've always wanted something like in typescript! But `functools.singledispatch` is just not possible in TS. I think working with dicts is still a little clunky in python, and there are some rough edges (eg mypy being a little weaker), but I genuinely think I like types in python better than TS!
  - davidatbu 4y ago
    > I genuinely think I like types in python better than TS! That's a step further than I'm willing to go :D. For example, the other day, I was able to do settingToToggle: KeysWithTypeOf<SomeObject, boolean> which (after properly defining `KeysWithTypeOf`) allowed me to specify that `settingToToggle` should be a string literal that is a key on `SomeObject` whose corresponding value is a boolean. Typescript comes closest to fully letting me express my intentions statically and curtly, and while I hope Python's static typing will grow to be as expressive as Typescript, I think that day is still afar off.
    - cdrini 4y ago
      For me the remaining pieces for Python are much more attainable than the missing pieces for typescript. I think basically all the helper TS things (eg KeyOf, etc) are addable to Python. Runtime types are not addable to TS. It goes against some of the core principles of TS. But I agree there are loads of missing features in Python :)
NonNefarious 4y ago
I would like to learn Python and would like to like it, but the significance of whitespace is a huge turn-off.
- xdfgh1112 4y ago
  That's fine, I can't get past Go error handling either.
- cardanome 4y ago
  Most editors have a "show whitespace" option that you can you use.
- toyg 4y ago
  1998 called, they want your objections back. Consider the statement "I would like to learn C, but the lack of significant indentation is a huge turn-off for me". It has the exact same validity as your statement.
  - NonNefarious 4y ago
    Actually, it doesn't. In C, you can use indentation as you see fit; braces allow you to define code blocks and scope as necessary. That gives you both worlds. Python apparently doesn't. But stay smug.
    - ptx 4y ago
      > That gives you both worlds. Or put another way, you're forced to specify the same thing twice: with whitespace for the human and with braces for the compiler, creating the possibility for these specifications to get out of sync. In Python, the compiler reads the same information you already put there for the human.
- kzrdude 4y ago
  The charitable way to express it is that Python has significant indentation.
- 4y ago
3np 4y ago
Not explicitly mentioned but demonstrated: reveal_type https://docs.python.org/3.11/library/typing.html#typing.reveal_type https://docs.python.org/3.11/library/typing.html#typing.reve...
- davidatbu 4y ago
  To be clear, this feature has forever been supported by the type checkers, it's just that running python code with `reveal_type` would fail since it's not defined at run-time. IIUC, this change makes it so that it doesn't fail at run time.
linsomniac 4y ago
For the longest time I was pretty meh about following newer Python releases, instead preferring to stick with a "python--" subset that would work across all the machines in my fleet. But recently I've really been enjoying the new features and have decided to make 3.10 my standard. In particular, the type annotations make my editor (LunarVim) a LOT smarter and let me find many errors before I even run the code. Part of that has been that Ubuntu 22.04 has 3.10 in it. So, while I'd like to go up to 3.11, I'm not sure there's a good story for adding 3.11 to a 22.04 system.
silent_cal 4y ago
Previously, if you had to define a class method that returned an object of the class itself, adding types for it was a bit weird, it would look something like this... To be able to say that a method returns the same type as the class itself, you had to define a TypeVar, and say that the method returns the same type T as the current class itself. This kind of stuff is so crazy to me. I see how types are useful for defining general categories of data (int, string, float), but isn't it better to have as few types as possible? It just makes reading and using code more confusing to keep multiplying "types" like this.
- wheelerof4te 4y ago
  "Types" beyond the basic ones, in general, are needed only when you have to define complex data structures that can hold multiple variables in one. What Python really needs is an easy, built-in way to create mutable structs, without an overhead of creating classes. I think it's a failure of language design to miss such a useful and basic feature. For everything else, use a function.
jonnycomputer 4y ago
A few of the decisions made over the last few years are questionable, imo, but on the whole, most of the changes have been very very welcome, and I just don't use the parts I don't like. For example, the changes to TypedDict in 3.11 might actually get me to try them out again.
woodrowbarlow 4y ago
regarding task groups: can this be used in places where we want to run several tasks until any one (or more) of the tasks exit? (i.e. instead of asyncio.gather, asyncio.wait with return_when=asyncio.FIRST_COMPLETED)
jenny91 4y ago
> "TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'" > It is crystal clear, that data['profits']['yearly'] was None. I wouldn't call that exactly "crystal clear" for someone not somewaht experienced with the language or software in general.
_0xdd 4y ago
…for Workgroups?

322 points by sanketsaurav 4y ago

Does anyone actually use asyncio for processing stuff? It's been around for a while now but I still find it very awkward compared to just creating a pool and calling map or starmap. Am I missing out?

heavyset_go 4y ago
All the time now for IO-bound workloads. It was very awkward a few years ago, but things like asyncio.run(), asyncio.gather(), asyncio.to_thread(), and asyncio.create_task() have made things less awkward. TaskGroups should make things even less awkward, too.
- mrfusion 4y ago
  Why not just use multi processing?
  - objektif 4y ago
    Yes curious why this package is needed at all?
    - heavyset_go 4y ago
      Single-threaded execution of IO-bound work can be faster than breaking it up between threads or processes, and it can use a lot less resources. Then there are the preemptive vs cooperative multitasking concerns and the pros and cons of processes/threads vs light-weight threads/coroutines/etc. Some IO-bound workloads are suited really well by the asyncio model, while other workloads might be better suited for processes and threads. They're three separate tools whose use cases might be similar, but they're not necessarily replacing one another. Multiple processes still have their place even while asyncio exists and vice versa.
  - nhumrich 4y ago
    If your task is io bound, aka, lots of network stuff, multi process is overkill. Also, asyncio can handle a _lot_ more tasks 100,000's as opposed to 10s. So it really shines in heavy io things. Also, multiprocessing can not share memory, and that can be a pretty busy g disadvantage depending on the task.
  - zackees 4y ago
    Probably because async io generally scales better.
  - hprotagonist 4y ago
    because pickle is an absolute shitshow.
  - quietbritishjim 4y ago
    One reason is that you often don't need to use locks. Between lines containing await (or async for or async with), you can be sure that this task won't be pre-empted to run another async task. Another reason, if you're using the Trio async library, is that managing and cancelling multiple tasks is really easy, and you can be sure that none get lost. This update to Python brings some of that to core asyncio (but I'll stick with Trio for now, thanks).
  - est 4y ago
    Why not just use Gevent?
    - d0mine 4y ago
      Explicit (await) vs. implicit (anything that uses patched I/O deep down) switch. Essentially, it makes the reasoning about the code almost as hard as with preemptive threading.
  - smitty1e 4y ago
    Asycio is really a list of tasks in the main interpreter thread. This is far easier to work with than multiprocessing. When doing, e.g. AWS work, multiprocessing is additional pain for little gain. Maybe 3.11 will make threads less painful.
    - heavyset_go 4y ago
      From what I've been following, it's going to take another release (or several) after 3.11 for Node-like worker threads to land in Python.
      - smitty1e 4y ago
        I'm not deep on Node, but this sounds like a GIL thing.
        heavyset_go 4y ago
        It is, and it's a subinterpreters thing. Node has separate interpreters that run in worker threads but also share memory, which is the route Python is planning on taking.
  - heavyset_go 4y ago
    It can use less resources and can be faster than passing data between processes. There's really no reason to use multiple processes for IO-bound workloads, either, and even for some IO-bound workloads, a thread pool can be faster than processes. The second you aren't just passing raw bytes around, you have to take into consideration what can and can't be sent between processes in Python, as some objects can't be pickled and thus can't be passed between processes. You can concurrently load a lot more coroutines than processes and threads, as well.
  - hermitdev 4y ago
    Asyncio actually plays really nicely with multiprocessing, too. The concurrent.futures.ProcessPoolExecutor can handle running tasks in child processes and handles the communication seemlessly for you. I've used it quite a bit. Can easily use all 32 cores on my server this way without the GIL getting in the way.
  - dragonwriter 4y ago
    > Why not just use multi processing? Multiprocessing provides parallelism up to what the machine supports, but no additional degree of concurrency, asyncio provides a fairly high degree of concurrency, but no parallelism. OF course, you can use them together to get both. https://github.com/omnilib/aiomultiprocess https://github.com/omnilib/aiomultiprocess
  - witcher_rat 4y ago
    One reason is if you need to launch a subprocess with a timeout but don't want to use up CPU in the python script while that subprocess runs. The regular subprocess module will busy-loop in such cases, consuming CPU, while asyncio's does not. The docs even warn about this for subprocess and suggest using asyncio to avoid it, although the docs are misleading - it only busy-loops if the timeout is not None, and only when running on Mac/Linux not Windows.
d0mine 4y ago
Cooperative multitasking is much simpler to reason about than the general preemptive multithreading. pool+map is useful but it is not general.
Sohcahtoa82 4y ago
It depends...are you looking for concurrency or parallelism? The two are similar, but not the same. Using multiprocessing creates a LOT of overhead and is really intended for parallelism and is only the better solution if you're CPU-bound. If you're I/O-bound, then what you're really looking for is concurrency, and using asyncio will be more performant.

beisner 4y ago

Variadic Generics has the potential to enable all the scientific computing libraries to provide really nice type hints. One step closer to eliminating insidious broadcasting bugs… or running training for 10 hrs only to have a validation loop fail because of a shape mismatch.

deleted 4y ago
[deleted]
MichaelMoser123 4y ago
i would say it makes sense for a typed language, not sure if you can have it both in python. Why do they keep adding features upon features to the syntax of poor python? do they want to turn it into another kind of c++?
- beisner 4y ago
  It's an optional typechecking tool. Speaking as an ML researcher who has authored my fair share of linear algebra bugs due to broadcasting, being able to optionally typecheck specific critical regions of the codebase would be a killer feature.
- bbkane 4y ago
  I think they want Python to fit into every niche, like C++, which means a lot of knobs to turn
davidatbu 4y ago
THIS. OH MY GOSH. THIS. The only issue now is that pytorch has their own "shapes" solution[0], and last I checked, were kinda reserved about participating in the standardization of variadic generics because they don't expect to use it. I really really hope that the ML community comes together to use variadic generics because I believe it will save researchers and devs so many debug cycles (as well as compute resources, tbh). [0] https://pytorch.org/docs/stable/named_tensor.html https://pytorch.org/docs/stable/named_tensor.html
aix1 4y ago

gpm 4y ago

I love that they're improving the error messages, that looks like a huge productivity improvement for beginners and experts alike.

di 4y ago

Straight from the horse's mouth, if you prefer: https://docs.python.org/3.11/whatsnew/3.11.html https://docs.python.org/3.11/whatsnew/3.11.html

rjh29 4y ago
I do, thank you for the link.

barosl 4y ago

The addition of a TOML parser to the standard library is really welcome. I've always wanted to use TOML in my Python scripts but having to import an external library made me use JSON instead, which was awkward due to the lack of comments.

mixmastamyk 4y ago
.ini configparser worked fine for decades, especially as a simple project config.
- barosl 4y ago
  PEP 518 lists some reasons not to use configparser and I agree with them wholeheartedly.[1] Personally I don't want to use ill-defined formats, even for simple projects. [1] https://peps.python.org/pep-0518/#configparser https://peps.python.org/pep-0518/#configparser
  - mixmastamyk 4y ago
    Their reason was obsolete shortly after it was written, Python 2 was EOL years before TOML included. Toml is ill-defined as well. ini files work fine for these trivial uses. https://hitchdev.com/strictyaml/why-not/toml/ https://hitchdev.com/strictyaml/why-not/toml/ So now you have two flawed ways to do it, congrats.
  - ptx 4y ago
    The only reason offered is that there are (or might be) differences between version of Python, mainly 2 and 3, as far as I can see. Python 2 is EOL, so that's no longer a concern. As for differences between Python 3 releases, isn't there a fairly large difference in TOML support as well, since in versions before 3.11 it doesn't work at all? Wouldn't specifying the behaviour of the INI parser as whatever 3.11 is doing (and raising an exception on earlier versions) amount to essentially the same thing?
- bulatb 4y ago
  Newer packaging-related PEPs require TOML, and many developer tools use TOML by choice, but a fresh install of Python couldn't actually read it. That required workarounds from tool and library maintainers, making it more complicated to support as many users' preferences and ways of working as possible. However you prefer to work, your tools may now be easier to test and maintain. This change is good for everyone.

throw0101a 4y ago

Crickey: > Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.25x speedup on the standard benchmark suite. See Faster CPython for details. * https://docs.python.org/3.11/whatsnew/3.11.html https://docs.python.org/3.11/whatsnew/3.11.html * https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpyth...

jamesfinlayson 4y ago
Among other things: "In Python 3.11, the core modules essential for Python startup are “frozen”. This means that their code objects (and bytecode) are statically allocated by the interpreter." Very nice.
- throwaway81523 4y ago
  I'd prefer that they had done something like unexec to allow loading some arbitrary set of modules into a bootstrapping image, then dumping a runnable executable that would launch with all those modules already present. The old Emacs approaches are now mostly busted, but there are alternatives.
  - eslaught 4y ago
    Ye olde Lisp dump image. A neat approach, but results in an absolutely horrific cross compilation experience. As someone who worked briefly on a Common Lisp implementation, including trying to get it to cross-compile, I'm very glad this technique hasn't caught on in the broader community.
    - throwaway81523 4y ago
      Yeah, it is pretty unworkable now, and Emacs uses an object serialization approach that still works pretty well, but there are other methods too. There is some OpenVZ-derived way to make process snapshots in Linux whose name I've forgotten, but I've been wanting to look into it for this.

deleted 4y ago

[deleted]

bulatb 4y ago

This unblocks or simplifies so many of my projects. Variadic generics, better exceptions, and TOML in the standard library are huge, self-types are great, and extra speed is always appreciated. Thanks to everyone involved.

daenz 4y ago
Can you elaborate on how your projects are unblocked? My reading of it was that these improvements are nice, but nothing groundbreaking.
- bulatb 4y ago
  I wouldn't quite say they're gamechangers, but they're exactly what I happened to need. Variadic generics will (I hope) be very nice for typing pre-2.0 SQLAlchemy code. `Query[*Ts]` or similar would be a natural type for many values from the ORM API, and hopefully allow expressing types for query transformations like adding or removing columns, joining to specific query shapes, adding arbitrary subqueries, etc. The self-type might unblock a testing tool I haven't shipped because the DX isn't where I want it. IIRC: normal TypeVar binding plus the TypeVar hack for self-types, interacting with Mypy's particular way of detecting the type of a descriptor, interacting with the way the project uses mixins, sometimes produce types that technically aren't wrong but lead to spurious type errors in user-defined data fixtures. So I'm probably the only one on earth with this particular problem. I don't know, it's been a while. Maybe it won't work. But the self-type hack is noisy anyway and getting rid of it is nice. Shipping TOML in the standard library means I can drop a bunch of Tox gymnastics in projects that support pyproject.toml or TOML-based configuration in addition to the older standards, and now I'm not stuck either forcing certain TOML libraries on people or working out a way to plug in their choice. It's not that hard, but complexity multiplies, and now it's one less thing to deal with.

salmo 4y ago

I’ll just say. Since about 3.6/3.8 the language has been growing huge. Like C++ huge. I liked f-strings. Asyncio has its place but is waaay overused. Typing was a cool idea, but so awkward in practice. All of it has really turned me off, personally. It’s getting to where reading libraries is just painful. The simplicity of the language was really beautiful and when I wanted type safety, etc. I’d use something else. I can still write simple Python, but it’s more all the other code I need to grok. I find myself going to Go more and more for stuff I used to use Python for. It’s easier to set up a dev environment for other people, easier to distribute code, and gives me type safety and concurrency as first class citizens while being a very small language. I miss dictionary comprehensions and other shortcuts from time to time. But it actually feels more ergonomic now. All personal opinion. I’m fine shifting languages. I still write C from time to time to bang bits. I play with various LISPs to exercise my brain. I never used Python for performance critical code. I guess ML is changing that need. But I know others that want Java to have functional features and Python to have this stuff. I’m not saying I’m “right”, just uncomfortable in a place I used to love to hang out in.

Night_Thastus 4y ago
As someone who mainly sits in the C++ and not the Python space - why? Can't you still use all those old features just as you did before? If you don't like the complexity of newer libraries and features, they're still optional aren't they? Or did something fundamental shift (aside from the obvious P2->P3 switch) that has made existing features harder to use?
- Galanwe 4y ago
  > If you don't like the complexity of newer libraries and features, they're still optional aren't they? Chances are you're not programming in a vacuum. These features will be used by third party libraries and you will have to read/understand this code. Even in your in house codebase, there will always be co-workers that want to use these features. The optionality of language features is not real. If they exist, they will be used, you will have to understand them, they will leak from other libraries, etc.

6gvONxR4sf7o 4y ago

Better error messages is something that literally every single language/framework/library should prioritize. It improves users lives by more than almost everything else besides correctness. It’s been great to see the improvements in python.

chrisma0 4y ago
I agree that the exact error locations in tracebacks seem like the most impactful change here. Particularly in the light that Python is often used as an introductory language and unhelpful error messages are incredibly frustrating. Like this it feels like they can become more of a tool to improve!
dominotw 4y ago
best i've seen so far are in elm lang.
drited 4y ago
Totally agree! It's hard to fathom why so many languages leave you to be some kind of detective trying to determine exactly where the issue is rather than just telling you.
dazzawazza 4y ago
.. and they have the highest impact when you are starting to learn a language. Many a student turns away from a language because the developers underestimate the importance of error messages.

asicsp 4y ago

The `re` module in 3.11 will support "Atomic grouping and possessive quantifiers". I recently wrote a blog post about these features: https://learnbyexample.github.io/python-regex-possessive-quantifier/ https://learnbyexample.github.io/python-regex-possessive-qua...

TeeMassive 4y ago

Glad to see they ported Anyio's TaskGroups (or was it from trio?) to the stdlib. Starting tasks so they run concurrently was a huge pain in vanilla Python.

tusharsadhwani 4y ago
The idea came from trio's nurseries I believe.
erdewit 4y ago
Is the TaskGroup really in the 3.11 release though? It's not to be found anywhere in the Python documentation or release notes: https://docs.python.org/3.11/whatsnew/3.11.html https://docs.python.org/3.11/whatsnew/3.11.html It has been talked about for a long time, probably 4 years now.
- Mehdi2277 4y ago
  It's implemented in 3.11. Documentation still hasn't been written for it yet though and that's why the issue is still open, https://github.com/python/cpython/issues/90908 https://github.com/python/cpython/issues/90908.

j16sdiz 4y ago

The note in exception design is interesting. This allow more contextual information to be embedded.

stinos 4y ago
Embedding more information isn't really the problem: alternatives have always existed, it's just that none of them were as brief and good as this one. E.g. normally you'd catch then re-raise another exception with a new message with extra information and the original one as inner exception. The issue with that is that the resulting message/stacktrace isn't as readable as these notes.

assignmenthelp0 4y ago

davidatbu 4y ago

I'm really excited for where Python's static typing is heading! I doubt that typing Python code will ever become as ergonomic as typescript, simply because typescript doesn't have the constraint of modifying the syntax of it's "target language" (JS) when it wants to add a feature or whatever, whereas typed python must still be Python, and syntactic changes therefore need to be added much more conservatively. But still, the typing story is improving before my very eyes with every release, and given that Python types are actually available at runtime (allowing for things like pydantic and FastAPI), and that Python has massive adoption in the ML space, all this is making Python a very enticing language to get back to using!

cdrini 4y ago
I love that Python has types available at runtime! The ability to have those type-generated CLIs or APIs is fantastic. I learned from this article about functools.singledispatch, and can't wait to use it. I've always wanted something like in typescript! But `functools.singledispatch` is just not possible in TS. I think working with dicts is still a little clunky in python, and there are some rough edges (eg mypy being a little weaker), but I genuinely think I like types in python better than TS!
- davidatbu 4y ago
  > I genuinely think I like types in python better than TS! That's a step further than I'm willing to go :D. For example, the other day, I was able to do settingToToggle: KeysWithTypeOf<SomeObject, boolean> which (after properly defining `KeysWithTypeOf`) allowed me to specify that `settingToToggle` should be a string literal that is a key on `SomeObject` whose corresponding value is a boolean. Typescript comes closest to fully letting me express my intentions statically and curtly, and while I hope Python's static typing will grow to be as expressive as Typescript, I think that day is still afar off.
  - cdrini 4y ago
    For me the remaining pieces for Python are much more attainable than the missing pieces for typescript. I think basically all the helper TS things (eg KeyOf, etc) are addable to Python. Runtime types are not addable to TS. It goes against some of the core principles of TS. But I agree there are loads of missing features in Python :)

NonNefarious 4y ago

I would like to learn Python and would like to like it, but the significance of whitespace is a huge turn-off.

xdfgh1112 4y ago
That's fine, I can't get past Go error handling either.
cardanome 4y ago
Most editors have a "show whitespace" option that you can you use.
toyg 4y ago
1998 called, they want your objections back. Consider the statement "I would like to learn C, but the lack of significant indentation is a huge turn-off for me". It has the exact same validity as your statement.
- NonNefarious 4y ago
  Actually, it doesn't. In C, you can use indentation as you see fit; braces allow you to define code blocks and scope as necessary. That gives you both worlds. Python apparently doesn't. But stay smug.
  - ptx 4y ago
    > That gives you both worlds. Or put another way, you're forced to specify the same thing twice: with whitespace for the human and with braces for the compiler, creating the possibility for these specifications to get out of sync. In Python, the compiler reads the same information you already put there for the human.
kzrdude 4y ago
The charitable way to express it is that Python has significant indentation.
4y ago

3np 4y ago

Not explicitly mentioned but demonstrated: reveal_type https://docs.python.org/3.11/library/typing.html#typing.reveal_type https://docs.python.org/3.11/library/typing.html#typing.reve...

davidatbu 4y ago
To be clear, this feature has forever been supported by the type checkers, it's just that running python code with `reveal_type` would fail since it's not defined at run-time. IIUC, this change makes it so that it doesn't fail at run time.

linsomniac 4y ago

For the longest time I was pretty meh about following newer Python releases, instead preferring to stick with a "python--" subset that would work across all the machines in my fleet. But recently I've really been enjoying the new features and have decided to make 3.10 my standard. In particular, the type annotations make my editor (LunarVim) a LOT smarter and let me find many errors before I even run the code. Part of that has been that Ubuntu 22.04 has 3.10 in it. So, while I'd like to go up to 3.11, I'm not sure there's a good story for adding 3.11 to a 22.04 system.

silent_cal 4y ago

Previously, if you had to define a class method that returned an object of the class itself, adding types for it was a bit weird, it would look something like this... To be able to say that a method returns the same type as the class itself, you had to define a TypeVar, and say that the method returns the same type T as the current class itself. This kind of stuff is so crazy to me. I see how types are useful for defining general categories of data (int, string, float), but isn't it better to have as few types as possible? It just makes reading and using code more confusing to keep multiplying "types" like this.

wheelerof4te 4y ago
"Types" beyond the basic ones, in general, are needed only when you have to define complex data structures that can hold multiple variables in one. What Python really needs is an easy, built-in way to create mutable structs, without an overhead of creating classes. I think it's a failure of language design to miss such a useful and basic feature. For everything else, use a function.

jonnycomputer 4y ago

A few of the decisions made over the last few years are questionable, imo, but on the whole, most of the changes have been very very welcome, and I just don't use the parts I don't like. For example, the changes to TypedDict in 3.11 might actually get me to try them out again.

woodrowbarlow 4y ago

regarding task groups: can this be used in places where we want to run several tasks until any one (or more) of the tasks exit? (i.e. instead of asyncio.gather, asyncio.wait with return_when=asyncio.FIRST_COMPLETED)

jenny91 4y ago

> "TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'" > It is crystal clear, that data['profits']['yearly'] was None. I wouldn't call that exactly "crystal clear" for someone not somewaht experienced with the language or software in general.

_0xdd 4y ago

…for Workgroups?