6 ms·

Pydantic V2 leverages Rust's Superpowers [video]

122 points by BerislavLopac 4y ago

hotfixguru 4y ago
Samuel also presented a slightly altered talk at PyCon US yesterday[0], which was awesome! GitHub link to pydantic[1], and pydantic-core[2]. [0]: https://twitter.com/samuel_colvin/status/1649928041462915072 https://twitter.com/samuel_colvin/status/1649928041462915072 (slides in comments) [1]: https://github.com/pydantic/pydantic https://github.com/pydantic/pydantic [2]: https://github.com/pydantic/pydantic-core https://github.com/pydantic/pydantic-core
fbdab103 4y ago
Have not watched the video, but I did find the slides from a different version[0]. [0] https://slides.com/samuelcolvin/how-pydantic-v2-leverages-rust-s-superpowers https://slides.com/samuelcolvin/how-pydantic-v2-leverages-ru... Edit: Realized that the linked slides were from a different iteration of the ~same content.
- zikohh 4y ago
  This is the one from the video https://slides.com/samuelcolvin/deck-0e6306 https://slides.com/samuelcolvin/deck-0e6306
keithasaurus 4y ago
As someone who built a pure python validation library[0] that's much faster than pydantic (~1.5x - 12x depending on the benchmark), I have to say that this whole focus on Rust seems premature. There's clearly a lot of room for pydantic to optimize its Python implementation. Beyond that, rust seems like a great fit for tooling (i.e. ruff), but as a library used at runtime, it seems a little odd to make a validation library (which can expect to receive any kind of legal python data) to also be constrained by a separate set of data types which are legal in rust. [0]: https://github.com/keithasaurus/koda-validate https://github.com/keithasaurus/koda-validate
- jammycrisp 4y ago
  While I agree that there are ways to write a faster validation library in python, there are also benefits to moving the logic to native code. msgspec[1] is another parsing/validation library, written in C. It's on average 50-80x faster than pydantic for parsing and validating JSON [2]. This speedup is only possible because we make use of native code, letting us parse JSON directly and efficiently into the proper python types, removing any unnecessary allocations. It's my understanding that pydantic V2 currently doesn't do this (they still have some unnecessary intermediate allocations during parsing), but having the validation logic already in compiled code makes integrating this with the parser theoretically possible later on. With the logic in python this efficiency gain wouldn't be possible. [1]: https://github.com/jcrist/msgspec https://github.com/jcrist/msgspec [2]: https://jcristharif.com/msgspec/benchmarks.html#benchmark-schema-validation https://jcristharif.com/msgspec/benchmarks.html#benchmark-sc...
  - keithasaurus 4y ago
    Definitely true. I've just soured on the POV that native code is the first thing one should reach for. I was surprised that it only took a few days of optimizations to convert my validation library to being significantly faster than pydantic, when pydantic as already largely compiled via cython. If you're interested in both efficiency and maintainability, I think you need to start by optimizing the language of origin. It seems to me that with pydantic, the choice has consistently been to jump to compilation (cython, now rust) without much attempt at optimizing within Python. I'm not super-familiar with how things are being done on an issue-to-issue / line-to-line basis, but I see this rust effort taking something like a year+, when my intuition is some simpler speedups in python could have been in a matter of days or weeks (which is not to say they would be of the same magnitude of performance gains).
    - nine_k 4y ago
      Two things may preclude optimization in pure Python when producing a library for general public. Having a nice / ergonomic interface is one. Keeping things backwards-compatible is another.
satvikpendem 4y ago
Rust is the future of tooling [0]. While [0] is about JS tooling specifically, we're seeing the same effects in other languages as well. Turns out, you probably don't want to write infrastructure tooling in slow, dynamically typed languages when faster, more safe languages exist. Python knows this already, with much of the scientific computing libraries being just wrappers over the core C++ codebases. JS is beginning to catch up as well, with swc (speedy web compiler), stc (speedy type checker), Turbopack (Webpack successor) and so on, with Vercel leading the charge mainly. [0] https://leerob.io/blog/rust#the-future-of-javascript-tooling https://leerob.io/blog/rust#the-future-of-javascript-tooling
- nine_k 4y ago
  I'd say that you don't want to write the second crop of the tooling in a language like JS or Python. This is because the first crop, like mypy or babel or jslint existed, and has shown the general direction. But for the first crop, a slow-running but fast-turnaround language was essential, to my mind. The first iteration had to move fast, and change the direction fast, because it wasn't yet clear what direction was going to be right.
- Already__Taken 4y ago
  even just knocking up a cli too in python I'm seeing so many cases and states I could be in that python just isn't warning me about.
bratao 4y ago
Someone recommended here msgspec as a Pydantic alternative for serialization/validation and wow. It is fantastic. I really recommend it https://github.com/jcrist/msgspec https://github.com/jcrist/msgspec
- jammycrisp 4y ago
  Thanks, glad you like it!
jonatron 4y ago
Subtitled Video: https://jonatron.github.io/fosdem2023whisper/files/rust_how_pydantic_v2_leverages_rusts_superpowers.webm.html https://jonatron.github.io/fosdem2023whisper/files/rust_how_... Transcript: https://jonatron.github.io/fosdem2023whisper/files/rust_how_pydantic_v2_leverages_rusts_superpowers.webm.txt https://jonatron.github.io/fosdem2023whisper/files/rust_how_... (transcribed with whisper)
nightski 4y ago
I'm rather amazed at the sheer number of human hours invested (wasted?) into making a terrible language(s) better just so things are slightly easier for beginners.
- _zamorano_ 4y ago
  You're being downvoted, but I sort of agree... How in the world can I stop worrying about errors like using a non existant class member and having a runtime exception? How can I refactor? Ok, the core devs add annotations, then type hints to the language. Some other people create MyPy, the JetBrains people use some other linting thing that does well, but does not match MyPy. The Pydantic appear and some other people says its awesome, but yet again, different to the others... And all that to enforce things we already had 20 years ago in Java 1.0 How about using dynamic typing for small script and prototyping and using better tools for bigger projects?
  - ptx 4y ago
    Java 1.0 didn't have generics, so collection elements were only type-checked at runtime, and Java still doesn't have null-safety in the type system. The Python type annotations handle both of these things.
- aobdev 4y ago
  I’m someone who spent a lot of time focusing on finding the most efficient tools, writing the most elegant code, using the most “pure” frameworks, etc. and I realized that this did very little to help me obtain my goals. Unless your objectives are not commercial or will never scale beyond the ability of a single person, the overhead of human communication and collaboration will be enormous compared the inefficiencies of shuffling electrons and flipping bits with imperfect instructions. Sometimes this means using technically-inferior tools better than idealistic tools improperly to get the job done.
- mjr00 4y ago
  Assuming you're talking about Python, it's a product of its time. At the time it was released, it was very much following the newest "best practices" from "experts". You had lots of well-known people in the industry going to conferences and talking about how static typing was Actually Bad and unit testing was the one true way. (I won't bother explaining the retrospective idiocy of that.) It's not a great language in a vacuum... but it is the most popular backend language in the world, by most estimations; there's a library for literally everything, and the entire data science ecosystem lives in the Python world. If you're on the management side, hiring a Python developer is orders of magnitudes easier than finding a Rust developer. So you've really only got a few options... 1) throw everything out to rewrite it in Rust (et al); 2) accept that things suck and do nothing about it; 3) accept that things suck, but make some tooling to make it suck less. The rewrite strategy is a nonstarter for a lot of massive legacy codebases, so 3 is the only option that makes sense. So it's really not that amazing how much time people invest in improving Python, when you think about it. Similar situation with PHP and Javascript.
AdilZtn 4y ago
PyO3 is the key actor, in this rust-binding trend.
the__alchemist 4y ago
Code style in Pydantic user code reminds me of Rust's Serde serialization lib. Not directly relevant to the point of this video, but it makes me curious if it was inspired by Serde.
drhagen 4y ago
Pydantic is so cool and I am really excited to migrate to v2. My biggest pain point in Pydantic is that it has no abstract base class for a serializer/deserializer. There is just BaseModel. This means that there is no "Pydantic interface" to implement. If I need to serialize/deserialize something other than an object with attributes, I basically can't do it in Pydantic, or at least I couldn't figure out how to do it. In contrast, when I designed Serialite [1], I designed it to have a Serializer base class with two abstract methods: to_data and from_data. Then I added a @serializable decorator that can be applied to any dataclass, which injects concrete implementations of to_data and from_data. The decorator is what I usually use, but I sometimes implement the base class directly when it gets too complicated. Making Serialite work with FastAPI was almost impossible because there is no interface to implement, duck type or otherwise. In the end, Serialite monkey patches some key FastAPI functions to make them understand Serialite Serializers and that seems to work. [1] https://github.com/drhagen/serialite https://github.com/drhagen/serialite

122 points by BerislavLopac 4y ago

Samuel also presented a slightly altered talk at PyCon US yesterday[0], which was awesome! GitHub link to pydantic[1], and pydantic-core[2]. [0]: https://twitter.com/samuel_colvin/status/1649928041462915072 https://twitter.com/samuel_colvin/status/1649928041462915072 (slides in comments) [1]: https://github.com/pydantic/pydantic https://github.com/pydantic/pydantic [2]: https://github.com/pydantic/pydantic-core https://github.com/pydantic/pydantic-core

fbdab103 4y ago

Have not watched the video, but I did find the slides from a different version[0]. [0] https://slides.com/samuelcolvin/how-pydantic-v2-leverages-rust-s-superpowers https://slides.com/samuelcolvin/how-pydantic-v2-leverages-ru... Edit: Realized that the linked slides were from a different iteration of the ~same content.

zikohh 4y ago
This is the one from the video https://slides.com/samuelcolvin/deck-0e6306 https://slides.com/samuelcolvin/deck-0e6306

keithasaurus 4y ago

As someone who built a pure python validation library[0] that's much faster than pydantic (~1.5x - 12x depending on the benchmark), I have to say that this whole focus on Rust seems premature. There's clearly a lot of room for pydantic to optimize its Python implementation. Beyond that, rust seems like a great fit for tooling (i.e. ruff), but as a library used at runtime, it seems a little odd to make a validation library (which can expect to receive any kind of legal python data) to also be constrained by a separate set of data types which are legal in rust. [0]: https://github.com/keithasaurus/koda-validate https://github.com/keithasaurus/koda-validate

jammycrisp 4y ago
While I agree that there are ways to write a faster validation library in python, there are also benefits to moving the logic to native code. msgspec[1] is another parsing/validation library, written in C. It's on average 50-80x faster than pydantic for parsing and validating JSON [2]. This speedup is only possible because we make use of native code, letting us parse JSON directly and efficiently into the proper python types, removing any unnecessary allocations. It's my understanding that pydantic V2 currently doesn't do this (they still have some unnecessary intermediate allocations during parsing), but having the validation logic already in compiled code makes integrating this with the parser theoretically possible later on. With the logic in python this efficiency gain wouldn't be possible. [1]: https://github.com/jcrist/msgspec https://github.com/jcrist/msgspec [2]: https://jcristharif.com/msgspec/benchmarks.html#benchmark-schema-validation https://jcristharif.com/msgspec/benchmarks.html#benchmark-sc...
- keithasaurus 4y ago
  Definitely true. I've just soured on the POV that native code is the first thing one should reach for. I was surprised that it only took a few days of optimizations to convert my validation library to being significantly faster than pydantic, when pydantic as already largely compiled via cython. If you're interested in both efficiency and maintainability, I think you need to start by optimizing the language of origin. It seems to me that with pydantic, the choice has consistently been to jump to compilation (cython, now rust) without much attempt at optimizing within Python. I'm not super-familiar with how things are being done on an issue-to-issue / line-to-line basis, but I see this rust effort taking something like a year+, when my intuition is some simpler speedups in python could have been in a matter of days or weeks (which is not to say they would be of the same magnitude of performance gains).
  - nine_k 4y ago
    Two things may preclude optimization in pure Python when producing a library for general public. Having a nice / ergonomic interface is one. Keeping things backwards-compatible is another.

satvikpendem 4y ago

Rust is the future of tooling [0]. While [0] is about JS tooling specifically, we're seeing the same effects in other languages as well. Turns out, you probably don't want to write infrastructure tooling in slow, dynamically typed languages when faster, more safe languages exist. Python knows this already, with much of the scientific computing libraries being just wrappers over the core C++ codebases. JS is beginning to catch up as well, with swc (speedy web compiler), stc (speedy type checker), Turbopack (Webpack successor) and so on, with Vercel leading the charge mainly. [0] https://leerob.io/blog/rust#the-future-of-javascript-tooling https://leerob.io/blog/rust#the-future-of-javascript-tooling

nine_k 4y ago
I'd say that you don't want to write the second crop of the tooling in a language like JS or Python. This is because the first crop, like mypy or babel or jslint existed, and has shown the general direction. But for the first crop, a slow-running but fast-turnaround language was essential, to my mind. The first iteration had to move fast, and change the direction fast, because it wasn't yet clear what direction was going to be right.
Already__Taken 4y ago
even just knocking up a cli too in python I'm seeing so many cases and states I could be in that python just isn't warning me about.

bratao 4y ago

Someone recommended here msgspec as a Pydantic alternative for serialization/validation and wow. It is fantastic. I really recommend it https://github.com/jcrist/msgspec https://github.com/jcrist/msgspec

jammycrisp 4y ago
Thanks, glad you like it!

jonatron 4y ago

Subtitled Video: https://jonatron.github.io/fosdem2023whisper/files/rust_how_pydantic_v2_leverages_rusts_superpowers.webm.html https://jonatron.github.io/fosdem2023whisper/files/rust_how_... Transcript: https://jonatron.github.io/fosdem2023whisper/files/rust_how_pydantic_v2_leverages_rusts_superpowers.webm.txt https://jonatron.github.io/fosdem2023whisper/files/rust_how_... (transcribed with whisper)

nightski 4y ago

I'm rather amazed at the sheer number of human hours invested (wasted?) into making a terrible language(s) better just so things are slightly easier for beginners.

_zamorano_ 4y ago
You're being downvoted, but I sort of agree... How in the world can I stop worrying about errors like using a non existant class member and having a runtime exception? How can I refactor? Ok, the core devs add annotations, then type hints to the language. Some other people create MyPy, the JetBrains people use some other linting thing that does well, but does not match MyPy. The Pydantic appear and some other people says its awesome, but yet again, different to the others... And all that to enforce things we already had 20 years ago in Java 1.0 How about using dynamic typing for small script and prototyping and using better tools for bigger projects?
- ptx 4y ago
  Java 1.0 didn't have generics, so collection elements were only type-checked at runtime, and Java still doesn't have null-safety in the type system. The Python type annotations handle both of these things.
aobdev 4y ago
I’m someone who spent a lot of time focusing on finding the most efficient tools, writing the most elegant code, using the most “pure” frameworks, etc. and I realized that this did very little to help me obtain my goals. Unless your objectives are not commercial or will never scale beyond the ability of a single person, the overhead of human communication and collaboration will be enormous compared the inefficiencies of shuffling electrons and flipping bits with imperfect instructions. Sometimes this means using technically-inferior tools better than idealistic tools improperly to get the job done.
mjr00 4y ago
Assuming you're talking about Python, it's a product of its time. At the time it was released, it was very much following the newest "best practices" from "experts". You had lots of well-known people in the industry going to conferences and talking about how static typing was Actually Bad and unit testing was the one true way. (I won't bother explaining the retrospective idiocy of that.) It's not a great language in a vacuum... but it is the most popular backend language in the world, by most estimations; there's a library for literally everything, and the entire data science ecosystem lives in the Python world. If you're on the management side, hiring a Python developer is orders of magnitudes easier than finding a Rust developer. So you've really only got a few options... 1) throw everything out to rewrite it in Rust (et al); 2) accept that things suck and do nothing about it; 3) accept that things suck, but make some tooling to make it suck less. The rewrite strategy is a nonstarter for a lot of massive legacy codebases, so 3 is the only option that makes sense. So it's really not that amazing how much time people invest in improving Python, when you think about it. Similar situation with PHP and Javascript.

AdilZtn 4y ago

PyO3 is the key actor, in this rust-binding trend.

the__alchemist 4y ago

Code style in Pydantic user code reminds me of Rust's Serde serialization lib. Not directly relevant to the point of this video, but it makes me curious if it was inspired by Serde.

drhagen 4y ago

Pydantic is so cool and I am really excited to migrate to v2. My biggest pain point in Pydantic is that it has no abstract base class for a serializer/deserializer. There is just BaseModel. This means that there is no "Pydantic interface" to implement. If I need to serialize/deserialize something other than an object with attributes, I basically can't do it in Pydantic, or at least I couldn't figure out how to do it. In contrast, when I designed Serialite [1], I designed it to have a Serializer base class with two abstract methods: to_data and from_data. Then I added a @serializable decorator that can be applied to any dataclass, which injects concrete implementations of to_data and from_data. The decorator is what I usually use, but I sometimes implement the base class directly when it gets too complicated. Making Serialite work with FastAPI was almost impossible because there is no interface to implement, duck type or otherwise. In the end, Serialite monkey patches some key FastAPI functions to make them understand Serialite Serializers and that seems to work. [1] https://github.com/drhagen/serialite https://github.com/drhagen/serialite