3 ms·

We use this extensively as probably half of the Python programmers on HN do. It's great, but there are plenty of alternatives out there as well, especially fo

by JPKab 5y ago

We use this extensively as probably half of the Python programmers on HN do. It's great, but there are plenty of alternatives out there as well, especially for performance. Our specific use case is IO limited so it's perfect for that.

jamil7 5y ago
Care to list some? I disabled it on a few endpoints of our backend because the performance was awful for GeoJSON.
- lazypenguin 5y ago
  https://www.attrs.org/en/stable/ https://www.attrs.org/en/stable/ would be the main alternative
  - johnwheeler 5y ago
    Marshmallow is another if you’re talking schemas, validation, and json serialization.
  - hackandtrip 5y ago
    Are those alternatives faster according to benchmark? In my experience, the bottleneck is either: - JSON parsing and dumping; the solution for me is ORJSON, fantastic wrapper to use fast JSON serialisation for most common fields, and also datetime. - Validation - if you choose to validate your data, pydantic can indeed be slow... But it's not Pydantic the problem, but the validation that you apply to your data.
    - progval 5y ago
      > But it's not Pydantic the problem, but the validation that you apply to your data. Indeed, this kind of validation is usually based on 'isinstance', which is really slow in Python, because you often need to call it many times. More than once, I doubled of tripled the throughput of some data pipelines (not microbenchmarks) just by replacing 'isinstance' calls with something else. When you really need something like isinstance, type equality sometimes works and is much faster. For example, this works as a replacement for attrs-strict's type checking on a limited subset of types (non-generic classes, Any, Optional, Tuple, and Union): https://archive.softwareheritage.org/swh:1:cnt:7f4f1ea32eacef1317c49297430c15aa2dbea08c;origin=https://forge.softwareheritage.org/source/swh-model.git;visit=swh:1:snp:91a1f51040d40b57e164493ee95d474e0599b79e;anchor=swh:1:rev:a0f5436273ef6f5b62a388ae131ed5afa7287d00;path=/swh/model/model.py;lines=95-160 https://archive.softwareheritage.org/swh:1:cnt:7f4f1ea32eace... The downside is that you can't use subclasses of the specified types.
- mulmboy 5y ago
  I've found that using construct() to skip validation is great when a model has e.g. a `List[float]` attribute with a length of 10k+ items. But this has to be done judiciously. https://pydantic-docs.helpmanual.io/usage/models/#creating-models-without-validation https://pydantic-docs.helpmanual.io/usage/models/#creating-m... In the case of GeoJSON it might be worth using a custom data type with specific faster validation. https://pydantic-docs.helpmanual.io/usage/types/#custom-data-types https://pydantic-docs.helpmanual.io/usage/types/#custom-data...

by JPKab 5y ago

jamil7 5y ago

Care to list some? I disabled it on a few endpoints of our backend because the performance was awful for GeoJSON.

lazypenguin 5y ago
https://www.attrs.org/en/stable/ https://www.attrs.org/en/stable/ would be the main alternative
- johnwheeler 5y ago
  Marshmallow is another if you’re talking schemas, validation, and json serialization.
- hackandtrip 5y ago
  Are those alternatives faster according to benchmark? In my experience, the bottleneck is either: - JSON parsing and dumping; the solution for me is ORJSON, fantastic wrapper to use fast JSON serialisation for most common fields, and also datetime. - Validation - if you choose to validate your data, pydantic can indeed be slow... But it's not Pydantic the problem, but the validation that you apply to your data.
  - progval 5y ago
    > But it's not Pydantic the problem, but the validation that you apply to your data. Indeed, this kind of validation is usually based on 'isinstance', which is really slow in Python, because you often need to call it many times. More than once, I doubled of tripled the throughput of some data pipelines (not microbenchmarks) just by replacing 'isinstance' calls with something else. When you really need something like isinstance, type equality sometimes works and is much faster. For example, this works as a replacement for attrs-strict's type checking on a limited subset of types (non-generic classes, Any, Optional, Tuple, and Union): https://archive.softwareheritage.org/swh:1:cnt:7f4f1ea32eacef1317c49297430c15aa2dbea08c;origin=https://forge.softwareheritage.org/source/swh-model.git;visit=swh:1:snp:91a1f51040d40b57e164493ee95d474e0599b79e;anchor=swh:1:rev:a0f5436273ef6f5b62a388ae131ed5afa7287d00;path=/swh/model/model.py;lines=95-160 https://archive.softwareheritage.org/swh:1:cnt:7f4f1ea32eace... The downside is that you can't use subclasses of the specified types.
mulmboy 5y ago
I've found that using construct() to skip validation is great when a model has e.g. a `List[float]` attribute with a length of 10k+ items. But this has to be done judiciously. https://pydantic-docs.helpmanual.io/usage/models/#creating-models-without-validation https://pydantic-docs.helpmanual.io/usage/models/#creating-m... In the case of GeoJSON it might be worth using a custom data type with specific faster validation. https://pydantic-docs.helpmanual.io/usage/types/#custom-data-types https://pydantic-docs.helpmanual.io/usage/types/#custom-data...