5 ms·

This is a python wrapper around libuv not a pure python solution (not that it’s bad but it explains the click bait title).

korijn 4y ago
Would a framework like uvicorn/fastapi be able to achieve similar performance if it were backed by libuv (as opposed to e.g. asyncio)?
- cirospaciari 4y ago
  uvicorn and fastapi is backed by libuv :) it uses uvloop
- matsemann 4y ago
  Pydantic is unreasonably slow for big datastructures, though. For simple performance tests with simple requests/response it's probably fast. But with lots of the data marshalling happening in python some things are really slow in real life use cases.
  - cirospaciari 4y ago
    Yeah i have a lot of things to do, to minimize this types of things. But Caching Tools that never cross GIL will boost a lot of performance in real life scenarios too. See https://github.com/cirospaciari/socketify.py/issues/60 https://github.com/cirospaciari/socketify.py/issues/60 I plan to create an native Redis cache and memcache too.
  - wdroz 4y ago
    pydandic-core [0] will hopefully solve this issue (written in Rust) [0] -- https://github.com/pydantic/pydantic-core https://github.com/pydantic/pydantic-core
  - LtWorf 4y ago
    Author of typedload here. Pydantic is indeed really slow (https://ltworf.github.io/typedload/performance.html https://ltworf.github.io/typedload/performance.html) typedload is faster and written in pure python (unlike pydantic). I know they are rewriting it in rust… but I don't know what will come out of it. After all my pure python implementation already manages to beat pydantic's binary, and when unions are in use, also apischema's binary.
    - cirospaciari 4y ago
      I will take a look! pure python will perform great on PyPy!
    - no_circuit 4y ago
      The perf part of the tests just seems to be a microbenchmark for seeing how fast the various frameworks can parse a 30000x300 dict of strings representing numbers [1]. If that is all one's application does, and can use your library in their organization/team, that's great. However a 2-3x performance boost for the parsing stage for a use case like an API call might not matter when that could be overshadowed by validation and/or upstream API calls. A realistic app would likely use a validation library like Pydantic's [2] to throw a custom typed-error that can be processed, e.g., localization, before returning it downstream. [1] https://github.com/ltworf/typedload/blob/37c72837e0a8fd5f3502182e8d7716f13045e4d7/perftest/load%20big%20dictionary.py#L70 https://github.com/ltworf/typedload/blob/37c72837e0a8fd5f350... [2] https://docs.pydantic.dev/usage/validators/ https://docs.pydantic.dev/usage/validators/
      - LtWorf 4y ago
        > a microbenchmark for seeing how fast the various frameworks can parse a 30000x300 dict of strings representing numbers [1]. Did you somehow miss all the other tests, even thought they are higher on the page? The important test is shown first: loading objects. I'm not trying to benchmark my wifi or my disk. The IO time is not included there on purpose. Of course a bigger application that does other things wouldn't see this huge difference in performance. But I'm testing the performance of a library here. typedload can use validators, but since it doesn't reimplement attrs/dataclass, I had no interest in testing those… it'd be a race between libraries I didn't write. typedload's exceptions contain enough detail to tell the end user what went wrong. when encountering errors in lists, typedload slows down due to keeping compatibility with python 3.7. However in my test with loading objects, despite the slowdown it remains faster than pydantic.
cirospaciari 4y ago
uvicorn uses uvloop that is an wrapper to libuv ;) almost any performance focused package use native extensions (CFFI, Cython, HPy, Python CAPI etc)
Ensorceled 4y ago
I don't really understand this criticism nor why it's "clickbait". I have 12 "unpure" python packages in production. If my app ever needs WSGI, I'll have 13.
- ModernMech 4y ago
  The reason this is clickbait is because you can wrap libuv in some other language and get the same results. So what’s the purpose of this article? That a language can wrap libuv is not a new or interesting idea.
  - pantsforbirds 4y ago
    Why do people only get upset about this sort of thing when it comes to web servers? I don't see the same nitpicks when its a BLAS library, or when it uses a CUDA kernel under the hood. Obviously python is a slow language and needs the C implementation to be fast. You could say the same thing for the vast majority of the python code base.
commitpizza 4y ago
Node.js uses libuv also but it is way slower and tbh, can't you compare node to other things because of that it is using v8 and libuv? I don't understand this this logic.

korijn 4y ago

Would a framework like uvicorn/fastapi be able to achieve similar performance if it were backed by libuv (as opposed to e.g. asyncio)?

cirospaciari 4y ago
uvicorn and fastapi is backed by libuv :) it uses uvloop
matsemann 4y ago
Pydantic is unreasonably slow for big datastructures, though. For simple performance tests with simple requests/response it's probably fast. But with lots of the data marshalling happening in python some things are really slow in real life use cases.
- cirospaciari 4y ago
  Yeah i have a lot of things to do, to minimize this types of things. But Caching Tools that never cross GIL will boost a lot of performance in real life scenarios too. See https://github.com/cirospaciari/socketify.py/issues/60 https://github.com/cirospaciari/socketify.py/issues/60 I plan to create an native Redis cache and memcache too.
- wdroz 4y ago
  pydandic-core [0] will hopefully solve this issue (written in Rust) [0] -- https://github.com/pydantic/pydantic-core https://github.com/pydantic/pydantic-core
- LtWorf 4y ago
  Author of typedload here. Pydantic is indeed really slow (https://ltworf.github.io/typedload/performance.html https://ltworf.github.io/typedload/performance.html) typedload is faster and written in pure python (unlike pydantic). I know they are rewriting it in rust… but I don't know what will come out of it. After all my pure python implementation already manages to beat pydantic's binary, and when unions are in use, also apischema's binary.
  - cirospaciari 4y ago
    I will take a look! pure python will perform great on PyPy!
  - no_circuit 4y ago
    The perf part of the tests just seems to be a microbenchmark for seeing how fast the various frameworks can parse a 30000x300 dict of strings representing numbers [1]. If that is all one's application does, and can use your library in their organization/team, that's great. However a 2-3x performance boost for the parsing stage for a use case like an API call might not matter when that could be overshadowed by validation and/or upstream API calls. A realistic app would likely use a validation library like Pydantic's [2] to throw a custom typed-error that can be processed, e.g., localization, before returning it downstream. [1] https://github.com/ltworf/typedload/blob/37c72837e0a8fd5f3502182e8d7716f13045e4d7/perftest/load%20big%20dictionary.py#L70 https://github.com/ltworf/typedload/blob/37c72837e0a8fd5f350... [2] https://docs.pydantic.dev/usage/validators/ https://docs.pydantic.dev/usage/validators/
    - LtWorf 4y ago
      > a microbenchmark for seeing how fast the various frameworks can parse a 30000x300 dict of strings representing numbers [1]. Did you somehow miss all the other tests, even thought they are higher on the page? The important test is shown first: loading objects. I'm not trying to benchmark my wifi or my disk. The IO time is not included there on purpose. Of course a bigger application that does other things wouldn't see this huge difference in performance. But I'm testing the performance of a library here. typedload can use validators, but since it doesn't reimplement attrs/dataclass, I had no interest in testing those… it'd be a race between libraries I didn't write. typedload's exceptions contain enough detail to tell the end user what went wrong. when encountering errors in lists, typedload slows down due to keeping compatibility with python 3.7. However in my test with loading objects, despite the slowdown it remains faster than pydantic.

cirospaciari 4y ago

uvicorn uses uvloop that is an wrapper to libuv ;) almost any performance focused package use native extensions (CFFI, Cython, HPy, Python CAPI etc)

Ensorceled 4y ago

I don't really understand this criticism nor why it's "clickbait". I have 12 "unpure" python packages in production. If my app ever needs WSGI, I'll have 13.

ModernMech 4y ago
The reason this is clickbait is because you can wrap libuv in some other language and get the same results. So what’s the purpose of this article? That a language can wrap libuv is not a new or interesting idea.
- pantsforbirds 4y ago
  Why do people only get upset about this sort of thing when it comes to web servers? I don't see the same nitpicks when its a BLAS library, or when it uses a CUDA kernel under the hood. Obviously python is a slow language and needs the C implementation to be fast. You could say the same thing for the vast majority of the python code base.

commitpizza 4y ago

Node.js uses libuv also but it is way slower and tbh, can't you compare node to other things because of that it is using v8 and libuv? I don't understand this this logic.