3 ms·

Yes, that's essentially the point being made here. It's a fair comparison if your intent is to run this kind of job as quickly as possible.

chaps 4y ago
No it's not. It tells me exactly nothing useful about the postgresml performance because it's impossible for me to rule out redis and the http server when factoring in performance. It's two hops, with a guaranteed delay that the postgres setup won't have. If they wanted it to be a fair comparison they should have used FDWs to connect to the same redis and http server that the python benchmarks tested against.
- vasco 4y ago
  The point is you don't need those hops if you use postgresML. It's like if I told you to move to a place where you can walk 5mins to work, and you tell me it's not a fair comparison because right now you have to drive to the station and then get on a train and you're interested in a comparison where you walk to the train instead. You don't need the train because you're already there! You don't need the network hops exactly because the data is already there in the right way.
  - chaps 4y ago
    I get the point of the post, but I still don't see how it's remotely useful to understand the performance in postgresml, as someone who's interested in using it for my own tooling. Maybe I don't spend enough time in ML space to know how often they use HTTP/redis in their flows as much as they do. Most of my stuff is just data on-disk, where adding two additional services would be embarrassingly overkill. Don't you think it would be incredibly useful as a baseline if they included a third test with FDWs against redis and the http server?
    - theamk 4y ago
      Are there any other FDWs that do ML inference? Remember, this is not plain file serving -- this is actually invoking XGBoost library which does complex mathematical operations. The user does not get data from disk, they get inference results. Unless you know of any other solution which can invoke XGBoost (or some other inference library), I don't see anything "embarrassingly overkill" there.
      - chaps 4y ago
        My issue isn't with the inference step or even the reading step, it's the fetching step.
        montanalow 4y ago
        How are you doing online ML inference, without fetching data?
  - FreakLegion 4y ago
    You don't need those hops if you use Python either. Python runs inside Postgres. https://www.postgresql.org/docs/current/plpython.html https://www.postgresql.org/docs/current/plpython.html Naturally Rust or C functions will still be faster.
    - levkk 4y ago
      PostgresML v1.0 was doing exactly that. When we rewrote in Rust for v2.0, we improved 35x: https://postgresml.org/blog/postgresml-is-moving-to-rust-for-our-2.0-release/ https://postgresml.org/blog/postgresml-is-moving-to-rust-for...
- darksaints 4y ago
  The article very clearly was pointing out that the performance advantage of postgresml comes from the architecture. Hell, they are even using the same exact algorithm. What benefit is to be had from benchmarking the same algorithm on the same architecture? Do we also need to make sure Teslas have internal combustion engines when we compare their performance to ICE cars?
pushedx 4y ago
I also don’t think it’s a fair comparison. There’s nothing stopping me from loading the model into the memory of each Flask process (or some shmem), and getting the same performance or possibly better than the Postgres implementation, considering coroutines are being used in the Python case. Calling this Postgres vs Flask is misleading at best. It’s more like “1 tier architecture vs 2 tier architecture”
- montanalow 4y ago
  You get it. 1 tier is better than 2 tier. Python can't be 1 tier, unless it loads the full dataset which is not generally feasible for production online inference cases. PostgresML is 1 tier, and supports the traditional Python use cases.
  - xapata 4y ago
    Why can't Python be 1 tier? It's a general-purpose, extensible language. It can do anything that PostgreSQL can do.

chaps 4y ago

No it's not. It tells me exactly nothing useful about the postgresml performance because it's impossible for me to rule out redis and the http server when factoring in performance. It's two hops, with a guaranteed delay that the postgres setup won't have. If they wanted it to be a fair comparison they should have used FDWs to connect to the same redis and http server that the python benchmarks tested against.

vasco 4y ago
The point is you don't need those hops if you use postgresML. It's like if I told you to move to a place where you can walk 5mins to work, and you tell me it's not a fair comparison because right now you have to drive to the station and then get on a train and you're interested in a comparison where you walk to the train instead. You don't need the train because you're already there! You don't need the network hops exactly because the data is already there in the right way.
- chaps 4y ago
  I get the point of the post, but I still don't see how it's remotely useful to understand the performance in postgresml, as someone who's interested in using it for my own tooling. Maybe I don't spend enough time in ML space to know how often they use HTTP/redis in their flows as much as they do. Most of my stuff is just data on-disk, where adding two additional services would be embarrassingly overkill. Don't you think it would be incredibly useful as a baseline if they included a third test with FDWs against redis and the http server?
  - theamk 4y ago
    Are there any other FDWs that do ML inference? Remember, this is not plain file serving -- this is actually invoking XGBoost library which does complex mathematical operations. The user does not get data from disk, they get inference results. Unless you know of any other solution which can invoke XGBoost (or some other inference library), I don't see anything "embarrassingly overkill" there.
    - chaps 4y ago
      My issue isn't with the inference step or even the reading step, it's the fetching step.
      - montanalow 4y ago
        How are you doing online ML inference, without fetching data?
- FreakLegion 4y ago
  You don't need those hops if you use Python either. Python runs inside Postgres. https://www.postgresql.org/docs/current/plpython.html https://www.postgresql.org/docs/current/plpython.html Naturally Rust or C functions will still be faster.
  - levkk 4y ago
    PostgresML v1.0 was doing exactly that. When we rewrote in Rust for v2.0, we improved 35x: https://postgresml.org/blog/postgresml-is-moving-to-rust-for-our-2.0-release/ https://postgresml.org/blog/postgresml-is-moving-to-rust-for...
darksaints 4y ago
The article very clearly was pointing out that the performance advantage of postgresml comes from the architecture. Hell, they are even using the same exact algorithm. What benefit is to be had from benchmarking the same algorithm on the same architecture? Do we also need to make sure Teslas have internal combustion engines when we compare their performance to ICE cars?

pushedx 4y ago

I also don’t think it’s a fair comparison. There’s nothing stopping me from loading the model into the memory of each Flask process (or some shmem), and getting the same performance or possibly better than the Postgres implementation, considering coroutines are being used in the Python case. Calling this Postgres vs Flask is misleading at best. It’s more like “1 tier architecture vs 2 tier architecture”

montanalow 4y ago
You get it. 1 tier is better than 2 tier. Python can't be 1 tier, unless it loads the full dataset which is not generally feasible for production online inference cases. PostgresML is 1 tier, and supports the traditional Python use cases.
- xapata 4y ago
  Why can't Python be 1 tier? It's a general-purpose, extensible language. It can do anything that PostgreSQL can do.