3 ms·

Let's not forget that Postgres has first-class support for Python [0] (unlike JavaScript etc which are maintained, or not, by third parties). I don't advocate g

Let's not forget that Postgres has first-class support for Python [0] (unlike JavaScript etc which are maintained, or not, by third parties). I don't advocate going for Python for simple stuff but when you do use it, it has the added benefit of having plain ol' Python functions that you can unit test thoroughly (including with the wonderful Hypothesis [1] property testing framework). [0] https://www.postgresql.org/docs/12/plpython-funcs.html https://www.postgresql.org/docs/12/plpython-funcs.html [1] https://hypothesis.works/ https://hypothesis.works/

mlthoughts2018 7y ago
The Postgres support of Python is usually way overhyped. I’ve done a lot of crazy things with it, like even compiling my own extension modules using Postgres C libraries to write Python native functions that vectorized business logic over columns like the way numpy works. It’s extremely hard to manage the Python environment itself with Postgres, as Postgres has to be compiled with it. If you’re working on scientific or analytics applications where you are spinning up a new conda environment all the time, adding & changing package dependencies, etc., and you need to keep your external-to-database Python app logic synchronized with the internal-to-database Python, it’s virtually unusable. Imagine needing to ship new versions of an in-house machine learning library into the database, test it, update that library’s own dependencies inside the database’s python, etc. It becomes a crazy packaging nightmare very fast. The plpython extension is basically just a cute toy. Occasionally useful or a small script or single transformation that needs Python standard library functions, but anything beyond that and it becomes very unscalable & unmaintainable very fast.
- steve-chavez 7y ago
  That seems more related to a lacking devops approach than to plpython capabilities. pg coupled with Nix[1] could potentially solve all of these issues. nixpkgs includes a large collection of python libs that you can use to get reproducible environments. [1]: https://nixos.org/nix/ https://nixos.org/nix/
  - mlthoughts2018 7y ago
    I think you are misunderstanding. There’s plenty of tooling for good Python env / package management, but when you need a custom compiled extension module with dependencies to maintain parity both in the database (recompiled with the custom extension packages each round trip) and outside the database, it’s a big problem. The problem is not generally making an environment, it’s that different projects need different environments but need the same database.

mlthoughts2018 7y ago

The Postgres support of Python is usually way overhyped. I’ve done a lot of crazy things with it, like even compiling my own extension modules using Postgres C libraries to write Python native functions that vectorized business logic over columns like the way numpy works. It’s extremely hard to manage the Python environment itself with Postgres, as Postgres has to be compiled with it. If you’re working on scientific or analytics applications where you are spinning up a new conda environment all the time, adding & changing package dependencies, etc., and you need to keep your external-to-database Python app logic synchronized with the internal-to-database Python, it’s virtually unusable. Imagine needing to ship new versions of an in-house machine learning library into the database, test it, update that library’s own dependencies inside the database’s python, etc. It becomes a crazy packaging nightmare very fast. The plpython extension is basically just a cute toy. Occasionally useful or a small script or single transformation that needs Python standard library functions, but anything beyond that and it becomes very unscalable & unmaintainable very fast.

steve-chavez 7y ago
That seems more related to a lacking devops approach than to plpython capabilities. pg coupled with Nix[1] could potentially solve all of these issues. nixpkgs includes a large collection of python libs that you can use to get reproducible environments. [1]: https://nixos.org/nix/ https://nixos.org/nix/
- mlthoughts2018 7y ago
  I think you are misunderstanding. There’s plenty of tooling for good Python env / package management, but when you need a custom compiled extension module with dependencies to maintain parity both in the database (recompiled with the custom extension packages each round trip) and outside the database, it’s a big problem. The problem is not generally making an environment, it’s that different projects need different environments but need the same database.