5 ms·

How does this compare/contrast to polars? Seems pretty similar, anybody tried both?

by bionhoward 2y ago

thebuilderjr 2y ago
[flagged]
- donor20 2y ago
  And the thing is - single node can still scale ridiculously high without the orchestration overheads of distributed stuff. You can do dual AMD 192 core CPU's (384 cores / 768 threads) with 9 TB of memory and a 24 disk SSD array in a 2U box.
  - spratzt 2y ago
    The vast majority of businesses will never need more than single node architecture. Hardware advances are continually increasing that percentage. SPARK and its modern counterpart Databricks are essentially obsolete for these organizations. Whatever justification they may have had in the past is no longer true. I’ve recently closed down several in house SPARK clusters and replaced them with single nodes. In addition to the simplicity of the design and reduction in cost there was a massive increase in performance. I expect this will become more common in the future; leaving distributed architecture for a small and increasingly niche group.
- elasticventures 2y ago
  Exactly, datafusion is implied batteries included apache bigdata ecosystem. Polars is chasing the Python Pandas crowd and uses python syntax, handy if you're already comfortable with ipython.
  - lidavidm 2y ago
    Can't you use DataFusion single node/without any Apache ecosystem stuff? They have a Python library and DataFusion is "just" a query engine. (If anything, I'd call Pandas the batteries included option...) I think the difference is more that DataFusion is built as a library so you can plug it into the product you're building (e.g. Comet, which plugs it into Spark, or pg_lakehouse, which plugs it into Postgres). Polars could be used that way, but it's also a functional package you can pip install and use as a Pandas alternative right now.
    - deleted 2y ago
      [deleted]
    - Epa095 2y ago
      "pg_analytics (formerly named pg_lakehouse) puts DuckDB inside Postgres" https://github.com/paradedb/pg_analytics https://github.com/paradedb/pg_analytics
      - lidavidm 2y ago
        It used to use DataFusion. https://www.paradedb.com/blog/iceberg_lakehouse https://www.paradedb.com/blog/iceberg_lakehouse
        philippemnoel 2y ago
        That's true. We have some more ideas for DataFusion in the works, though... Stay tuned!
Omega359 2y ago
When I started on my path to get my application off of Spark late 2023 I started with Polars because it seemed to have more community velocity and seemed more approachable. Unfortunately, at least at that time, the lazy evaluation Rust api was very much a wip and didn't work for my use case. Switching to DataFusion enabled me to port/rewrite my application into Rust and drastically improve it's performance.

How does this compare/contrast to polars? Seems pretty similar, anybody tried both?

thebuilderjr 2y ago

[flagged]

donor20 2y ago
And the thing is - single node can still scale ridiculously high without the orchestration overheads of distributed stuff. You can do dual AMD 192 core CPU's (384 cores / 768 threads) with 9 TB of memory and a 24 disk SSD array in a 2U box.
- spratzt 2y ago
  The vast majority of businesses will never need more than single node architecture. Hardware advances are continually increasing that percentage. SPARK and its modern counterpart Databricks are essentially obsolete for these organizations. Whatever justification they may have had in the past is no longer true. I’ve recently closed down several in house SPARK clusters and replaced them with single nodes. In addition to the simplicity of the design and reduction in cost there was a massive increase in performance. I expect this will become more common in the future; leaving distributed architecture for a small and increasingly niche group.
elasticventures 2y ago
Exactly, datafusion is implied batteries included apache bigdata ecosystem. Polars is chasing the Python Pandas crowd and uses python syntax, handy if you're already comfortable with ipython.
- lidavidm 2y ago
  Can't you use DataFusion single node/without any Apache ecosystem stuff? They have a Python library and DataFusion is "just" a query engine. (If anything, I'd call Pandas the batteries included option...) I think the difference is more that DataFusion is built as a library so you can plug it into the product you're building (e.g. Comet, which plugs it into Spark, or pg_lakehouse, which plugs it into Postgres). Polars could be used that way, but it's also a functional package you can pip install and use as a Pandas alternative right now.
  - deleted 2y ago
    [deleted]
  - Epa095 2y ago
    "pg_analytics (formerly named pg_lakehouse) puts DuckDB inside Postgres" https://github.com/paradedb/pg_analytics https://github.com/paradedb/pg_analytics
    - lidavidm 2y ago
      It used to use DataFusion. https://www.paradedb.com/blog/iceberg_lakehouse https://www.paradedb.com/blog/iceberg_lakehouse
      - philippemnoel 2y ago
        That's true. We have some more ideas for DataFusion in the works, though... Stay tuned!

Omega359 2y ago

When I started on my path to get my application off of Spark late 2023 I started with Polars because it seemed to have more community velocity and seemed more approachable. Unfortunately, at least at that time, the lazy evaluation Rust api was very much a wip and didn't work for my use case. Switching to DataFusion enabled me to port/rewrite my application into Rust and drastically improve it's performance.