4 ms·

Pg_lakehouse: A DuckDB Alternative in Postgres

67 points by eatonphil 2y ago

nikhilsimha 2y ago
duckdb is mit licensed. [1] datafusion is apache v2 licensed. [2] pg_lakehouse built on top of data fusion is AGPL v3 + business licensed. [3] [1] https://github.com/duckdb/duckdb/blob/main/LICENSE https://github.com/duckdb/duckdb/blob/main/LICENSE [2] https://github.com/apache/datafusion/blob/main/LICENSE.txt https://github.com/apache/datafusion/blob/main/LICENSE.txt [3] https://github.com/paradedb/paradedb/blob/dev/LICENSE https://github.com/paradedb/paradedb/blob/dev/LICENSE Most companies won't touch AGPL v3 license. Maybe not a bad thing, but FYI.
- swasheck 2y ago
  off-topic but i’m so licensing ignorant that each time something like this comes up i have no idea what’s being said. is there a good ELI5 resource where i can get started with licensing permissiveness?
  - corytheboyd 2y ago
    I like the website GitHub put together, here’s the AGPL v3 page: https://choosealicense.com/licenses/agpl-3.0 https://choosealicense.com/licenses/agpl-3.0
    - twelvechairs 2y ago
      For an explanation in practice see below. From someone pro-AGPL seeking enforcement of the licence. https://raymii.org/s/blog/I_enforced_the_AGPL_on_my_code_heres_how_it_went.html https://raymii.org/s/blog/I_enforced_the_AGPL_on_my_code_her...
  - duncan-donuts 2y ago
    GNU is a good source for copyleft license info. Iirc they also address other open source licenses. https://www.gnu.org/licenses/licenses.html https://www.gnu.org/licenses/licenses.html. Also https://choosealicense.com/ https://choosealicense.com/ is good for some tldr info
  - gavindean90 2y ago
    You can build closed source products into and on top of MIT and Apache licenses. AGPL code can’t be a part of a project unless you license everything that you write as a GPL kind of license as well.
- dietr1ch 2y ago
  I hate that copyleft licensed software should be viable for companies, but they are often too scared to tap into open source as a mistake they might end up forcing companies to open source everything according to faint hearted busy/lazy legal teams. At Google I was somehow allowed to use Emacs for development, but new copyleft software was immediately dismissed by legal, even if it's source was not getting into /google3 nor leaving my laptop.
- braza 2y ago
  how this type of licencing affects people that are not google et. al. in big companies? e.g. I am bootstrap/indie dev doing a small SaaS? Should I be concerned?
  - fmajid 2y ago
    It will bite you when you try to sell your company and they do due diligence. CitusDB was also AGPL-licensed at one point, not sure if they still are. Still, AGPL is a proper open-source license, unlike the sleazy fake ones adopted by Redis, Elastic or MongoDB.
  - cpleppert 2y ago
    Short answer: yes. The AGPL should be avoided at all costs because it has never been robustly tested in court and its unclear what the licensing implications actually are. There are various explanations in plain english sometimes offered about how the AGPL will apply. None of these are true. Companies that have a made a business decision to provide AGPL licensed code do so with the understanding that no serious business will ever consider using such a product in their software stack. If you choose an AGPL licensed product it will (rightly) become a gigantic headache at some point. It will certainly become a problem if anyone does due diligence.
    - philippemnoel 2y ago
      Most companies that provide AGPL, including us (ParadeDB) also offer a commercial license for interested companies. Several successful software companies (Grafana, MinIO, Citus, etc.) have chosen to be AGPL to thread the needle between being true OSS while also managing to monetize their offering :)
nitinreddy88 2y ago
I don't understand how they imagine the business usecase around this and build these products which are almost negligible In real world, I need a system which must be writing data into parquet files. If such system exist, why would I ever choose Postgres to query. I can pretty much use same data lakes/systems which are writing the data to query the data. If I want to scale horizontally, I would use something like duckdb for in-process or even simpler solutions instead of eating the transactional database process workers.
- philippemnoel 2y ago
  Many people write Parquet files to S3 from Kafka, and read it from DuckDB, Databricks, etc. And now Postgres! It's already configured with access control, caching, etc. and is where your operational data lives
gregw2 2y ago
A lakehouse solution without iceberg support? (And postgres sql DML writing to iceberg would be even better…) Poking around, the Postgres/iceberg options recently were articulated here: https://medium.com/datareply/unlocking-the-power-of-iceberg-within-postgresql-14e8ed2d8ccb https://medium.com/datareply/unlocking-the-power-of-iceberg-...
- philippemnoel 2y ago
  Iceberg support is on the horizon. The iceberg-rust crate is still nascent, but it's coming!
deleted 2y ago
[deleted]

67 points by eatonphil 2y ago

duckdb is mit licensed. [1] datafusion is apache v2 licensed. [2] pg_lakehouse built on top of data fusion is AGPL v3 + business licensed. [3] [1] https://github.com/duckdb/duckdb/blob/main/LICENSE https://github.com/duckdb/duckdb/blob/main/LICENSE [2] https://github.com/apache/datafusion/blob/main/LICENSE.txt https://github.com/apache/datafusion/blob/main/LICENSE.txt [3] https://github.com/paradedb/paradedb/blob/dev/LICENSE https://github.com/paradedb/paradedb/blob/dev/LICENSE Most companies won't touch AGPL v3 license. Maybe not a bad thing, but FYI.

swasheck 2y ago
off-topic but i’m so licensing ignorant that each time something like this comes up i have no idea what’s being said. is there a good ELI5 resource where i can get started with licensing permissiveness?
- corytheboyd 2y ago
  I like the website GitHub put together, here’s the AGPL v3 page: https://choosealicense.com/licenses/agpl-3.0 https://choosealicense.com/licenses/agpl-3.0
  - twelvechairs 2y ago
    For an explanation in practice see below. From someone pro-AGPL seeking enforcement of the licence. https://raymii.org/s/blog/I_enforced_the_AGPL_on_my_code_heres_how_it_went.html https://raymii.org/s/blog/I_enforced_the_AGPL_on_my_code_her...
- duncan-donuts 2y ago
  GNU is a good source for copyleft license info. Iirc they also address other open source licenses. https://www.gnu.org/licenses/licenses.html https://www.gnu.org/licenses/licenses.html. Also https://choosealicense.com/ https://choosealicense.com/ is good for some tldr info
- gavindean90 2y ago
  You can build closed source products into and on top of MIT and Apache licenses. AGPL code can’t be a part of a project unless you license everything that you write as a GPL kind of license as well.
dietr1ch 2y ago
I hate that copyleft licensed software should be viable for companies, but they are often too scared to tap into open source as a mistake they might end up forcing companies to open source everything according to faint hearted busy/lazy legal teams. At Google I was somehow allowed to use Emacs for development, but new copyleft software was immediately dismissed by legal, even if it's source was not getting into /google3 nor leaving my laptop.
braza 2y ago
how this type of licencing affects people that are not google et. al. in big companies? e.g. I am bootstrap/indie dev doing a small SaaS? Should I be concerned?
- fmajid 2y ago
  It will bite you when you try to sell your company and they do due diligence. CitusDB was also AGPL-licensed at one point, not sure if they still are. Still, AGPL is a proper open-source license, unlike the sleazy fake ones adopted by Redis, Elastic or MongoDB.
- cpleppert 2y ago
  Short answer: yes. The AGPL should be avoided at all costs because it has never been robustly tested in court and its unclear what the licensing implications actually are. There are various explanations in plain english sometimes offered about how the AGPL will apply. None of these are true. Companies that have a made a business decision to provide AGPL licensed code do so with the understanding that no serious business will ever consider using such a product in their software stack. If you choose an AGPL licensed product it will (rightly) become a gigantic headache at some point. It will certainly become a problem if anyone does due diligence.
  - philippemnoel 2y ago
    Most companies that provide AGPL, including us (ParadeDB) also offer a commercial license for interested companies. Several successful software companies (Grafana, MinIO, Citus, etc.) have chosen to be AGPL to thread the needle between being true OSS while also managing to monetize their offering :)

nitinreddy88 2y ago

I don't understand how they imagine the business usecase around this and build these products which are almost negligible In real world, I need a system which must be writing data into parquet files. If such system exist, why would I ever choose Postgres to query. I can pretty much use same data lakes/systems which are writing the data to query the data. If I want to scale horizontally, I would use something like duckdb for in-process or even simpler solutions instead of eating the transactional database process workers.

philippemnoel 2y ago
Many people write Parquet files to S3 from Kafka, and read it from DuckDB, Databricks, etc. And now Postgres! It's already configured with access control, caching, etc. and is where your operational data lives

gregw2 2y ago

A lakehouse solution without iceberg support? (And postgres sql DML writing to iceberg would be even better…) Poking around, the Postgres/iceberg options recently were articulated here: https://medium.com/datareply/unlocking-the-power-of-iceberg-within-postgresql-14e8ed2d8ccb https://medium.com/datareply/unlocking-the-power-of-iceberg-...

philippemnoel 2y ago
Iceberg support is on the horizon. The iceberg-rust crate is still nascent, but it's coming!

deleted 2y ago

[deleted]