pgguru

4 ms·

1.
by pgguru 1y ago
I think we have recently merged (or are getting ready to merge) REST catalog support, so that will open some things up in this department.
2.
by pgguru 1y ago
For testing, we at least have a Dockerfile to automate the setup of the pgduck_server and a minio instance so it Just Works™ with the extensions installed in your local Postgres cluster (after installing the extensions). The configuration m
3.
by pgguru 1y ago
Hypertables definitely had the arrays columns auto-expanding with the custom node type. Not sure what else it would look like for what you describe. That said, don't sleep on the "this is awesome" parts in this project... my
4.
by pgguru 1y ago
Boils down to design decisions; see: https://news.ycombinator.com/item?id=45813631
5.
by pgguru 1y ago
Well, dealing with large analytics queries will always perform better with larger amounts of memory... :D You can perhaps tune things to perform based on the amount of system memory (IME 80% is what DuckDB targets if not otherwise configur
6.
by pgguru 1y ago
We have some level of external iceberg table read-only support, but it is limited at the moment. See this example/caveat: https://github.com/Snowflake-Labs/pg_lake/blob/main/docs/fil...
7.
by pgguru 1y ago
DuckDB provided a lot of infrastructure for reading/writing parquet files and other common formats here. It also was inherently multi-threaded and supported being embedded in a larger program (similar to sqllite), so made it a good ba
8.
by pgguru 1y ago
In any query engine you can execute the same query in different ways. The more restrictions that you can apply on the DuckDB side the less data you need to return to Postgres. For instance, you could compute a `SELECT COUNT(*) FROM mytable
9.
by pgguru 1y ago
Hi, what types are you expecting to see that aren't supported? I believe we had support for most/all builtin postgres types.
10.
by pgguru 1y ago
DuckDB secrets management supports custom IAM roles and the like; at this point we are basically treating the pgduck_server external system as a black box. For the postgres grants themselves, we provide privs to allow read/write to the
11.
by pgguru 1y ago
You create foreign tables in postgres using either the pg_lake_table wrapper or pg_lake_iceberg. Once those tables exist, queries against them are able to either push down entirely to the remote tables and uses a Custom Scan to execute and
12.
by pgguru 1y ago
What has been pointed out from the README; also: - Separation of concerns, since with a single external process we can share object store caches without complicated locking dances between multiple processes. - Memory limits are easier to re
13.
by pgguru 1y ago
Hi, one of the developers here. You define credentials that can access the S3 buckets and use those as DuckDB secrets, usually in an init script for pgduck_server. (You can see some examples of this in the testing framework.) I'll se