3 ms·

Can Partitioning be used to move data to S3 Storage, for long term archiving?

• ignaciovdk 2mo ago
  No, and with timescaledb is a feature of their cloud platform. For on prem you can use something like Arc: https://github.com/Basekick-Labs/arc https://github.com/Basekick-Labs/arc
  • aleks_me2 2mo ago
    Thank you. I have decided to use clickhouse with that config because of missing S3 for logs and metrics for long term store. <clickhouse> <storage_configuration> <disks> <audit_s3> <type>s3</type> <endpoint>https://S3-EndPoint/{{ audit_bucket_name }}/clickhouse/</endpoint> <access_key_id>{{ clickhouse_audit_s3_access_key }}</access_key_id> <secret_access_key>{{ clickhouse_audit_s3_secret_key }}</secret_access_key> </audit_s3> </disks> <policies> <audit_tiered> <volumes> <default> <disk>default</disk> </default> <audit_s3> <disk>audit_s3</disk> </audit_s3> </volumes> </audit_tiered> </policies> </storage_configuration> </clickhouse>
• __s 2mo ago
  In theory yes, just need s3 fdw I demonstrated this with ClickHouse: https://github.com/ClickHouse/pg_clickhouse/blob/main/doc/offload-partition.sql https://github.com/ClickHouse/pg_clickhouse/blob/main/doc/of... We're working on a chdb based mechanism to copy to/from s3, maybe with fdw on top we can back table in s3 You can try similar things with pg_duckdb & pg_lake