jconline543

18 ms·

1. 1.▲
   by jconline543 2y ago

   Also, for querying both recent and historical data together, you wouldn't need to modify this tool at all. You could just add a separate periodic job (e.g. hourly/daily) that copies recent data to S3: sqlCopyCOPY (SELECT * FROM io
2. 2.▲
   by jconline543 2y ago

   Thanks for your interest! Let me address your questions: Querying the data: Yes, you can easily query the Parquet files with DuckDB. The files are stored in a year/month partitioned structure (e.g., year=2024/month=03/iot_dat
3. 3.▲
   by jconline543 2y ago

   If you just need the Postgres -> S3 archival pattern you described, I built a simpler focused tool: pg-archiver ( https://github.com/johnonline35/pg-archiver ) It: - Auto-archives old Postgres data to Parquet files on