4 ms·

You're exactly right about log streaming to S3. We open sourced the tool that we use to perform that: https://github.com/heroku/wal-e https://github.com/heroku/

You're exactly right about log streaming to S3. We open sourced the tool that we use to perform that: https://github.com/heroku/wal-e https://github.com/heroku/wal-e

bdonlan 15y ago
S3 doesn't support continual logging - you need to decide on a file size up-front. Are you creating a new object for each log entry, or is there a window in which commits can be lost before the log gets flushed out, or do you have some other workaround...?
- joevandyk 15y ago
  There's a window of a couple minutes in which data could be lost (in the worst case scenario).
  - sector 15y ago
    So it's postgres log shipping, with S3 as the target?
    - deleted 15y ago
      [deleted]
    - joevandyk 15y ago
      Yup.
jaylevitt 15y ago
So would I need to (learn and) use PG9's replication with this? Or is one ephemeral-stored database plus shipped logs sufficient, assuming I can live with some downtime if the instance disappears? As an experienced techie but a PG novice, what do I need to go learn about PG recovery procedures to safely use this?