4 ms·

It looks with what is currently implemented you'd have to drop the tables from the destination if you want repeated copies, probably not quite what you want. Cl

It looks with what is currently implemented you'd have to drop the tables from the destination if you want repeated copies, probably not quite what you want. Close enough to your use case that it still might be worth testing it out though.

rileymichael 2y ago
yeah that’s not an immediate deal breaker for me, I’m essentially looking for pgdump/restore + transformations. I’ll give it a look and see how it performs
- shayonj 2y ago
  Thank you for giving this a spin. That's correct, today you'd need to drop the table before another sync with pg_flo. That said, I have given delta syncs some thought, and also looking into a control plane that can make some of these things easier. Would love to hear your feedback.
  - shayonj 2y ago
    I think this also might be of interest to you - just one time copies and apply transformations - https://github.com/shayonj/pg_flo/issues/6 https://github.com/shayonj/pg_flo/issues/6. I will look into shipping that very soon.
    - rileymichael 2y ago
      Oh awesome, yeah that’d work perfect then. I’d just prep beforehand and run the one-time copy.
      - shayonj 2y ago
        amazing! Will plan a release by Tuesday latest. You can experiment with `--copy-and-stream ` in the meantime [1] [1] https://github.com/shayonj/pg_flo?tab=readme-ov-file#streaming-modes https://github.com/shayonj/pg_flo?tab=readme-ov-file#streami...
      - shayonj 2y ago
        Heya! v0.0.4 is now out with `--copy` support, which will stream all the data from the table and the transformation/routing will get applied before the changes land in destination. [1] https://github.com/shayonj/pg_flo?tab=readme-ov-file#streaming-modes https://github.com/shayonj/pg_flo?tab=readme-ov-file#streami... [2] https://github.com/shayonj/pg_flo/releases https://github.com/shayonj/pg_flo/releases
        rileymichael 2y ago
        Awesome turn around — I’ll give it a go this week!

rileymichael 2y ago

yeah that’s not an immediate deal breaker for me, I’m essentially looking for pgdump/restore + transformations. I’ll give it a look and see how it performs

shayonj 2y ago
Thank you for giving this a spin. That's correct, today you'd need to drop the table before another sync with pg_flo. That said, I have given delta syncs some thought, and also looking into a control plane that can make some of these things easier. Would love to hear your feedback.
- shayonj 2y ago
  I think this also might be of interest to you - just one time copies and apply transformations - https://github.com/shayonj/pg_flo/issues/6 https://github.com/shayonj/pg_flo/issues/6. I will look into shipping that very soon.
  - rileymichael 2y ago
    Oh awesome, yeah that’d work perfect then. I’d just prep beforehand and run the one-time copy.
    - shayonj 2y ago
      amazing! Will plan a release by Tuesday latest. You can experiment with `--copy-and-stream ` in the meantime [1] [1] https://github.com/shayonj/pg_flo?tab=readme-ov-file#streaming-modes https://github.com/shayonj/pg_flo?tab=readme-ov-file#streami...
    - shayonj 2y ago
      Heya! v0.0.4 is now out with `--copy` support, which will stream all the data from the table and the transformation/routing will get applied before the changes land in destination. [1] https://github.com/shayonj/pg_flo?tab=readme-ov-file#streaming-modes https://github.com/shayonj/pg_flo?tab=readme-ov-file#streami... [2] https://github.com/shayonj/pg_flo/releases https://github.com/shayonj/pg_flo/releases
      - rileymichael 2y ago
        Awesome turn around — I’ll give it a go this week!