4 ms·

The snapshots already deliver your desired functionality. This dumps it as a different format. In fact, it probably depends on restoring the snapshot to a datab

by zten 7y ago

The snapshots already deliver your desired functionality. This dumps it as a different format. In fact, it probably depends on restoring the snapshot to a database in order to implement this format change. I think this is a replacement for rolling your own database exports for analytics applications with Sqoop or Spark.

jungturk 7y ago
Exporting to parquet has existed for a bit for AWS RDS (via AWS Data Migration Service), but this should make doing so more straightforward (since it doesn't require managing any DMS compute). AWS DMS also supports incrementals (using the change-data-capture features in the DB).
- zten 7y ago
  Hmm, I wonder why they didn't push people towards DMS instead? This S3 export offering certainly commands a premium price for the privilege.
  - anbotero 7y ago
    I just wish DMS supported real-time data synchronization for utf8mb4 on MySQL... That really destroyed my workflow with errors when I moved to utf8mb4 (requirement) and then had to stop using DMS altogether. It’s pretty much the only thing I used from that services.
    - jungturk 7y ago
      There are a number of unfortunate shortcomings depending on what your DMS source/target are. Another that bit us was (the lack of) support for JSON types in MySQL sources. Was there no option to add a transformation to your DMS routine to handle that?
sandGorgon 7y ago
there is a ridiculous price difference between vanilla s3 and managed backup storage. Its close to 10x. backup storage on postgresql is billed at $ $0.095 per GiB-month same on Aurora is billed at $0.023 per GB-month If you include Glacier, then it is 100x. I truly believe that AWS charges these costs because they make a crazy amount of premium with that. And that is why these exports are architected to be non-restorable.

4 ms·

The snapshots already deliver your desired functionality. This dumps it as a different format. In fact, it probably depends on restoring the snapshot to a datab

by zten 7y ago

jungturk 7y ago
Exporting to parquet has existed for a bit for AWS RDS (via AWS Data Migration Service), but this should make doing so more straightforward (since it doesn't require managing any DMS compute). AWS DMS also supports incrementals (using the change-data-capture features in the DB).
- zten 7y ago
  Hmm, I wonder why they didn't push people towards DMS instead? This S3 export offering certainly commands a premium price for the privilege.
  - anbotero 7y ago
    I just wish DMS supported real-time data synchronization for utf8mb4 on MySQL... That really destroyed my workflow with errors when I moved to utf8mb4 (requirement) and then had to stop using DMS altogether. It’s pretty much the only thing I used from that services.
    - jungturk 7y ago
      There are a number of unfortunate shortcomings depending on what your DMS source/target are. Another that bit us was (the lack of) support for JSON types in MySQL sources. Was there no option to add a transformation to your DMS routine to handle that?
sandGorgon 7y ago
there is a ridiculous price difference between vanilla s3 and managed backup storage. Its close to 10x. backup storage on postgresql is billed at $ $0.095 per GiB-month same on Aurora is billed at $0.023 per GB-month If you include Glacier, then it is 100x. I truly believe that AWS charges these costs because they make a crazy amount of premium with that. And that is why these exports are architected to be non-restorable.