3 ms·

I did that exact migration. Unfortunately, to my knowledge, there's no way to do it with zero downtime. You need to make your app read only until the RDS instan

I did that exact migration. Unfortunately, to my knowledge, there's no way to do it with zero downtime. You need to make your app read only until the RDS instance has ingested your data, then you can cut over. For me, that was roughly one gigabyte of data and took about forty seconds. My best advice is to automate the whole thing. You can automate it with the Heroku and AWS CLIs. Test on your staging site until you can run through the whole process end to end a few times with no interruptions.

tomhallett 5y ago
Thank you for this - extremely helpful in validating the current approach and de-risking the developer time.
oauea 5y ago
Yep, absolutely garbage that these clouds (Azure is another one) don't allow you to replicate with external systems. Pretty much devalues their entire hosted postgresql offering if you ask me, since it's just designed to keep you locked in (duh). If you have any significant amount of data where you're worried about a migration, stay far away from hosted postgres offerings. You'll never get your data out without significant downtime.
barrkel 5y ago
There are other ways to handle this at the application level, to be clear, using dual read & write and backfill. More relevant when you have TB+++ of data.
- tomhallett 5y ago
  Interesting. I've done dual-writes at the application level to migrate the datastore for a smaller feature (branch by abstraction), but never for an entire application. And the code path was quite simple, so it was easy to think about all of the edge cases at one time in your head. Do you have any resources which talk through the read/write/backfill approach? Here's what I found so far: * https://medium.com/google-cloud/online-database-migration-by-dual-write-this-is-not-for-everyone-cb4307118f4b https://medium.com/google-cloud/online-database-migration-by... * https://aws.amazon.com/blogs/architecture/middleware-assisted-zero-downtime-live-database-migration-to-aws/ https://aws.amazon.com/blogs/architecture/middleware-assiste...
  - bastawhiz 5y ago
    Jumping in again... Your post reminded me that I actually typed my migration up! https://mattbasta.medium.com/migrating-from-heroku-to-aws-6d334d1640fd https://mattbasta.medium.com/migrating-from-heroku-to-aws-6d... Hopefully it's somewhat helpful!
laurent92 5y ago
So, basically, Postgres would have a replication port which can be used for both replication/clustering and transfer across cloud providers. And sharding. </dreaming>
- bastawhiz 5y ago
  I mean, it essentially does. Heroku's managed postgres has it disabled.

tomhallett 5y ago

Thank you for this - extremely helpful in validating the current approach and de-risking the developer time.

oauea 5y ago

Yep, absolutely garbage that these clouds (Azure is another one) don't allow you to replicate with external systems. Pretty much devalues their entire hosted postgresql offering if you ask me, since it's just designed to keep you locked in (duh). If you have any significant amount of data where you're worried about a migration, stay far away from hosted postgres offerings. You'll never get your data out without significant downtime.

barrkel 5y ago

There are other ways to handle this at the application level, to be clear, using dual read & write and backfill. More relevant when you have TB+++ of data.

tomhallett 5y ago
Interesting. I've done dual-writes at the application level to migrate the datastore for a smaller feature (branch by abstraction), but never for an entire application. And the code path was quite simple, so it was easy to think about all of the edge cases at one time in your head. Do you have any resources which talk through the read/write/backfill approach? Here's what I found so far: * https://medium.com/google-cloud/online-database-migration-by-dual-write-this-is-not-for-everyone-cb4307118f4b https://medium.com/google-cloud/online-database-migration-by... * https://aws.amazon.com/blogs/architecture/middleware-assisted-zero-downtime-live-database-migration-to-aws/ https://aws.amazon.com/blogs/architecture/middleware-assiste...
- bastawhiz 5y ago
  Jumping in again... Your post reminded me that I actually typed my migration up! https://mattbasta.medium.com/migrating-from-heroku-to-aws-6d334d1640fd https://mattbasta.medium.com/migrating-from-heroku-to-aws-6d... Hopefully it's somewhat helpful!

laurent92 5y ago

So, basically, Postgres would have a replication port which can be used for both replication/clustering and transfer across cloud providers. And sharding. </dreaming>

bastawhiz 5y ago
I mean, it essentially does. Heroku's managed postgres has it disabled.