4 ms·

We tried to work with DMS to migrate mysql to postgres, and it was a nightmare. Support was useless, and would often just not get back to us without prodding. T

We tried to work with DMS to migrate mysql to postgres, and it was a nightmare. Support was useless, and would often just not get back to us without prodding. Then them giving us canned responses unrelated to our queries. The whole thing is nigh on un-debuggable. Stay away.

rjh29 3y ago
Tried it for mysql and it was flat out broken and silently corrupted data.
- Topgamer7 3y ago
  For us it seemed to be trying to load data from the wrong table on mysql, into the wrong table on postgres.
mey 3y ago
I worked on migrating our MySQL system to PostgreSQL using pgloader ( https://pgloader.io/ https://pgloader.io/ ). There were some hiccups, things that needed clarification in documentation, and some additional processes that needed to be done outside of the system to get everything we need in place, it was a amazing help. Not sure the project would've been possible without it. Data mapping from PostgreSQL to PostgreSQL as in the article isn't nearly as bad as going between systems. We took a full extended outage and didn't preload any data. There were many dry runs before hand and validation before hand, but the system wasn't so mission critical that we couldn't afford to shutoff the system for a couple of hours.
- Topgamer7 3y ago
  We also ended up using pgloader. Its not without its friction either. For example the concurrency/number of rows setting seems broken out the gate, and its like playing darts with a blindfold on to get it to run without running out of memory. But being able to read the source, github actions, and overall at least I could debug my problems, or find others who had the same issue. Would recommend pgloader.
Spivak 3y ago
This was my experience as well. We thought it was a nice managed way to move pg to pg with minimal setup but we ran into so many issues we just did logical replication with bespoke fixes for things it didn't handle well.
NomDePlum 3y ago
Surprised by that. I've used AWS DMS quite a lot to do both on-prem to AWS and AWS (MySQL) to AWS Postgres migrations and long term ongoing replication. Whilst there is some complexity/gotchas there it's always been more than up to the task. Takes a little bit of validation/testing to understand but it's very well documented too. What sort of issues did you hit? In all honesty I'm not sure I've been more impressed by another AWS service.
- tibbon 3y ago
  One issue we hit were any schema changes totally messed it up. I don't have my notes in front of me, but we were constantly hitting data that wouldn't migrate, or that things suddenly broke whenever things changed.
  - NomDePlum 3y ago
    Interesting we managed several schema changes when using AWS DMS for replicating data over a long period between MySQL and Postgres clusters. We treated these carefully and tested as we made them but never had any real issues with them. From memory DMS could cope with adding columns pretty transparently. One setting we invested in configuring and understanding was to allow DMS to attempt to recover from replication failures. This allowed it to error on DDL changes and attempt to recover. This usually involved restarting the task, but it would do this transparently as part of the recovery.
- Topgamer7 3y ago
  It seemed to try to load data into the wrong table on postgres. That was the one that immediately comes to mind. Honestly poor support is what really killed it for us. But we had other technical problems with it. We burned 3 weeks just trying to get support to provide a sensible response. I never got the sense anyone replying to us knew any more than the surface level about the infrastructure of how DMS worked.
  - NomDePlum 3y ago
    We kept our table mappings pretty much mirrors of the source tables. Any data transformation we managed on the target cluster, not through AWS DMS. I've used it fairly frequently over a number of years so maybe the issues we hit on the learning curve have dimmed. We also deliberately kept things as simple as possible at first and focused on DMS's ability to move the data from source to target and really only tweaked settings that increased security or reliability. We stayed well away from any of the data transformation or more complex functionality.
extesy 3y ago
I can confirm that. Both DMS reliability and support quality were terrible.
tibbon 3y ago
We also tried to use DMS for a few things (Rackspace to AWS migration, replication out to data lakes, etc) and it has been consistently undersupported, buggy and ate months of time before we went to other solutions. While a lot of AWS support has been good; not for DMS. It feels entirely half baked .
troublebucket 3y ago
+1 DMS is very half baked. Silent, unbuggable errors. Tons of unsupported LOB and CLOB data types. Built-in Postgres logical replication is way easier.
deleted 3y ago
[deleted]

rjh29 3y ago

Tried it for mysql and it was flat out broken and silently corrupted data.

Topgamer7 3y ago
For us it seemed to be trying to load data from the wrong table on mysql, into the wrong table on postgres.

mey 3y ago

I worked on migrating our MySQL system to PostgreSQL using pgloader ( https://pgloader.io/ https://pgloader.io/ ). There were some hiccups, things that needed clarification in documentation, and some additional processes that needed to be done outside of the system to get everything we need in place, it was a amazing help. Not sure the project would've been possible without it. Data mapping from PostgreSQL to PostgreSQL as in the article isn't nearly as bad as going between systems. We took a full extended outage and didn't preload any data. There were many dry runs before hand and validation before hand, but the system wasn't so mission critical that we couldn't afford to shutoff the system for a couple of hours.

Topgamer7 3y ago
We also ended up using pgloader. Its not without its friction either. For example the concurrency/number of rows setting seems broken out the gate, and its like playing darts with a blindfold on to get it to run without running out of memory. But being able to read the source, github actions, and overall at least I could debug my problems, or find others who had the same issue. Would recommend pgloader.

Spivak 3y ago

This was my experience as well. We thought it was a nice managed way to move pg to pg with minimal setup but we ran into so many issues we just did logical replication with bespoke fixes for things it didn't handle well.

NomDePlum 3y ago

Surprised by that. I've used AWS DMS quite a lot to do both on-prem to AWS and AWS (MySQL) to AWS Postgres migrations and long term ongoing replication. Whilst there is some complexity/gotchas there it's always been more than up to the task. Takes a little bit of validation/testing to understand but it's very well documented too. What sort of issues did you hit? In all honesty I'm not sure I've been more impressed by another AWS service.

tibbon 3y ago
One issue we hit were any schema changes totally messed it up. I don't have my notes in front of me, but we were constantly hitting data that wouldn't migrate, or that things suddenly broke whenever things changed.
- NomDePlum 3y ago
  Interesting we managed several schema changes when using AWS DMS for replicating data over a long period between MySQL and Postgres clusters. We treated these carefully and tested as we made them but never had any real issues with them. From memory DMS could cope with adding columns pretty transparently. One setting we invested in configuring and understanding was to allow DMS to attempt to recover from replication failures. This allowed it to error on DDL changes and attempt to recover. This usually involved restarting the task, but it would do this transparently as part of the recovery.
Topgamer7 3y ago
It seemed to try to load data into the wrong table on postgres. That was the one that immediately comes to mind. Honestly poor support is what really killed it for us. But we had other technical problems with it. We burned 3 weeks just trying to get support to provide a sensible response. I never got the sense anyone replying to us knew any more than the surface level about the infrastructure of how DMS worked.
- NomDePlum 3y ago
  We kept our table mappings pretty much mirrors of the source tables. Any data transformation we managed on the target cluster, not through AWS DMS. I've used it fairly frequently over a number of years so maybe the issues we hit on the learning curve have dimmed. We also deliberately kept things as simple as possible at first and focused on DMS's ability to move the data from source to target and really only tweaked settings that increased security or reliability. We stayed well away from any of the data transformation or more complex functionality.

extesy 3y ago

I can confirm that. Both DMS reliability and support quality were terrible.

tibbon 3y ago

We also tried to use DMS for a few things (Rackspace to AWS migration, replication out to data lakes, etc) and it has been consistently undersupported, buggy and ate months of time before we went to other solutions. While a lot of AWS support has been good; not for DMS. It feels entirely half baked .

troublebucket 3y ago

+1 DMS is very half baked. Silent, unbuggable errors. Tons of unsupported LOB and CLOB data types. Built-in Postgres logical replication is way easier.

deleted 3y ago

[deleted]