3 ms·
This is a perfect example of doing something row by row instead of in a data sets is bad for anything with over a million rows. Bulk Dump / Copy Load and then r
by jtwebman 12y ago
This is a perfect example of doing something row by row instead of in a data sets is bad for anything with over a million rows. Bulk Dump / Copy Load and then run the rules to clean up data.
- zenogais 12y agoThis would have been great, but the lack of a 1-to-1 mapping between data models complicated this. To migrate a single "record" required several queries in the old database, some restructuring, and then several inserts into the new database. We were restructuring nested records in mongo into several separate relations in Postgres.
- jtwebman 12y agoYou could still copy raw and then run selecting inserts to load the real tables. This could have been done in under an hour I am sure of it.
- zenogais 12y agoWould honestly be great to know about, but how do you raw copy a nested mongo document? The big hurdle was the lack of 1-to-1 mapping between primitive types in the databases.
- jtwebman 12y agoMongoexport will let you export into flat files using --fields and --csv http://docs.mongodb.org/v2.2/reference/mongoexport/ http://docs.mongodb.org/v2.2/reference/mongoexport/
- zenogais 12y agoTotally agree that would've likely worked too, though I'm not convinced it would have been significantly faster doing a dump, import, and data processing. Dumps from mongo typically took upwards of two hours not to mention transferring the file and then importing it then processing the data. Just a different way to skin the cat.