3 ms·

The biggest individual CSV file is 48.5MB. Together uncompressed is ~750MB.

• tixocloud 9y ago
  Are we talking about stitching 5*x CSVs that are each close to 1GB each? I would love to help you out as this is what I do on a daily basis. Maybe I am misunderstanding the issue but couldn't you just load all the CSVs and then build queries to according to your preferred schema?
  • SnowingXIV 9y ago
    I actually managed to load all CSVs into a postgres database. I have one database now with 5 tables that contain all the information. Now the tricky part is doing the restructure, stripping out unneeded columns, some if statements to choose the correct column to use, connecting the relationships, and redoing the IDs and making that as painless as possible (hopefully a query I can repeat) since I'll likely have a new fresh batch of data coming in a month. All together the CSVs totaled 750MBs.
    • tixocloud 9y ago
      Gotcha. Sure, I can help you out. Send me an email and we can connect.