What is the issue with the ID structure? I would think generating your own ID's would make the process of updating rows when you get new data much more difficul

What is the issue with the ID structure? I would think generating your own ID's would make the process of updating rows when you get new data much more difficult. Also, is there a reason why dropping the database and adding these records is an issue. Updating records could certainly add some complexity in terms of the data in the file changing. I would consider going one of two ways: A: Do all of the data transformation in Postgres. Load it into a staging table using COPY and load the data into a target table. B: Do the data transformation in Python and just load it into a target table in Postgres. I have had some pretty good success using pandas and pandas.toSQL to do this sort of thing with files > 1 MB but <1GB. It basically comes down to whether you are more comfortable manipulating the data in SQL or Python