3 ms·
Our application has an article catalog. There's an article number (primary key), along with description, classification codes, weights and other things, in addi
by magicalhippo 6y ago
Our application has an article catalog. There's an article number (primary key), along with description, classification codes, weights and other things, in addition to specific stuff for our application.
Our users have integrations which can update the catalogs "common" data. Descriptions can change, classification codes can change etc. A few users have several million articles, and they simply hand us a dump of them each night.
I tried initially handling this client side, load the articles, check if each article already existed, if so update else insert. If no changes don't do anything. I quickly discovered it used 4-5 hours each night when one of these big catalog changes started rolling in.
Then I rewrote it to issuing "insert ... on conflict" and just overwrote everything. Now it uses 15-20 minutes tops.