We have a need to upsert millions of records in Rails on an ongoing basis (weekly). Biggest problem we ran into was attempting to use Sidekiq and using too man

We have a need to upsert millions of records in Rails on an ongoing basis (weekly). Biggest problem we ran into was attempting to use Sidekiq and using too many workers each making smallish inserts. You don't run out of connections in that case - PG cannot handle that many parallel inserts into the same table(s). After discussing with a friend that frequently works with large data sets, he suggested that single very large upserts/inserts (50k to 100k) records would be handled much better by Postgresql, and they were. Our final setup still has all those Sidekiq workers, but they dump the records as JSON blobs into Kafka. From there a single consumer pulls them out and buffers. Once it has 50k, it dumps them all in a single ActiveRecord Upsert. Its been working very well for us for a while now.