5 ms·
Bulk loading into PostgreSQL: Options and comparison
- mbell 6y agoI feel like the article misses most of the 'meat' of bulk loading setups. e.g. Using partitioned tables, attaching partitions with little to no locking, indexing after loading, `COPY ... FREEZE`, etc. One other note: > Goto solution for bulk loading into PostgreSQL is the native copy command. But one limitation with the copy command is that it requires the CSV file to be placed on the server. You can use `COPY ... FROM STDIN` and stream the data from the client, this is basically what `/copy` does in psql.
- hamandcheese 6y agoThe article did compare loading with index vs index after load. What good do partitions do? Loading in parallel?
- mbell 6y agoOne use case I've used them for is when you have a table that is 'under fire', and you need to add bulk data to it. Doing a large insert into a table like that can cause various issues. Instead I a partitioned table and each bulk load is a separate partition. This allows doing the bulk load to a 'clean' table out of the line of fire. You can then apply indexes, do cache warming or whatever else you may want to do to it, then attach the partition to the parent table to make it 'active'. As long as you have covering constraint on the partition that matches the partition constraint, this results is minimal impact to query performance. We do bulk loads of partitions with ~100-200M rows this way and it's barely noticeable to the query side latency.
- dilatedmind 6y agoThe best use case I've had for partitioning tables is when there is a lot of churn that would otherwise result in significant vacuuming. For example, a table holding the past 7 days of data would require deleting 1/7 of its rows every day. Swapping out a child table and dropping the old one is free in that sense, but does require an exclusive lock which can cause issues if you have long running queries.
- momothereal 6y agoWhat about cases where there's multiple tables with foreign keys between them? Should you just drop the constraints, import tables individually, and then re-apply the constraints? Any alternatives?
- deleted 6y ago[deleted]
- latch 6y agoNot sure if this solves it in all cases, but foreign key constraint checks can be deferred until the end of the transaction. https://www.postgresql.org/docs/9.1/sql-set-constraints.html https://www.postgresql.org/docs/9.1/sql-set-constraints.html
- paulrjones 6y agoPostgres allows for deferring constraints, so you can do an alter command at the start to defer them, then upon commit all of the foreign keys are checked. Makes it hugely easier to load when you don’t need to do it in dependency order.
- gigatexal 6y agoWhat happens to rows that fail the constraint check after load?
- momothereal 6y agoIt'll just throw an error when COMMITing, most likely.
- systemvoltage 6y agoNit: The plots are hard to follow to the point of being useless. I need to skip the entire group (4 bars) to compare against each other. Bar charts work best when the bars you want to compare are adjacent, not organized by groups. Anyways, the table is far more useful here.
- ohnoesjmr 6y agoAlso, the choice of colours is incredibly poor, different shades of blue don't work with people for some forms of color blindness.
- chris_overseas 6y agoThere are some good tips from a previous HN post on the same topic here: https://news.ycombinator.com/item?id=20403284 https://news.ycombinator.com/item?id=20403284
- zipidedoda 6y agoNot logging with pg_bulkload will win every time. However, not logging transactions is not something that should be taken lightly nor used for every bulk load operation. For instance, if replicating or synchronizing data to another db or using logs for such activity than you'll need the load transactions to perform the same operation on that other db.
- mslot 6y agoThe important thing to remember is to use COPY .. FROM STDIN, not insert, for bulk loading into PostgreSQL. Most PostgreSQL drivers support COPY from the client, though it's not always available through generic database APIs. COPY commands typically write hundreds of thousands of rows per second on a large server/cluster. It's useful to write over multiple connections, but rarely more than ~16.
- tlocke 6y agoThis is true. I'm the maintainer of pg8000, a Python driver for PostgreSQL. Here are the docs for how you'd do this using pg8000: https://github.com/tlocke/pg8000#copy-from-and-to-a-file https://github.com/tlocke/pg8000#copy-from-and-to-a-file The standard API for database access in Python is DB-API 2 and it doesn't include support for COPY , so each driver may implement it differently. There's another aspect to this, and that's the format of the file to be ingested as it can be 'text', 'CSV' or 'binary'. If you're generating the file yourself then you have a choice, but whether 'binary' is faster than 'CSV' I just don't know.
- devbug 6y agoWorth noting that you pass in data in a proprietary binary format. [1] I've done it, not out of performance concerns, but because transcoding between proprietary binary formats with types is a lot saner than the alternative. [1]: https://www.postgresql.org/docs/current/sql-copy.html#id-1.9.3.55.9.4 https://www.postgresql.org/docs/current/sql-copy.html#id-1.9...
- mslot 6y agoOne caveat to keep in mind when using the binary format is that arrays of custom types are not portable across databases because the serialized array contains the OID of the custom type, which may be different on the other end. The other thing to keep in mind is that text or CSV can be much more compact for data sets with many small integers or NULLs. On the other hand, the binary format is much more compact for timestamps and floating point numbers. In general, binary format has lower parsing overhead.
- phoe-krk 6y agoThis article, for whatever reason, doesn't mention pgloader! https://pgloader.io/ https://pgloader.io/
- dunefox 6y agoWritten in Common Lisp which is pretty cool.
- nemoniac 6y agoCould you comment on how it compares to the methods benchmarked?
- phoe-krk 6y agoSadly can't - I have never needed to use any bulk loading Postgres utilities, so I don't have the necessary experience. Still, I have had some glances at pgloader as a Common Lisp program, and I'm aware that it can migrate from X to PostgreSQL, where X includes multiple distinct databases and CSV files. That's why I've mentioned it here.
- jskrablin 6y agoDepending on use case one can consider creating target table as UNLOGGED, then altering it after bulk data load. It makes for a really fast load, but it won't survive a server crash so altering it back to 'set logged' after the data loading is recommended.
- ilikepi 6y agoI'm slightly surprised to not see any mention of materialized views. We're in the middle of a project in which we use file_fdw to provide an interface to CSV files which are replaced nightly, but we then also have indexed materialized views to provide faster access to the data in those files. The CSVs are only read once per day when REFRESHing the materialized views after a new set of files is dropped in. One of the nice things file_fdw is that it allows you to create foreign tables even when the source files don't exist on the filesystem. PG only cares about the files at the moment it's trying to read them. There is one downside, though, which is that the files have to be specified via absolute path. This is something that has to be considered given the heterogeneous machine environments on which the application runs.