3 ms·
Do you have any simple mechanism for bulk import, either from CSV or JSON files, or existing Postgres? Googled a bit and didn't see how to bulk load an existing
by burlesona 5y ago
Do you have any simple mechanism for bulk import, either from CSV or JSON files, or existing Postgres? Googled a bit and didn't see how to bulk load an existing dataset.
- RedCrowbar 5y agoThere aren't any special "bulk" interfaces at the moment, you can load data via a small program using our language bindings (INSERT in a loop). We'll add support for batched `executemany()` soon to cater to very large loads specifically. A similar optimization in asyncpg [1] led to a 10x improvement in throughput (~100Krec/sec per connection), and we expect similar performance from EdgeDB once the protocol and the bindings are updated. Loading arbitrary JSON by introspecting the data, and auto-generating the schema and the necessary DML statements is fairly straightforward too. I have a rough Python proof-of-concept, will need to find time to finish and publish it. [1] https://github.com/MagicStack/asyncpg/pull/295 https://github.com/MagicStack/asyncpg/pull/295