Yeah I imagine it depends where the data is coming from and what exactly it looks like (num fields, dtypes...?). What I did was source data -> Numpy Structured

Yeah I imagine it depends where the data is coming from and what exactly it looks like (num fields, dtypes...?). What I did was source data -> Numpy Structured Array [0] -> Postgres binary [1]. Bit of a pain getting it into the required shape, but if you follow the links the code should get you going (sorry no type hints!). [0] https://rdrn.me/optimising-sampling/#round-10-off-the-deep-end https://rdrn.me/optimising-sampling/#round-10-off-the-deep-e... [1] In the original blog I linked.