3 ms·
Perhaps look into using dlt from https://dlthub.com https://dlthub.com, using pyarrow or polars. It handles large datasets well, especially when using generator
by bluedemon 2y ago
Perhaps look into using dlt from https://dlthub.com https://dlthub.com, using pyarrow or polars. It handles large datasets well, especially when using generators to process the data in chunks.