4 ms·
Not sure which DB you are using, but you can load the csv file into the DB directly on a single thread using something like LOAD DATA INFILE. If you have some
by tkyjonathan 7y ago
Not sure which DB you are using, but you can load the csv file into the DB directly on a single thread using something like LOAD DATA INFILE.
If you have some good indexes and do some push-down work (give the database aggregation tasks to do instead of your python code), you should probably be more than fine.
For a 250Gb file.. should be ok.. maybe add some partitioning too.
- hnews_account_1 7y agoI'm open to using any db that I can query over some engine with a python implementation. So any SQL db should be fine. However, I don't know how to convert a csv to an SQL directly. Is the command you mentioned part of some SQL server package? Sounds like it's exactly what I need.
- nwallin 7y agopandas can read from a CSV file and then write to SQL. Even if you don't go the SQL route, you'd probably gain significant benefits by working with HDF instead of CSV. https://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.read_csv.html#pandas.read_csv https://pandas.pydata.org/pandas-docs/version/0.22/generated... https://pandas.pydata.org/pandas-docs/version/0.22/io.html#writing-dataframes https://pandas.pydata.org/pandas-docs/version/0.22/io.html#w...