3 ms·

You can do a shitload of data cleanup and transformation stuff with python and pandas (or more precise: using data frames) which only does not give you the idea

You can do a shitload of data cleanup and transformation stuff with python and pandas (or more precise: using data frames) which only does not give you the idea of having super powers but also makes the whole process actually fun. Loading the data is simply: df = pd.read_csv(...) http://pandas.pydata.org/pandas-docs/stable/io.html#io-read-csv-table http://pandas.pydata.org/pandas-docs/stable/io.html#io-read-... where pd is an alias for pandas (import pandas as pd), df is the var in which the data frame is stored and the ... is the set of parameters required to load and parse your csv (see the docs) then for example if you for example need to find all rows where a certain column containing an empty string, something like: df_without_empty_my_columns = df[my_column != ''] http://pandas.pydata.org/pandas-docs/stable/comparison_with_sql.html http://pandas.pydata.org/pandas-docs/stable/comparison_with_... Finally, as an example, to export the refined data frame to some sql database supported by sqlalchemy: refined_df.to_sql(...) http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html#pandas.DataFrame.to_sql http://pandas.pydata.org/pandas-docs/stable/generated/pandas... Background: My team is currently doing ETL through pandas for a data analytics platform and we kinda love it. Check out http://pandas.pydata.org/pandas-docs/stable/ http://pandas.pydata.org/pandas-docs/stable/ for the entire set of docs if you are interested. Have fun!

BrandonBradley 9y ago
I have recently finished (today) some ETL work with Pandas. Pandas touts time series resampling features that are hard to match anywhere else. Disclaimer: Contributed performance improvements to Pandas around two years ago.
auxym 9y ago
Pandas is however limited to what can fit in memory, and can hit that limit pretty quick, unfortunately. If your CSVs are pretty big, or your machine is limited in memory, you could consider dask dataframes. It is supposed to have a similar api to Pandas, but it can keep most data on disk if needed, I believe. Edit: just saw your other post. You shouldn't have any problems using plain pandas with a 750 mb dataset on any sort of modern computers with a few Gb's of memory.

BrandonBradley 9y ago

I have recently finished (today) some ETL work with Pandas. Pandas touts time series resampling features that are hard to match anywhere else. Disclaimer: Contributed performance improvements to Pandas around two years ago.

auxym 9y ago

Pandas is however limited to what can fit in memory, and can hit that limit pretty quick, unfortunately. If your CSVs are pretty big, or your machine is limited in memory, you could consider dask dataframes. It is supposed to have a similar api to Pandas, but it can keep most data on disk if needed, I believe. Edit: just saw your other post. You shouldn't have any problems using plain pandas with a 750 mb dataset on any sort of modern computers with a few Gb's of memory.