4 ms·

Ask HN: Recommended language to learn data manipulation

10 points by digdugdirk 10y ago

I have a few very large datasets (in csv format) that I need to work with that completely overwhelm Excel. I'm looking for recommendations on a single language (or minimal number of languages) that I could learn in order to take these datasets in and start running various calculations on. Simplicity and future usefulness of the programming environment I'll be learning are my highest ranked criteria. I'm not necessarily looking for a how-to book (though I wouldn't turn it down) I'm really just looking for language advice. There's a lot out there on the Internet, and much of the advice is conflicting.

dozzie 10y ago
SQL. PostgreSQL has decent CSV support.
- apohn 10y ago
  100% agree. I've loaded multi-gigabyte datasets into Postgres on my laptop and SQL is a great language for manipulating and aggregating data. Plus, when you use a database and SQL you can access that data from a wide range of applications and languages via drivers (JDBC and ODBC). You can even access Postgres from Excel!
eternalban 10y ago
https://www.r-bloggers.com/handling-large-datasets-in-r/ https://www.r-bloggers.com/handling-large-datasets-in-r/
Jugurtha 10y ago
>Simplicity and future usefulness of the programming environment I'll be learning are my highest ranked criteria. Given this criteria, I think SQL would be a good choice: simple to get started with, useful in the future. SQLite doesn't get in your way.. https://sqlite.org/ https://sqlite.org/
throw_away_777 10y ago
What do you mean by large? Probably as your data is in csv format it is small enough that python or R are both good choices. Personally I'd recommend python as it has a large community using it so googling questions about python libraries such as pandas usually provides good answers.
ptype 10y ago
Will it fit in ram? Python + pandas. Pandas is really amazing. Otherwise, try postgresql or SQLite.
jackgolding 10y ago
Python Pandas is absolutely amazing for this. Have a look at the cookbook and see how you feel about the language: http://pandas.pydata.org/pandas-docs/stable/cookbook.html http://pandas.pydata.org/pandas-docs/stable/cookbook.html I also did a presentation/workshop on data analysis with Python you might enjoy here: https://github.com/jackgolding/FullStackDataAnalysis https://github.com/jackgolding/FullStackDataAnalysis
mindcrash 10y ago
Any language is suitable for data processing, but if you want something which is written especially for data processing and analysis, and especially data processing and analysis for large scala data sets, I personally would go for R: https://www.r-project.org/ https://www.r-project.org/.
tedmiston 10y ago
Working with CSVs in Python is pretty simple. It also has plenty of libraries for deeper processing depending on what exactly you want to accomplish. https://docs.python.org/3/library/csv.html https://docs.python.org/3/library/csv.html

10 points by digdugdirk 10y ago

dozzie 10y ago

SQL. PostgreSQL has decent CSV support.

apohn 10y ago
100% agree. I've loaded multi-gigabyte datasets into Postgres on my laptop and SQL is a great language for manipulating and aggregating data. Plus, when you use a database and SQL you can access that data from a wide range of applications and languages via drivers (JDBC and ODBC). You can even access Postgres from Excel!

eternalban 10y ago

https://www.r-bloggers.com/handling-large-datasets-in-r/ https://www.r-bloggers.com/handling-large-datasets-in-r/

Jugurtha 10y ago

>Simplicity and future usefulness of the programming environment I'll be learning are my highest ranked criteria. Given this criteria, I think SQL would be a good choice: simple to get started with, useful in the future. SQLite doesn't get in your way.. https://sqlite.org/ https://sqlite.org/

throw_away_777 10y ago

What do you mean by large? Probably as your data is in csv format it is small enough that python or R are both good choices. Personally I'd recommend python as it has a large community using it so googling questions about python libraries such as pandas usually provides good answers.

ptype 10y ago

Will it fit in ram? Python + pandas. Pandas is really amazing. Otherwise, try postgresql or SQLite.

jackgolding 10y ago

Python Pandas is absolutely amazing for this. Have a look at the cookbook and see how you feel about the language: http://pandas.pydata.org/pandas-docs/stable/cookbook.html http://pandas.pydata.org/pandas-docs/stable/cookbook.html I also did a presentation/workshop on data analysis with Python you might enjoy here: https://github.com/jackgolding/FullStackDataAnalysis https://github.com/jackgolding/FullStackDataAnalysis

mindcrash 10y ago

Any language is suitable for data processing, but if you want something which is written especially for data processing and analysis, and especially data processing and analysis for large scala data sets, I personally would go for R: https://www.r-project.org/ https://www.r-project.org/.

tedmiston 10y ago

Working with CSVs in Python is pretty simple. It also has plenty of libraries for deeper processing depending on what exactly you want to accomplish. https://docs.python.org/3/library/csv.html https://docs.python.org/3/library/csv.html