3 ms·

Same thing happened with Python for data science in 2023-2015. ie reinvent SQL using Python DSL.

yagodragon 6y ago
Have you tried jupyter notebook + pandas dataframes? You just import a csv and you can explore, clean, manipulate data and make any sort of calculation using a real programming language. On top of that you can use something like seaborn and create visualizations. Then you click print pdf and you're ready. It almost feels like cheating. You can say all you want about how fast,optimized and superior SQL is, but the python experience is way better for almost any time you need to make sense of some data.
- disgruntledphd2 6y ago
  Not the OP, but I have. I've spent years doing the work that you specify above, and pandas & notebooks are probably one of the least good tools for it. SQL is super fast and useful (and everyone understands it, at least in the data world), while dplyr and ggplot2 are basically a DSL for this work, while pandas is an irritatingly bad clone of base-R which manages to keep the bad parts of both Python and R, while having the good parts of neither. I really feel like you should try Rmd with R if you like the notebook flow, as it's similar but much much better. If you really want to stick with Python, then org-mode is a much better notebook than anything else.
- pmart123 6y ago
  I have a significant amount of experience with both of them. For pandas, it is fine for initial exploratory analysis (input, plot, reshape and export, etc). However, it's API has inconsistencies and subtle "features" around silent data coercion that make it hard to use in production. Seaborn is much nicer to work with than matplotlib and Jupyter is useful for making interactive presentations. So I believe Pandas/Jupyter have a place, but then there was a tendency to create a Pandas-like wrapper for all data retrieval such as: https://github.com/ibis-project/ibis https://github.com/ibis-project/ibis https://github.com/blaze/blaze https://github.com/blaze/blaze

Have you tried jupyter notebook + pandas dataframes? You just import a csv and you can explore, clean, manipulate data and make any sort of calculation using a real programming language. On top of that you can use something like seaborn and create visualizations. Then you click print pdf and you're ready. It almost feels like cheating. You can say all you want about how fast,optimized and superior SQL is, but the python experience is way better for almost any time you need to make sense of some data.

disgruntledphd2 6y ago
Not the OP, but I have. I've spent years doing the work that you specify above, and pandas & notebooks are probably one of the least good tools for it. SQL is super fast and useful (and everyone understands it, at least in the data world), while dplyr and ggplot2 are basically a DSL for this work, while pandas is an irritatingly bad clone of base-R which manages to keep the bad parts of both Python and R, while having the good parts of neither. I really feel like you should try Rmd with R if you like the notebook flow, as it's similar but much much better. If you really want to stick with Python, then org-mode is a much better notebook than anything else.
pmart123 6y ago
I have a significant amount of experience with both of them. For pandas, it is fine for initial exploratory analysis (input, plot, reshape and export, etc). However, it's API has inconsistencies and subtle "features" around silent data coercion that make it hard to use in production. Seaborn is much nicer to work with than matplotlib and Jupyter is useful for making interactive presentations. So I believe Pandas/Jupyter have a place, but then there was a tendency to create a Pandas-like wrapper for all data retrieval such as: https://github.com/ibis-project/ibis https://github.com/ibis-project/ibis https://github.com/blaze/blaze https://github.com/blaze/blaze