4 ms·

All these articles always say that Python has better preprocessing compared with R. Where? I find tidyverse/data.table much more elegant than pandas, scipy et a

by padthai 10y ago

All these articles always say that Python has better preprocessing compared with R. Where? I find tidyverse/data.table much more elegant than pandas, scipy et al. The only thing that I like more in Python it is how it handles streams/generators.

__s 10y ago
The last time I preprocessed with Python I only used csvreader. Client had sent a tree structure as two column table, so in one pass I outputted SQL for the recursive table with a bit of if-second-column-empty-then-figure-out-depth-of-first-column logic Maybe R would work, but I'm familiar with Python. So Python stayed out of my way
itschekkers 10y ago
Ya, completely agree with this (having also used both extensively). dplyr can also connect to remote SQL server so data don't have to be local. Maybe pandas does this now too but in my experience SQL connections were generally more painful in python
- Analog24 10y ago
  This is what SQLAlchemy is for. It's one of the best ORMs out there and makes interacting with a SQL server a breeze.
laGrenouille 10y ago
I use both R and Python extensively, and also was confused for a long time why the conventional wisdom is that one of Python's strengths is a data cleaning and preprocessing. I too always found data cleaning to actually be one of R's strongest features. My hypothesis: I think there is an inconsistency between what is meant by "cleaning/processing" between stats- and CS-oriented people. To the latter, it often just means turning unstructured or tree-like data (JSON/XML) into a tabular format. As a statistician, I think of it being a much larger set of operations; specifically those supported by packages such as dplyr, tidyr, and stringr... I'm not sure if that interpretation is correct, but Python does excel in the XML/JSON => CSV step and R is great at all the processing that happens after that.

by padthai 10y ago

__s 10y ago

The last time I preprocessed with Python I only used csvreader. Client had sent a tree structure as two column table, so in one pass I outputted SQL for the recursive table with a bit of if-second-column-empty-then-figure-out-depth-of-first-column logic Maybe R would work, but I'm familiar with Python. So Python stayed out of my way

itschekkers 10y ago

Ya, completely agree with this (having also used both extensively). dplyr can also connect to remote SQL server so data don't have to be local. Maybe pandas does this now too but in my experience SQL connections were generally more painful in python

Analog24 10y ago
This is what SQLAlchemy is for. It's one of the best ORMs out there and makes interacting with a SQL server a breeze.

laGrenouille 10y ago

I use both R and Python extensively, and also was confused for a long time why the conventional wisdom is that one of Python's strengths is a data cleaning and preprocessing. I too always found data cleaning to actually be one of R's strongest features. My hypothesis: I think there is an inconsistency between what is meant by "cleaning/processing" between stats- and CS-oriented people. To the latter, it often just means turning unstructured or tree-like data (JSON/XML) into a tabular format. As a statistician, I think of it being a much larger set of operations; specifically those supported by packages such as dplyr, tidyr, and stringr... I'm not sure if that interpretation is correct, but Python does excel in the XML/JSON => CSV step and R is great at all the processing that happens after that.