5 ms·

Lately, `pip install pandas` is my first step after making a new virtual environment. Its read_sql and read_csv methods are magic. The resulting DataFrames are

Lately, `pip install pandas` is my first step after making a new virtual environment. Its read_sql and read_csv methods are magic. The resulting DataFrames are just like DataTables in C#. And for complex joins and aggregations, I can DataFrame.to_sql into an in-memory SQLite database. Pandas feels like the wrong tool for this job. I don't use multi-indexes or any statistical methods. I don't chart anything. But it's so darn convenient. If the time comes to optimize I can `import csv` directly and improve performance. But nothing beats it for prototyping. Are there better options in this space?

maest 7y ago
Genuine question - would you be willing to spend money for a better version of pandas? Better in some generic sense of lighter, faster, better API. I share your implied concern that pandas can be quite large and I personally disagree with a lot of the design decisions when it comes to the pandas API, but building an alternative tool would be a full time job. Unfortunately, there is no mechanism to support Python library developers and the expectation is for Python libraries to be free. I'm curious how many people would be ok paying for a Python library.
- _coveredInBees 7y ago
  I think that would be an uphill battle and very hard to succeed financially. I agree with you regarding the API being a mess, but pandas is so heavily entrenched in the datascience space (in Python land) that it is almost impossible for a free replacement to take over, let a lone a paid library.
  - whitehouse3 7y ago
    And pandas is valuable particularly because it has so many users. I'd worry that a paid product would stagnate over time, or change to meet the needs of its largest customers, leaving me behind. I go out of my way to support open source projects. Closed source would be a much harder sell.
- deleted 7y ago
  [deleted]
- luckydata 7y ago
  I have so many things to say about this but I also want to remind you that Wes is working on pandas 2. https://dev.pandas.io/pandas2/goals.html https://dev.pandas.io/pandas2/goals.html
  - sireat 7y ago
    Good to know! I thought that Wes had said a while ago that he was taking a break from working on Pandas
danpelota 7y ago
On occasion, I've fired up pandas just to sanitize a CSV file and drop malformed rows as preparation to bulk ingesting into a database: import pandas as pd pd.read_csv('bad_file.csv', error_bad_lines=False).to_csv('good_file.csv') It's not efficient (reads everything into memory), but read_csv is robust when it comes to handling embedded unescaped quotes/commas/etc., and supports dropping rows with the incorrect number of columns due to anomalies it can't handle.
datascientist 7y ago
If you need to scale out or speedup pandas, there's Modin https://modin.readthedocs.io/en/latest/ https://modin.readthedocs.io/en/latest/ (which uses Ray from)

maest 7y ago

Genuine question - would you be willing to spend money for a better version of pandas? Better in some generic sense of lighter, faster, better API. I share your implied concern that pandas can be quite large and I personally disagree with a lot of the design decisions when it comes to the pandas API, but building an alternative tool would be a full time job. Unfortunately, there is no mechanism to support Python library developers and the expectation is for Python libraries to be free. I'm curious how many people would be ok paying for a Python library.

_coveredInBees 7y ago
I think that would be an uphill battle and very hard to succeed financially. I agree with you regarding the API being a mess, but pandas is so heavily entrenched in the datascience space (in Python land) that it is almost impossible for a free replacement to take over, let a lone a paid library.
- whitehouse3 7y ago
  And pandas is valuable particularly because it has so many users. I'd worry that a paid product would stagnate over time, or change to meet the needs of its largest customers, leaving me behind. I go out of my way to support open source projects. Closed source would be a much harder sell.
deleted 7y ago
[deleted]
luckydata 7y ago
I have so many things to say about this but I also want to remind you that Wes is working on pandas 2. https://dev.pandas.io/pandas2/goals.html https://dev.pandas.io/pandas2/goals.html
- sireat 7y ago
  Good to know! I thought that Wes had said a while ago that he was taking a break from working on Pandas

danpelota 7y ago

On occasion, I've fired up pandas just to sanitize a CSV file and drop malformed rows as preparation to bulk ingesting into a database: import pandas as pd pd.read_csv('bad_file.csv', error_bad_lines=False).to_csv('good_file.csv') It's not efficient (reads everything into memory), but read_csv is robust when it comes to handling embedded unescaped quotes/commas/etc., and supports dropping rows with the incorrect number of columns due to anomalies it can't handle.

datascientist 7y ago

If you need to scale out or speedup pandas, there's Modin https://modin.readthedocs.io/en/latest/ https://modin.readthedocs.io/en/latest/ (which uses Ray from)