5 ms·

(Disclaimers: I don't have much experience using Python to build data science products; potentially silly questions) In industry, does Pandas tend to power the

(Disclaimers: I don't have much experience using Python to build data science products; potentially silly questions) In industry, does Pandas tend to power the application layer, or does it find more use as an exploratory data tool? If the latter, do people prefer to push computation down into OLAP databases for performance reasons? And if so, what impact will the convergence of libraries and database functionality have on product development? These features strike me as things that you'd find in a database, e.g. query optimizer. I know in the past couple years there have been a couple commercial acquisitions of in-memory execution engines, e.g. Hyper by Tableau.

nigelcleland 9y ago
We currently use a combination of Pandas and Scikit-Learn to run our production models. We're not in the big data space, instead, creating small tightly tuned models for a very specific purpose in a large energy company. At the moment the general work flow is: * Internal library based over Pandas which abstracts our mess of internal databases * Application specific model code that utilises the internal library to pull data in. This is then fed into a trained scikit-learn model and then further processed by Pandas. * Internal monitoring tools (dashboards based upon Ploty and Flask as well as an alerting system) are built using the internal library and Pandas as the glue. From a design decision we focused upon Pandas as the root source of all data. Everything is a DataFrame throughout the entire application. Painpoints: * Writing to a database is pretty painful (SQL Server here as Windows shop). * Minor API changes can be irritating. * Pandas MultiIndexing is both very painful and mind bending at the same time trying to get the slice syntax to work. Overall though, Pandas is a huge value add and we've gradually rolled out from 2 people to approximately 9-10 people who hadn't used python in anger before. Almost all reporting functionality is being migrated into Pandas instead of SQL stored procs, excel, tableau etc for the additional flexibility it provides.
ves 9y ago
We use pandas as a last-mile library for offline exploration. Typically, datasets have been sampled or aggregated enough that they can fit inside 10gb so you can work with them comfortably using pandas. I don't like using pandas in prod because the performance is really sensitive to stuff like missing a type declaration or calling the wrong method and the API is really convoluted. E: that's not to say pandas isn't good. It's really good. Thanks for the software, Wes!
tanilama 9y ago
> In industry, does Pandas tend to power the application layer, or does it find more use as an exploratory data tool? My experience echos yours, Pandas from my observation, is more like a post-modeling tool, that people use to further process data that they digest from certain DB query or Spark jobs. After reading through Arrow homepage, I am left somewhat baffled about where it seats. If my reading is correct, it is a client-side protocol that abstracts away the underlying data storage implementations? If so, isn't it still limited by how much data the client machine can handle? Or the benefits is about the unified interface of accessing different storage system? No matter what, it seems pretty ambitious. Looking forward to see how it goes.
- wesm 9y ago
  I recommend my JupyterCon keynote to explain how it fits into the picture: https://www.youtube.com/watch?v=wdmf1msbtVs https://www.youtube.com/watch?v=wdmf1msbtVs Data processing systems need runtime memory formats. Arrow is an efficient one for analytical data processing. It has the additional benefit of zero-copy data interchange layer for sharing memory between processes written in any language.
  - RHSman2 9y ago
    I'll be honest, it took me a while to really 'get' what it's for. I think an example use case would really help. I now get it and it applies to my use case 100% (probably as it evolved from mirroring your lead) but it took a while to understand this somewhat intangible concept (to a non computer scientist)
wesm 9y ago
Wes here. From what I understand, pandas is the middleware layer powering about 90% (if not more) of analytical applications in Python. It is what people use for data ingest, data prep, and feature engineering for machine learning models. The existence of other database systems that perform equivalent tasks isn't useful if they are not accessible to Python programmers with a convenient API.
- FridgeSeal 9y ago
  This is exactly how I use it. I pull the relevant data out of a production database, clean it, add relevant columns, filter out trash data, use seaborn to produce some simple plots to see what my data approximately looks/structured like then off to sklearn.
  - atupis 9y ago
    Definitely this and I like how simple plotting is at pandas, usually df['some_column'].plot() gives decent plot out of box.
atupis 9y ago
I usually use pandas as exploratory data tool and then rewrite code using numpy because pandas needs much more memory and is lot slower.

nigelcleland 9y ago

We currently use a combination of Pandas and Scikit-Learn to run our production models. We're not in the big data space, instead, creating small tightly tuned models for a very specific purpose in a large energy company. At the moment the general work flow is: * Internal library based over Pandas which abstracts our mess of internal databases * Application specific model code that utilises the internal library to pull data in. This is then fed into a trained scikit-learn model and then further processed by Pandas. * Internal monitoring tools (dashboards based upon Ploty and Flask as well as an alerting system) are built using the internal library and Pandas as the glue. From a design decision we focused upon Pandas as the root source of all data. Everything is a DataFrame throughout the entire application. Painpoints: * Writing to a database is pretty painful (SQL Server here as Windows shop). * Minor API changes can be irritating. * Pandas MultiIndexing is both very painful and mind bending at the same time trying to get the slice syntax to work. Overall though, Pandas is a huge value add and we've gradually rolled out from 2 people to approximately 9-10 people who hadn't used python in anger before. Almost all reporting functionality is being migrated into Pandas instead of SQL stored procs, excel, tableau etc for the additional flexibility it provides.

ves 9y ago

We use pandas as a last-mile library for offline exploration. Typically, datasets have been sampled or aggregated enough that they can fit inside 10gb so you can work with them comfortably using pandas. I don't like using pandas in prod because the performance is really sensitive to stuff like missing a type declaration or calling the wrong method and the API is really convoluted. E: that's not to say pandas isn't good. It's really good. Thanks for the software, Wes!

tanilama 9y ago

> In industry, does Pandas tend to power the application layer, or does it find more use as an exploratory data tool? My experience echos yours, Pandas from my observation, is more like a post-modeling tool, that people use to further process data that they digest from certain DB query or Spark jobs. After reading through Arrow homepage, I am left somewhat baffled about where it seats. If my reading is correct, it is a client-side protocol that abstracts away the underlying data storage implementations? If so, isn't it still limited by how much data the client machine can handle? Or the benefits is about the unified interface of accessing different storage system? No matter what, it seems pretty ambitious. Looking forward to see how it goes.

wesm 9y ago
I recommend my JupyterCon keynote to explain how it fits into the picture: https://www.youtube.com/watch?v=wdmf1msbtVs https://www.youtube.com/watch?v=wdmf1msbtVs Data processing systems need runtime memory formats. Arrow is an efficient one for analytical data processing. It has the additional benefit of zero-copy data interchange layer for sharing memory between processes written in any language.
- RHSman2 9y ago
  I'll be honest, it took me a while to really 'get' what it's for. I think an example use case would really help. I now get it and it applies to my use case 100% (probably as it evolved from mirroring your lead) but it took a while to understand this somewhat intangible concept (to a non computer scientist)

wesm 9y ago

Wes here. From what I understand, pandas is the middleware layer powering about 90% (if not more) of analytical applications in Python. It is what people use for data ingest, data prep, and feature engineering for machine learning models. The existence of other database systems that perform equivalent tasks isn't useful if they are not accessible to Python programmers with a convenient API.

FridgeSeal 9y ago
This is exactly how I use it. I pull the relevant data out of a production database, clean it, add relevant columns, filter out trash data, use seaborn to produce some simple plots to see what my data approximately looks/structured like then off to sklearn.
- atupis 9y ago
  Definitely this and I like how simple plotting is at pandas, usually df['some_column'].plot() gives decent plot out of box.

atupis 9y ago

I usually use pandas as exploratory data tool and then rewrite code using numpy because pandas needs much more memory and is lot slower.