5 ms·

I started with Python because of (or rather, thanks to) pandas, it was my gateway drug. Over the past ~5 years I've done all sorts of things with it, including

by drej 9y ago

I started with Python because of (or rather, thanks to) pandas, it was my gateway drug. Over the past ~5 years I've done all sorts of things with it, including converting the whole company I worked at. At one of my employer's, I sampled our big data platform, because it was tedious and slow to work with and used pandas instead. All that being said, I'd stress pretty clearly that I never let a single line of pandas into production. There are a few reasons that I've long wanted to summarise, but just real quick: 1) It's a heavy dependency and things can go wrong, 2) It can act in unexpected ways - throw in an empty value in a list of integers and you suddenly get floats (I know why, but still), or increase the number of rows beyond a certain threshold and type inference works differently. 3) It can be very slow, especially if your workflow is write heavy (at the same time it's blazing fast for reads and joins in most cases, thanks to its columnar data structure). 4) The API evolves and breaking changes are not infrequent - that's a great thing for exploratory work, but not when you want to update libs on your production. pandas is an amazing library, the best at exploratory work, bar none. But I would not let it power some unsupervised service.

krit_dms 9y ago
so you prototyped in pandas, and build production code around numpy arrays?
- drej 9y ago
  Production wouldn't usually be in Python but if it was, it'd probably be numpy (if it was numerical). It's also fairly heavy (we'd usually exclude MKL for that reason), but less 'smart' (fewer defaults, more explicit in most places), so it's a lot safer.
- jawilson2 9y ago
  That's what we have done (algo trading). Our research backend uses pandas, but we ended up taking about a month removing it from prod code. It does surprising things with memory usage, and the functionality we needed was more or less wrappers around numpy anyway. Most of our performance critical code is in cython as well. For this trading application, speed obviously isn't the biggest concern, so python+numpy is fine. It is C++/Java everywhere else though.
  - sandGorgon 9y ago
    any opinions between cython vs numba ? especially now that numba has gpu acceleration
    - jawilson2 9y ago
      Never tried numba. I write all of our cuda stuff by hand anyway, and wrap that into cython from c++ where needed.
- nas 9y ago
  We went through sort of a similar exercise. The features that pandas provided was compelling. For example, our main research guy uses R and so something like data frames were wanted. My conclusion was that pandas was too heavy to add as a dependency however. sloccount says about 200k lines of code. Instead, I wrote a small wrapper around numpy to provide a data frame like object (850 lines of code by sloccount). So far, this has worked well for us.
aldanor 9y ago
Ditto, same here (prop trading); pandas in notebooks and for quick (to hack together, not to run) scripts; pure NumPy or C++ via pybind11 for releases
deleted 9y ago
[deleted]
filmor 9y ago
1) Conda helps with that quite a bit, Pandas is not a much heavier dependency than NumPy itself. 2) Depends a bit on your background, but to me this is not really unexpected. Integers don't have a well-defined "missing" value while Floats do, so Pandas is trying to help you by not using python objects and instead converting to the "most useful" array type. It only does so if it can convert the integers without loss of precision. 3) This one I totally get, I wrote a custom, msgpack-based serialisation due to that for our usage (before Arrow was around, seriously considering that for data exchange now). 4) Apart from the changes to `resample` all of those breaking changes had a prior `DeprecationWarning`, IIRC.
- RHSman2 9y ago
  3) msgpack. Can you expand on that? I use this method and store/cache it for temporary passing between tasks in wf. Looking to arrow to be able to extend capability of this utility.
  - filmor 9y ago
    I needed a binary, stable format to pass large dataframes efficiently to our JS frontend as well as to other Python processes. The builtin msgpack converter in Pandas was very pandas-centric in that it was basically just a msgpack representation of Pandas's internal structures, which didn't fit our purposes. In the end, I implemented a format that is essentially a list of columns, each having a specific type and some other metadata. The format for each column depends on the stored type, fallback is a msgpack sequence of strings/ints/dates mixed with nils, but array data is stored as-is and datetimes are stored either as an array of int64 with the unit attached or as begin, end, frequency.
    - RHSman2 9y ago
      Gotcha. Thanks. I see how arrow will help!
    - RHSman2 9y ago
      I just got a x3 speedup from initial tests using arrow. Then I found plasma and that has blown my mind. @wesm, how hard have you pushed plasma?
chasedehan 9y ago
> pandas is an amazing library, the best at exploratory work, I will add ... "in python." That is definitely true, but to call it "the best at exploratory work" is not accurate. I might be opening up a completely separate debate, but for down and dirty exploratory work nothing beats R's dplyr and ggplot. With that being said, I now do most of my work in python because of putting models into production. I also haven't had any issues with pandas in production; maybe because I'm not doing high throughput operations and our ML application is relatively lightweight.
- newen 9y ago
  You mean R's data.table and ggplot ;)
pmart123 9y ago
pandas is a great example of data science code at its best and its worst. If you look at the source code you will see every object and every function allows for way too many variations in input options and therefore about 20 conditional statements. For instance, I believe DataFrame's init method can take a dictionary, DataFrame, Series, etc. versus a class method for each one. Contrast that to requests where the public interface is a nice requests.get, requests.post. Yet, have a csv file you are only uploading once or twice to peak at? Then its super efficient. I think my biggest issue is all the effort that goes into pandas-like api's, i.e. https://github.com/ibis-project/ibis https://github.com/ibis-project/ibis. To me, it doesn't make sense to take something stable and known (SQL) and build a complex DSL so it works like pandas.

by drej 9y ago

krit_dms 9y ago

so you prototyped in pandas, and build production code around numpy arrays?

drej 9y ago
Production wouldn't usually be in Python but if it was, it'd probably be numpy (if it was numerical). It's also fairly heavy (we'd usually exclude MKL for that reason), but less 'smart' (fewer defaults, more explicit in most places), so it's a lot safer.
jawilson2 9y ago
That's what we have done (algo trading). Our research backend uses pandas, but we ended up taking about a month removing it from prod code. It does surprising things with memory usage, and the functionality we needed was more or less wrappers around numpy anyway. Most of our performance critical code is in cython as well. For this trading application, speed obviously isn't the biggest concern, so python+numpy is fine. It is C++/Java everywhere else though.
- sandGorgon 9y ago
  any opinions between cython vs numba ? especially now that numba has gpu acceleration
  - jawilson2 9y ago
    Never tried numba. I write all of our cuda stuff by hand anyway, and wrap that into cython from c++ where needed.
nas 9y ago
We went through sort of a similar exercise. The features that pandas provided was compelling. For example, our main research guy uses R and so something like data frames were wanted. My conclusion was that pandas was too heavy to add as a dependency however. sloccount says about 200k lines of code. Instead, I wrote a small wrapper around numpy to provide a data frame like object (850 lines of code by sloccount). So far, this has worked well for us.

aldanor 9y ago

Ditto, same here (prop trading); pandas in notebooks and for quick (to hack together, not to run) scripts; pure NumPy or C++ via pybind11 for releases

deleted 9y ago

[deleted]

filmor 9y ago

1) Conda helps with that quite a bit, Pandas is not a much heavier dependency than NumPy itself. 2) Depends a bit on your background, but to me this is not really unexpected. Integers don't have a well-defined "missing" value while Floats do, so Pandas is trying to help you by not using python objects and instead converting to the "most useful" array type. It only does so if it can convert the integers without loss of precision. 3) This one I totally get, I wrote a custom, msgpack-based serialisation due to that for our usage (before Arrow was around, seriously considering that for data exchange now). 4) Apart from the changes to `resample` all of those breaking changes had a prior `DeprecationWarning`, IIRC.

RHSman2 9y ago
3) msgpack. Can you expand on that? I use this method and store/cache it for temporary passing between tasks in wf. Looking to arrow to be able to extend capability of this utility.
- filmor 9y ago
  I needed a binary, stable format to pass large dataframes efficiently to our JS frontend as well as to other Python processes. The builtin msgpack converter in Pandas was very pandas-centric in that it was basically just a msgpack representation of Pandas's internal structures, which didn't fit our purposes. In the end, I implemented a format that is essentially a list of columns, each having a specific type and some other metadata. The format for each column depends on the stored type, fallback is a msgpack sequence of strings/ints/dates mixed with nils, but array data is stored as-is and datetimes are stored either as an array of int64 with the unit attached or as begin, end, frequency.
  - RHSman2 9y ago
    Gotcha. Thanks. I see how arrow will help!
  - RHSman2 9y ago
    I just got a x3 speedup from initial tests using arrow. Then I found plasma and that has blown my mind. @wesm, how hard have you pushed plasma?

chasedehan 9y ago

> pandas is an amazing library, the best at exploratory work, I will add ... "in python." That is definitely true, but to call it "the best at exploratory work" is not accurate. I might be opening up a completely separate debate, but for down and dirty exploratory work nothing beats R's dplyr and ggplot. With that being said, I now do most of my work in python because of putting models into production. I also haven't had any issues with pandas in production; maybe because I'm not doing high throughput operations and our ML application is relatively lightweight.

newen 9y ago
You mean R's data.table and ggplot ;)

pmart123 9y ago

pandas is a great example of data science code at its best and its worst. If you look at the source code you will see every object and every function allows for way too many variations in input options and therefore about 20 conditional statements. For instance, I believe DataFrame's init method can take a dictionary, DataFrame, Series, etc. versus a class method for each one. Contrast that to requests where the public interface is a nice requests.get, requests.post. Yet, have a csv file you are only uploading once or twice to peak at? Then its super efficient. I think my biggest issue is all the effort that goes into pandas-like api's, i.e. https://github.com/ibis-project/ibis https://github.com/ibis-project/ibis. To me, it doesn't make sense to take something stable and known (SQL) and build a complex DSL so it works like pandas.