15 ms·

Apache Arrow and the “Things I Hate About Pandas”

394 points by jbredeche 9y ago

wpietri 9y ago
A tip for authors: briefly explain what you're talking about in the first paragraph. Or at least link to it. Because then people don't have to go hunting all over to find out that, e.g., pandas is a python data analysis library: http://pandas.pydata.org/ http://pandas.pydata.org/
- Blackthorn 9y ago
  This is directly from the blog of the Pandas creator. "What is Pandas" is assumed to be known, for good reason, and that bikeshed is the correct color.
  - phaemon 9y ago
    > "What is Pandas" is assumed to be known, for good reason What good reason? Why on earth would every single person on HN who happened to click on that link know what `pandas` is? How would they even know whose blog it is on? Unless your blog is specifically, only for people who are already familiar with your `thing` (in which case, a mailing list might be better), then it simply makes sense to always have a header with a tagline explaining what your `thing` is and a link back to the main project website. Just in case it gets featured on HN or something.
    - TheCowboy 9y ago
      Usually I ignore comments like this, but the author of the post is actually coming through and responding to comments, and I appreciate that. You're leveling the criticism that since his blog deals with a niche that he should drop to a mailing list. Seriously? HN has been developing a bad reputation, and nitpicky comments presented without decorum create an incentive for people to stay away. It's perfectly reasonably for the author of a blog post to assume that people familiar with his blog, his usual readers, will know what's up. It's not too unreasonable to suggest that the author provide a summary if it's not clear what the piece or blog is about, except in this case it is described on the blog's About page on the list of open source projects. Redefining all of these things in every post would get annoying very fast. It's reasonable to say that people working with statistical and data computing are likely as aware of Pandas as a web developer is aware of several of the largest JavaScript libraries.
      - phaemon 9y ago
        > You're leveling the criticism No, I'm not. I'm offering a suggestion. You're being absurdly thin-skinned on someone else's behalf. > Redefining all of these things in every post would get annoying very fast. So put it in a header so it automatically appears at the top of every post. As I suggested originally. This is simply good practise for any project where you want to attract people who haven't heard of your software before. If you don't care about these people, then a mailing list is a better idea so you can hide it away from everyone. Again, just a suggestion...
        Blackthorn 9y ago
        Maybe he doesn't need to attract people who haven't heard of his software before because his software is literally the default for data science. It's like complaining that someone didn't define what jQuery is. > No, I'm not. I'm offering a suggestion. You're being absurdly thin-skinned on someone else's behalf No, we're being annoyed at the crazy level of bike shedding that hacker news is starting to see.
    - Blackthorn 9y ago
      > Why on earth would every single person on HN who happened to click on that link know what `pandas` is? How would they even know whose blog it is on? It's the author of Pandas's blog. He didn't submit it to HN, someone else did. It's totally reasonable to expect that if people are reading his, the author of Pandas, blog, then they already know what Pandas is.
      - phaemon 9y ago
        > He didn't submit it to HN, someone else did. What difference does that make? It's still here, right? So people are still reading it from here, right? >It's totally reasonable to expect that if people are reading his, the author of Pandas, blog, then they already know what Pandas No, not when it's linked from elsewhere. Which is what happens with blogs. Like it was right now.
      - wpietri 9y ago
        Sorry, but I don't think it's reasonable to expect that. It's the web. People link to things. Which is why I think it's reasonable to add a link somewhere in the first paragraph to context.
sandGorgon 9y ago
This is why I was hoping ONNX (Facebook+Microsoft's new machine learning serialization format) was built on top of Arrow rather than proto2. Just like Feather is built on top of Arrow, ONNX can be based on top of Arrow.
- chubot 9y ago
  Wow I totally missed that it's based on protobuf 2. I thought Facebook used thrift and not protobufs. Well it's good to see that open source works and competitors can benefit from each other's work.
kornish 9y ago
(Disclaimers: I don't have much experience using Python to build data science products; potentially silly questions) In industry, does Pandas tend to power the application layer, or does it find more use as an exploratory data tool? If the latter, do people prefer to push computation down into OLAP databases for performance reasons? And if so, what impact will the convergence of libraries and database functionality have on product development? These features strike me as things that you'd find in a database, e.g. query optimizer. I know in the past couple years there have been a couple commercial acquisitions of in-memory execution engines, e.g. Hyper by Tableau.
- nigelcleland 9y ago
  We currently use a combination of Pandas and Scikit-Learn to run our production models. We're not in the big data space, instead, creating small tightly tuned models for a very specific purpose in a large energy company. At the moment the general work flow is: * Internal library based over Pandas which abstracts our mess of internal databases * Application specific model code that utilises the internal library to pull data in. This is then fed into a trained scikit-learn model and then further processed by Pandas. * Internal monitoring tools (dashboards based upon Ploty and Flask as well as an alerting system) are built using the internal library and Pandas as the glue. From a design decision we focused upon Pandas as the root source of all data. Everything is a DataFrame throughout the entire application. Painpoints: * Writing to a database is pretty painful (SQL Server here as Windows shop). * Minor API changes can be irritating. * Pandas MultiIndexing is both very painful and mind bending at the same time trying to get the slice syntax to work. Overall though, Pandas is a huge value add and we've gradually rolled out from 2 people to approximately 9-10 people who hadn't used python in anger before. Almost all reporting functionality is being migrated into Pandas instead of SQL stored procs, excel, tableau etc for the additional flexibility it provides.
- ves 9y ago
  We use pandas as a last-mile library for offline exploration. Typically, datasets have been sampled or aggregated enough that they can fit inside 10gb so you can work with them comfortably using pandas. I don't like using pandas in prod because the performance is really sensitive to stuff like missing a type declaration or calling the wrong method and the API is really convoluted. E: that's not to say pandas isn't good. It's really good. Thanks for the software, Wes!
wodenokoto 9y ago
Does anyone know how dataframe in R compares on these 10/11 points?
- huac 9y ago
  I'd say native data frames in R aren't great at these (maybe #4, maybe #9). I'm excited to see how Arrow can perform, and hopefully we'll see solid bindings to R as well. The data.table package (https://github.com/Rdatatable/data.table/wiki https://github.com/Rdatatable/data.table/wiki) does make progress on some of these - I'd say #1, #3, maybe #7, #8. Dplyr has a query planner too, fwiw.
- sin7 9y ago
  R data frames have excellent support for missing and categorical data. Everything else is just as much of a problem.
  - wesm 9y ago
    Disagree. Factors (categorical data) in R only support strings
    - sin7 9y ago
      But they can be ordered and are represented as integers. So you can turn your numerics to factors and keep them in order and do whatever goofy math you want to do on your categorical data.
      - wesm 9y ago
        Right, but I don't think it's fair to say that R's support for categorical data is "excellent" if only strings can be category labels/levels. Categories (aka dictionary-encoded data in other systems) semantically may be any type in practice (strings, timestamps, numbers, etc.). The function as.factor in R is lossy because the input type is coerced to string.
        9y ago
dogruck 9y ago
I really like this post. Is there a list of major projects that are leveraging Apache Arrow?
- lmeyerov 9y ago
  Graphistry is pretty under the radar as we're focusing more on big enterprise/federal security users and certain data science users before opening to others, but we've been investing big on Arrow for our V2. Ex: We recently released https://github.com/apache/arrow/tree/master/js https://github.com/apache/arrow/tree/master/js and are generally using Arrow as a way to compose interactive-time columnar CPU/GPU visual analytics technologies, including our own: https://devblogs.nvidia.com/parallelforall/goai-open-gpu-accelerated-data-analytics/ https://devblogs.nvidia.com/parallelforall/goai-open-gpu-acc... . I'm hoping we'll have the cycles to start describing how the piece fit together for how we're rethinking visual analytics web apps (and interactive-time ETL in general), but you can start to guess one aspect of it based on the above.
- wesm 9y ago
  we're due to add a "Powered By" page to the website, the list of users is growing pretty quickly (e.g. Spark has Arrow support now)
- olympus 9y ago
  It's not even two years old and doesn't have a 1.0 release. It wouldn't be smart for "major" projects to be using it. There are probably folks using it for exploration but probably not a production system.
  - lmeyerov 9y ago
    It's useful to separate the Arrow file format for standardizing passing columnar data between systems, and the native c library for not reinventing the wheel when doing so. Likewise, who this is to be used by: framework devs, and rather indirectly & niche by data analysts/engineers. For example, Apache Drill and the follow-on startup Dremio is in use in various places, and I believe they use the Arrow runtime. In contrast, when we worked on Graphistry/NodeJS <> MapD bindings, we did zero-copy data interop by agreeing on just the file format. For people used to building frameworks like the above, Arrow is at a fine point. We hoped to use the runtime, but it wasn't necessary so far. More importantly, as framework builders, it got us past the typical decision of roll-your-own format via ~flatbuffers vs. dealing with orc/parquet. As a user, you'd be leveraging Pandas, Spark, etc., and only calling Arrow when occasionally talking between systems using your framework's internally supported interop layer. Data engineers will eventually be more exposed to this as they focus more on say streaming arrows vs non-streaming parquets, but that seems early for now.
drej 9y ago
I started with Python because of (or rather, thanks to) pandas, it was my gateway drug. Over the past ~5 years I've done all sorts of things with it, including converting the whole company I worked at. At one of my employer's, I sampled our big data platform, because it was tedious and slow to work with and used pandas instead. All that being said, I'd stress pretty clearly that I never let a single line of pandas into production. There are a few reasons that I've long wanted to summarise, but just real quick: 1) It's a heavy dependency and things can go wrong, 2) It can act in unexpected ways - throw in an empty value in a list of integers and you suddenly get floats (I know why, but still), or increase the number of rows beyond a certain threshold and type inference works differently. 3) It can be very slow, especially if your workflow is write heavy (at the same time it's blazing fast for reads and joins in most cases, thanks to its columnar data structure). 4) The API evolves and breaking changes are not infrequent - that's a great thing for exploratory work, but not when you want to update libs on your production. pandas is an amazing library, the best at exploratory work, bar none. But I would not let it power some unsupervised service.
- krit_dms 9y ago
  so you prototyped in pandas, and build production code around numpy arrays?
  - drej 9y ago
    Production wouldn't usually be in Python but if it was, it'd probably be numpy (if it was numerical). It's also fairly heavy (we'd usually exclude MKL for that reason), but less 'smart' (fewer defaults, more explicit in most places), so it's a lot safer.
  - jawilson2 9y ago
    That's what we have done (algo trading). Our research backend uses pandas, but we ended up taking about a month removing it from prod code. It does surprising things with memory usage, and the functionality we needed was more or less wrappers around numpy anyway. Most of our performance critical code is in cython as well. For this trading application, speed obviously isn't the biggest concern, so python+numpy is fine. It is C++/Java everywhere else though.
stdbrouw 9y ago
Wes seems to be very focused on performance and big data applications these days, and of course it'd be great if Pandas could be used for bigger datasets, but when I hear people complain about Pandas they complain about: 1. the weird handling of types and null values (#4) 2. the verbosity of filtering like `dataframe[dataframe.column == x]` and transformations like `dataframe.col_a - dataframe.col_b`, compared to `dplyr` in R 3. warts on the indexing system (including MultiIndex, which is very powerful but confusing) For those of us who use Pandas as an alternative to R, these usability shortcomings matter way more than memory efficiency.
- vonnik 9y ago
  People interested in perf and big data ETL for the JVM may want to check out DataVec: https://github.com/deeplearning4j/datavec https://github.com/deeplearning4j/datavec https://deeplearning4j.org/datavec https://deeplearning4j.org/datavec It vectorizes/tensorizes most major data types to put them in shape for machine learning. It also lets you save the data pipeline as a reusable object.
  - deleted 9y ago
    [deleted]
- timClicks 9y ago
  I too would welcome a friendlier pandas library, but every time I've tried to think of an API that would work I fail. Well actually, I keep on wanting pandas to understand SQL.
  - lars512 9y ago
    There’s a pandasql library which lets you execute SQL on dataframes. It’s a little slower because it needs to serialize via SQLite, but it’s a quick way to get going. https://pypi.python.org/pypi/pandasql https://pypi.python.org/pypi/pandasql
- chasedehan 9y ago
  I will definitely echo (2). dplyr is amazing and works in far, far fewer lines of code than pandas. That was my largest issue when migrating over from R. There is dplython, but it doesn't quite work the same so I don't use it much. https://github.com/dodger487/dplython https://github.com/dodger487/dplython
StreamBright 9y ago
This is huge. Performance matters (even in 2017) and we need to do things the right way. Project like Julia and Apache Arrow are paving the way for high performance analytics even for large data sets.
geocar 9y ago
I often want one of these middlewares. Strings are a killer -- indeed any variable-length object makes array programming tricky when it's nested, so a sound strategy is to intern your strings first (in some way; KDB has enumerations, but language support isn't necessary: hash the strings and save an inverted index works good enough for a lot of applications). Interning strings means you see integers in your data operations, which is about as un-fun to program in as it sounds. People want to be able to write something like: ….str.extract('([ab])(\d)', expand=False) and then get disappointed that it's slow. Everything is slow when you do it a few trillion times, but slow things are really slow when you do them a few trillion times. If we think about how we build our tables, we can store these as a single-byte column (or even a bitmask) and an int (or long) column, then we get fast again. However it is clear "fast" and "let's use JSON" are incompatible, and a good middleware or storage system isn't going to make me trade.
- lifeisstillgood 9y ago
  Could you expand on this (I'm asking because you clearly intend something you have thought about a lot, but I am missing the point - it's me not you) As far as I understand you want to handle nested arrays of strings in your data. Ok The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs This way our arrays are doing numbers and we handwavy search for or use strings through some wrapper Is this right? And I am guessing the middleware you want does this transparently? Maybe storing the index alongside the data in some fashion
  - geocar 9y ago
    > The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs I don't think the rule is that firm: You don't have to use longs because if there are only 256 unique values, why waste so many bytes? Meanwhile if there are so many unique values, is longs enough? > And I am guessing the middleware you want does this transparently? Maybe. I personally think explicit is fine, as long as it's the most easiest and obvious way to do it. But I get that there's a lot of "data scientists" (probably the gross majority) that really struggle with (what I think are) basic data structures -- they'll rarely produce an efficient solution, and we'll see "yet another" article about how awk+sort is faster than a 160 node hadoop cluster... > Maybe storing the index alongside the data in some fashion Maybe not. What works for 30k strings falls over with 300m unique strings. The real trick with nested objects is to invert the query -- to convert the select statements you want to run, into an insert/upsert statement you run when you're loading the data.
twic 9y ago
Noob question: what is the relationship between Arrow, Parquet, and ORC? Do we need all three?
- jamesblonde 9y ago
  Parquet and ORC are columnar on-disk data formats that power Hadoop-on-SQL engines (Impala/SparkSQL and Hive, respectively). Arrow is an in-memory representation (Parquet/Orc are on-disk). The idea is that you can have workflows in different languages or frameworks using the same in-memory representation, not having to rebuild it just because you're going from Spark to another framework
adwhit 9y ago
I would be interested to know what Wes thinks of the Weld project, which seems to have some similar goals, but takes the 'query planner' concept much further. https://weld-project.github.io/ https://weld-project.github.io/
- timClicks 9y ago
  A deeper link outlining the Weld-pandas integration https://github.com/weld-project/weld/blob/master/python/grizzly/README.md https://github.com/weld-project/weld/blob/master/python/griz...
- wesm 9y ago
  I see Weld as an embeddable component in systems like pandas, not a replacement.
quotemstr 9y ago
Pandas is definitely powerful, if somewhat mind-bending at first if you're used to a relational, SQL world. It's never been clear to me why Pandas wasn't more copy-on-write from the start: it's difficult to predict which operations copy.
goatlover 9y ago
How does Julia DataFrames compare to R & Pandas for the 11 issues he mentioned?
anentropic 9y ago
> Logical operator graphs for graph dataflow-style execution (think TensorFlow or PyTorch, but for data frames) > A multicore schedular for parallel evaluation of operator graphs Does anything like this already exist somewhere?
- deleted 9y ago
  [deleted]
- andrioni 9y ago
  Spark SQL does exactly that: declarative graph dataflow computations + a multicore (also distributed) scheduler and executor, although it doesn't really scale down that well to a single machine and the execution layer happens mostly on the JVM, even when using PySpark (with an ungodly mix of of serialized Python and JVM objects and code). Ibis (also by Wes McKinney) does the first part, but it offloads scheduling and execution to the underlying database you are using.
- almostkorean 9y ago
  We are using Airflow for this
  - RHSman2 9y ago
    We use a luigi type app for this.
Dowwie 9y ago
Anyone here who is smart enough to write alternative solutions to the problems that pandas solves is capable of making a meaningful contribution to the pandas project, yet it seems that once pandas reaches the limits of its usefulness people go off and write a proprietary solution, never giving back. What stopped you from contributing improvements to pandas? Have you taken alternate routes to open source your work?
- antod 9y ago
  Not very familiar with pandas, but it looks like the author of this post is the creator of pandas.
  - Dowwie 9y ago
    Yes, Wes is. My comment isn't about him.
- mojoe 9y ago
  There are pretty huge barriers to open sourcing work in certain large companies.
- coliveira 9y ago
  You probably never took a deep look at pandas. It is a very complex library with lots of dependencies. It is not surprising that it is easier to implement an alternative rather than change the existing one.
robochat42 9y ago
This is exciting stuff but will it have any downsides for the majority (??) of users who don't use pandas for big data? Also, this all sounds very similar to the Blaze ecosystem, whatever happened to that? Finally, will arrow/feather replace hdf5 and bcolz in the future?
- misnome 9y ago
  Blaze was also my thought - I'd love to know how this/these proposals match up with what Blaze is doing/planning to do.
  - wesm 9y ago
    The first talk about Blaze was in November 2012. It was marketed as many things over the years, including "pandas for Big Data". My understanding is that Anaconda (fka Continuum Analytics) is no longer working on it.
rkwasny 9y ago
Apache Arrow is the next big thing in data analytics, imagine doing a SQL query in MAPD ( executes in ms on GPU ) then passing the data zero-copy into python, doing some calculations in pandas and outputting the result into web interface, because everything is zero-copy it can be done faster than ever before.
- F_J_H 9y ago
  What is your "go to" web interface for displaying/visualizing data? I use Oracle APEX because it has a killer "interactive report" feature (ie a data grid on steroids), which enables non-programers to easily filter, aggregate,export,report on, etc, the data. However, although APEX is a free option that comes with the DB, it ties you to Oracle. It would be great if there was a similar, database independent, low-code tool like APEX out there, so am curious what you have seen to work well.
  - tmostak 9y ago
    Have you tried the native visual analytics client that comes with MapD, MapD Immerse? (https://www.mapd.com/platform/immerse/ https://www.mapd.com/platform/immerse/). It’s not a full BI tool but is very good for interactively slicing and dicing datasets from 1M to 100B rows. Here’s a demo of it on 11.6B rows of US Ship data (https://www.mapd.com/demos/ships https://www.mapd.com/demos/ships).
    - F_J_H 9y ago
      No, I have not - thank you for pointing that out. I'll take a look.
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
pleasecalllater 9y ago
The pandas memory consumption is hilarious. The last time I was trying to use pandas, it was the hackernews data dump. It wasn't big. However when pandas started using the memory, my 32GB was just too little. I just ended to convert the data within postgres, much faster, with sensible memory usage.
- jjawssd 9y ago
  At what point did your memory consumption go too high? https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas https://stackoverflow.com/questions/25962114/how-to-read-a-6...
  - pleasecalllater 9y ago
    At the point when it was loading csv file with data. I think there was 5-6GB. My machine has 32GB of ram, and when usage of the python process alone increased to 35GB, I just stopped that. I've tried a couple of times, with similar results. I have loaded that csv file to postgres, the database had similar size. With indexing it was 15GB on disk. All queries were quite fast. So instead of loading the data to pandas, and making searches, I just wrote some SQL, and got the same results. However much faster, and with much smaller memory usage.
- F_J_H 9y ago
  I'm always a little baffled as to why people just don't dump the data into a database, and then use SQL for further data manipulation and analysis. I recently moved some data processing from Python/pandas into a database, and with SQL, the processing time went from several minutes to a couple seconds, (and that on a tiny VM). I understand that not everyone is familiar with databases and SQL, and so default to the toolset they know. But, the performance gains can make learning databases and SQL highly worthwhile. (And, much can be learned in a just few days, especially for those already familiar with working with data.)
  - jampekka 9y ago
    SQL is a total pain for many types of data. Eg time-series analysis is horrible and large datasets are almost impossible to fetch to the application due to the hilariously inefficient serialization formats.
jampekka 9y ago
I would love to see some sort of "smarter" indexing in the engine. I use pandas quite a bit, but I've never really understood the rationale behind the indexing, especially why indexes are treated so separately from data columns. I seem to be resetting and recreating indexes all the time, and use the .values a lot. More SQL-style indexing would be a lot more intuitive at least for me.
- nerdponx 9y ago
  I used to hate it, but I've come around to its usefulness in some cases. However I do prefer the R data.table model, which is what you descibe. You can set an index on one or more columns in the table, and that's that.
sevensor 9y ago
Pandas is the library I wish I'd had in the late '00s when my employer decided our site license for JMP was too costly. Well, really pandas plus matplotlib plus Jupyter notebooks. My job frequently involved creating plots and putting them in Powerpoint. Often the same plot, day in and day out, with new data from the production line. An interactive tool that can automate this, with a low barrier to entry, can save an incredible amount of time. Since I discovered pandas, I've been recommending it to anybody who works in a putting-plots-in-powerpoint job. And there are a lot of people who have jobs like that.
- taeric 9y ago
  That sounds more amenable to an Excel sheet, honestly. Which I suppose is not that surprising, since spreadsheets were the original freeform notebook style program.
  - sevensor 9y ago
    Excel is what we had to fall back on when they pulled our JMP license. There's a lot you can do with Excel, but automating Excel is incredibly error-prone. I came away from that experience with the conclusion that Excel is great as long as you stick to writing formulas, but as soon as you start writing macros things go bad in a hurry. And if you're not using macros for automation, that means pasting data in by hand every day, which is quite possibly worse.
    - taeric 9y ago
      I feel like I could replace "Excel" with "Jupyter" and not much changes there, honestly. Having a data ingest process in a notebook worries me because it seems to take a ton of the engineering practices we have managed to get in software, and completely ignore them. And I get that I am being a little harsher than reality dictates. However, the testing and "build" process that surrounds most of "notebooks" is laughably like what we specifically avoided in software when we said your build should be standardized in an external file. And not scripting the main IDE that you happen to be using. Indeed, I am perplexed by folks that don't know how to move between IDEs or who won't bother to understand how they are pulling dependencies into their system. Notebooks, though, seem to embrace that. Which, as I've indicated elsewhere, is great for interactive use, but seems a major step backwards for serious solutions.
fish2000 9y ago
tl;dr this article has no Native American weaponry review nor far-Eastern bear-hate listicles, meh
Myrmornis 9y ago
As has been said before, the problem with pandas is the confusing and hard-to-remember python API.

394 points by jbredeche 9y ago

A tip for authors: briefly explain what you're talking about in the first paragraph. Or at least link to it. Because then people don't have to go hunting all over to find out that, e.g., pandas is a python data analysis library: http://pandas.pydata.org/ http://pandas.pydata.org/

Blackthorn 9y ago
This is directly from the blog of the Pandas creator. "What is Pandas" is assumed to be known, for good reason, and that bikeshed is the correct color.
- phaemon 9y ago
  > "What is Pandas" is assumed to be known, for good reason What good reason? Why on earth would every single person on HN who happened to click on that link know what `pandas` is? How would they even know whose blog it is on? Unless your blog is specifically, only for people who are already familiar with your `thing` (in which case, a mailing list might be better), then it simply makes sense to always have a header with a tagline explaining what your `thing` is and a link back to the main project website. Just in case it gets featured on HN or something.
  - TheCowboy 9y ago
    Usually I ignore comments like this, but the author of the post is actually coming through and responding to comments, and I appreciate that. You're leveling the criticism that since his blog deals with a niche that he should drop to a mailing list. Seriously? HN has been developing a bad reputation, and nitpicky comments presented without decorum create an incentive for people to stay away. It's perfectly reasonably for the author of a blog post to assume that people familiar with his blog, his usual readers, will know what's up. It's not too unreasonable to suggest that the author provide a summary if it's not clear what the piece or blog is about, except in this case it is described on the blog's About page on the list of open source projects. Redefining all of these things in every post would get annoying very fast. It's reasonable to say that people working with statistical and data computing are likely as aware of Pandas as a web developer is aware of several of the largest JavaScript libraries.
    - phaemon 9y ago
      > You're leveling the criticism No, I'm not. I'm offering a suggestion. You're being absurdly thin-skinned on someone else's behalf. > Redefining all of these things in every post would get annoying very fast. So put it in a header so it automatically appears at the top of every post. As I suggested originally. This is simply good practise for any project where you want to attract people who haven't heard of your software before. If you don't care about these people, then a mailing list is a better idea so you can hide it away from everyone. Again, just a suggestion...
      - Blackthorn 9y ago
        Maybe he doesn't need to attract people who haven't heard of his software before because his software is literally the default for data science. It's like complaining that someone didn't define what jQuery is. > No, I'm not. I'm offering a suggestion. You're being absurdly thin-skinned on someone else's behalf No, we're being annoyed at the crazy level of bike shedding that hacker news is starting to see.
  - Blackthorn 9y ago
    > Why on earth would every single person on HN who happened to click on that link know what `pandas` is? How would they even know whose blog it is on? It's the author of Pandas's blog. He didn't submit it to HN, someone else did. It's totally reasonable to expect that if people are reading his, the author of Pandas, blog, then they already know what Pandas is.
    - phaemon 9y ago
      > He didn't submit it to HN, someone else did. What difference does that make? It's still here, right? So people are still reading it from here, right? >It's totally reasonable to expect that if people are reading his, the author of Pandas, blog, then they already know what Pandas No, not when it's linked from elsewhere. Which is what happens with blogs. Like it was right now.
    - wpietri 9y ago
      Sorry, but I don't think it's reasonable to expect that. It's the web. People link to things. Which is why I think it's reasonable to add a link somewhere in the first paragraph to context.

sandGorgon 9y ago

This is why I was hoping ONNX (Facebook+Microsoft's new machine learning serialization format) was built on top of Arrow rather than proto2. Just like Feather is built on top of Arrow, ONNX can be based on top of Arrow.

chubot 9y ago
Wow I totally missed that it's based on protobuf 2. I thought Facebook used thrift and not protobufs. Well it's good to see that open source works and competitors can benefit from each other's work.

kornish 9y ago

(Disclaimers: I don't have much experience using Python to build data science products; potentially silly questions) In industry, does Pandas tend to power the application layer, or does it find more use as an exploratory data tool? If the latter, do people prefer to push computation down into OLAP databases for performance reasons? And if so, what impact will the convergence of libraries and database functionality have on product development? These features strike me as things that you'd find in a database, e.g. query optimizer. I know in the past couple years there have been a couple commercial acquisitions of in-memory execution engines, e.g. Hyper by Tableau.

nigelcleland 9y ago
We currently use a combination of Pandas and Scikit-Learn to run our production models. We're not in the big data space, instead, creating small tightly tuned models for a very specific purpose in a large energy company. At the moment the general work flow is: * Internal library based over Pandas which abstracts our mess of internal databases * Application specific model code that utilises the internal library to pull data in. This is then fed into a trained scikit-learn model and then further processed by Pandas. * Internal monitoring tools (dashboards based upon Ploty and Flask as well as an alerting system) are built using the internal library and Pandas as the glue. From a design decision we focused upon Pandas as the root source of all data. Everything is a DataFrame throughout the entire application. Painpoints: * Writing to a database is pretty painful (SQL Server here as Windows shop). * Minor API changes can be irritating. * Pandas MultiIndexing is both very painful and mind bending at the same time trying to get the slice syntax to work. Overall though, Pandas is a huge value add and we've gradually rolled out from 2 people to approximately 9-10 people who hadn't used python in anger before. Almost all reporting functionality is being migrated into Pandas instead of SQL stored procs, excel, tableau etc for the additional flexibility it provides.
ves 9y ago
We use pandas as a last-mile library for offline exploration. Typically, datasets have been sampled or aggregated enough that they can fit inside 10gb so you can work with them comfortably using pandas. I don't like using pandas in prod because the performance is really sensitive to stuff like missing a type declaration or calling the wrong method and the API is really convoluted. E: that's not to say pandas isn't good. It's really good. Thanks for the software, Wes!

wodenokoto 9y ago

Does anyone know how dataframe in R compares on these 10/11 points?

huac 9y ago
I'd say native data frames in R aren't great at these (maybe #4, maybe #9). I'm excited to see how Arrow can perform, and hopefully we'll see solid bindings to R as well. The data.table package (https://github.com/Rdatatable/data.table/wiki https://github.com/Rdatatable/data.table/wiki) does make progress on some of these - I'd say #1, #3, maybe #7, #8. Dplyr has a query planner too, fwiw.
sin7 9y ago
R data frames have excellent support for missing and categorical data. Everything else is just as much of a problem.
- wesm 9y ago
  Disagree. Factors (categorical data) in R only support strings
  - sin7 9y ago
    But they can be ordered and are represented as integers. So you can turn your numerics to factors and keep them in order and do whatever goofy math you want to do on your categorical data.
    - wesm 9y ago
      Right, but I don't think it's fair to say that R's support for categorical data is "excellent" if only strings can be category labels/levels. Categories (aka dictionary-encoded data in other systems) semantically may be any type in practice (strings, timestamps, numbers, etc.). The function as.factor in R is lossy because the input type is coerced to string.
      - 9y ago

dogruck 9y ago

I really like this post. Is there a list of major projects that are leveraging Apache Arrow?

lmeyerov 9y ago
Graphistry is pretty under the radar as we're focusing more on big enterprise/federal security users and certain data science users before opening to others, but we've been investing big on Arrow for our V2. Ex: We recently released https://github.com/apache/arrow/tree/master/js https://github.com/apache/arrow/tree/master/js and are generally using Arrow as a way to compose interactive-time columnar CPU/GPU visual analytics technologies, including our own: https://devblogs.nvidia.com/parallelforall/goai-open-gpu-accelerated-data-analytics/ https://devblogs.nvidia.com/parallelforall/goai-open-gpu-acc... . I'm hoping we'll have the cycles to start describing how the piece fit together for how we're rethinking visual analytics web apps (and interactive-time ETL in general), but you can start to guess one aspect of it based on the above.
wesm 9y ago
we're due to add a "Powered By" page to the website, the list of users is growing pretty quickly (e.g. Spark has Arrow support now)
olympus 9y ago
It's not even two years old and doesn't have a 1.0 release. It wouldn't be smart for "major" projects to be using it. There are probably folks using it for exploration but probably not a production system.
- lmeyerov 9y ago
  It's useful to separate the Arrow file format for standardizing passing columnar data between systems, and the native c library for not reinventing the wheel when doing so. Likewise, who this is to be used by: framework devs, and rather indirectly & niche by data analysts/engineers. For example, Apache Drill and the follow-on startup Dremio is in use in various places, and I believe they use the Arrow runtime. In contrast, when we worked on Graphistry/NodeJS <> MapD bindings, we did zero-copy data interop by agreeing on just the file format. For people used to building frameworks like the above, Arrow is at a fine point. We hoped to use the runtime, but it wasn't necessary so far. More importantly, as framework builders, it got us past the typical decision of roll-your-own format via ~flatbuffers vs. dealing with orc/parquet. As a user, you'd be leveraging Pandas, Spark, etc., and only calling Arrow when occasionally talking between systems using your framework's internally supported interop layer. Data engineers will eventually be more exposed to this as they focus more on say streaming arrows vs non-streaming parquets, but that seems early for now.

drej 9y ago

I started with Python because of (or rather, thanks to) pandas, it was my gateway drug. Over the past ~5 years I've done all sorts of things with it, including converting the whole company I worked at. At one of my employer's, I sampled our big data platform, because it was tedious and slow to work with and used pandas instead. All that being said, I'd stress pretty clearly that I never let a single line of pandas into production. There are a few reasons that I've long wanted to summarise, but just real quick: 1) It's a heavy dependency and things can go wrong, 2) It can act in unexpected ways - throw in an empty value in a list of integers and you suddenly get floats (I know why, but still), or increase the number of rows beyond a certain threshold and type inference works differently. 3) It can be very slow, especially if your workflow is write heavy (at the same time it's blazing fast for reads and joins in most cases, thanks to its columnar data structure). 4) The API evolves and breaking changes are not infrequent - that's a great thing for exploratory work, but not when you want to update libs on your production. pandas is an amazing library, the best at exploratory work, bar none. But I would not let it power some unsupervised service.

krit_dms 9y ago
so you prototyped in pandas, and build production code around numpy arrays?
- drej 9y ago
  Production wouldn't usually be in Python but if it was, it'd probably be numpy (if it was numerical). It's also fairly heavy (we'd usually exclude MKL for that reason), but less 'smart' (fewer defaults, more explicit in most places), so it's a lot safer.
- jawilson2 9y ago
  That's what we have done (algo trading). Our research backend uses pandas, but we ended up taking about a month removing it from prod code. It does surprising things with memory usage, and the functionality we needed was more or less wrappers around numpy anyway. Most of our performance critical code is in cython as well. For this trading application, speed obviously isn't the biggest concern, so python+numpy is fine. It is C++/Java everywhere else though.

stdbrouw 9y ago

Wes seems to be very focused on performance and big data applications these days, and of course it'd be great if Pandas could be used for bigger datasets, but when I hear people complain about Pandas they complain about: 1. the weird handling of types and null values (#4) 2. the verbosity of filtering like `dataframe[dataframe.column == x]` and transformations like `dataframe.col_a - dataframe.col_b`, compared to `dplyr` in R 3. warts on the indexing system (including MultiIndex, which is very powerful but confusing) For those of us who use Pandas as an alternative to R, these usability shortcomings matter way more than memory efficiency.

vonnik 9y ago
People interested in perf and big data ETL for the JVM may want to check out DataVec: https://github.com/deeplearning4j/datavec https://github.com/deeplearning4j/datavec https://deeplearning4j.org/datavec https://deeplearning4j.org/datavec It vectorizes/tensorizes most major data types to put them in shape for machine learning. It also lets you save the data pipeline as a reusable object.
- deleted 9y ago
  [deleted]
timClicks 9y ago
I too would welcome a friendlier pandas library, but every time I've tried to think of an API that would work I fail. Well actually, I keep on wanting pandas to understand SQL.
- lars512 9y ago
  There’s a pandasql library which lets you execute SQL on dataframes. It’s a little slower because it needs to serialize via SQLite, but it’s a quick way to get going. https://pypi.python.org/pypi/pandasql https://pypi.python.org/pypi/pandasql
chasedehan 9y ago
I will definitely echo (2). dplyr is amazing and works in far, far fewer lines of code than pandas. That was my largest issue when migrating over from R. There is dplython, but it doesn't quite work the same so I don't use it much. https://github.com/dodger487/dplython https://github.com/dodger487/dplython

StreamBright 9y ago

This is huge. Performance matters (even in 2017) and we need to do things the right way. Project like Julia and Apache Arrow are paving the way for high performance analytics even for large data sets.

geocar 9y ago

I often want one of these middlewares. Strings are a killer -- indeed any variable-length object makes array programming tricky when it's nested, so a sound strategy is to intern your strings first (in some way; KDB has enumerations, but language support isn't necessary: hash the strings and save an inverted index works good enough for a lot of applications). Interning strings means you see integers in your data operations, which is about as un-fun to program in as it sounds. People want to be able to write something like: ….str.extract('([ab])(\d)', expand=False) and then get disappointed that it's slow. Everything is slow when you do it a few trillion times, but slow things are really slow when you do them a few trillion times. If we think about how we build our tables, we can store these as a single-byte column (or even a bitmask) and an int (or long) column, then we get fast again. However it is clear "fast" and "let's use JSON" are incompatible, and a good middleware or storage system isn't going to make me trade.

lifeisstillgood 9y ago
Could you expand on this (I'm asking because you clearly intend something you have thought about a lot, but I am missing the point - it's me not you) As far as I understand you want to handle nested arrays of strings in your data. Ok The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs This way our arrays are doing numbers and we handwavy search for or use strings through some wrapper Is this right? And I am guessing the middleware you want does this transparently? Maybe storing the index alongside the data in some fashion
- geocar 9y ago
  > The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs I don't think the rule is that firm: You don't have to use longs because if there are only 256 unique values, why waste so many bytes? Meanwhile if there are so many unique values, is longs enough? > And I am guessing the middleware you want does this transparently? Maybe. I personally think explicit is fine, as long as it's the most easiest and obvious way to do it. But I get that there's a lot of "data scientists" (probably the gross majority) that really struggle with (what I think are) basic data structures -- they'll rarely produce an efficient solution, and we'll see "yet another" article about how awk+sort is faster than a 160 node hadoop cluster... > Maybe storing the index alongside the data in some fashion Maybe not. What works for 30k strings falls over with 300m unique strings. The real trick with nested objects is to invert the query -- to convert the select statements you want to run, into an insert/upsert statement you run when you're loading the data.

twic 9y ago

Noob question: what is the relationship between Arrow, Parquet, and ORC? Do we need all three?

jamesblonde 9y ago
Parquet and ORC are columnar on-disk data formats that power Hadoop-on-SQL engines (Impala/SparkSQL and Hive, respectively). Arrow is an in-memory representation (Parquet/Orc are on-disk). The idea is that you can have workflows in different languages or frameworks using the same in-memory representation, not having to rebuild it just because you're going from Spark to another framework

adwhit 9y ago

I would be interested to know what Wes thinks of the Weld project, which seems to have some similar goals, but takes the 'query planner' concept much further. https://weld-project.github.io/ https://weld-project.github.io/

timClicks 9y ago
A deeper link outlining the Weld-pandas integration https://github.com/weld-project/weld/blob/master/python/grizzly/README.md https://github.com/weld-project/weld/blob/master/python/griz...
wesm 9y ago
I see Weld as an embeddable component in systems like pandas, not a replacement.

quotemstr 9y ago

Pandas is definitely powerful, if somewhat mind-bending at first if you're used to a relational, SQL world. It's never been clear to me why Pandas wasn't more copy-on-write from the start: it's difficult to predict which operations copy.

goatlover 9y ago

How does Julia DataFrames compare to R & Pandas for the 11 issues he mentioned?

anentropic 9y ago

> Logical operator graphs for graph dataflow-style execution (think TensorFlow or PyTorch, but for data frames) > A multicore schedular for parallel evaluation of operator graphs Does anything like this already exist somewhere?

deleted 9y ago
[deleted]
andrioni 9y ago
Spark SQL does exactly that: declarative graph dataflow computations + a multicore (also distributed) scheduler and executor, although it doesn't really scale down that well to a single machine and the execution layer happens mostly on the JVM, even when using PySpark (with an ungodly mix of of serialized Python and JVM objects and code). Ibis (also by Wes McKinney) does the first part, but it offloads scheduling and execution to the underlying database you are using.
almostkorean 9y ago
We are using Airflow for this
- RHSman2 9y ago
  We use a luigi type app for this.

Dowwie 9y ago

Anyone here who is smart enough to write alternative solutions to the problems that pandas solves is capable of making a meaningful contribution to the pandas project, yet it seems that once pandas reaches the limits of its usefulness people go off and write a proprietary solution, never giving back. What stopped you from contributing improvements to pandas? Have you taken alternate routes to open source your work?

antod 9y ago
Not very familiar with pandas, but it looks like the author of this post is the creator of pandas.
- Dowwie 9y ago
  Yes, Wes is. My comment isn't about him.
mojoe 9y ago
There are pretty huge barriers to open sourcing work in certain large companies.
coliveira 9y ago
You probably never took a deep look at pandas. It is a very complex library with lots of dependencies. It is not surprising that it is easier to implement an alternative rather than change the existing one.

robochat42 9y ago

This is exciting stuff but will it have any downsides for the majority (??) of users who don't use pandas for big data? Also, this all sounds very similar to the Blaze ecosystem, whatever happened to that? Finally, will arrow/feather replace hdf5 and bcolz in the future?

misnome 9y ago
Blaze was also my thought - I'd love to know how this/these proposals match up with what Blaze is doing/planning to do.
- wesm 9y ago
  The first talk about Blaze was in November 2012. It was marketed as many things over the years, including "pandas for Big Data". My understanding is that Anaconda (fka Continuum Analytics) is no longer working on it.

rkwasny 9y ago

Apache Arrow is the next big thing in data analytics, imagine doing a SQL query in MAPD ( executes in ms on GPU ) then passing the data zero-copy into python, doing some calculations in pandas and outputting the result into web interface, because everything is zero-copy it can be done faster than ever before.

F_J_H 9y ago
What is your "go to" web interface for displaying/visualizing data? I use Oracle APEX because it has a killer "interactive report" feature (ie a data grid on steroids), which enables non-programers to easily filter, aggregate,export,report on, etc, the data. However, although APEX is a free option that comes with the DB, it ties you to Oracle. It would be great if there was a similar, database independent, low-code tool like APEX out there, so am curious what you have seen to work well.
- tmostak 9y ago
  Have you tried the native visual analytics client that comes with MapD, MapD Immerse? (https://www.mapd.com/platform/immerse/ https://www.mapd.com/platform/immerse/). It’s not a full BI tool but is very good for interactively slicing and dicing datasets from 1M to 100B rows. Here’s a demo of it on 11.6B rows of US Ship data (https://www.mapd.com/demos/ships https://www.mapd.com/demos/ships).
  - F_J_H 9y ago
    No, I have not - thank you for pointing that out. I'll take a look.

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]

pleasecalllater 9y ago

The pandas memory consumption is hilarious. The last time I was trying to use pandas, it was the hackernews data dump. It wasn't big. However when pandas started using the memory, my 32GB was just too little. I just ended to convert the data within postgres, much faster, with sensible memory usage.

jjawssd 9y ago
At what point did your memory consumption go too high? https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas https://stackoverflow.com/questions/25962114/how-to-read-a-6...
- pleasecalllater 9y ago
  At the point when it was loading csv file with data. I think there was 5-6GB. My machine has 32GB of ram, and when usage of the python process alone increased to 35GB, I just stopped that. I've tried a couple of times, with similar results. I have loaded that csv file to postgres, the database had similar size. With indexing it was 15GB on disk. All queries were quite fast. So instead of loading the data to pandas, and making searches, I just wrote some SQL, and got the same results. However much faster, and with much smaller memory usage.
F_J_H 9y ago
I'm always a little baffled as to why people just don't dump the data into a database, and then use SQL for further data manipulation and analysis. I recently moved some data processing from Python/pandas into a database, and with SQL, the processing time went from several minutes to a couple seconds, (and that on a tiny VM). I understand that not everyone is familiar with databases and SQL, and so default to the toolset they know. But, the performance gains can make learning databases and SQL highly worthwhile. (And, much can be learned in a just few days, especially for those already familiar with working with data.)
- jampekka 9y ago
  SQL is a total pain for many types of data. Eg time-series analysis is horrible and large datasets are almost impossible to fetch to the application due to the hilariously inefficient serialization formats.

jampekka 9y ago

I would love to see some sort of "smarter" indexing in the engine. I use pandas quite a bit, but I've never really understood the rationale behind the indexing, especially why indexes are treated so separately from data columns. I seem to be resetting and recreating indexes all the time, and use the .values a lot. More SQL-style indexing would be a lot more intuitive at least for me.

nerdponx 9y ago
I used to hate it, but I've come around to its usefulness in some cases. However I do prefer the R data.table model, which is what you descibe. You can set an index on one or more columns in the table, and that's that.

sevensor 9y ago

Pandas is the library I wish I'd had in the late '00s when my employer decided our site license for JMP was too costly. Well, really pandas plus matplotlib plus Jupyter notebooks. My job frequently involved creating plots and putting them in Powerpoint. Often the same plot, day in and day out, with new data from the production line. An interactive tool that can automate this, with a low barrier to entry, can save an incredible amount of time. Since I discovered pandas, I've been recommending it to anybody who works in a putting-plots-in-powerpoint job. And there are a lot of people who have jobs like that.

taeric 9y ago
That sounds more amenable to an Excel sheet, honestly. Which I suppose is not that surprising, since spreadsheets were the original freeform notebook style program.
- sevensor 9y ago
  Excel is what we had to fall back on when they pulled our JMP license. There's a lot you can do with Excel, but automating Excel is incredibly error-prone. I came away from that experience with the conclusion that Excel is great as long as you stick to writing formulas, but as soon as you start writing macros things go bad in a hurry. And if you're not using macros for automation, that means pasting data in by hand every day, which is quite possibly worse.
  - taeric 9y ago
    I feel like I could replace "Excel" with "Jupyter" and not much changes there, honestly. Having a data ingest process in a notebook worries me because it seems to take a ton of the engineering practices we have managed to get in software, and completely ignore them. And I get that I am being a little harsher than reality dictates. However, the testing and "build" process that surrounds most of "notebooks" is laughably like what we specifically avoided in software when we said your build should be standardized in an external file. And not scripting the main IDE that you happen to be using. Indeed, I am perplexed by folks that don't know how to move between IDEs or who won't bother to understand how they are pulling dependencies into their system. Notebooks, though, seem to embrace that. Which, as I've indicated elsewhere, is great for interactive use, but seems a major step backwards for serious solutions.

fish2000 9y ago

tl;dr this article has no Native American weaponry review nor far-Eastern bear-hate listicles, meh

Myrmornis 9y ago

As has been said before, the problem with pandas is the confusing and hard-to-remember python API.