7 ms·

In the context of pandas, 3 GB of (raw, uncompressed) data could easily require 30 GB of RAM, and that kind of overhead adds up quickly.

• willseth 6y ago
  Pandas is not some mysterious black box. If you need predictable runtime performance or bounded memory usage, you have to figure it out. Pandas doesn't inherently have a staggering or unpredictable amount of overhead, given that it's a statistical analysis package. There are ways to mitigate Pandas memory usage (10x is a sign that something has gone very horribly wrong), and sometimes Pandas is simply the wrong tool for the job.
  • ihnorton 6y ago
    10x reflects both experience and expert recommendations. You may recognize the author [1]: > Nowadays, my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset [1] https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/
    • willseth 6y ago
      I don't doubt Wes' upper bound for Pandas OOTB, without optimization. The context was web applications. If you're seeing 10x on a web app, either something is wrong or you probably shouldn't be using Pandas.
      • mlyle 6y ago
        I think most of us use Pandas for data exploration and one-offs, so it is completely reasonable to discuss what our likely use of RAM is going to be in this circumstance. Web application using Pandas and "highly optimized web application" would seem to be nearly disjoint sets...