11 ms·

Pandas 2.0

325 points by calpaterson 4y ago

bb1234 4y ago
If interested in benchmarks comparing different dataframe implementations, here is one: https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/
HopenHeyHi 4y ago
You want to link to the actual release notes: https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v2.0.0.html https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v...
mattrighetti 4y ago
What's new here [0], saved you a click. [0]: https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v2.0.0.html https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v...
- kinow 4y ago
  Your link is broken for me, but going to their website and clicking on the 2.0 what's new link takes me to the same URL. They might be updating it... the closest I found was the Sphinx docs source for that: https://github.com/pandas-dev/pandas/blob/main/doc/source/whatsnew/v2.0.0.rst https://github.com/pandas-dev/pandas/blob/main/doc/source/wh...
  - huskyr 4y ago
    Link works fine for me.
- deleted 4y ago
  [deleted]
wodenokoto 4y ago
strings as objects and integers turning into floats when NaNs are introduced have been a much bigger annoyance to me, than it ought to. I'm excited to try out the new pyarrow dtypes, but it also sounds confusing that there are now 2 classes of types
- hospitalJail 4y ago
  >strings as objects and integers turning into floats when NaNs are introduced have been a much bigger annoyance to me, than it ought to. Nah, you rightly are annoyed. When I am writing unit tests, it is especially annoying to fix the type.
- agent281 4y ago
  Yeah, I stopped using pandas entirely for ETL for this exact reason. If you are trying to maintain the fidelity of the data while cleaning, automatic casting is awful. If the new backend prevents automatic casting, it might be worth reconsidering for me.
  - wodenokoto 4y ago
    what do you use instead?
    - agent281 4y ago
      We're using AWS glue (basically pyspark) right now. I used standard python before. I've implemented a function for schema based processing JSON documents for both vanilla python and pyspark that makes the process really easy. It'll take a schema and a document and product a list of flat dictionaries for python or a data frame for pyspark. Vanilla python is really streamable and keeps memory overhead low so it was actually faster than the pandas based workflows that it replaced.
villgax 4y ago
How I wished they just changed the .apply for adding progress and parallelization by default instead of resorting to tqdm & swifter/dask or what have you
0cf8612b2e1e 4y ago
A quick skim shows a lot of quality of life improvements. Unless I am misreading, it looks like it is still a numpy backed by default. I thought one of the drivers for the 2.0 was to make Arrow the default.
- miohtama 4y ago
  Is there any comparison, performance and feature wise, between different backends yet? Or is it too early?
- bsg75 4y ago
  Pre-release this was a reason [1]. Not sure if its still a compatibility reason: > There is also an option to let pandas know we want Arrow backed types by default. The option at the time of writing this article is partially implemented and has a confusing API. In particular, it's not yet working when creating data with pandas.Series or pandas.DataFrame. And for loading data from files it will only work when the parameter use_nullable_dtypes is set to True. For example, to load a CSV file with PyArrow directly into PyArrow backed pandas Series, you can use the next code: > pandas.options.mode.dtype_backend = 'pyarrow' [1] https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i https://datapythonista.me/blog/pandas-20-and-the-arrow-revol...
fauxpause_ 4y ago
> Accessing a single column of a DataFrame as a Series (e.g. df["col"]) now always returns a new object every time it is constructed when Copy-on-Write is enabled (not returning multiple times an identical, cached Series object). This ensures that those Series objects correctly follow the Copy-on-Write rules (GH49450) Is this going to mean I can’t do df[‘a’] = 2 to set all values in column a to 2?
- miohtama 4y ago
  Copy-on-write is not enabled by default, but needs to be explicitly enabled with an option or a context manager.
  - fauxpause_ 4y ago
    But if it were enabled, am I correct
henrydark 4y ago
Say what you will about sql, polars, pyspark, or whatever else. But nothing beats pandas' df[col].value_counts().value_counts()
- humanistbot 4y ago
  Can your entire dataframe fit in memory? Then you're good with pandas. If not, that's why those alternatives exist.
lvl102 4y ago
I still prefer Stata if I am completely honest with myself.
- binarymax 4y ago
  I mean that’s fine, but is there any reason? Just throwing your half second opinion out there on a product release announcement isn’t really helpful.
- spaniard89277 4y ago
  Why?
- humanistbot 4y ago
  You're comparing apple pie and orange juice here (even worse than comparing apples and oranges). Pandas is an open-source library for people who know python programming. Stata is an expensive GUI-based suite for non-programmers. If you're not in university, government, or non-profit, Stata's single-CPU license is $840/year in the US. Even a single-CPU student license is $94/year.
  - lvl102 4y ago
    Have you ever used Stata? You sound like someone who never used Stata for serious data analysis.
    - crop_rotation 4y ago
      Do you have any counter point to the above mentioned pricing? If not, that illustrates a huge difference.
    - postexitus 4y ago
      you sound more like someone who doesn't know what they are talking about.
    - crimsoneer 4y ago
      The only people still using stata are 40+ year old economics professors and the poor lab students they force into obsolescence. Everybody else is using R (if you're into econometrics) or Python (if you're into ML).
benrutter 4y ago
I know arrow support is only part way their with this release - but this is a huge deal for Pandas, for standardisation as whole, but also speed ups. Benchmarking that was shared a while back here suggests 2x speed ups in some cases, 30x if you count strings since pandas uses python's in-built string data type[1] [1]https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i https://datapythonista.me/blog/pandas-20-and-the-arrow-revol...
- 0cf8612b2e1e 4y ago
  Not just speed, but I thought one of the big Arrow wins relative to numpy processing was better memory usage.
- wenc 4y ago
  Polars is already built on Arrow and the advantages are tremendous.
  - benrutter 4y ago
    Yeah Polars is an awesome library! I don't know a whole bunch about its internals, but I think it implements a bunch of additional speed ups through memory allocation and lazy evaluation, so its unlikely Pandas is will get to a similar speed without some huge changes elsewhere
    - wenc 4y ago
      Yup here’s a list of optimizations that Polars does to achieve the speed that it does https://pola-rs.github.io/polars-book/user-guide/#current-status https://pola-rs.github.io/polars-book/user-guide/#current-st... Polars’ lazy evaluation is a big deal — this lets it do query plan optimization. Whereas in Pandas every step is eager so it can’t look ahead to eliminate redundant steps. You basically can’t do a lot of query optimization in a multi step transform.
mynameisash 4y ago
I'm curious if there will be any appreciable performance gains here that are worthwhile. FWIW, last I checked[0], Polars still smokes Pandas in basically every way. [0] https://www.pola.rs/benchmarks.html https://www.pola.rs/benchmarks.html
- westurner 4y ago
  One could run the benchmarks with the new version of the software under concern and report back
- wenc 4y ago
  I’ve moved entirely to Polars (which is essentially Pandas written in Rust with new design decisions) with DuckDB as my SQL query engine. Since both are backed by Arrow, there is zero copy and performance on large datasets is super fast (due to vectorization, not just parallelization) I keep Pandas around for quick plots and legacy code. I will always be grateful for Pandas because there truly was no good dataframe library during its time. It has enabled an entire generation of data scientists to do what they do and built a foundation — a foundation which Polars and DuckDB are now building on and have surpassed.
  - kylebarron 4y ago
    DuckDB is not backed by Arrow, but by something similar enough that it can be zero copy to Arrow in many cases https://duckdb.org/2021/12/03/duck-arrow.html https://duckdb.org/2021/12/03/duck-arrow.html
  - RockyMcNuts 4y ago
    TFA: Pandas 2.0 is also backed by Arrow as an option, yielding a large speed improvement, although not all the way to Polars
  - next_xibalba 4y ago
    How well does Polars play with many of the other standard data tools in Python (scikit learn, etc.)? Do the performance gains carry over?
modriano 4y ago
Jeff Reback gave a presentation on the roadmap for Pandas at PyData NYC 2022 [0]. In it, he basically says that pandas is used so widely in industry that big breaking changes are a non-starter, there won't be any radical changes to the API, but more performant implementations can/will be built into the library (although not set as defaults, at least not for a long while). Not a revolutionary leap, but a move towards making Wes McKinney's Arrow work more accessible through pandas. [0] https://www.youtube.com/watch?v=85XdsWz_Q_o https://www.youtube.com/watch?v=85XdsWz_Q_o
- __mharrison__ 4y ago
  This is why Modin and Ponder are game-changers for Pandas folks. You keep the API but get scaleout.
cmcconomy 4y ago
I'd love to know when geopandas snaps into alignment
timcavel 4y ago
[dead]
mongol 4y ago
Is it correct that I associate to a type of Chinese bears when I read about this project? Or is Pandas an acronym?
- Guybrush_T 4y ago
  Apparently it's taken from panel data.
- ghshephard 4y ago
  "The term Panel data is derived from econometrics and is partially responsible for the name pandas − pan(el)-da(ta)-s" https://www.tutorialspoint.com/python_pandas/python_pandas_panel.htm https://www.tutorialspoint.com/python_pandas/python_pandas_p...
- deleted 4y ago
  [deleted]
nonfamous 4y ago
Not a lot of people realize that Pandas was inspired by R, and in particular the Tidyverse model of handling rectangular data frames, created originally by Hadley Wickham. These days R is primarily used by data scientists in academia and certain niche industries like pharma, but its impact goes way beyond its core user base.
- next_xibalba 4y ago
  > and in particular the Tidyverse model This isn't really true. R took data frames from S, which was using the concept at least as early as 1991. Pandas itself predates most if not all of the tidyverse. Pandas original release occurred in 2008, whereas the first release of dplyr (one of the original packages of the tidyverse), didn't come until 2014.
  - milliams 4y ago
    In fact I seem to recall that the Tidyverse paper references Pandas.
  - nonfamous 4y ago
    It's definitely true that R's base data frame (a rectangular structure with columns of mixed types, which R in turn inherited from S) was the inspiration for Pandas. The concept of verbs operating on those structures IMO was inspired by plyr (the antecedent to dplyr, first published in 2008, which introduced composability for those verbs). data.table was also an inspiration, as another commenter points out.
    - kgwgk 4y ago
      > The concept of verbs operating on those structures IMO was inspired by plyr Was it? (I have no idea. But R already had verbs operating on data frames.)
    - deleted 4y ago
      [deleted]
  - kgwgk 4y ago
edublancas 4y ago
How are people managing the existence of data frame APIs like pandas/polars with SQL engines like BigQuery, Snowflake, and DuckDB? Most of my notebooks are a mix of SQL and Python: SQL for most processing, dump the results as a pandas dataframe (via https://github.com/ploomber/jupysql https://github.com/ploomber/jupysql) and then use Python for operations that are difficult to express with SQL (or that I don't know how to do it), so I end up with 80% SQL, 20% Python. Unsure if this is the best workflow but it's the most efficient one I've come up with. Disclaimer: my team develops JupySQL.
- Helmut10001 4y ago
  I checked most solutions and I am sticking with plain old SQL in triple-qoutes in jupyter, e.g. [1]. I don't need auto-completion in SQL and I don't really need syntax highlighting. It's also very nice to have the combination of f-strings/variables substitution and SQL. Yet, my SQL needs are very basic. [1]: https://kartographie.geo.tu-dresden.de/ad/wip/ephemeral_events/html/00_raw_hll_conversion.html#Import-Milvus-milvus-example https://kartographie.geo.tu-dresden.de/ad/wip/ephemeral_even...
- __mharrison__ 4y ago
  You should check out https://ponder.io https://ponder.io . It is created by the folks who made Mondin (a scale out version of Pandas with API compatibility as a goal). Can use dask or ray as a backend. Ponder is the enterprise version that runs on Snowflake and BigQuery. Again, same goal, API compatibility with Pandas. You can scale out your Pandas workflow by changing the import and leaving the Pandas code. (Full disclosure I'm an advisor.)
- wswope 4y ago
  As a fellow evangelist of the SQL + Pandas hybrid workflow, I’m a happy camper with Pandas’ built-in read_sql_query and to_sql. Only big pain points are having to ship around boilerplate to construct SQLAlchemy create_engine URIs, and the performance limitations of SQLAlchemy’s inserts (if moving anything larger than a few gigs, it typically pays to ditch to_sql, and write a db-specific bulk insert process instead).

325 points by calpaterson 4y ago

If interested in benchmarks comparing different dataframe implementations, here is one: https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/

HopenHeyHi 4y ago

You want to link to the actual release notes: https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v2.0.0.html https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v...

mattrighetti 4y ago

What's new here [0], saved you a click. [0]: https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v2.0.0.html https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v...

kinow 4y ago
Your link is broken for me, but going to their website and clicking on the 2.0 what's new link takes me to the same URL. They might be updating it... the closest I found was the Sphinx docs source for that: https://github.com/pandas-dev/pandas/blob/main/doc/source/whatsnew/v2.0.0.rst https://github.com/pandas-dev/pandas/blob/main/doc/source/wh...
- huskyr 4y ago
  Link works fine for me.
deleted 4y ago
[deleted]

wodenokoto 4y ago

strings as objects and integers turning into floats when NaNs are introduced have been a much bigger annoyance to me, than it ought to. I'm excited to try out the new pyarrow dtypes, but it also sounds confusing that there are now 2 classes of types

hospitalJail 4y ago
>strings as objects and integers turning into floats when NaNs are introduced have been a much bigger annoyance to me, than it ought to. Nah, you rightly are annoyed. When I am writing unit tests, it is especially annoying to fix the type.
agent281 4y ago
Yeah, I stopped using pandas entirely for ETL for this exact reason. If you are trying to maintain the fidelity of the data while cleaning, automatic casting is awful. If the new backend prevents automatic casting, it might be worth reconsidering for me.
- wodenokoto 4y ago
  what do you use instead?
  - agent281 4y ago
    We're using AWS glue (basically pyspark) right now. I used standard python before. I've implemented a function for schema based processing JSON documents for both vanilla python and pyspark that makes the process really easy. It'll take a schema and a document and product a list of flat dictionaries for python or a data frame for pyspark. Vanilla python is really streamable and keeps memory overhead low so it was actually faster than the pandas based workflows that it replaced.

villgax 4y ago

How I wished they just changed the .apply for adding progress and parallelization by default instead of resorting to tqdm & swifter/dask or what have you

0cf8612b2e1e 4y ago

A quick skim shows a lot of quality of life improvements. Unless I am misreading, it looks like it is still a numpy backed by default. I thought one of the drivers for the 2.0 was to make Arrow the default.

miohtama 4y ago
Is there any comparison, performance and feature wise, between different backends yet? Or is it too early?
bsg75 4y ago
Pre-release this was a reason [1]. Not sure if its still a compatibility reason: > There is also an option to let pandas know we want Arrow backed types by default. The option at the time of writing this article is partially implemented and has a confusing API. In particular, it's not yet working when creating data with pandas.Series or pandas.DataFrame. And for loading data from files it will only work when the parameter use_nullable_dtypes is set to True. For example, to load a CSV file with PyArrow directly into PyArrow backed pandas Series, you can use the next code: > pandas.options.mode.dtype_backend = 'pyarrow' [1] https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i https://datapythonista.me/blog/pandas-20-and-the-arrow-revol...

fauxpause_ 4y ago

> Accessing a single column of a DataFrame as a Series (e.g. df["col"]) now always returns a new object every time it is constructed when Copy-on-Write is enabled (not returning multiple times an identical, cached Series object). This ensures that those Series objects correctly follow the Copy-on-Write rules (GH49450) Is this going to mean I can’t do df[‘a’] = 2 to set all values in column a to 2?

miohtama 4y ago
Copy-on-write is not enabled by default, but needs to be explicitly enabled with an option or a context manager.
- fauxpause_ 4y ago
  But if it were enabled, am I correct

henrydark 4y ago

Say what you will about sql, polars, pyspark, or whatever else. But nothing beats pandas' df[col].value_counts().value_counts()

humanistbot 4y ago
Can your entire dataframe fit in memory? Then you're good with pandas. If not, that's why those alternatives exist.

lvl102 4y ago

I still prefer Stata if I am completely honest with myself.

binarymax 4y ago
I mean that’s fine, but is there any reason? Just throwing your half second opinion out there on a product release announcement isn’t really helpful.
spaniard89277 4y ago
Why?
humanistbot 4y ago
You're comparing apple pie and orange juice here (even worse than comparing apples and oranges). Pandas is an open-source library for people who know python programming. Stata is an expensive GUI-based suite for non-programmers. If you're not in university, government, or non-profit, Stata's single-CPU license is $840/year in the US. Even a single-CPU student license is $94/year.
- lvl102 4y ago
  Have you ever used Stata? You sound like someone who never used Stata for serious data analysis.
  - crop_rotation 4y ago
    Do you have any counter point to the above mentioned pricing? If not, that illustrates a huge difference.
  - postexitus 4y ago
    you sound more like someone who doesn't know what they are talking about.
  - crimsoneer 4y ago
    The only people still using stata are 40+ year old economics professors and the poor lab students they force into obsolescence. Everybody else is using R (if you're into econometrics) or Python (if you're into ML).

benrutter 4y ago

I know arrow support is only part way their with this release - but this is a huge deal for Pandas, for standardisation as whole, but also speed ups. Benchmarking that was shared a while back here suggests 2x speed ups in some cases, 30x if you count strings since pandas uses python's in-built string data type[1] [1]https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i https://datapythonista.me/blog/pandas-20-and-the-arrow-revol...

0cf8612b2e1e 4y ago
Not just speed, but I thought one of the big Arrow wins relative to numpy processing was better memory usage.
wenc 4y ago
Polars is already built on Arrow and the advantages are tremendous.
- benrutter 4y ago
  Yeah Polars is an awesome library! I don't know a whole bunch about its internals, but I think it implements a bunch of additional speed ups through memory allocation and lazy evaluation, so its unlikely Pandas is will get to a similar speed without some huge changes elsewhere
  - wenc 4y ago
    Yup here’s a list of optimizations that Polars does to achieve the speed that it does https://pola-rs.github.io/polars-book/user-guide/#current-status https://pola-rs.github.io/polars-book/user-guide/#current-st... Polars’ lazy evaluation is a big deal — this lets it do query plan optimization. Whereas in Pandas every step is eager so it can’t look ahead to eliminate redundant steps. You basically can’t do a lot of query optimization in a multi step transform.

mynameisash 4y ago

I'm curious if there will be any appreciable performance gains here that are worthwhile. FWIW, last I checked[0], Polars still smokes Pandas in basically every way. [0] https://www.pola.rs/benchmarks.html https://www.pola.rs/benchmarks.html

westurner 4y ago
One could run the benchmarks with the new version of the software under concern and report back
wenc 4y ago
I’ve moved entirely to Polars (which is essentially Pandas written in Rust with new design decisions) with DuckDB as my SQL query engine. Since both are backed by Arrow, there is zero copy and performance on large datasets is super fast (due to vectorization, not just parallelization) I keep Pandas around for quick plots and legacy code. I will always be grateful for Pandas because there truly was no good dataframe library during its time. It has enabled an entire generation of data scientists to do what they do and built a foundation — a foundation which Polars and DuckDB are now building on and have surpassed.
- kylebarron 4y ago
  DuckDB is not backed by Arrow, but by something similar enough that it can be zero copy to Arrow in many cases https://duckdb.org/2021/12/03/duck-arrow.html https://duckdb.org/2021/12/03/duck-arrow.html
- RockyMcNuts 4y ago
  TFA: Pandas 2.0 is also backed by Arrow as an option, yielding a large speed improvement, although not all the way to Polars
- next_xibalba 4y ago
  How well does Polars play with many of the other standard data tools in Python (scikit learn, etc.)? Do the performance gains carry over?

modriano 4y ago

Jeff Reback gave a presentation on the roadmap for Pandas at PyData NYC 2022 [0]. In it, he basically says that pandas is used so widely in industry that big breaking changes are a non-starter, there won't be any radical changes to the API, but more performant implementations can/will be built into the library (although not set as defaults, at least not for a long while). Not a revolutionary leap, but a move towards making Wes McKinney's Arrow work more accessible through pandas. [0] https://www.youtube.com/watch?v=85XdsWz_Q_o https://www.youtube.com/watch?v=85XdsWz_Q_o

__mharrison__ 4y ago
This is why Modin and Ponder are game-changers for Pandas folks. You keep the API but get scaleout.

cmcconomy 4y ago

I'd love to know when geopandas snaps into alignment

timcavel 4y ago

[dead]

mongol 4y ago

Is it correct that I associate to a type of Chinese bears when I read about this project? Or is Pandas an acronym?

Guybrush_T 4y ago
Apparently it's taken from panel data.
ghshephard 4y ago
"The term Panel data is derived from econometrics and is partially responsible for the name pandas − pan(el)-da(ta)-s" https://www.tutorialspoint.com/python_pandas/python_pandas_panel.htm https://www.tutorialspoint.com/python_pandas/python_pandas_p...
deleted 4y ago
[deleted]

nonfamous 4y ago

Not a lot of people realize that Pandas was inspired by R, and in particular the Tidyverse model of handling rectangular data frames, created originally by Hadley Wickham. These days R is primarily used by data scientists in academia and certain niche industries like pharma, but its impact goes way beyond its core user base.

next_xibalba 4y ago
> and in particular the Tidyverse model This isn't really true. R took data frames from S, which was using the concept at least as early as 1991. Pandas itself predates most if not all of the tidyverse. Pandas original release occurred in 2008, whereas the first release of dplyr (one of the original packages of the tidyverse), didn't come until 2014.
- milliams 4y ago
  In fact I seem to recall that the Tidyverse paper references Pandas.
- nonfamous 4y ago
  It's definitely true that R's base data frame (a rectangular structure with columns of mixed types, which R in turn inherited from S) was the inspiration for Pandas. The concept of verbs operating on those structures IMO was inspired by plyr (the antecedent to dplyr, first published in 2008, which introduced composability for those verbs). data.table was also an inspiration, as another commenter points out.
  - kgwgk 4y ago
    > The concept of verbs operating on those structures IMO was inspired by plyr Was it? (I have no idea. But R already had verbs operating on data frames.)
  - deleted 4y ago
    [deleted]
- kgwgk 4y ago

edublancas 4y ago

How are people managing the existence of data frame APIs like pandas/polars with SQL engines like BigQuery, Snowflake, and DuckDB? Most of my notebooks are a mix of SQL and Python: SQL for most processing, dump the results as a pandas dataframe (via https://github.com/ploomber/jupysql https://github.com/ploomber/jupysql) and then use Python for operations that are difficult to express with SQL (or that I don't know how to do it), so I end up with 80% SQL, 20% Python. Unsure if this is the best workflow but it's the most efficient one I've come up with. Disclaimer: my team develops JupySQL.

Helmut10001 4y ago
I checked most solutions and I am sticking with plain old SQL in triple-qoutes in jupyter, e.g. [1]. I don't need auto-completion in SQL and I don't really need syntax highlighting. It's also very nice to have the combination of f-strings/variables substitution and SQL. Yet, my SQL needs are very basic. [1]: https://kartographie.geo.tu-dresden.de/ad/wip/ephemeral_events/html/00_raw_hll_conversion.html#Import-Milvus-milvus-example https://kartographie.geo.tu-dresden.de/ad/wip/ephemeral_even...
__mharrison__ 4y ago
You should check out https://ponder.io https://ponder.io . It is created by the folks who made Mondin (a scale out version of Pandas with API compatibility as a goal). Can use dask or ray as a backend. Ponder is the enterprise version that runs on Snowflake and BigQuery. Again, same goal, API compatibility with Pandas. You can scale out your Pandas workflow by changing the import and leaving the Pandas code. (Full disclosure I'm an advisor.)
wswope 4y ago
As a fellow evangelist of the SQL + Pandas hybrid workflow, I’m a happy camper with Pandas’ built-in read_sql_query and to_sql. Only big pain points are having to ship around boilerplate to construct SQLAlchemy create_engine URIs, and the performance limitations of SQLAlchemy’s inserts (if moving anything larger than a few gigs, it typically pays to ditch to_sql, and write a db-specific bulk insert process instead).