I've been thinking about this lately at work, where we do a bunch of stuff with a pile of python scripts built around csv files and pandas. This post prompted

I've been thinking about this lately at work, where we do a bunch of stuff with a pile of python scripts built around csv files and pandas. This post prompted me to give pyarrow a spin and WOW did it load the CSVs fast compared to what I'm used to. That said, I'm not entirely sure what to do with them from there. When I tried table["col"] it gave back a list of lists (maybe because of the multi-threaded loading?). And while the article talked about "querying" the arrow files with duckdb, how easy is it to work with them otherwise, in terms of updating them? And what about non-SQL style updates? Sometimes you just have to do an itertuples or apply and do random non-declarative modifications to the data.