3 ms·

Not a data engineer / scientist here. What am I missing with data-frames? I've written products that utilized them before (years ago), but it's been long enou

Not a data engineer / scientist here. What am I missing with data-frames? I've written products that utilized them before (years ago), but it's been long enough that I don't know what I would even use them for in daily programming.

shirakawasuna 8y ago
They're for tables and doing table-ie things: an imperative programming method for the same tasks people use Excel or SQL for. This dovetails nicely with most data science-ie tasks, like statistical analysis, engineering features, simple machine learning techniques (the tables can be easily transformed into numpy arrays), and plotting results. You can use them any time you feel the need to do table manipulations on non-insanely-large data: aggregate, transform, combine, split, reshape, join...
- vonseel 8y ago
  Got it, I was aware of this and many of the performance benefits another user mentioned, but wasn't sure how it could be useful in daily operation or quick Django shell_plus style scripting (one-off commands) when inspecting something like a User's table, as the original comment above indicated.
Qwertystop 8y ago
From NumPy, you get fast (vectorized) operations on large multidimensional arrays of numbers. This bit not be so useful outside of a data-science context but it depends on what it is you _are_ doing. Most graphics stuff is also arrays of numbers, for example. Also lots of useful indexing/slicing/masking/filtering on those structures. You can also have non-numeric multidimensional arrays, but the performance part might not be there depending on the type. Pandas adds utility methods like join, groupby, and fromsql. Also hierarchical indexing, and a bunch of other stuff that I don't know much about - I'm not in data-science either, but I work with some of them. Overall, lots of useful stuff for processing tables locally once you've fetched them from SQL or similar. Also, the simple fact of having a tabular data structure without needing to define a custom Row class. Lists of dicts or tuples have a lot of overhead by comparison.

shirakawasuna 8y ago

They're for tables and doing table-ie things: an imperative programming method for the same tasks people use Excel or SQL for. This dovetails nicely with most data science-ie tasks, like statistical analysis, engineering features, simple machine learning techniques (the tables can be easily transformed into numpy arrays), and plotting results. You can use them any time you feel the need to do table manipulations on non-insanely-large data: aggregate, transform, combine, split, reshape, join...

vonseel 8y ago
Got it, I was aware of this and many of the performance benefits another user mentioned, but wasn't sure how it could be useful in daily operation or quick Django shell_plus style scripting (one-off commands) when inspecting something like a User's table, as the original comment above indicated.

Qwertystop 8y ago

From NumPy, you get fast (vectorized) operations on large multidimensional arrays of numbers. This bit not be so useful outside of a data-science context but it depends on what it is you _are_ doing. Most graphics stuff is also arrays of numbers, for example. Also lots of useful indexing/slicing/masking/filtering on those structures. You can also have non-numeric multidimensional arrays, but the performance part might not be there depending on the type. Pandas adds utility methods like join, groupby, and fromsql. Also hierarchical indexing, and a bunch of other stuff that I don't know much about - I'm not in data-science either, but I work with some of them. Overall, lots of useful stuff for processing tables locally once you've fetched them from SQL or similar. Also, the simple fact of having a tabular data structure without needing to define a custom Row class. Lists of dicts or tuples have a lot of overhead by comparison.