3 ms·

You have to be sure, like always. You can't write the code dealing with the data you have no clue about. The type reflects invariants, constraints given for yo

You have to be sure, like always. You can't write the code dealing with the data you have no clue about. The type reflects invariants, constraints given for you data. The less you know, the more general type your data would have. Dynamic typing gives no advantages here safe for providing a super-general ANY type. In case of statically typed language you would need to parse your data and construct an object representing that data just like in case of dynamically typed one. But the more you know about your data in advance, the better, especially when you are dealing with protocols and communication.

goatlover 7y ago
> But the more you know about your data in advance, the better, especially when you are dealing with protocols and communication. The point of a library like Pandas is that you will be cleaning and reshaping your data until it's in a format that can be properly analyzed. And that data can be very dirty and very large. But if you prefer to clean it up in Excel or with some bash scripts, or custom written functions in PL of choice, then go for it. You still have to deal with the cleaning and wrangling. The point of mentioning Pandas is that you don't always have a guarantee on properly structured data where you can predefine the types and read it in. And you don't always have control over how that data comes to you.
- bitwize 7y ago
  All data has SOME structural metainformation that can be captured in a type. If you are importing a CSV file, you know -- at a minimum -- that it's a CSV and can be parsed as a set of tuples of variants (or else there's something really wrong with the input and an exception continuation should be invoked to handle it). From there you can do analysis on the data to further refine its type. But ALL data is typed, and encoding its type early in the code that processes it makes your assumptions clear and compiler-enforceable. You're right that a typed approach to handling arbitrary input data is not right for a library like Pandas. It's a fundamentally different approach than the one Pandas takes, but it's no less valid and brings with it certain advantages.

goatlover 7y ago

> But the more you know about your data in advance, the better, especially when you are dealing with protocols and communication. The point of a library like Pandas is that you will be cleaning and reshaping your data until it's in a format that can be properly analyzed. And that data can be very dirty and very large. But if you prefer to clean it up in Excel or with some bash scripts, or custom written functions in PL of choice, then go for it. You still have to deal with the cleaning and wrangling. The point of mentioning Pandas is that you don't always have a guarantee on properly structured data where you can predefine the types and read it in. And you don't always have control over how that data comes to you.

bitwize 7y ago
All data has SOME structural metainformation that can be captured in a type. If you are importing a CSV file, you know -- at a minimum -- that it's a CSV and can be parsed as a set of tuples of variants (or else there's something really wrong with the input and an exception continuation should be invoked to handle it). From there you can do analysis on the data to further refine its type. But ALL data is typed, and encoding its type early in the code that processes it makes your assumptions clear and compiler-enforceable. You're right that a typed approach to handling arbitrary input data is not right for a library like Pandas. It's a fundamentally different approach than the one Pandas takes, but it's no less valid and brings with it certain advantages.