3 ms·

Most modern languages have tools to work with "messy data". The difference is that strongly typed languages have the tools to provide more structure once you d

by skohan 6y ago

Most modern languages have tools to work with "messy data". The difference is that strongly typed languages have the tools to provide more structure once you do get the data into a reasonable state - which is hopefully something you would want before you start doing serious analysis.

mumblemumble 6y ago
There's a chicken and egg problem. To figure out the structure of the data, you often need to get it into a programming environment so that you can more easily interrogate it. Unless you're already working with data that are well structured and well documented, that is often much easier to do in a dynamic language, because dynamic languages make it much easier to shift your types around on the fly. It's often even convenient to be able to modify them at run time, which is something I often end up doing, and also something that is, by definition, incompatible with static typing. By the time you have the structure grokked enough to easily work with it in a statically typed language, you've already got a program that is well-typed and strongly typed. Porting it to a static language would sort of be missing the point, because you certainly aren't going to get any static type checking to help you make sure you've correctly ported the code. And data wrangling code is rarely complex enough that I'm particularly worried about safe refactoring. And I'm not particularly excited to jump to a static language before I start doing serious analysis, because there still aren't any static languages with top-shelf data analysis packages. Partially, I think, because static typing also becomes a hindrance on this end of things. It's really convenient, for example, to be able to tell your graphing package, "Give me a splot matrix of these five columns," and have it be able to do that. A hypothetical static language with two datatypes would need 32 different overloads of the function for drawing splot matrix, or it would need to operate on a specialized data structure that puts a lot of effort into Greenspunning a limited dynamic typing facility into the language, or alternatively (and this seems to be the more popular option), it would need to simply forego having useful things like splot matrices, and instead just stick to the basics.

by skohan 6y ago

mumblemumble 6y ago

There's a chicken and egg problem. To figure out the structure of the data, you often need to get it into a programming environment so that you can more easily interrogate it. Unless you're already working with data that are well structured and well documented, that is often much easier to do in a dynamic language, because dynamic languages make it much easier to shift your types around on the fly. It's often even convenient to be able to modify them at run time, which is something I often end up doing, and also something that is, by definition, incompatible with static typing. By the time you have the structure grokked enough to easily work with it in a statically typed language, you've already got a program that is well-typed and strongly typed. Porting it to a static language would sort of be missing the point, because you certainly aren't going to get any static type checking to help you make sure you've correctly ported the code. And data wrangling code is rarely complex enough that I'm particularly worried about safe refactoring. And I'm not particularly excited to jump to a static language before I start doing serious analysis, because there still aren't any static languages with top-shelf data analysis packages. Partially, I think, because static typing also becomes a hindrance on this end of things. It's really convenient, for example, to be able to tell your graphing package, "Give me a splot matrix of these five columns," and have it be able to do that. A hypothetical static language with two datatypes would need 32 different overloads of the function for drawing splot matrix, or it would need to operate on a specialized data structure that puts a lot of effort into Greenspunning a limited dynamic typing facility into the language, or alternatively (and this seems to be the more popular option), it would need to simply forego having useful things like splot matrices, and instead just stick to the basics.