3 ms·

But arguably some of that issue could be helped if you had good type systems on both ends. How much data is poorly formatted because somebody just hacked it ou

by skohan 6y ago

But arguably some of that issue could be helped if you had good type systems on both ends. How much data is poorly formatted because somebody just hacked it out of some system with whatever scripting language?

PeterisP 6y ago
Sure, it would be nice if everyone would have a pony, but in such scenarios you're often working with data where you have no control over the data sources and limited or no contact with the people maintaining them, if these sources are even maintained at all. In some cases (not all) it is possible to assign resources to improve the system from which you're getting data, but that generally is conditional on you being able to demonstrate that this use of data is valuable, so that would only happen after an extensive data analysis is already done on the data as it is now. Exploratory data analysis (and much of data science as a whole) requires you to work with data it is, not data as you'd want or data as it should be; and a tool which can't conveniently work with bad, dirty data is simply not a suitable tool for the task.
- skohan 6y ago
  Most modern languages have tools to work with "messy data". The difference is that strongly typed languages have the tools to provide more structure once you do get the data into a reasonable state - which is hopefully something you would want before you start doing serious analysis.
  - mumblemumble 6y ago
    There's a chicken and egg problem. To figure out the structure of the data, you often need to get it into a programming environment so that you can more easily interrogate it. Unless you're already working with data that are well structured and well documented, that is often much easier to do in a dynamic language, because dynamic languages make it much easier to shift your types around on the fly. It's often even convenient to be able to modify them at run time, which is something I often end up doing, and also something that is, by definition, incompatible with static typing. By the time you have the structure grokked enough to easily work with it in a statically typed language, you've already got a program that is well-typed and strongly typed. Porting it to a static language would sort of be missing the point, because you certainly aren't going to get any static type checking to help you make sure you've correctly ported the code. And data wrangling code is rarely complex enough that I'm particularly worried about safe refactoring. And I'm not particularly excited to jump to a static language before I start doing serious analysis, because there still aren't any static languages with top-shelf data analysis packages. Partially, I think, because static typing also becomes a hindrance on this end of things. It's really convenient, for example, to be able to tell your graphing package, "Give me a splot matrix of these five columns," and have it be able to do that. A hypothetical static language with two datatypes would need 32 different overloads of the function for drawing splot matrix, or it would need to operate on a specialized data structure that puts a lot of effort into Greenspunning a limited dynamic typing facility into the language, or alternatively (and this seems to be the more popular option), it would need to simply forego having useful things like splot matrices, and instead just stick to the basics.

by skohan 6y ago

PeterisP 6y ago

Sure, it would be nice if everyone would have a pony, but in such scenarios you're often working with data where you have no control over the data sources and limited or no contact with the people maintaining them, if these sources are even maintained at all. In some cases (not all) it is possible to assign resources to improve the system from which you're getting data, but that generally is conditional on you being able to demonstrate that this use of data is valuable, so that would only happen after an extensive data analysis is already done on the data as it is now. Exploratory data analysis (and much of data science as a whole) requires you to work with data it is, not data as you'd want or data as it should be; and a tool which can't conveniently work with bad, dirty data is simply not a suitable tool for the task.

skohan 6y ago
Most modern languages have tools to work with "messy data". The difference is that strongly typed languages have the tools to provide more structure once you do get the data into a reasonable state - which is hopefully something you would want before you start doing serious analysis.
- mumblemumble 6y ago
  There's a chicken and egg problem. To figure out the structure of the data, you often need to get it into a programming environment so that you can more easily interrogate it. Unless you're already working with data that are well structured and well documented, that is often much easier to do in a dynamic language, because dynamic languages make it much easier to shift your types around on the fly. It's often even convenient to be able to modify them at run time, which is something I often end up doing, and also something that is, by definition, incompatible with static typing. By the time you have the structure grokked enough to easily work with it in a statically typed language, you've already got a program that is well-typed and strongly typed. Porting it to a static language would sort of be missing the point, because you certainly aren't going to get any static type checking to help you make sure you've correctly ported the code. And data wrangling code is rarely complex enough that I'm particularly worried about safe refactoring. And I'm not particularly excited to jump to a static language before I start doing serious analysis, because there still aren't any static languages with top-shelf data analysis packages. Partially, I think, because static typing also becomes a hindrance on this end of things. It's really convenient, for example, to be able to tell your graphing package, "Give me a splot matrix of these five columns," and have it be able to do that. A hypothetical static language with two datatypes would need 32 different overloads of the function for drawing splot matrix, or it would need to operate on a specialized data structure that puts a lot of effort into Greenspunning a limited dynamic typing facility into the language, or alternatively (and this seems to be the more popular option), it would need to simply forego having useful things like splot matrices, and instead just stick to the basics.