3 ms·

I’m not sure I really get this argument - you need to know something about the type you’re getting to explore it, so why does specifying that type slow things d

by exdsq 6y ago

I’m not sure I really get this argument - you need to know something about the type you’re getting to explore it, so why does specifying that type slow things down more than any gains you would with run time errors from hacking around?

PeterisP 6y ago
You're often reading and processing a data source that's explicitly not "well typed" (heterogenous 'columns', heterogenous deep nested json structures, null values in places where they shouldn't be, garbage rows, etc), where getting them to fit any reasonable type assertions would be a lot of work in correcting and/or filtering that data, which is something that you would write after your exploration is done. Also, it's not uncommon to not be sure about anything in the data structure you're getting; it's so often that the schema or documentation of that data is wrong or outdated or simply missing, so understanding the properties of the data types is a key part of that exploration. You start with eyeballing the data with your code; making some assumptions, and making code to process and analyze the data that doesn't meet your type assumptions, instead of throwing a runtime error as soon as your assumptions are wrong and conflict with the data.
- skohan 6y ago
  But arguably some of that issue could be helped if you had good type systems on both ends. How much data is poorly formatted because somebody just hacked it out of some system with whatever scripting language?
  - PeterisP 6y ago
    Sure, it would be nice if everyone would have a pony, but in such scenarios you're often working with data where you have no control over the data sources and limited or no contact with the people maintaining them, if these sources are even maintained at all. In some cases (not all) it is possible to assign resources to improve the system from which you're getting data, but that generally is conditional on you being able to demonstrate that this use of data is valuable, so that would only happen after an extensive data analysis is already done on the data as it is now. Exploratory data analysis (and much of data science as a whole) requires you to work with data it is, not data as you'd want or data as it should be; and a tool which can't conveniently work with bad, dirty data is simply not a suitable tool for the task.
    - skohan 6y ago
      Most modern languages have tools to work with "messy data". The difference is that strongly typed languages have the tools to provide more structure once you do get the data into a reasonable state - which is hopefully something you would want before you start doing serious analysis.
      - mumblemumble 6y ago
        There's a chicken and egg problem. To figure out the structure of the data, you often need to get it into a programming environment so that you can more easily interrogate it. Unless you're already working with data that are well structured and well documented, that is often much easier to do in a dynamic language, because dynamic languages make it much easier to shift your types around on the fly. It's often even convenient to be able to modify them at run time, which is something I often end up doing, and also something that is, by definition, incompatible with static typing. By the time you have the structure grokked enough to easily work with it in a statically typed language, you've already got a program that is well-typed and strongly typed. Porting it to a static language would sort of be missing the point, because you certainly aren't going to get any static type checking to help you make sure you've correctly ported the code. And data wrangling code is rarely complex enough that I'm particularly worried about safe refactoring. And I'm not particularly excited to jump to a static language before I start doing serious analysis, because there still aren't any static languages with top-shelf data analysis packages. Partially, I think, because static typing also becomes a hindrance on this end of things. It's really convenient, for example, to be able to tell your graphing package, "Give me a splot matrix of these five columns," and have it be able to do that. A hypothetical static language with two datatypes would need 32 different overloads of the function for drawing splot matrix, or it would need to operate on a specialized data structure that puts a lot of effort into Greenspunning a limited dynamic typing facility into the language, or alternatively (and this seems to be the more popular option), it would need to simply forego having useful things like splot matrices, and instead just stick to the basics.
- exdsq 6y ago
  But isn’t this caused by a lack of decent typing? If I had a well-typed API and a public schema, wouldn’t that be the better option? Also you could just stick that in as an array or string (or var) to start with

by exdsq 6y ago

PeterisP 6y ago

You're often reading and processing a data source that's explicitly not "well typed" (heterogenous 'columns', heterogenous deep nested json structures, null values in places where they shouldn't be, garbage rows, etc), where getting them to fit any reasonable type assertions would be a lot of work in correcting and/or filtering that data, which is something that you would write after your exploration is done. Also, it's not uncommon to not be sure about anything in the data structure you're getting; it's so often that the schema or documentation of that data is wrong or outdated or simply missing, so understanding the properties of the data types is a key part of that exploration. You start with eyeballing the data with your code; making some assumptions, and making code to process and analyze the data that doesn't meet your type assumptions, instead of throwing a runtime error as soon as your assumptions are wrong and conflict with the data.

skohan 6y ago
But arguably some of that issue could be helped if you had good type systems on both ends. How much data is poorly formatted because somebody just hacked it out of some system with whatever scripting language?
- PeterisP 6y ago
  Sure, it would be nice if everyone would have a pony, but in such scenarios you're often working with data where you have no control over the data sources and limited or no contact with the people maintaining them, if these sources are even maintained at all. In some cases (not all) it is possible to assign resources to improve the system from which you're getting data, but that generally is conditional on you being able to demonstrate that this use of data is valuable, so that would only happen after an extensive data analysis is already done on the data as it is now. Exploratory data analysis (and much of data science as a whole) requires you to work with data it is, not data as you'd want or data as it should be; and a tool which can't conveniently work with bad, dirty data is simply not a suitable tool for the task.
  - skohan 6y ago
    Most modern languages have tools to work with "messy data". The difference is that strongly typed languages have the tools to provide more structure once you do get the data into a reasonable state - which is hopefully something you would want before you start doing serious analysis.
    - mumblemumble 6y ago
      There's a chicken and egg problem. To figure out the structure of the data, you often need to get it into a programming environment so that you can more easily interrogate it. Unless you're already working with data that are well structured and well documented, that is often much easier to do in a dynamic language, because dynamic languages make it much easier to shift your types around on the fly. It's often even convenient to be able to modify them at run time, which is something I often end up doing, and also something that is, by definition, incompatible with static typing. By the time you have the structure grokked enough to easily work with it in a statically typed language, you've already got a program that is well-typed and strongly typed. Porting it to a static language would sort of be missing the point, because you certainly aren't going to get any static type checking to help you make sure you've correctly ported the code. And data wrangling code is rarely complex enough that I'm particularly worried about safe refactoring. And I'm not particularly excited to jump to a static language before I start doing serious analysis, because there still aren't any static languages with top-shelf data analysis packages. Partially, I think, because static typing also becomes a hindrance on this end of things. It's really convenient, for example, to be able to tell your graphing package, "Give me a splot matrix of these five columns," and have it be able to do that. A hypothetical static language with two datatypes would need 32 different overloads of the function for drawing splot matrix, or it would need to operate on a specialized data structure that puts a lot of effort into Greenspunning a limited dynamic typing facility into the language, or alternatively (and this seems to be the more popular option), it would need to simply forego having useful things like splot matrices, and instead just stick to the basics.
exdsq 6y ago
But isn’t this caused by a lack of decent typing? If I had a well-typed API and a public schema, wouldn’t that be the better option? Also you could just stick that in as an array or string (or var) to start with