9 ms·

Data science. Static typing is a complete PITA when you're doing exploratory data work. Type inference only fixes things on a superficial level. For data engin

Data science. Static typing is a complete PITA when you're doing exploratory data work. Type inference only fixes things on a superficial level. For data engineering, I haven't had a chance to try this due to lack of language options, but I strongly suspect that structural typing would be much more useful than nominal typing.

skohan 6y ago
Yeah but even then you can usually deserialize a CSV or json into “plain” data structures like arrays and dicts to explore it, and then switch over to a more rhobist, automatically serializable one when you have a better idea what the data looks like
- mumblemumble 6y ago
  You can, but, IME, the practical benefit of doing so generally isn't proportional to the cost. It's just something you do if you have strong feelings about type discipline and are willing to sink a bunch of time into soothing them. It may have to do with the types you're using. When I'm doing EDA, I'm usually zooming past the arrays and dicts as quickly as I can, and making a beeline for something like Pandas that lets me manipulate the data more easily. Pandas is fairly pretty resistant to static typing, and getting over to something that does play nicer with static typing tends to also necessitate a lot of wheel reinvention. Tends to also bring a performance cost, too, since I'm often losing a lot of nice fast multithreaded C++ implementation for a mess of pointer chasing and GIL.
exdsq 6y ago
I’m not sure I really get this argument - you need to know something about the type you’re getting to explore it, so why does specifying that type slow things down more than any gains you would with run time errors from hacking around?
- PeterisP 6y ago
  You're often reading and processing a data source that's explicitly not "well typed" (heterogenous 'columns', heterogenous deep nested json structures, null values in places where they shouldn't be, garbage rows, etc), where getting them to fit any reasonable type assertions would be a lot of work in correcting and/or filtering that data, which is something that you would write after your exploration is done. Also, it's not uncommon to not be sure about anything in the data structure you're getting; it's so often that the schema or documentation of that data is wrong or outdated or simply missing, so understanding the properties of the data types is a key part of that exploration. You start with eyeballing the data with your code; making some assumptions, and making code to process and analyze the data that doesn't meet your type assumptions, instead of throwing a runtime error as soon as your assumptions are wrong and conflict with the data.
  - skohan 6y ago
    But arguably some of that issue could be helped if you had good type systems on both ends. How much data is poorly formatted because somebody just hacked it out of some system with whatever scripting language?
    - PeterisP 6y ago
      Sure, it would be nice if everyone would have a pony, but in such scenarios you're often working with data where you have no control over the data sources and limited or no contact with the people maintaining them, if these sources are even maintained at all. In some cases (not all) it is possible to assign resources to improve the system from which you're getting data, but that generally is conditional on you being able to demonstrate that this use of data is valuable, so that would only happen after an extensive data analysis is already done on the data as it is now. Exploratory data analysis (and much of data science as a whole) requires you to work with data it is, not data as you'd want or data as it should be; and a tool which can't conveniently work with bad, dirty data is simply not a suitable tool for the task.
      - skohan 6y ago
mountainriver 6y ago
I type all my data science/ml code and I absolutely love it. Now when I explore packages or data I have a more clear understanding of what I’m dealing with
- skohan 6y ago
  I feel like moving toward types is like organizing a messy chest of drawers. At first taking the extra care feels like more work than just rifling through the drawers like you're used to, but once the structure is in place you can't imagine going back.

skohan 6y ago

Yeah but even then you can usually deserialize a CSV or json into “plain” data structures like arrays and dicts to explore it, and then switch over to a more rhobist, automatically serializable one when you have a better idea what the data looks like

mumblemumble 6y ago
You can, but, IME, the practical benefit of doing so generally isn't proportional to the cost. It's just something you do if you have strong feelings about type discipline and are willing to sink a bunch of time into soothing them. It may have to do with the types you're using. When I'm doing EDA, I'm usually zooming past the arrays and dicts as quickly as I can, and making a beeline for something like Pandas that lets me manipulate the data more easily. Pandas is fairly pretty resistant to static typing, and getting over to something that does play nicer with static typing tends to also necessitate a lot of wheel reinvention. Tends to also bring a performance cost, too, since I'm often losing a lot of nice fast multithreaded C++ implementation for a mess of pointer chasing and GIL.

exdsq 6y ago

I’m not sure I really get this argument - you need to know something about the type you’re getting to explore it, so why does specifying that type slow things down more than any gains you would with run time errors from hacking around?

PeterisP 6y ago
You're often reading and processing a data source that's explicitly not "well typed" (heterogenous 'columns', heterogenous deep nested json structures, null values in places where they shouldn't be, garbage rows, etc), where getting them to fit any reasonable type assertions would be a lot of work in correcting and/or filtering that data, which is something that you would write after your exploration is done. Also, it's not uncommon to not be sure about anything in the data structure you're getting; it's so often that the schema or documentation of that data is wrong or outdated or simply missing, so understanding the properties of the data types is a key part of that exploration. You start with eyeballing the data with your code; making some assumptions, and making code to process and analyze the data that doesn't meet your type assumptions, instead of throwing a runtime error as soon as your assumptions are wrong and conflict with the data.
- skohan 6y ago
  But arguably some of that issue could be helped if you had good type systems on both ends. How much data is poorly formatted because somebody just hacked it out of some system with whatever scripting language?
  - PeterisP 6y ago
    Sure, it would be nice if everyone would have a pony, but in such scenarios you're often working with data where you have no control over the data sources and limited or no contact with the people maintaining them, if these sources are even maintained at all. In some cases (not all) it is possible to assign resources to improve the system from which you're getting data, but that generally is conditional on you being able to demonstrate that this use of data is valuable, so that would only happen after an extensive data analysis is already done on the data as it is now. Exploratory data analysis (and much of data science as a whole) requires you to work with data it is, not data as you'd want or data as it should be; and a tool which can't conveniently work with bad, dirty data is simply not a suitable tool for the task.
    - skohan 6y ago

mountainriver 6y ago

I type all my data science/ml code and I absolutely love it. Now when I explore packages or data I have a more clear understanding of what I’m dealing with

skohan 6y ago
I feel like moving toward types is like organizing a messy chest of drawers. At first taking the extra care feels like more work than just rifling through the drawers like you're used to, but once the structure is in place you can't imagine going back.