31 ms·

This is why 90% of data science is the messy, grunt work of just simply cleaning/normalizing the data. The analysis is the easy part.

screye 8y ago
> grunt work Honestly, it is an insult to call good data preprocessing skills grunt work. It may not be as attractive, but requires a lot of understanding of the nature of data and way in which patterns present themselves.
- minimaxir 8y ago
  That's the point of the comment. Modern data science tutorials assume the source data is a nice and tidy CSV, when the reality is that data is very often a mess, and it's a constraint often not reflected in the glamorization of the field. (it's also a reason that I recommend that code always be published when making posts such as these, as it's very easy to do wrong!)
  - alexhutcheson 8y ago
    Hadley Wickham's book "R for Data Science"[1] does a good job emphasizing the data cleaning and reshaping steps in the analysis process. [1] https://r4ds.had.co.nz/ https://r4ds.had.co.nz/
  - tfehring 8y ago
    >Modern data science tutorials assume the source data is a nice and tidy CSV For what it's worth, I don't really see this as a problem in and of itself. Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data. But it's good that tutorials segment these out, since not everyone needs to learn both at the same time. Anecdotally, I had plenty of real-world experience dealing with messy data by the time I started learning data science. As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation. Granted, they're still really time-consuming as it is, but there's probably some room for optimization there.
    - minimaxir 8y ago
      > Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data. True, although ideally the data is cleaned/structured before it hits a data warehouse and the data scientist starts working with it. It's an iterative process (DS finds messy data, flags the problem upstream, repeat). I do wish there were more tutorials with messy data; the ones I make I deliberately try to use complex datasets and highly relational ones, although as a result the tutorials are more complicated and more imposing for beginners. > As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation. When I was interviewing for DS jobs, I got a lot of the implement-binary-search-on-a-whiteboard questions which annoyed the hell out of me. But for the takehome assignments, they required some data ETL, which I felt was more representative. (some assignments had deliberately flawed datapoints the user was expected to identify and remove while not explicitly being told to do so)
- mrleinad 8y ago
  Grunt work noun [ U ] uk /ˈɡrʌnt ˌwɜːk/ us /ˈɡrʌnt ˌwɝːk/ informal. the basic, hard work, often physical or boring work, that is necessary for something to succeed. They're just calling it like it is.
- jMyles 8y ago
  It's insulting to insinuate that grunt work is insulting. (lightheartedly; I'm not like disgusted over here :-) )
- donaldknuth123 8y ago
  Eventually it will be split from actual Data Science work which is essentially either Machine Learning or Statistics. Imagine making a master sculptor mine his own Marble, it would be absolutely absurd similarly it would be silly to make a PhD Machine Learning expert work on data cleaning. A massive waste of talent on trivial work. Edit: trivial isn't the right word, it's obviously an important step in the process but quite different from what you are really paying a "Data Scientist/ML Engineer/etc" to do.
  - traek 8y ago
    This is why most large tech companies have data engineer roles that are separate from the data science role.
  - jmalicki 8y ago
    It's quite different... I would hope the marble miner would know quite a bit about geology, mining equipment, etc.... It's not that putting the sculptor on the task would be a waste, nearly as much as that the sculptor doesn't possess any of the necessary skills. How many people know how to find a source of marble, run a mine, and sculpt a great work of art? Roughly nil, but it's how many organizations seem to hire.
  - xapata 8y ago
    On the contrary, if the statisticians aren't intimately familiar with the data and the cleaning process, it's likely that they'll overlook some important features. When I interview potential hires, I make sure they'll be self-sufficient when writing data extraction code.
- scarejunba 8y ago
  My God. This "it is an insult" thing is insufferable. Please stop doing it. It's literally what the phrase means.
rhizome 8y ago
I worked for a company, which is still in business, 80% of whose work involves "cleaning/normalizing" data, basically everything that isn't sales and client services. Market research is all about getting different kinds of data and massaging it into queryable and productizable form (reports).
- AznHisoka 8y ago
  Where do they usually get the market research data from?
  - rhizome 8y ago
    Depends on the market being researched. Focus groups are one source, POS data is another. Large operators like Nielsen collect all kinds of data for all kinds of industries, which can be purchased. You collect what exists where it lies for the stuff you want to look at.
z3t4 8y ago
Often it can be faster to enter data manually then to make a script collect it automatically with the same accuracy.

screye 8y ago

> grunt work Honestly, it is an insult to call good data preprocessing skills grunt work. It may not be as attractive, but requires a lot of understanding of the nature of data and way in which patterns present themselves.

minimaxir 8y ago
That's the point of the comment. Modern data science tutorials assume the source data is a nice and tidy CSV, when the reality is that data is very often a mess, and it's a constraint often not reflected in the glamorization of the field. (it's also a reason that I recommend that code always be published when making posts such as these, as it's very easy to do wrong!)
- alexhutcheson 8y ago
  Hadley Wickham's book "R for Data Science"[1] does a good job emphasizing the data cleaning and reshaping steps in the analysis process. [1] https://r4ds.had.co.nz/ https://r4ds.had.co.nz/
- tfehring 8y ago
  >Modern data science tutorials assume the source data is a nice and tidy CSV For what it's worth, I don't really see this as a problem in and of itself. Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data. But it's good that tutorials segment these out, since not everyone needs to learn both at the same time. Anecdotally, I had plenty of real-world experience dealing with messy data by the time I started learning data science. As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation. Granted, they're still really time-consuming as it is, but there's probably some room for optimization there.
  - minimaxir 8y ago
    > Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data. True, although ideally the data is cleaned/structured before it hits a data warehouse and the data scientist starts working with it. It's an iterative process (DS finds messy data, flags the problem upstream, repeat). I do wish there were more tutorials with messy data; the ones I make I deliberately try to use complex datasets and highly relational ones, although as a result the tutorials are more complicated and more imposing for beginners. > As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation. When I was interviewing for DS jobs, I got a lot of the implement-binary-search-on-a-whiteboard questions which annoyed the hell out of me. But for the takehome assignments, they required some data ETL, which I felt was more representative. (some assignments had deliberately flawed datapoints the user was expected to identify and remove while not explicitly being told to do so)
mrleinad 8y ago
Grunt work noun [ U ] uk /ˈɡrʌnt ˌwɜːk/ us /ˈɡrʌnt ˌwɝːk/ informal. the basic, hard work, often physical or boring work, that is necessary for something to succeed. They're just calling it like it is.
jMyles 8y ago
It's insulting to insinuate that grunt work is insulting. (lightheartedly; I'm not like disgusted over here :-) )
donaldknuth123 8y ago
Eventually it will be split from actual Data Science work which is essentially either Machine Learning or Statistics. Imagine making a master sculptor mine his own Marble, it would be absolutely absurd similarly it would be silly to make a PhD Machine Learning expert work on data cleaning. A massive waste of talent on trivial work. Edit: trivial isn't the right word, it's obviously an important step in the process but quite different from what you are really paying a "Data Scientist/ML Engineer/etc" to do.
- traek 8y ago
  This is why most large tech companies have data engineer roles that are separate from the data science role.
- jmalicki 8y ago
  It's quite different... I would hope the marble miner would know quite a bit about geology, mining equipment, etc.... It's not that putting the sculptor on the task would be a waste, nearly as much as that the sculptor doesn't possess any of the necessary skills. How many people know how to find a source of marble, run a mine, and sculpt a great work of art? Roughly nil, but it's how many organizations seem to hire.
- xapata 8y ago
  On the contrary, if the statisticians aren't intimately familiar with the data and the cleaning process, it's likely that they'll overlook some important features. When I interview potential hires, I make sure they'll be self-sufficient when writing data extraction code.
scarejunba 8y ago
My God. This "it is an insult" thing is insufferable. Please stop doing it. It's literally what the phrase means.

rhizome 8y ago

I worked for a company, which is still in business, 80% of whose work involves "cleaning/normalizing" data, basically everything that isn't sales and client services. Market research is all about getting different kinds of data and massaging it into queryable and productizable form (reports).

AznHisoka 8y ago
Where do they usually get the market research data from?
- rhizome 8y ago
  Depends on the market being researched. Focus groups are one source, POS data is another. Large operators like Nielsen collect all kinds of data for all kinds of industries, which can be purchased. You collect what exists where it lies for the stuff you want to look at.

z3t4 8y ago

Often it can be faster to enter data manually then to make a script collect it automatically with the same accuracy.