3 ms·

The high theory stuff is great, but a significant portion of the job is being a data janitor. Being experienced and fast at manipulating data structures, recogn

by daemonk 10y ago

The high theory stuff is great, but a significant portion of the job is being a data janitor. Being experienced and fast at manipulating data structures, recognizing patterns in text datasets, understanding common formats used in the field and just having domain knowledge in what you are analyzing should be more emphasized in my opinion.

crispyambulance 10y ago
I sort-of agree but the "data janitor" knowledge can be learned "on the job" ad-hoc and as-needed. Mastering basic theory, on the other hand, needs a coherent and structured study-plan which requires extended focus and single-minded emphasis (at least for most folks).
- sgt101 10y ago
  The on the job learning of "data janitoring" may be a contributing factor to why so much janitor action is required!
- coldtea 10y ago
  >I sort-of agree but the "data janitor" knowledge can be learned "on the job" ad-hoc and as-needed. Logically yes. But it's amusing how many scienty types (PhDs et al) who know all the fundamentals in theory can't do such practical tasks if their life was depended on it. It's like they thought the theorems and abstract objects they've learned would never be encountered in the wild.
  - Ar-Curunir 10y ago
    Yet PhDs are disproportionately more likely to get the interesting jobs that require stats: quabts, experimental.positions at Google, etc.etc.
brudgers 10y ago
Listening to Software Engineering Daily, I heard that the there is a trend toward developing 'data engineering' as a discipline that handles sanitizing the data pipeline so that data scientists can spend more time working at business abstraction layers. There's probably a scale at which that works better, two pizza data science teams for example.
- daemonk 10y ago
  My issue with these sanitizing pipelines is that they only really work for fields where the data generation is relatively stable. Meaning, whatever instrument/method used in generating the data doesn't change dramatically every year. It's extremely challenging to design a all-purpose sanitizing pipeline. So I can imagine a pipeline developed for use with well established social media APIs or standard scientific experiments that have been in used for decades. But it is hard to imagine a pipeline that can handle amorphous emerging high throughput instruments/methods.
- randcraw 10y ago
  Yeah, DE seems to have emerged as the title for those who manage large databases -- who regularly import raw data, cleanse, normalize, update, construct analysis pipelines, scale up and parallelize processes, and validate results. It lies somewhere between a DBA, sysadmin, and HPC engineer, with an awareness of basic stats -- where mastery of Hadoop's many components might converge. It seems like a natural evolution of DB admin for very large scale noSQL and DBs, esp. those with unstructured data and often non-commodity architecture. I've seen numerous companies looking for such folks and I suspect demand will rise. IMO, it's not a job you'd want to outsource. The role is too mission critical, the skills not predictable enough to be a commodity, and the penalty for screwing up is too great.
zengid 10y ago
As someone shooting for a 'Data Janitor' (or what I think of as a digital plumber) position upon graduation, It would be cool to speak to the Data Scientists in their own language and understand what the hell they mean when they mention higher dimensional vectors, etc. But I'd much rather leave the high math to an expert and they leave the plumbing to me!
shas3 10y ago
I have to disagree on this. Data scientist is a very fluid and fast changing term. I know companies who now call what would traditionally have been "machine learning researcher", "machine learning engineer", or "research scientist in <any data related stuff - data mining, computer vision, signal processing, machine learning, statistics>" as "data scientist" positions. Because it is sexy, in many places, management and job hunters get a kick out of the term "data science" even if it is actually a renaming of other traditional roles. This book perfectly captures the diverse nature of the term.

by daemonk 10y ago

crispyambulance 10y ago

I sort-of agree but the "data janitor" knowledge can be learned "on the job" ad-hoc and as-needed. Mastering basic theory, on the other hand, needs a coherent and structured study-plan which requires extended focus and single-minded emphasis (at least for most folks).

sgt101 10y ago
The on the job learning of "data janitoring" may be a contributing factor to why so much janitor action is required!
coldtea 10y ago
>I sort-of agree but the "data janitor" knowledge can be learned "on the job" ad-hoc and as-needed. Logically yes. But it's amusing how many scienty types (PhDs et al) who know all the fundamentals in theory can't do such practical tasks if their life was depended on it. It's like they thought the theorems and abstract objects they've learned would never be encountered in the wild.
- Ar-Curunir 10y ago
  Yet PhDs are disproportionately more likely to get the interesting jobs that require stats: quabts, experimental.positions at Google, etc.etc.

brudgers 10y ago

Listening to Software Engineering Daily, I heard that the there is a trend toward developing 'data engineering' as a discipline that handles sanitizing the data pipeline so that data scientists can spend more time working at business abstraction layers. There's probably a scale at which that works better, two pizza data science teams for example.

daemonk 10y ago
My issue with these sanitizing pipelines is that they only really work for fields where the data generation is relatively stable. Meaning, whatever instrument/method used in generating the data doesn't change dramatically every year. It's extremely challenging to design a all-purpose sanitizing pipeline. So I can imagine a pipeline developed for use with well established social media APIs or standard scientific experiments that have been in used for decades. But it is hard to imagine a pipeline that can handle amorphous emerging high throughput instruments/methods.
randcraw 10y ago
Yeah, DE seems to have emerged as the title for those who manage large databases -- who regularly import raw data, cleanse, normalize, update, construct analysis pipelines, scale up and parallelize processes, and validate results. It lies somewhere between a DBA, sysadmin, and HPC engineer, with an awareness of basic stats -- where mastery of Hadoop's many components might converge. It seems like a natural evolution of DB admin for very large scale noSQL and DBs, esp. those with unstructured data and often non-commodity architecture. I've seen numerous companies looking for such folks and I suspect demand will rise. IMO, it's not a job you'd want to outsource. The role is too mission critical, the skills not predictable enough to be a commodity, and the penalty for screwing up is too great.

zengid 10y ago

As someone shooting for a 'Data Janitor' (or what I think of as a digital plumber) position upon graduation, It would be cool to speak to the Data Scientists in their own language and understand what the hell they mean when they mention higher dimensional vectors, etc. But I'd much rather leave the high math to an expert and they leave the plumbing to me!

shas3 10y ago

I have to disagree on this. Data scientist is a very fluid and fast changing term. I know companies who now call what would traditionally have been "machine learning researcher", "machine learning engineer", or "research scientist in <any data related stuff - data mining, computer vision, signal processing, machine learning, statistics>" as "data scientist" positions. Because it is sexy, in many places, management and job hunters get a kick out of the term "data science" even if it is actually a renaming of other traditional roles. This book perfectly captures the diverse nature of the term.