4 ms·

I agree with you, is this the type of work a data engineer does? I actually kinda like that.

• static_noise 8y ago
  It may not be in the job description but usually somebody has to clean the data by hand or build specialized tools before it can be used. Something as simple as a text field for entering a date can produce interesting effects. Even if your data looks good on the surface, when you dig in, you might find interesting things. Like data entered when the system was definitely shut down for maintenance.
• kthejoker2 8y ago
  That's how we define it where I work. Data engineering cleanse data, manage workflows for ETL, build views, encapsulate rules and logic. Data science is taking that cleaned up data and testing hypotheses. Plenty of people can do both, and frankly it's very condescendimg when I run into a data scientist who prefers to outsource the engineering (because they're bad at it) so they can do "real work." Source: I hire data scientists for Fortune 100 companies.
  • geezerjay 8y ago
    > Data science is taking that cleaned up data and testing hypotheses. Dimensionality reduction is a big part of data science, and it tends to be far more important than data cleanup and outlier detection as it dictates which modeling methods are viable in practice given your resources.