6 ms·

Is anyone actually doing that? I wonder if there's a profession, or if there ought to be, specifically for "data wrangling". On the other hand, maybe you can'

Is anyone actually doing that? I wonder if there's a profession, or if there ought to be, specifically for "data wrangling". On the other hand, maybe you can't separate it from the work of the data scientist... perhaps the data scientist needs to be involved in the data wrangling in order to understand the source better?

romming 12y ago
> perhaps the data scientist needs to be involved in the data wrangling in order to understand the source better? Founder of an ETL startup here. This is exactly what we believe: the end-user of the data should be involved as early in the data pipeline as possible, including the wrangling. If you eliminate the engineer from the ETL process you remove a lot of painful back-and-forth and get more flexible pipelines.
- joncooper 12y ago
  I'd love to hear about your startup and/or beta test.
  - romming 12y ago
    Hit us up at info at etleap dot com.
lmm 12y ago
There are places that do that; at my first job we knew that the ETL code was core business functionality, and version-controlled it appropriately. And we knew that our ability to store raw data reliably was a competitive advantage. But in many cases it's a technical debt issue. In the early startup stages when you're struggling for product-market fit, cleaning or auditing your data is not a good use of time. When a business like that starts to get traction, it finds itself with clients that now want more reliability, scaling issues, and a bunch of data that no-one really remembers where it all came from or how it was generated. A lot of the big-data startups are hitting that stage now.