4 ms·

Speaking for the data janitors of the world, I can't believe there wasn't a mention of hiring some of us to do this so-called dirty work, if for nothing else th

Speaking for the data janitors of the world, I can't believe there wasn't a mention of hiring some of us to do this so-called dirty work, if for nothing else than to save these data scientists from using the word sexy. After twenty years of financial operations, including a decade in the back-office of a top hedge fund, I eventually accepted that I'm a bit backwards for my desire not to jump straight to a pivot table when encountering a new data set. Like a farmer reaching down to touch the soil, my first step is exploring the rows and columns with little tests here and there to find weaknesses within the information at hand rather than paving over them with instantly flawed reporting. Anyway, for all the big data scientists out there too sexy to clean up their own data, I'm looking for work and don't mind pushing a broom. Check my profile for more info.

deathanatos 12y ago
> Like a farmer reaching down to touch the soil, my first step is exploring the rows and columns with little tests here and there to find weaknesses within the information at hand rather than paving over them with instantly flawed reporting. This needs to be done by more people, more often. I've worked with more than one (often SQL, but it doesn't matter¹) database whereby someone will claim that X is always true. It X is something that can be put into a DB query, then you should do that, and run the query. Every time. In my experience, if it isn't enforced by constraints in the database software, then it isn't true. ¹The nice thing about SQL is that you can but constraints on the data, and use the database to keep various invariants invariant. Often, you'll find people making assumptions about the data based on the business logic, not the constraints in the database. Such thinking is flawed: you cannot reason about what you think the data is, you must reason from what the data actually is. Worse, you'll find people making assumptions about what they think the business logic is.
Terr_ 12y ago
Reminds me of http://en.wikipedia.org/wiki/Anscombe's_quartet http://en.wikipedia.org/wiki/Anscombe's_quartet
crdb 12y ago
We created a "Data Engineer" job (inspired by Amazon's name for the position) for precisely that purpose. The core schema behind Zalora (we sell clothes online) had over 5,000 tables and well, some of the stuff could definitely be improved. There were so many upstream issues we decided to make fixing them full time work (working with both the developers, and the end users like the people buying the products to put on the site). I can't comment too much on it publicly but well, it looks like it was worth it. I definitely agree that there is a split between statisticians, and "data engineers" in both personality and interests, and you should have both on your "data science"/"big data"/whatever they call analysts these days team. It is however bloody hard to convince upstairs to pay a lot of money for the work! (as with anything "without deliverables", really)

deathanatos 12y ago

> Like a farmer reaching down to touch the soil, my first step is exploring the rows and columns with little tests here and there to find weaknesses within the information at hand rather than paving over them with instantly flawed reporting. This needs to be done by more people, more often. I've worked with more than one (often SQL, but it doesn't matter¹) database whereby someone will claim that X is always true. It X is something that can be put into a DB query, then you should do that, and run the query. Every time. In my experience, if it isn't enforced by constraints in the database software, then it isn't true. ¹The nice thing about SQL is that you can but constraints on the data, and use the database to keep various invariants invariant. Often, you'll find people making assumptions about the data based on the business logic, not the constraints in the database. Such thinking is flawed: you cannot reason about what you think the data is, you must reason from what the data actually is. Worse, you'll find people making assumptions about what they think the business logic is.

Terr_ 12y ago

Reminds me of http://en.wikipedia.org/wiki/Anscombe's_quartet http://en.wikipedia.org/wiki/Anscombe's_quartet

crdb 12y ago

We created a "Data Engineer" job (inspired by Amazon's name for the position) for precisely that purpose. The core schema behind Zalora (we sell clothes online) had over 5,000 tables and well, some of the stuff could definitely be improved. There were so many upstream issues we decided to make fixing them full time work (working with both the developers, and the end users like the people buying the products to put on the site). I can't comment too much on it publicly but well, it looks like it was worth it. I definitely agree that there is a split between statisticians, and "data engineers" in both personality and interests, and you should have both on your "data science"/"big data"/whatever they call analysts these days team. It is however bloody hard to convince upstairs to pay a lot of money for the work! (as with anything "without deliverables", really)