4 ms·

According to this, I'm a data scientist. I've done and do everything on that list except for "put python in production" and "Scaling sharing of Jupyter noteboo

According to this, I'm a data scientist. I've done and do everything on that list except for "put python in production" and "Scaling sharing of Jupyter notebooks". I've put R in production (albeit not my code, I am responsible for making sure it runs correctly and surfacing errors to the system/developer of that code). I maintain a data lake, multiple SQL servers, deal with gobs of json, version control my SQL Schemas, vc our data types (admittedly they change quite rarely), etc. etc. But I'm really just a developer who's good at databases and ETL, along with my regular tasks of writing near-realtime background processing systems, web api's, SQL, etc. I think the data science industry seems to have been massively overhyped, and now they want people who can use AI and statistical learning methods and all this other stuff I don't know to do plain old data engineer work. A sad outcome for a discipline that once held so much promise.

gipp 8y ago
"Data science", as originally conceived, is something that requires an existing base of engineering infrastructure and scale (and decision-making culture) that's typically only found in a capital-T Tech Company. But 5 years of HBR think pieces and McKinsey studies later, execs all over think they can just hire a bunch of statisticians, put them on their own team far from engineers, and get intelligent software systems out of it. They, in turn, realize their predicament and set out building their own such infrastructure to the best of their abilities. Sometimes they do okay and squeeze out real value, but they're never going to produce the cutting edge ai and prescriptive analytics that execs think they will.
- kgwgk 8y ago
  As originally conceived by whom?
Bartweiss 8y ago
> I'm really just a developer who's good at databases and ETL On the other side of things, this might be rarer than you think. My experience is that a lot of newish programmers have very little database experience. What they do have is often centered on Mongo or other non-relational stores, used more for persistent storage than as interactive entities. The ability to get info out of a SQL database is pretty standard, obviously. But handling aggregated or joined tables are not entirely standard. (Interviewing for an entry-level backend dev job at a major company, I was pretty startled to have the databases section cap out at 'group by' and 'join'.) And anticipating error sources (e.g. MySQL's rollup handling), reading and responding to 'explain' plans, or knowing about backend issues like InnoDB settings is well outside a lot of developers' familiarity. I assume part of this is the heavy focus of bootcamps and some college programs on building web apps, and the optional status of databases classes in many college CS programs. But I could imagine a lot of other factors stopping people from picking it up elsewhere, like the changing divisions among DBA/SysAdmin/DevOps/SRE. So on one end, a data science boom turned out a lot of people with advanced skills in a field with lots of simple work, and at the other there's a gap in developer knowledge which makes it convenient to hire highly-trained people and dump them into roles that are a mix of analyst and DBA work.
- oarabbus_ 8y ago
  > The ability to get info out of a SQL database is pretty standard, obviously. I wouldn't have a job if this were true. And on a related note, I've found software engineers to be generally poor at writing queries (compared to DBAs/Analysts/Data Scientists).
perturbation 8y ago
I think they want data scientist to do plain-old-data-engineer work, but not just plain-old-data-engineer work. Getting / cleaning data is part of the job description, IMHO. You can't be a data scientist and entirely reliant on others to do data cleaning, or you'll programming through Slack messages.
thaumaturgy 8y ago
Not at all, actually. Cathy O'Neil and Rachel Schutt's book, Doing Data Science (http://shop.oreilly.com/product/0636920028529.do http://shop.oreilly.com/product/0636920028529.do) covers this almost immediately. Because the term developed sort of organically as a cross-discipline approach to solving certain challenging problems, students in their classes were often a mix of scientists, statisticians, and software/database engineers like yourself. So, you may not consider yourself "a data scientist", and that's fine, but there's certainly a role for your specialization in data science, and that doesn't at all indicate trouble in the field. On the contrary, it's exciting that there's a marriage of these specializations underway. If I had the time, I'd write a much more in-depth reply along the same lines to many of the criticisms in the article. The lack of a clear definition for "data science" or "data scientist" has caused some confusion, but at the same time, there is new technology available and new approaches to working with large-scale datasets that weren't available before, and that does represent new skillsets.

gipp 8y ago

"Data science", as originally conceived, is something that requires an existing base of engineering infrastructure and scale (and decision-making culture) that's typically only found in a capital-T Tech Company. But 5 years of HBR think pieces and McKinsey studies later, execs all over think they can just hire a bunch of statisticians, put them on their own team far from engineers, and get intelligent software systems out of it. They, in turn, realize their predicament and set out building their own such infrastructure to the best of their abilities. Sometimes they do okay and squeeze out real value, but they're never going to produce the cutting edge ai and prescriptive analytics that execs think they will.

kgwgk 8y ago
As originally conceived by whom?

Bartweiss 8y ago

> I'm really just a developer who's good at databases and ETL On the other side of things, this might be rarer than you think. My experience is that a lot of newish programmers have very little database experience. What they do have is often centered on Mongo or other non-relational stores, used more for persistent storage than as interactive entities. The ability to get info out of a SQL database is pretty standard, obviously. But handling aggregated or joined tables are not entirely standard. (Interviewing for an entry-level backend dev job at a major company, I was pretty startled to have the databases section cap out at 'group by' and 'join'.) And anticipating error sources (e.g. MySQL's rollup handling), reading and responding to 'explain' plans, or knowing about backend issues like InnoDB settings is well outside a lot of developers' familiarity. I assume part of this is the heavy focus of bootcamps and some college programs on building web apps, and the optional status of databases classes in many college CS programs. But I could imagine a lot of other factors stopping people from picking it up elsewhere, like the changing divisions among DBA/SysAdmin/DevOps/SRE. So on one end, a data science boom turned out a lot of people with advanced skills in a field with lots of simple work, and at the other there's a gap in developer knowledge which makes it convenient to hire highly-trained people and dump them into roles that are a mix of analyst and DBA work.

oarabbus_ 8y ago
> The ability to get info out of a SQL database is pretty standard, obviously. I wouldn't have a job if this were true. And on a related note, I've found software engineers to be generally poor at writing queries (compared to DBAs/Analysts/Data Scientists).

perturbation 8y ago

I think they want data scientist to do plain-old-data-engineer work, but not just plain-old-data-engineer work. Getting / cleaning data is part of the job description, IMHO. You can't be a data scientist and entirely reliant on others to do data cleaning, or you'll programming through Slack messages.

thaumaturgy 8y ago

Not at all, actually. Cathy O'Neil and Rachel Schutt's book, Doing Data Science (http://shop.oreilly.com/product/0636920028529.do http://shop.oreilly.com/product/0636920028529.do) covers this almost immediately. Because the term developed sort of organically as a cross-discipline approach to solving certain challenging problems, students in their classes were often a mix of scientists, statisticians, and software/database engineers like yourself. So, you may not consider yourself "a data scientist", and that's fine, but there's certainly a role for your specialization in data science, and that doesn't at all indicate trouble in the field. On the contrary, it's exciting that there's a marriage of these specializations underway. If I had the time, I'd write a much more in-depth reply along the same lines to many of the criticisms in the article. The lack of a clear definition for "data science" or "data scientist" has caused some confusion, but at the same time, there is new technology available and new approaches to working with large-scale datasets that weren't available before, and that does represent new skillsets.