3 ms·

I agree with the beginning of the article, which describes the present state pretty well, the part about "better engineers than statisticians and better statist

I agree with the beginning of the article, which describes the present state pretty well, the part about "better engineers than statisticians and better statisticians than engineers", etc. But then I disagree with the rest. The distinction between "Data Scientists" and "Engineers" is bogus, and the point about whether your data is "Big" is a red herring. In reality, there should not be any distinctions between "scientists" and "engineers", you must strive to be both a "doer" and a "thinker". You can't think without doing, and can't do without thinking. If you're in this field, and consider yourself an "engineer" but your math sucks, go read up on all you can about mathematics and statistics, just like you did back when you were learning about programming, operating systems and networking. If you consider yourself a "data scientist" but don't know anything other than R and basic Python, go study programming and operating systems and networking, like you studied math at some point. Somewhere on youtube I remember Dr. Donald Knuth (who is definitely an excellent programmer/engineer/computer scientist, arguably one of the best the world has known) saying that he considers himself primarily a mathematician. Or, if you've read (or at least heard of) "the dragon book", you might find it interesting and inspiring that one of its main authors Dr. Jeffrey Ullman (whom I'd place in the same league as Knuth) went on to write another excellent (and available freely online, BTW) book "Mining of Massive Datasets", which IMHO is the one fundamental "big data" book out there. So Data Scientists - go learn some programming languages like C and study UNIX and may be read "The Art of Computer Programming" and Engineers go read http://www.mmds.org/ http://www.mmds.org/. Then you'll all get along.

achompas 11y ago
> If you're in this field, and consider yourself an "engineer" but your math sucks, go read up on all you can about mathematics and statistics, just like you did back when you were learning about programming, operating systems and networking. This assumes availability of time. Obviously, given enough time, people could develop both top-tier engineering and DS skillsets! Of course, if lots of free time were common we'd all be full-stack developers who also field sales calls and work on product strategy etc. etc.
- gtrubetskoy 11y ago
  I don't know... Are you saying that someone like Donald Knuth had a lot of free time? Data problems are arguably the hardest problems out there and they require deep understanding of mathematics as well as computers, that's just the way it is. It does take time and effort and even may be a bit of talent as well, not everyone is cut out for it.
  - achompas 11y ago
    No, I'm saying that someone like Donald Knuth is obligated, by the terms of his employment as a premier academic, to sit on the cutting edge of both mathematics and computer science. /u/dxbydt commented on this very well, so I'll only reinforce the point that not everyone is Donald Knuth. If the distinction between DS and Engineer is blurred in your specific instance, and you're capable of Knuth-ian levels of work in both, you are almost certainly underpaid and need to lead a team or start a company yourself, stat, since a top-tier combination of those skills is exceedingly rare.
dxbydt 11y ago
This is a supremely ridiculous set of suggestions that has no merit whatsoever. Companies aren't libraries. They aren't paying you to sit and read books. There is an assigned dayjob, a set of tasks you have on your jira that you have to resolve by your deadlines, and that occupies the 8 hour workday ID you are doing any justice to it. So any reading you do is on the side, on your own time. Furthermore, people have these roles precisely because of their talents and their choices. As a Data Scientist, most of what I do is read ML literature, build ML models and write technical reports in Tex on what worked and what didn't. The skills to do this were acquired over many painful years of graduate work in math, statistics, ML. To suggest somebody can just read their way through that material is quite laudable, but you are underestimating the difficulty by orders of magnitude. Essentially, you are suggesting that all of the graduate study and mentoring and homeworks and assignments and all that went into the learning process be condensed into a book which one can just plow through and become a DS. Well, good luck with that. By the same token, expecting me to have the same level of efficiency and passion as a data engineer when faced with a Hadoop/Oozie/Presto/Pig/kafka or what have you is silly. I don't care for these technologies and how to work them. I know it takes a really long time to get good at them - that's why the engineers get paid a lot of money and also get yelled at when the ETL job fails. Because it's a set of seriously valuable skills that were no doubt acquired over lots of time and practice. It's not like I can buy a book on these things, just read through them and suddenly I am a DE! I neither have the interest nor the time to do that. >>the distinction between data scientists and data engineers is bogus Not at all! Both DS and DE professionals do distinctly different work and conflating everything under 1 umbrella buys you nothing.
- leblancfg 11y ago
  >> Companies aren't libraries. They aren't paying you to sit and read books. That stroke you've brushed is too wide. Smart employers will have some of the money they're paying an employee going towards learning... and if they're really smart, they can even measure their ROI. Leads to less turnover, and better long-term vision for their projects. I get your point, but give someone passionate enough 6 months in a new work environment, and with a decent mentor, and you might find they become surprisingly adept at it. The hard part is hiring for the capability to learn (fast).

achompas 11y ago

> If you're in this field, and consider yourself an "engineer" but your math sucks, go read up on all you can about mathematics and statistics, just like you did back when you were learning about programming, operating systems and networking. This assumes availability of time. Obviously, given enough time, people could develop both top-tier engineering and DS skillsets! Of course, if lots of free time were common we'd all be full-stack developers who also field sales calls and work on product strategy etc. etc.

gtrubetskoy 11y ago
I don't know... Are you saying that someone like Donald Knuth had a lot of free time? Data problems are arguably the hardest problems out there and they require deep understanding of mathematics as well as computers, that's just the way it is. It does take time and effort and even may be a bit of talent as well, not everyone is cut out for it.
- achompas 11y ago
  No, I'm saying that someone like Donald Knuth is obligated, by the terms of his employment as a premier academic, to sit on the cutting edge of both mathematics and computer science. /u/dxbydt commented on this very well, so I'll only reinforce the point that not everyone is Donald Knuth. If the distinction between DS and Engineer is blurred in your specific instance, and you're capable of Knuth-ian levels of work in both, you are almost certainly underpaid and need to lead a team or start a company yourself, stat, since a top-tier combination of those skills is exceedingly rare.

dxbydt 11y ago

This is a supremely ridiculous set of suggestions that has no merit whatsoever. Companies aren't libraries. They aren't paying you to sit and read books. There is an assigned dayjob, a set of tasks you have on your jira that you have to resolve by your deadlines, and that occupies the 8 hour workday ID you are doing any justice to it. So any reading you do is on the side, on your own time. Furthermore, people have these roles precisely because of their talents and their choices. As a Data Scientist, most of what I do is read ML literature, build ML models and write technical reports in Tex on what worked and what didn't. The skills to do this were acquired over many painful years of graduate work in math, statistics, ML. To suggest somebody can just read their way through that material is quite laudable, but you are underestimating the difficulty by orders of magnitude. Essentially, you are suggesting that all of the graduate study and mentoring and homeworks and assignments and all that went into the learning process be condensed into a book which one can just plow through and become a DS. Well, good luck with that. By the same token, expecting me to have the same level of efficiency and passion as a data engineer when faced with a Hadoop/Oozie/Presto/Pig/kafka or what have you is silly. I don't care for these technologies and how to work them. I know it takes a really long time to get good at them - that's why the engineers get paid a lot of money and also get yelled at when the ETL job fails. Because it's a set of seriously valuable skills that were no doubt acquired over lots of time and practice. It's not like I can buy a book on these things, just read through them and suddenly I am a DE! I neither have the interest nor the time to do that. >>the distinction between data scientists and data engineers is bogus Not at all! Both DS and DE professionals do distinctly different work and conflating everything under 1 umbrella buys you nothing.

leblancfg 11y ago
>> Companies aren't libraries. They aren't paying you to sit and read books. That stroke you've brushed is too wide. Smart employers will have some of the money they're paying an employee going towards learning... and if they're really smart, they can even measure their ROI. Leads to less turnover, and better long-term vision for their projects. I get your point, but give someone passionate enough 6 months in a new work environment, and with a decent mentor, and you might find they become surprisingly adept at it. The hard part is hiring for the capability to learn (fast).