6 ms·

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out int

• TXV 10y ago
  I think what you say is easily applicable to software engineering in general. Data science maybe is a field that is even more negatively impacted by bad hires because the threshold after which you start adding value to the company is higher.
• pcsanwald 10y ago
  I've only been hiring DS folks since 2012, but my experience matches what you've said exactly. The biggest differentiator I've seen is to be able to participate in actually building production quality systems vs being proficient enough in R or python to hack together a prototype on a very small dataset. The former kind of data scientists were very successful at our company, the latter, not so much. Both categories I described usually had a STEM type PhD.
  • user5994461 10y ago
    That sounds weird to me. Does American PhD don't have to work for a few years at real companies as part of the PhD curriculum?
    • ylem 10y ago
      I think the scenario they are describing is for say math/physics PhDs who are transitioning to data science. During their degrees, they concentrate on research, so they don't (on average) have real software engineering experience.
    • sigstoat 10y ago
      no. it's even a stereotype that they've never worked at a company. where is it required that PhDs work at companies to receive a degree?
      • user5994461 10y ago
        France. There are various internships dues during the PhD and the final paper is done while working 1-3 years at one place. Basically, everyone has experience when they graduate from a master or a PhD there. Not all experiences are born equal. e.g. Some work is at top tech companies, some work is at universities or low quality research centers with little work & expectations.
    • gumby 10y ago
      I am used to the PhD being an academic, not practice (trade) degree. This is true even in fields where the PhD is pretty much required for work in industry (medicinal chemistry, biology etc). And even in computer science where a PhD can even be a hindrance to getting an industry job, the PhD is pretty much university only. I am curious what countries it works the other way around.
• rafinha 10y ago
  Isn't it 2 profiles of engineers? Those who make production code and those who work in prototypes? One thing is to understand why your model isn't converging, another is how to scale it up...
  • audleman 10y ago
    A machine learning model is core to my company, and core to that is pulling in, cleaning, and working with large dataset from a variety of platforms. This means: - excellence in being able to clean data at the column level for millions of data points - knowing how to work with such large scale data in a time efficient manner. One of our newer hires worked at the Postgres/SSD level to optimize and got it to where he could produce a full set within 5 minutes. Before that it was once every few months. Being able to do these things is a prerequisite to building even a prototype of a model, and it requires substantial real-world programming experience to deal with those complexities.
  • user5994461 10y ago
    2 profiles: - Research departments that make prototypes that are consistently abandoned and never get any real world usage. - Actual companies that make prototypes for real business cases that are then shipped to production, maintained and improved as they bring in $$$. The first experience is of limited value to the world of real business.
  • elastic_church 10y ago
    the data science people I've worked with in multiple VC backed companies were not engineers at all. They were either "data scientists" or had engineer tacked on to the name. I'm continually exposed to new kinds of software engineering roles I never heard of at tech companies. (fwiw software engineer is sometimes still considered an inflated title.)
• hellogoodbyeeee 10y ago
  Do you have advice for someone in the second camp? I have a master's in econ and I'm comfortable in R but I don't have good development skills. I'm wondering if I need to give up on data science jobs for a bit and try and find an entry level software development job
  • anton_tarasenko 10y ago
    Target research-intensive jobs. Job descriptions are clear about what sort of machine learning professionals the company is looking for. In most cases, they want data engineers and developers who can make big, scalable production. Not researchers. Just find who's looking for economists, specifically. Consulting firms and government agencies hire economists, but they don't call them data scientists. Investment funds may be interested, though hedge funds prefer math and physics graduates because they are easier to train.
  • user5994461 10y ago
    Take whatever job you can... as far as it's related to the kind of things you want to do. Forget about the big companies that everyone knows, forget about the high pay that appear in the newspapers. It's not real, it's not for you. Get an entry or junior level position and get experience. There are many unknown places that will take people only because they're cheap. That's where everyone started.
    • hellogoodbyeeee 10y ago
      Thanks for your comment. It has eased some of the anxiety caused by reading through this thread. My current job fits your description. I currently have an analyst role and I spend maybe 25% of my time building predictive models in R. I just haven't had any success finding a role that ups that percentage (and the pay). I know I need to be more patient, but it is difficult.
    • nether 10y ago
      I'm looking to go through a DS boot camp this summer, have a master's in aerospace engineering. I'd be very content with mid-five figures (40% lower than what I made as an engineer) at a no-name company if it's in the city that I want (not SF). Are such lower-end data analyst positions in decent supply?
• shas3 10y ago
  Frankly, MOOC-type machine learning- being able to do plain vanilla logistic regression or black-box deep learning techniques is not enough to get a job. This knowledge and experience has to be paired with one or more strengths: excellence in programming, grad-level math/stat/numerical skills/theoretical machine learning, domain specific expertise or experience (e.g. vision, audio, natural language, networks, geophysics, biomed, finance, etc.), proven ability to learn and adapt very fast.
  • omginternets 10y ago
    > vision Could you elaborate on which elements of vision science are relevant? Do you have any examples of what these people end up working on?
    • shas3 10y ago
      I loosely used the term to refer to computer vision, which overlaps a bit with vision science (especially color, perception, etc.). But yeah, computer vision.
  • czep 10y ago
    > plain vanilla logistic regression Eh? It's not as easy as it sounds. Search for "mlelr" for a rather detailed illustration of how to code LR in C with only standard libraries. Now that was some fun to put together!
    • shas3 10y ago
      But that does mean that you have pretty good C programming skills! I think your comment reinforces my claim about pairing MOOC-depth ML with other skills. (Edit: I'm not saying MOOC-depth derisively. Serious learners and autodidacts can go pretty deep with just MOOCs for guidance. I just mean prima facie, based on content of some of the lighter and more popular MOOCs like Andrew Ng's. Abu-Mostafa's on EdX is meatier.)
      • czep 10y ago
• cardine 10y ago
  Even outside Silicon Valley this is very similar. When I post a programming job I get more applicants from data scientists than from actual programmers. When I post a marketing job I get more applicants from data scientists than from actual marketers. It is clear to me that supply has far outpaced demand.
  • dnautics 10y ago
    I'm getting burned because I have a STEM PhD and am a competent programmer (have been programming on the side for decades) and am looking for a not-necessarly-data-science job, but it seems like all the places I'm applying to want to bin me in the "data-science" category.
• huac 10y ago
  How do you test for programming ability in an interview? Especially for people without a lot of work experience (e.g. graduating from college) One data engineering role asked me to implement k-means from scratch and one data science role asked me to do some algorithms whiteboarding. But beyond this, people just asked SQL. From the rest of this thread it doesn't look like people would consider that to be 'SWE ability'.
  • nicoburns 10y ago
    Set someone the interviewee a programming task. This is pretty standard for developer jobs, and is absolutely fantastic for people without work experience (as it gives them another way to probe their skill).
    • huac 10y ago
      I mean, I did a bunch of take-home tests. But those were focused on doing an analysis on some dataset, focused on answering a question. Should data scientists be expected to do more than that? Or are you expecting a certain level of code quality from them? Within a 48 hour time window?
• LeanderK 10y ago
  I am currently studying Bachelor CS an am interested in ML/DS. How can i make sure i fall in the first group? I try to do some side-projects to expand my skill set and practice software dev. skills, but this whole thread seems pretty discouraging.
  • RA_Fisher 10y ago
    Given you're strong in CS, study as much statistical theory as possible. That'd be my advice. You need to be a great programmer, but statistics is what's really in short supply.
  • aub3bhat 10y ago
    Depending on where you are studying you can take ML courses. However the easiest way is to get a Master's degree in CS from Stanford, GA Tech, Cornell, CMU, etc. Few Universities offer a Data Science Masters degree e.g. I think NYU, Columbia & Harvard. You can also try joining a comapny and then starting the degree program a year later. As far as this thread, don't worry too much about it.
  • jps359 10y ago
    If your perspective of ML was based solely on the comments in this thread you'd think there's 0 opportunity in the field -- obviously not true. The people who comment on these kinds of posts tend to always be incredibly cynical, and I am not sure why. They offer a limited perspective on the industry and you should not let it discourage you from your desire to learn and become an ML engineer. If I were an undergraduate CS student interested in ML/DS I would start by taking an intro course for ML and creating a side project you're proud of.
• drew 10y ago
  The problem is DS is really 2-3 different disciplines under one nebulous title. What you're describing is folks who are prototyping and productionizing models. That's definitely in short supply, but random STEM PhDs are in no way competitive for those roles unless they're coming from CS programs + have work experience in production engineering. But that's by no means all of the DS field. There are lots of DS jobs where you're collecting and interpreting and communicating about complex data sets. An engineering mindset is occasionally helpful, but a bias towards building versus towards analyzing and writing can just as often be counter-productive. Not all problems are solved by systems; lots of problems are solved by better understanding the problem and then letting other specialists build the right solution. The bootcamps have contributed to the problem by focusing so much on building things. The idea that you can go from an econ undergrad to being a self-sufficient member of a production ML team in 6-12 weeks is nuts. What's less nuts (and what I wish programs like Insight focused on) is taking people from having data skills in one domain and with one set of tools (e.g. logitudinal medical record data, stored in CSVs and handled in Stata) to another set of tools (billions of rows of event-based product data stored in a data warehouse, processed in R or Python). But instead the bootcamps behave like the missing skillset is the ability to make a predictive random forest model on some arbitrary data set and build an AWS web app around it. THAT job market definitely doesn't exist and is completely over-saturated. But people who are smart communicators about data, can manipulate and make sense of massive data sets, can ask incisive questions about their data, and can use data to convince people of a complex argument are always going to have job opportunities, even if they're not production grade engineers. If that sounds like you, I'm hiring - hit me up on Twitter: @drewwww.
  • ma2rten 10y ago
    I was part of the Data Science team in my previous company. We mainly build models for production, but we also were responsible for generating both daily and ad hoc reports. We tried to hire someone to take over the reporting part, but we found out even that requires engineering skills. This role ended up to even more difficult to hire for because it's hard to find someone who has the engineering skills but wants to work only on reporting. Maybe if we had a dedicated data engineering team the story would be different.
• hiddencost 10y ago
  This is correct. I've worked in industry for about 3 years, much in a top industry lab that built a product you probably use. I've got a couple papers, including a really good one. I've worked as a software engineer. I hopped jobs twice in a relatively short frame, with a 20% and 40% raise each time. The market is on fire if you have software engineering competence and real industrial experience.
• wakkaflokka 10y ago
  I'm coming from a PhD in STEM where I did a lot of application of basic ML to my (neuroscience) research, and it took me a good while to get a data science position. But once I got the job, I have been inundated with interview requests, both from recruiters and from specific companies, as in multiple a week (granted most are blast-em-all style recruiters, that I'm sure anybody with any tech skills get). Maybe it's because I'm on the East Coast? I do feel like everybody in neuro is jumping on the bandwagon, and that generates a bit of "you don't belong here" feeling from the CS- or math- educated crowd. But over time I think this will all level-out. I'm not coming from a CS background and don't purport to know absolutely all of the details of all the mathematics and theory behind many of the machine learning algorithms that I use. I try my best daily to expand my knowledge, understand the algorithms, and apply them appropriately. I would hope that any company who is looking for someone who has a PhD-in-CS-or-ML could weed someone with lesser knowledge out during the interview process. With that being said, a couple lines of code using SciKit Learn and all the default parameters is enough to impress many non-tech companies that are looking for a way to use 'predictive' in their marketing materials. And they pay very well for it. I get the feeling that provokes the ire of people who think those types of basic implementations belong to the traditional label of 'data analyst'. For what it's worth, I work with data sets that aren't quite large enough to justify anything more than Python, Pandas, SKLearn, Luigi pipeline, and PySpark. The vast majority of my time is spent cleaning the data and generating features, must less on the hyperparameter tuning, model training side itself. Anyways, I think I'm rambling a bit. I just want to say that I LOVE this job, whatever the label is, or whatever the hype surrounding the label is, and I hope it's around for a while before it's automated...
  • rcpt 10y ago
    > a couple lines of code using SciKit Learn and all the default parameters is enough Except for when it's not - those are the cases you're hired for.
    • deleted 10y ago
      [deleted]
    • apohn 10y ago
• datascientist 10y ago
  A few months ago Mikio Braun wrote a great post on this topic (the need for data scientists with strong software development skills) https://www.oreilly.com/ideas/what-is-hardcore-data-science-in-practice https://www.oreilly.com/ideas/what-is-hardcore-data-science-...
• bjlkwpZxAygCIg 10y ago
  There's always room for smart, hard working and creative thinkers in any field. Except maybe marine biology. But as in any lucrative field, e.g. law, you have a wide variety of capabilities and work quality. After 10 years in quantititative work (I got an MS in Applied Math before they were calling Statistics "Data Science") I can say that it's just a tough field to work in. Experience goes further than education, but education (yes, a real accredited graduate level education) is necessary. The most challenging aspect of DS isn't the technical aspects, it's being able to have a thick enough skin to not let the skeptical and reluctant engineering types upon who you depend to implement your brilliant models get under your skin and enough patience to explain and convince the skeptical and not too savvy business types who cut your checks, that their intuition is wrong and your math is right. And then of course, there's the inevitable boredom that comes with solving yet another mundane business problem with the simplest and least sexy tools. I'm not complaining, just saying that almost all STEM graduates turned DS I've ever worked with have a small hollow spot in their soul that burns with passion for the astrophysics or theoretical math problems they traded in for the perk filled corporate life. 3 easy steps to get a job in DS if you want them though: Grad Degree in Math/Stats/CompSci; work on a bunch of hard to predict problems and then publish and present them to your local meetup community to gain experience; learn engineering tools and devops and be about 90% as good a software engineer as your team's actual engineers (git, hg, IDEs, java, pig)... your brilliant models are way less important than being able to help the already overwhelmed engineering team make them work.