3 ms·

In real life, problems are hard and require theoretical investigation to do well. That's the actual job! The theoretical investigation. Not the programming. Th

• PunchTornado 8y ago
  I would say the non-programming part, the theoretical investigation is really easy. Plus the majority of data science do data cleaning 80% of their time.
  • make3 8y ago
    If you're a programmer by formation, then indeed you will be given a programming intensive, theoretically non-intensive task (or be found to be qualified for such a job). If you are more theoretically inclined, unsurprisingly, you end up with a more theoretically heavy job.
• maxxxxx 8y ago
  How many data scientists really do theoretical investigation? From what I see not too many. It feels a little like looking only for devs who can write complete operating systems and then having them write CRUD apps.
  • gaius 8y ago
    90% of data science time is spent sourcing and then cleaning the data. 10% is spent preparing and delivering reports to stakeholders. The ML time is a rounding error. The job in which you are magically given a pristine dataset and can just tweak ML models all day doesn’t exist.
    • Puer 8y ago
      The running assumption on HN seems to be that ML/data science jobs are easy because you can just "plug in" already implemented algorithms and just expect things to work. I think the barrier for most of these jobs is a proper statistics education--one that involves active exploration of real world data that's imperfect and has to be cleaned. Very rarely is there an obvious solution to big data problems and much of the work is tedious parameter tuning that absolutely does require knowledge of the math and domain and not just how to write k-Means.
      • gaius 8y ago
        I think the barrier for most of these jobs is a proper statistics education--one that involves active exploration of real world data that's imperfect and has to be cleaned. It's a bit of both really. If the data science job is in the field of predictive maintenance for example, then a (relatively) simple model may be sufficient to add business value straight away, and the hard part is a deep understanding of the potential failure modes of the machinery you're predicting on and the kinds of sensors used to gather the raw data. There's no "one size fits all". This is one of my favourite papers on the subject, written by one of the lecturers on the DS course I did: https://users.cs.duke.edu/~cynthia/docs/RudinETAL2010.pdf https://users.cs.duke.edu/~cynthia/docs/RudinETAL2010.pdf
    • deleted 8y ago
      [deleted]
    • mmsimanga 8y ago
      This 1000 times. I work in data warehousing and Business Intelligence. The closest I have gotten to data scientists are actuaries. Most who don't have a clue about data management, the process to get your data to be consistent and clean so you can do your analysis. Instead of creating standardised processes and tables, each analysis is started from scratch leading to maintenance nightmare and inability to cross-reference the results with any other data sets. Whoever is running data scientist units needs to realise you need IT/DBA/programming people in the mix. Statisticians in my experience cannot design databases, concepts of code reuse and normalisation do not exist in their vocabulary. A great deal of time is wasted doing repetitive tasks that someone with a programming background would have assisted.
    • make3 8y ago
      If you're a programmer by formation, then indeed you will be given a programming intensive, theoretically non-intensive task (or be found to be qualified for such a job). If you are more theoretically inclined, unsurprisingly, you end up with a more theoretically heavy job.
  • superhuzza 8y ago
    That's basically the job - people who can get, transform and interpret the data correctly.