4 ms·

As a former data scientist I feel we try solve problems the hard way because one of two reasons (maybe both): 1. To feel smart 2. To justify our paycheck Most

As a former data scientist I feel we try solve problems the hard way because one of two reasons (maybe both): 1. To feel smart 2. To justify our paycheck Most of the time simple solutions like the one in the link will be more than enough but we just can't resist the urge to implement this new paper we just found. I remember thinking about using a NN for a problem we had, after looking it closely for two days all I needed was a simple linear regression and got the extra benefits of being able to explain what was going on. Eventually I started looking for the "elegant" solution because now that was the thing that made me feel smart. Although some times using brute force is the best approach. There's a balance you find once you gain enough experience... I think.

GarethGonzales 6y ago
I agree, but as a data scientist myself I've always known that part of my job is to choose the right tool out of the box. If that tool is a linear fit, or heck even adding a few columns in a spreadsheet, it's my job to realise that and choose appropriately. If it's a complex custom-built NN, then as long as the cost-benefit analysis justifies the build time, I'll choose that. Of course, as you mention, there's always the business-political aspects - of explaining or justifying your choice to people who don't understand any of those tools, and who often want to pretend that they're part of a "smart data-driven AI" company.
spratzt 6y ago
Genuinely curious as to what you moved into after working as a data scientist. I'm a data scientist and desperate to get out.
- Flashtoo 6y ago
  As a data scientist at the start of my career: why?
  - st1x7 6y ago
    The list is so long we might as well start another thread for it.
  - ccortes 6y ago
    For me it was unrealistic expectations. Tons of companies hire data scientists just because everyone else is doing it and/or they think somehow they'll make everything better just by being in the company. If you are lucky enough to be in a truly "data driven" startup then you'll most likely have fun a learn a lot.
- ccortes 6y ago
  I founded my own company after I had a problem that solved from myself and thought I could actually scale it. Just starting tho.
twelfthnight 6y ago
I've seen this pattern frequently on a few data science teams. However, I do think there is a lot of value to data science, but it more has to do with incorporating metrics and accountability rather than modeling. For example, I've seen many mission-critical rules-based systems with no accuracy metrics. A data scientist is asked to beat this system using machine learning and fails miserably because the rules were developed by experts over years and the data scientist has a month and is only a few years out of grad school. However, in the process of building the model that data scientist built data pipelines for measuring the rules-based system's accuracy and fixed a small number of major data integrity issues. Unfortunately this isn't considered a win and the data scientist usually leaves the company soon after. :shrug:
noobermin 6y ago
> 1. To feel smart 2. To justify our paycheck Finally someone on HN is being honest. It's also a good touch that this is downvoted somewhat while replies getting mad at essentially choice of words (of the title, nonetheless) are on top. Doing the elegant solution was the first thing I learned in grad school, it was literally a comment a professor made. In undergraduate, you feel smart for working out a long calculation. In grad school and beyond, you should feel smart by doing the least math possible and using intuition. It's not that long calculations aren't necessary, it's just that thinking a little first is important rather than just turning the hard work crank.

GarethGonzales 6y ago

I agree, but as a data scientist myself I've always known that part of my job is to choose the right tool out of the box. If that tool is a linear fit, or heck even adding a few columns in a spreadsheet, it's my job to realise that and choose appropriately. If it's a complex custom-built NN, then as long as the cost-benefit analysis justifies the build time, I'll choose that. Of course, as you mention, there's always the business-political aspects - of explaining or justifying your choice to people who don't understand any of those tools, and who often want to pretend that they're part of a "smart data-driven AI" company.

spratzt 6y ago

Genuinely curious as to what you moved into after working as a data scientist. I'm a data scientist and desperate to get out.

Flashtoo 6y ago
As a data scientist at the start of my career: why?
- st1x7 6y ago
  The list is so long we might as well start another thread for it.
- ccortes 6y ago
  For me it was unrealistic expectations. Tons of companies hire data scientists just because everyone else is doing it and/or they think somehow they'll make everything better just by being in the company. If you are lucky enough to be in a truly "data driven" startup then you'll most likely have fun a learn a lot.
ccortes 6y ago
I founded my own company after I had a problem that solved from myself and thought I could actually scale it. Just starting tho.

twelfthnight 6y ago

I've seen this pattern frequently on a few data science teams. However, I do think there is a lot of value to data science, but it more has to do with incorporating metrics and accountability rather than modeling. For example, I've seen many mission-critical rules-based systems with no accuracy metrics. A data scientist is asked to beat this system using machine learning and fails miserably because the rules were developed by experts over years and the data scientist has a month and is only a few years out of grad school. However, in the process of building the model that data scientist built data pipelines for measuring the rules-based system's accuracy and fixed a small number of major data integrity issues. Unfortunately this isn't considered a win and the data scientist usually leaves the company soon after. :shrug:

noobermin 6y ago

> 1. To feel smart 2. To justify our paycheck Finally someone on HN is being honest. It's also a good touch that this is downvoted somewhat while replies getting mad at essentially choice of words (of the title, nonetheless) are on top. Doing the elegant solution was the first thing I learned in grad school, it was literally a comment a professor made. In undergraduate, you feel smart for working out a long calculation. In grad school and beyond, you should feel smart by doing the least math possible and using intuition. It's not that long calculations aren't necessary, it's just that thinking a little first is important rather than just turning the hard work crank.