45 ms·

Goodbye, data science

• limbojunkie 4y ago
  Working as a data engineer for over 7 years, I can add my two cents into this: Mainstream data scientists don't do data science, they are either ML engineers or data analysts who use ML Python libraries and promoted to data scientists with bigger paycheck. With machine learning and AI being the trend, the job is sexy and it is a chance for the company to market itself as it uses AI and cutting-edge tech. I worked for a data solutions company and while I work together with data scientists to propose a design to our clients, it was the AI part narrated by the data scientists that makes clients ready to throw money. Even if the requirements seem impossible or at least very difficult to achieve. These projects often fail because the data scientists couldn't reach the accuracy goal written in the contract and the project ends up in the trash. These data scientists eventually leave the company or get fired, only to find another job within a short time with even bigger salaries. Now for the data engineering part; I wish the OP all the best with his career, but he is still in the honeymoon period and didn't witness the misery of being a data engineer. - The job can get very repetitive very quickly, unless he'll work on infrastructure, his tasks will be mainly focused on maintaining existing ETL pipelines or building ones, and both are labor tasks. You'll end up a data plumber who makes sure data goes from A to B and then C and that's it. You'll seldom find something new or revolutionary to work on and you'll keep using the same tools as long as you're in the same company. Even when moving to a new job, you'll pretty much be hired because of your knowledge of the same data warehouse or ETL tool you used in the previous job. - Data engineering is an underappreciated job. If things go right, nobody pats you in the shoulder. When shit gets loose, you'll be the one to clean up the mess. What makes it worse is, you can't leave this mess long because data is a snowball effect; if you leave it unprocessed, you'll end up with more data clogging your pipelines and what can be fixed within an hour can quickly take long nights and even days to resolve, and do you know what this means? Managers won't have their fancy dashboards updated and they'll start panicking. - Data engineering is not rewarding. Again, you're doing your job. No one cares. - Data engineering has nothing to show for. Yes, data is being crunched and processed and baked. It's the data analyst who builds the fancy dashboards for managers, and the data scientists who create fancy graphs for managers, and the ML engineers who create fancy products for managers. You're just a plumber who, instead of fixing toilets and sinks, fixes data pipelines. - Just recently, data engineering salaries are rising thanks to low supply and higher demand thanks to better awareness from CTOs and heads of data about the importance of the role, but until a few years ago, they were paid less than a software engineer.
• amai 4y ago
  Having worked as a data engineer before and now working as a data scientist I cannot agree with that article. Data Engineering was really boring. The most complex math I ever used was computing an average value of something. Most of the time your work is only ETL, just data in/data out. The most complex technology I ever used was a search engine. Everyone in IT who likes to do some math and statistics at his workplace, even if it simply a linear regression or some histograms should go for data science. Also instead of nonsense discussions about what is agile and what not, I enjoy talking with my colleagues about the newest papers in ML, even if nobody understands the details.
• LarsDu88 4y ago
  This sounds accurate from my experience. I was a data scientist who moved to engineering. In a lot of orgs data science is there to help decision making, but human nature often makes it such that the decisions are already made.
• jstx1 4y ago
  As someone who also wants to move away from data science, data engineering is the last thing I would want to do. I think DE comes with many of the same problems and it's also a very ill-defined career track; I wouldn't recommend it to anyone. ML engineer or backend developer seem like much more appealing job profiles.
  • aulin 4y ago
    would you highlight some of the biggest differences between ML engineering and data engineering? I believe they're sometimes used interchangebly especially if "data" is "datasets" for ML.
    • jstx1 4y ago
      Data engineers don't work with machine learning at all. In fact one of the reasons why it developed as a job title over time waas specifically to differentiate the people who work with data but don't do any statistics or ML. If a DE who is doing "datasets for ML" decides to call themselves an ML engineer, they're just getting a bit too creative with the job titles (maybe they want a career change, more money, they think it sounds better, all sorts of reasons). As an ML engineer you might need to do some data engineering work as part of your job but not the other way around.
      • omega3 4y ago
        > Data engineers don't work with machine learning at all. This very much depend on the company. From experience DE is used as a catch-all title.
        • jstx1 4y ago
          Data scientist is much more catch-all from what I've seen. But a lot of that varies a lot by geography too (for example in the US people very often use DS very differently from how the title is used in the UK).
          • 4y ago
• newaccount2021 4y ago
• samvher 4y ago
  > Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A common one I've seen quite many times is people using a flawed validation strategy (e.g. one which rewards the model for using data "leaked" from the future), or to rely on in-sample results too much in other ways. Because these issues are subtle, management will often not pick up on them or not be aware that this kind of thing can go wrong. With a short-term focus they also won't really care, because they can still put these results in marketing materials and impress most outsiders as well.
  • CapmCrackaWaka 4y ago
    I've seen this a LOT in my professional group. Many people (who often have PhDs!!) I interview for data science positions seem to know absolutely nothing about the algorithms they use professionally, or how to optimize them, or why they are a good fit for their use case, etc etc etc. I usually see through LinkedIn that these same people are now in impressive-sounding positions at other companies. I had one candidate who was in charge of a multi-armed-bandit project at their current company. I asked them how it worked, and how they settled on that. Their response was "you know, I'm not really sure, the code was set up when I got there". He had been there for over a year, and could tell me nothing! > A common one I've seen quite many times is people using a flawed validation strategy (e.g. one which rewards the model for using data "leaked" from the future), or to rely on in-sample results too much in other ways. It's funny you mention this, we have a direct competitor who does this and advertises flawed metrics to clients. Often times our clients will come back to us saying "XYZ says they can get better performance", the performance in this case being something which is simply impossible without data leakage or some flawed validation strategy.
    • adamsmith143 4y ago
      Where are these jobs where you can interview this badly and still get hired because in my experience DS interviews are extremely hard and often expect people to have very high Stats skills as well as Data Structures/Algo skills at FAANG level.
• frgtpsswrdlame 4y ago
  Read this yesterday and absolutely loved it. Especially feel the pain regarding working with management. I think there's an accountability that comes with evaluating management decisions with data that nobody really wants. I still have a lot of half-formed thoughts/opinions about this but it really feels like data-driven requires strict discipline but the data people who would be accountable for that discipline both 1. aren't empowered to wield it and 2. probably don't want to wield it anyways. Also agree about the simple tools but it's really hard from a career perspective. If I deploy XGBoost in production and put it on my resume, I'm making double my salary next year. If I can find a simple ruleset or linear regression that performs 90%+ as well as the XGBoost and put it in production then nobody cares even though it feels like distilling the complex down to the simple is really where the value is.
  • oneoff786 4y ago
    The thing is that anymore anything short of a deep net is equally difficult to implement as a linear regression, if not easier due to nulls and categoricals. Also, modern tooling makes a lot of these models more than explainable enough for a lot of cases… 10% is a lot
  • disgruntledphd2 4y ago
    Just as an FWIW, I've been interviewing data people for about a decade now, and I would definitely be far more impressed with the simpler approach, but I realise that people like me are a minority in the field.
• rib3ye 4y ago
  The upstream problems are real. Yes, there are a few very meaningful dashboards that are high value to the business, and then there is analysis meant to justify a project. After core dashboards have been built, a lot of data analysis is a political weapon and the data science people are designers of those weapons.
• p0pcult 4y ago
  I've written about this in other comments relevant to ad tech topic, but what the article says about twisting the data to support the pre-made decision (versus making decisions based on data-borne insights) is so true.
• agomez314 4y ago
  "Managers will say they want to make data-driven decisions, but they really want decision-driven data" Ooofff. This is too true. How often is the case that data is collected to test hypotheses vs confirming priors?
  • phkahler 4y ago
    >> "Managers will say they want to make data-driven decisions, but they really want decision-driven data" Ooofff. This is too true. How often is the case that data is collected to test hypotheses vs confirming priors? Find me some evidence of WMDs in Iraq! Yessss Sir!
  • A4ET8a8uTh0 4y ago
    That one stood out to me as well, but, to be fair, this predated current 'fashionable trend' for data driven decisions. It is, sadly, not a new development, but something to still be overcome.
  • Reimersholme 4y ago
    Rather than wanting to confirm priors, I believe this usually is a problem with neither the PM nor the data scientist ensuring that the problem formulation is good enough before diving in. I.e., what data would be needed to actually test the hypothesis? Do we have that data or not? Is the hypothesis even formulated in a way to be falsified in theory? I've seen so many analysis tasks where data scientists without questioning went away for a few weeks to crunch data and come back with some random graphs and statistics that are completely useless as decision support.
    • nisegami 4y ago
      You're overthinking it. Executives and managers quite literally want to see data that confirms their existing convictions and beliefs so they can act on those beliefs under the guise of it being "data-driven".
  • cobbzilla 4y ago
    So true. This article accurately describes DS at many companies. The preceding sentence is a hilariously cynical zinger: “Those who have seen my Twitter posts know that I believe the role of the data scientist in a scenario of insane management is not to provide real, honest consultation, but to launder these insane ideas as having some sort of basis in objective reality even if they don’t.”
• xyzelement 4y ago
  // it was often personally unfulfilling (e.g. tuning a parameter to make the business extra money). He lost me here. Something I've always loved about being an engineer (and now in product) is that something small we do/tweak can have big impact. If you tuned a parameter and that actually had tangible impact on the business, that's like the best case scenario and should be celebrated (vs doing some cool rocket science stuff that ends up unused and doesn't matter)
  • cmmeur01 4y ago
    And all that extra profit is hovered up by the people above you that had nothing to do with it. Validated engineering cost savings should be treated like sales, the engineer gets a percentage.
    • Closi 4y ago
      > Validated engineering cost savings should be treated like sales, the engineer gets a percentage. If you want to really follow the same compensation structure, we would then give engineers a really low base salary and make 80% of their compensation performance dependent. Be careful what you wish for :) Besides - this would drive some strange incentive structures. If you incentivise people based on cloud savings for instance, it will really only be the teams with unnecessarily large cloud spend in the first place that ‘get’ that bonus. If you incentivise on sales, engineers doing great work on back office tools don’t get any cake. Etc.
      • clevergadget 4y ago
        Oh yeah also the pressure on a continuous basis where your value is only ever your last 3 mo average
        • Closi 4y ago
          Yep :) Big performance related bonuses sound great until you realise it also means you are heavily performance managed and most of your livelihood depends on your pipeline and actually closing sales. You are constantly looking off a cliff. Maybe you are on $150k per year today, but in three months time you are back to $45k per year because you didn’t make some minimum sales threshold. Might be fine for some people, but depending on your mortgage…
• jmhammond 4y ago
  > But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I dunno, man. This is true everywhere. As a professor, every semester I’m baffled by students who aren’t curious. But I’ve come to terms that there is a difference between those who will graduate and go on to be readers of hacker news and write this kind of article, and those who won’t.
  • oneoff786 4y ago
    Seems pretentious to me. I’ve never bothered to look through many things I use. I look extensively at how to use them and what the API offers. I have a good intuition for how most models work. I don’t really care about the specifics of the implementations. I have more important things to do. The hacker mentality, imo, is about identifying what’s useful for you to explore to accomplish whatever you need. Often that’s a lot of glue between things that other people built. Other times it’s tweaking the internals to do something a bit different.
    • mark_l_watson 4y ago
      I agree with you. Super powers are seeing value in doing something, and then finding the easiest and most efficient path to get there. That said, sometimes I do like to read the code in libraries I use but often this is more for enjoyment with occasionally learning something interesting.
      • chris_wot 4y ago
        How can you see the value in doing something if you don’t understand the underlying concepts?
        • mark_l_watson 4y ago
• spaniard89277 4y ago
  Correct me if I'm wrong because I'm on the receiving end of such models, but I feel that many times a couple of linear regressions, surveys and qualitative work with customers could land much better results. I say so because I've had time to read some of the reports that DS teams produce to drive decisions in my BIGCORP and it makes very little sense most of the times. And we suffer from it because we have direct contact with clients, but nobody cares about my department opinion, they will rather believe in some model where I can see insane dispersion in datapoints when they plot em in reports, conclussions by people who clearly has zero understanding of our business. I'm forced to make decisions on how to treat certain customers, by entering data into some software and being given an output I can't challenge, which produces lots of insane and unfair situations. Also, IDK how they clean and treat their data, but if they're relying on our ERP's data, good luck. Our CRM if full of BS because most employees rush to put whatever it allows to continue as they need to keep up with KPIs, so they aren't trying to make nice comments and check everything is ok.
  • webnrrd2k 4y ago
    The word "conclussion" is especially appropriate in this context -- it could easily be pronounced like a combination of "conclusion" and "concussion".
  • SkyPuncher 4y ago
    Actual, human based decisions will almost always win out. Data is only helpful when it is directly and clearly tied to the problem. * Good: "Our customers are complaining of random drop-outs. We've noticed X% of requests to Y service take longer than Z time. We believe that's the problem". * Bad: "Companies who are most successful on our platform upload X things in their first Z days. We must find a way for everyone to upload X things in Z days".
• Icathian 4y ago
  This hit all the same high notes I was feeling when I quit Data Science to become a software engineer. It's an infinitely better gig and I encourage all my colleagues with enough chops to make the same switch.
  • sebastiansm 4y ago
    How did you do the transition from DS to SWE?
    • time_to_smile 4y ago
      Be a strong software engineer? It's not hard to extend existing data science duties to include more engineering work. There's always a demand in any company for more strong engineers so it's not hard to find ways you can contribute more seriously to the data engineering/MLE part of your team's work. I've been a data scientist for quite awhile now at many different places, but every time I start interviewing again I always make sure to include a few pure software engineer roles in the positions I'm interviewing for. Even for some pretty elite teams, I'm still able to get to the final rounds but so far have always realized I still personally prefer the DS roles I'm looking at. Any data scientist who wants to keep working on quantitative problems in the future should aim to be a solid software engineer.
    • Icathian 4y ago
      Another comment here mentioned the on the job steps you can take, and that mirrors my experience. I also enrolled in Georgia Tech's OMSCS after a year or so of self study. About 1 year in I took a role using Python for network topology analysis software. I went from there to using go and C to develop a distributed database product. It's been incremental steps lower on the stack and towards more "pure" dev work. I'm now where I wanted to be and will keep doing this kind of work for as long as I can get away with it.
  • _dain_ 4y ago
    yep, exact same feeling here. I had several years as a "data scientist" and it was a an almost totally bullshit job. the org bought into the hype and hired a cohort of us straight out of university, but then couldn't find anything data-science-y for us to actually do. what I actually ended up doing 95% of the time was taping together dodgy excel-based workflows using python scripts. it gave me a visceral appreciation for Conway's Law. the other 5% was when I got to do some genuinely interesting mathematical work, but that wasn't "data science" either, it was more like operations research. I lived for that stuff, but there wasn't enough of it. so I jumped ship and became a software engineer. better pay and more interesting problems.
• Dragony 4y ago
  Many things in this article, especially about the problems with the Data Science role, resonate with me (low value work with low expectations for quality). Funny thing is I have never worked in Data Science. Rather I've worked in Software Development. The summary at the bottom about Data engineering seems like the dream job to me. But I don't think it's because I'm interested in doing Data engineering specifically. I think it's because doing things that actually have an impact day to day is fullfilling. The last job I had totally lost me after ignoring security problems in favor of surface level things like updating CTA labels or similar. Have other Software Devs had this experience?
• boringg 4y ago
  Coles notes: Data Engineer - more money, more clout, less analysis / interesting projects, more job security, more infra style work Data Science - less money, a lot of random projects (Sometimes totally overqualified for), more analytical, don't have as much clout / confusion & lots of people don't actually understand capabilities.
  • zhdc1 4y ago
    Checks my CV: 50k+ lines of R, 10k+ lines of Julia, 5k+ in Python, C, and who knows what else. Most of it for what is, essentially, data engineering work. Where do researchers with social science degrees fall on this scale? Less money, less clout. The projects are certainly interesting though (which is why I do what I do).
• cjf4 4y ago
  One of the things that always sort of annoys me about complaints that "management doesn't listen to data (science)" is the lack of awareness they consist of. It turns out that data work is limited by all the same things every other part of the business is limited by: the need to make quick decisions, institutional imperative, the beliefs of decision makers, the ability to communicate well/influence, and so on. Having better access or skill with data doesn't give you a pass on these things, despite the suggestions otherwise from laments such as this.
• marcinzm 4y ago
  In my experience Data Science is based either on optimizing short term easily measurable KPIs or producing impressive looking BS. So if you're joining a new team and they can't explain in one sentence what they're optimizing for you're probably going to be tasked with producing impressive looking BS.
• college_physics 4y ago
  "Data Science" was always a vague term, purposefully so. Useful mostly as a vendor / consultant battle-cry and hype term to "encourage" a number of new business domains to adopt digitization and automated information processing / decision support. Various older information intensive fields (medicine, insurance, finance etc) knew the benefits and pitfals long ago. These examples show also the survival strategy for the generic "data scientist": specialization. The role of the human in the loop is to blow some context and relevance into an otherwise dead body of data. You can only do that if you really know your domain.
• drgiggles 4y ago
  Unfortunately it seemed pretty clear from the start that this is what data science would turn into. Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result. For research and analysis data scientists must have a strong understanding of underlying statistical theory and at least a decent ability write passable code. With regard to engineering ability, certainly people exists with both skill sets, but its an awfully high bar. It is similar in my field (quant finance), the number of people that understand financial theory, valuation, etc and have the ability to design and implement robust production systems are few and you need to pay them. I don't see data science openings paying anywhere near what you would need to pay a "unicorn", you can't really expect the folks that fill those roles to perform at that level.
  • adamsmith143 4y ago
    On the flip side you used to have statisticians writing code that is frankly unusable in a Production environment. You would weep at the R code I've seen and had to turn into something to actually produce business value.
    • esparrohack 4y ago
      Yeah but in the end it’s just code. And even better, just R. The business value comes from the stats guy.
      • adamsmith143 4y ago
        When the R/stats guy quits and you have to figure out which of his 7 notebooks to run in which order and which local files need to be in which local directories to run correctly and which versions of each package are now broken and which code you need to rewrite to fix it you start to realize the value he produced was clicking a lot of buttons in the right order and that overall this doesn't scale at all.
        • 4y ago
• itissid 4y ago
  I have never understood the what a good ML engineer couldn't do and a Data scientist could in _majority_ situations. When you need a decision to be made based on data its just common sense risk analysis added together with basic statistics. I feel some good field training in statistics(Look up Andrew Gelman) a couple of good courses on Linear, Bayesian Regression is all you need, rest is just engineering skill. The dichotomy between ML Engg and Datascience is as stupid as was between Systems Engg and Application Engg before Devops came along.
  • TrackerFF 4y ago
    IMO a data scientist should also be a domain expert, in the same way analysts are. But of course, too many view DS as some abstract skill where domain knowledge is not needed, and where the methodology will solve all problems / provide insight.
    • _Wintermute 4y ago
      I agree completely, but if a data scientist should be a domain expert, surely we should just focus more on programming and quantitative skills in these fields?
  • adamsmith143 4y ago
    Can you make pretty graphs in Matplotlib and sound smart in front of Executives?
  • time_to_smile 4y ago
    I think the qualifying term here is "good". I've worked with a surprising number of MLEs that don't really understand gradient descent or how most models really work under the hood. They certainly couldn't implement most things from scratch if they needed to (neither could most data scientists). I used to think an MLE was a solid engineer who also had a strong quantitative and numerical computing background. The kind of engineer that always has a copy of Numerical Recipes handy, and if needed, could reimplement core components of statsmodels and sklearn in javascript. I think after this current contraction in tech is over we'll see that most of the remaining "data scientists/MLEs" will be the type of engineer I imagine an MLE to be.
• lajosbacs 4y ago
  I've been a 'data scientist' for years, and I probably will be again at some point as it is the biggest item in my CV. It was in a company, where data science was not the bread and butter, but just something extra to show to the clients. For me therefore, data science is the epitome of Graber's 'bullshit job' -- if the position didn't exist, the company would go on just the same.
• farhanhubble 4y ago
  Not only are bad ideas not dropped, in the name of data science, no one ever does a postmortem of why an experiment failed.
• gillesjacobs 4y ago
  > Managers will say they want to make data-driven decisions, but they really want decision-driven data. Has been my experience as ML engineer too. Decision making being intuition- and not data-driven was one of the largest shocks to me when I went from academia into industry. How upper management and the board determine the course of the company was based more on emotion than anything else.
• starwind 4y ago
  I made this same transition from data science to data engineering about 18 months ago and I've never looked back. I hated working with bad code and dealing with arrogant phds who don't value good code. I've seen so many terrible Jupyter Notebooks just copied and pasted into VS Code and the data scientist just washed their hands of it calling it "production ready." Here's a conversation I've had multiple times: Me: have you ever considered not making every variable global scope Them: that's just software engineering. We do machine learning Me: if it's just software engineering, then why can't you do it? Meanwhile, automated data science tools are getting halfway decent. If you know what algorithm to pick and you don't need to run millions of records through the model every minute, your standard business analyst could probably get a solid model going--at least as well as most data scientists for all the reasons the article mentions. And I like that I know I can do data engineering. With data science you can never really know if you can hit your target metrics given the data you have. So data scientists end up encouraged to fudge their results or make sloppy decisions. With data engineering I can say "yes this is doable or no that's not" and people believe me. My prediction: there's value in the massive volume of data but most of it can be had through standard dashboards, some summary statistics, a graph network, or maybe a linear/logistic regression. Most data science is BS and companies aren't getting the return they need to pay for these guys. (And good God, you almost certainly don't need a neural network.) Meanwhile, data engineering will get integrated into software development, and machine learning—by virtue of its proliferation through academia—will just become another tool for software developers. Data scientists won't get laid off enmass but they will go the way of the webmaster: either pick up new skills and evolve or move on til they end up with new titles
  • srajabi 4y ago
    This resonates with me so much, I stumbled into data science out of University a decade ago. Left it to do SWE and came back to it in the last 3 years. So many data scientists are full of themselves thinking they are magicians and software developers are blacksmiths who are beneath them. Incrementally at my company the SWE's have automated so much of the data scientists workflow that they end up just as you describe, using the tooling and being relegated to becoming analysts. After 3 years coming back to this field, I see the writing on the wall: In the 90's most models were created by software developers, in the 2030's most models will be created by software developers.
• angry_moose 4y ago
  In a recent past life, I was a HPC (high performance computing) administrator for a mid size company (just barely S&P400) who was in the transportation industry, so I had a lot of interactions with the "data science" team and it was just a fascinating delusion to watch. Our CTO did the "Quick, this is the future! I'll be fired if I don't hop on this trend" panic thing and picked up a handful of recent grads and gave them an obscene budget by our company's standard. The main problem they were expected to solve - forecasting future sales - was functionally equivalent to "predict the next 20 years of ~25% of the world economy". Somehow these 4 guys with a handful of GPUs were expected to out-predict the entirety of the financial sector. The amazing part was they knew it was crap. All of their stakeholders knew it was crap. Everyone else who heard about it knew it was crap. But our CTO kept paying them a fortune and giving them more hardware every year with almost no expectation of results or performance. It was a common joke (behind the scenes) that if they actually got it right, we'd shut down our original business and becomes the world's largest bank overnight. At least it finally gave the physics modelers access to some decent GPUs which led to some breakthrough products, as they finally were able to sneak onto some modern hardware.
  • fnands 4y ago
    Honestly, I can't tell you how many jobs ads I saw where I was wondering: "What would they expect me to bring to the table here?" Some companies just don't have the data, or heck even the need, for data scientist yet try and hire them anyway. Give smart people a fundamentally ill-posed problem and they won't get anywhere anyway.
    • SilverBirch 4y ago
      It’s a great skill to walk in to a job and say “hey I’m the expert, that’s not a reasonable proposal, here’s the problem we can solve and here’s what we’ll do”. Much more value to the company, but hard to do.
      • fnands 4y ago
        Yeah I feel a lot of companies could do with running their problems past a consultant first. Also, w.r.t hiring in cases like these, I think often the experienced candidates can smell that this won't be a good gig so don't apply, while the less experienced (or desperate) ones apply. This means the workers get stuck with an intractable problem, and the company gets stuck with workers who are too inexperienced to know better.
• fly_b0y 4y ago
  i am just reading this and thinking "but why?"... there are different flavors of DS, there are people who are doing diffusion models, doing top stuff that may or may not yield anything, but they are doing it because they know math well and enough code to put new maths stuff into new products. deep knowledge. so called ML engineers. maybe they are even good at coding at the lowest level, but in their point of view, why? these people are at huge companies that have vast resources downstream, they make people like OP work their work actually.. there is T shaped folks (unicorns, everybody wants one even if politically not ready [most arent]), where they know some concepts of many topics, perhaps so called full stack DS, which i consider myself to be... and i wouldn't be able to read thru most scientific papers, but I can put stuff together from start to finish including deploying it as an API that's scalable to top performance because of cloud. i do go back to basics often and I think its only natural! i think its like being a pilot, why not check the basics that actually, if forgotten, will take everything down lol... and you will use that the most as well! i think also many people who are too much into one thing, math, code, whatever it is, start to call non basic things that are basic to them, well --- basic... BUT THERE IS NOTHING BASIC about multi linear reg and how to set it up all proper and how humanity spent thousands of years getting to this point.. there is also DS thats like data analyst on steroids, knowing middle basic and middle tier algos and stats well and can deliver mad value with a bit of business knowledge. hell, they could even use excel for their stuff, but proper understanding of the question at hand will most likely allow you to downgrade to lower, simpler tools. and simple is awesome! people often misinterpret complicated for advanced, not the case whatsoever. once you know the land you accept your weak points and strong points and points you need to know enough to put stuff together. at the end if you know how to make sure stuff works and it works, hey, it works. and the only thing at that point between messing around and science, is "writing it down"... ;) push that code up , make it reproducible end to end.
• csimon80 4y ago
  It's so buzz word heavy. I had a manager that wanted me to solve a problem using the monte-carlo method when it fact the problem had a closed form solution...
  • r-zip 4y ago
    Why do math on paper when you can write code and look cool (and not like a useless academic)?
    • FridgeSeal 4y ago
      I’ll take this one step further: Why bother to understand how either software development has solved a problem, or how maths+stats has solved a problem when you could just ignore operational practices and “train a neural network to do it”?
  • CarbonCycles 4y ago
    Now that's both funny and sad. Mostly sad....
• HuwFulcher 4y ago
  WARNING: Stream of consciousness ahead As someone who has just moved from Data Science to Software Engineering I feel very much the same way, liberated. I worked in DS for 5 years and had varying degrees of success in working at companies that understood the proper use and application of Data Science. What killed my passion for it was a few things: 1. Data Science is a dubious field - Data Science can certainly be applied correctly but I and others have used the underlying statistical methods gung-ho at times. Part of this comes down to something that W.D said. That Data Scientists are generally early on in their career. We have been captivated by the shiny new field and want to use it as quickly as possible without fully understanding it. Throughout my career I've been met with varying degrees of scepticism about my profession by people because Data Science offers more than it can give. 2. Data Science professional development is poorly understood/completely neglected - If you look for resources to grow in your Data Science skills online you are invariably drowned out by the sheer volume of crappy "Intro to Data Science" courses online. As far as I can find there is very little advanced Data Science professional development resources out there. Compounding the problem is that Data Science teams are invariably managed by people who aren't native to the field. This has the effect of the manager letting Data Scientists self direct their learning which will hit the problem mentioned previously. 3. Support for MLOps is non-existent - I think this problem will change over the next couple of years but Data Science has had to go through cycles of being integrated into a business. The first "wave" of Data Science was met with the realisation by companies that they couldn't get Data Scientists to magic money out of the poorly maintained data they kept. This has caused a huge increase in Data Engineers (not just Data Science has spawned this), now we have Data Scientists who have access to nice data (thanks Data Engineers!), they can build some interesting models but how do they get it deployed? This second "wave" is seeing the rise of MLOps tools, engineers, etc but Data Scientists currently don't have the know-how to get their own models in to production. This inability is incredibly demoralizing from my experience. 4. Educating fellow Data Scientists is too difficult - Unfortunately the perception that is given to people coming in to Data Science is that you can just do model engineering and call it a day. Bootcamps, courses, tutorials are all geared towards getting people good at building models, not about considering how those models fit into the bigger picture. There is little to no knowledge about good programming practices, source control (a lot of Data Scientists I worked with only knew git as a swear word) or deployment strategies. You could argue that a Data Scientist's should only be concerned with building models, I would agree but the reality is that companies will hire a team of Data Scientists but will likely not provide complementing teams to get models in to production. When trying to upskill others on my team it's been an uphill battle. Either people don't care as they just want to build models or they have come from an adjacent field with no software engineering experience. Apologies for the stream of consciousness but it feels good to get it off my chest. My move to Software Engineering started in my last role where I was a Lead for a Data Science team. Thankfully my boss (head of Data) understood the need for developing a whole data system from good Data Engineering all the way through to MLOps for deployments. I was very fortunate to be able to move to being the Lead MLOps Engineer and develop our capability to deploy models with CI/CD mechanisms using AWS. That really gave me the taste for building systems rather than models. I really do think Data Science has a place and can provide great value but it's still a long way off. If we can make it so that Data Science teams can deploy to production quickly and safely we can really start to reap the rewards. For Software Engineers looking at getting in to Data Science I would suggest looking at MLOps first. You get to combine existing experience with tackling new problems (how do we keep models live and continuously learning? how do we ensure the tracking of experiments?) and will have a tremendous impact.
• tgtweak 4y ago
  I feel like in the near future there will be a more formal hybrid role between data engineering and data science - like devops or full stack developers. The best data scientists I have worked with (ML mostly) have been incredible data engineers as well - some of them former sysadmins, backend developers or DBAs themselves. They know where to get the data, how to set up pipelines and jobs, how to make sure they run properly, best practices for reading and writing to databases so they don't fall over, error reporting and logging, hosting their inference models on APIs, security by design... The amount of back-and-forth that gets cut out to go from raw data to product is huge when you compare it to a traditional siloed setup of business analysis/stakeholder, data science, infrastructure/security and systems engineering. I know some people cringe (mostly infra) when they think of data scientists having direct access to databases and infrastructure but honestly you should have a level of understanding and responsibility to get there. The data scientists that do data engineering are usually much more valuable to the company and definitely earn more.
  • rjtavares 4y ago
    I agree, up to a point. I feel that companies up to a few billion dollars in stable revenues in a non data intensive business don't need teams of Data Scientists, but would benefit from some people having Data Science skills (both Data Engineers and Business Analysts).
• rcarmo 4y ago
  My $DAYJOB has a very high match to this. In fact, I used to do data and analytics full-time for a long while (a lot of Spark, basically, mostly cutting through the usual lambda and K-architecture hype, bulding data lakes and <omg> "lakehouses") in several F500 companies. I could not agree more with the overall sentiment that data science is overblown in terms of reproducible results because the people doing it just don't have an actual process or good leadership focus (which is not just a startup problem...). So much so that I stayed stauncihily on the "data engineering" track because it was much more concrete in terms of technology, performance drivers, and business outcomes than the folk who sold pipedreams of magical AI models that would provide amazing analytics overnight. Turns out that if you can't get at, scrub and actually _use_ the data, figuring out trends or training models doesn't happen, so I focused on making at least that 50% of the project happen and leave nice, tidy infrastructure, workflows and schemas for the data science folk to go through. I also had the good fortune to work with some very organized, knowledgeable ML folk who actually understood how things worked, but some partners and customers had... incredibly disorganized "data scientists" that would leave stuff scattered all over the place (including private copies of datasets on their laptops when we had nice, secure remote sandboxes for them that even did data masking to avoid leaking sensitive data). Personally, I blame a lot of this on lack of certifications or professional training that emphasises _process_. Otherwise it's exactly the same problem we've had for the past 20 years in BI departments: People doing their own Excel sheets because "SQL is hard" and nobody can do ETL properly. (Full disclosure: I am an MS FTE, spent something like 10 years doing analytics almost full time, and have presented on how to do Data Science at scale a few times: https://carmo.io/talks https://carmo.io/talks)
• throwawaaarrgh 4y ago
  I've never met a data scientist who could do anything more than basic statistics combined with the Python skills of a fifth grader (that's probably insulting to today's fifth graders tho). I honestly have no idea what they're supposed to be doing or why they're paid so much money. Pay a high school junior for the same and get better work. And where's the scientific method? Where's the experiments and rigor?
  • r-zip 4y ago
    > where's the scientific method? Where's the experiments and rigor? No time for that, buddy. CXOs need results, ASAP! (This was basically the attitude of my managers at the last place I worked).
• deleted 4y ago
  [deleted]
• tgibbster 4y ago
  > how much ostensible interest in more advanced topics is pure sophistry This line is some amazing writing! Also extremely applicable to other areas of tech.
• scifibestfi 4y ago
  Calling it Data Science was a tell. Have you noticed how non-scientific things add "science" to the name to make it sound like it has scientific rigor? Data Science, Political Science, Social Science, Scientology Compare to Physics, Biology, Math.
  • lajosbacs 4y ago
    I too am critical of data science, but I think that this is a bit unfair. The scientific part could be called just 'statistics'.
    • laichzeit0 4y ago
      Should we call deep learning “statistics”? It’s mostly empirical. Should recommendation systems be called “statistics”? What about multi objective non convex optimization? Not everyone is doing regression and classification all day. The above were studied in the Computer Science curriculum at my university. I work with a statistician, like someone with a degree in mathematical statistics. They know nothing about any of those.
      • disgruntledphd2 4y ago
        > Not everyone is doing regression and classification all day. Yeah, some of them are doing unsupervised learning (recommender systems) too! I dunno, I personally think we'd all have been better off if we'd called it statistics as at least then people would realise that the field wasn't created yesterday.
  • r-zip 4y ago
    Computer science...
• fnands 4y ago
  Tangentially, I also think the term data scientist has been so abused as to almost be meaningless at this point. When I was applying for jobs it could range from anything from "knows how to use MS Excel" to "Can train large language models at scale". Personally I went for ML Engineering. My company at some point hired people as data scientists (some of my more senior colleagues still have the title, despite doing the same work I do), but started hiring people as ML engineers, i.e. people who can do half-decent SW engineering and also do ML. Just a filtering thing I guess. I have a suspicion the term will start to fall out of fashion as things become more specialised.
  • tomrod 4y ago
    Agreed. I've run a "data science consultancy" in some form or fashion for three years now. When people say "data science" they mean one of three things: (1) MLE (2) Data Management (3) Data Analysis or Business Intelligence (applications of the same skillsets). (1) has a lot of ongoing innovation, be it in MLOps, autoML, mapping frontier ML to business cases, etc. Innovation is expensive if the investment strategy is unprincipled. (2) is a critical and essential part of making data a usable asset. Management is expensive if it exists solely as a control process and gatekeeps access and use. (3) is core and will never get away from the adhocs and the standard flows, but the inferences are often dubious or not logically justifiable and requires depth of statistical knowledge (rare) to do well -- and courage to call out BS. Very few people have the depth to do all three. What I have found is that many businesses hope for capacity in all three, plus some basic SWE, in the hope that they can decrease labor expenses. Not an irrational hope, to be frank, but ultimate the iron law of business holds: you can have it good, fast, or cheap -- pick two and be happy with one. My core observation (and one I see validated based on client interest and experience) is that this is not new and has happened before -- it is the hype cycle in action. The digitization process (including moving to digital and then moving to Web) had a similar cycle. When you treat "data science" like its a silver bullet it will generally fail to do anything but suck budget. When you embed it with your technology teams and treat it as an iterative add, as useful as devops, etc., you have a better chance for value add.
    • glomgril 4y ago
• baobabKoodaa 4y ago
  > The median data scientist is horrible at coding and engineering in general. The few who are remotely decent at coding are often not good at engineering in the sense that they tend to over-engineer solutions, have a sense of self-grandeur, and want to waste time building their own platform stuff (folks, do not do this). I can relate to this so much. I've worked in multiple projects with great people who shifted away from solving the problem at hand, to instead construct some sort of generic problem-solving platform. In one project this actually happened twice: after refactoring the beef out of our SpecificProblemSolvingService into GenericProblemSolvingService, the generic problem-solving platform was then rewritten with one extra level of abstraction, so it could run any models designed to solve any task. As far as I know, neither service was ever used to solve any other problem except the SpecificProblem that we were solving the first place. But it's fun writing platforms, I guess?
• itissid 4y ago
  1. Fact: The reality of most ML modeling on data is a lot of models don't need to be giant ass interaction machines with 10^80 features. Many non linear models with a Naive Bayes or other simple non parametric models and adding a few Boolean Random Indicator Variables to your regression will go a long way to make your models non linear and get you to 80% F1 score. These are all practical problems and none of them require just a pure DS but also ML skills 2. Fact: You need to deal with bigger problems in ML like data set class imbalance, calibrating responses to the right scalar range(figuring out what that range even is in terms of domain). This is not taught in schools, just like writing Software is not taught in schools. One needs to be in the field to learn these skills and an ML Engineer can pick these up as much as a DS.
• chaosbutters314 4y ago
  physics-based modeling with some supplemental data science/ML will always be superior. Every data scientist I've worked with has so little domain knowledge anything they do is already useless. They also have little to no presentation skills especially in a business setting. obviously there are some examples like computer vision that require ML.
• d23 4y ago
  > Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. This seems to be a problem with the industry as a whole. I'm speaking as a SWE, but I've observed similar things with PMs. I don't think it's impossible or even very hard to appreciate the right things, it just requires a bit of thought and the correct value system. Both of those seem to be a bit too far of a reach though.
  • r-zip 4y ago
    I think a big part of the problem is that most PMs are non-technical, and at some point up the chain so are most managers. Data science, when done thoughtfully, requires you to appreciate minute issues from data collection and management all the way through feature engineering, model selection/design, and validation. The biggest, hardest bridge to cross was an appreciation of the importance of metrics. For some reason, getting a business person to grok something as simple as precision/recall/F-beta is a near-impossible task. You can do multiple presentations on it (after having honed those presentations over years with multiple manager audiences), and it never sticks. It's always "what's the accuracy?" It's impossible to do good work when your bosses insist on measuring and therefore optimizing for the wrong thing (which in my experience consulting for multiple Fortune 500 businesses, they always do). Even worse, many organizations have such broken politics/cultures that the managers can't even tell you the big picture of what the project is trying to accomplish. Once you finally piece it together from the people who know their roles in-depth, it becomes clear that what they're trying to do is totally infeasible. At least that was my experience more than half the time.
• ChrisMarshallNY 4y ago
  I have no axe to grind, irt data science/engineering, as I have no experience in either. However, it seems this person's biggest gripe is with good old crap management; the bane of business for hundreds of years. This line stood out: > Companies all over were consistently pursuing things that could be reasoned about a priori as being insane ideas– ideas any decently smart person should know wouldn’t work before they’re tried. That pretty much summarizes why I have been told that today's companies want only young people. Us "olds," are "negative naysayers," who say things like "You know that the laws of physics forbid this, right?" or "I tried that, a couple of years ago. It didn't work out, and here's why...". Apparently, young people are able to do the impossible, because they haven't been told it's impossible, and mixing "olds" with them, spoils the soup, by telling them it's impossible (or maybe a lot more difficult that they imagine).
  • deleted 4y ago
    [deleted]
  • fritzo 4y ago
    This resonates. I wish it didn't. I'm experimenting with ever more delicate communications to avoid spoiling the soup.
• usgroup 4y ago
  There are good research jobs in industry which are serious and mathematical. However they also require you to be serious and mathematical. I’d venture to say at this stage that most “data scientists” are either self taught segues from adjacent fields or have a shallow relevant background. The serious places don’t want you… so you end up at the place that can’t tell the difference, and the self fulfilling prophecy begins.
  • r-zip 4y ago
    Can you recommend a few such places? :)
    • clircle 4y ago
      Look up the technical presenters at your favorite math or stats conference and look at the orgs they work for. That's how you can tell which orgs are serious about solving research problems and spreading solutions.
      • caddemon 4y ago
        You could also look into the successful quant trading shops, if you're alright with not sharing results outside the firm. The nice thing about them is success or failure of your work is apparent much more quickly than the timescale of science, and it's harder to spin than other businesses. So the data science that gets done is generally pretty technically legit.
• deleted 4y ago
  [deleted]
• deleted 4y ago
  [deleted]
• jesterlg 4y ago
  As someone who recently switched from DS to platform engineering, this post really captures why I also jumped ship. > Shitty code & shitty data science In my opinion the bar should be higher for code quality but also general engineering know-how in data science. You'd be surprised how many are uncomfortable with git, using the command line, interacting with APIs, managing environments, etc. Being able to only work within a jupyter notebook is not good enough, at all. Otherwise, you end up with people who's entire job it is to productionize and deploy the code which is a waste of time and effort. > Poor mentorship There is either a lack of quality leadership and mentorship or an inability for upper management to see the value in hiring for it. What ends up happening is you have a team who doesn't know how to grow, scale, or work together. They instead focus on building models and learning statistics when they should be focusing on building systems and process for helping the business scale analytics and building models when appropriate. I enjoyed data science but found it to also not matter in the implementation that everyone thinks it should be. Data science isn't building nothing but ML models. In most companies, in my opinion, it is actually about scaling analytics. Being able to reach further up into data engineering, get raw data, explore it, shape it, give it back to DE to automate, and then automate the delivery of data to upper management and guide them through using it. If the team thinks their job is to just build models, everyone is going to have a miserable, miserable time.
• nerpderp82 4y ago
  This blog post is directed at me, personally. Thanks W.D. This isn't just Data Science, I'd say that the gripes of the author are valid for about 60% of activity in tech companies. Not saying we can just eliminate 60% of it, but a lot of it supplies non-quantitative value that is driven by fashion (subset of politics) and direct politics. There are a lot of naked emperors walking around with lots of folks standing as close as they can to shield them from the cool winter wind.
  • jrochkind1 4y ago
    Not just tech companies either. Go bigger. It's possibly 60% of activity in US white collar economy. (I can't speak for those who work with their bodies, could be or not).
    • asdff 4y ago
      Former body worker. 100% of us worked, because if you had nothing to do the boss will invent something for you to do or send you home.
      • jrochkind1 4y ago
        Are you saying that meant you did a lot of useless work?
        • asdff 4y ago
          It was more like backlogged maintenance. Like "Ok its too wet to mow today, lets get the backhoe and finally redo the storm drain so it doesn't clog"
          • jrochkind1 4y ago
            So that is maybe different. I think the "60%" we're talking about above, to me, is people doing useless work, or work whose only use is internal "political".
  • deleted 4y ago
• euix 4y ago
  I have to agree with a lot of this - I started my career as a data scientist right out of a STEM PhD back when the term just started coming into existence. At the time, anyone who wanted to get hired as a Data Scientist needed to be trained as a professional scientist, i.e. have a PhD - at first my expectation that the purpose of my job was to apply the scientific method to solve business problems by leveraging the companies own data as the empirical evidence - whether I did this using machine learning, excel tables or a chalkboard didn't matter. ML was a barely used term at the time, the first version of Tensorflow wasn't released until later that year. But over time, the higher up I climbed the more I realized the job had marginal business impact. Usually a big company would hire a bunch of PhDs with fancy degrees and stick them in some "Advance Analysis" department and leverage them as internal consultants, which just meant creating some models, writing a powerpoint deck, get a pat on the back from the execs - not a single model would ever see the day of light. I got all the way up to Director this way, before calling it quits this January, at the end I had basically nothing to do except work on "corporate AI strategy", which meant writing presentations and white papers for upper management. It was comparatively easy job, one could coast their entire life in some of these corporations - especially in government sanctioned oligopolies like banking.
  • wheelinsupial 4y ago
    > the higher up I climbed the more I realized the job had marginal business impact Do you have any observations why? I'm a pretty lowly business analyst, but my observation is if you don't own the decision making (usually by having profit and loss responsibility), you can't have much impact. Possibly it's the companies and industries I've worked at, but at the end of the day if the results don't meet expectations, it's the business owner that gets fired and not the people providing the recommendations.
    • mjburgess 4y ago
      For the same reason science takes 100s (,1000s) of years to develop. All the "intelligence" takes place in the humans that design experiments to collect unambiguous data. "data" absent a profoundly intelligent (, expensive, fraught, ...) experimental design is basically useless.
• gpresot 4y ago
  I realise the article was written for a specific audience for which this may be obvious, but what is the difference between data scientist and data engineer (in terms of what their job is)?
  • kingant 4y ago
    Data scientist actually cook up and run the statistical/ML models on data and write reports about their "findings". However, the data that data scientists want to use is often messy and comes from varied sources. Hence, data engineers do supporting infra work like cleaning/loading data from different databases, etc.
  • teraflop 4y ago
    Generally speaking: "Data engineering" means building systems that can manipulate data (e.g. storing, retrieving, and delivering it). There are usually fairly well-defined functional requirements about what the system is supposed to do, plus goals about performance and reliability that might be slightly more nebulous. "Data science" means building systems that can draw conclusions from data. The functional requirement is usually some form of "accuracy", as measured somehow against some kind of human evaluation of the same conclusion. Concretely: a data engineer might be asked to build a system that can ingest every tweet posted to Twitter, and return the 10 most widely-used hashtags in the last hour. A data scientist might be asked to build a system that looks at a tweet and figures out what language it's written in, or whether it's spam, or whether an attached image is pornographic.
• hintymad 4y ago
  > Shitty management & insane projects > Shitty code & shitty data science Shouldn't the author conclude that "Goodbye, shitty companies"? How were the two problems unique to data science?
• LAC-Tech 4y ago
  For years I was low key obsessed with the idea that I should ditch regular programming, and become a data science. My thought process was very crude. Data science was mathsy and paid more, therefore it was a great career move! After a few abortive attempts to learn statistics and linear algebra in isolation, I decided to sign up to that famous Andrew Ng online course. I was bored out of my mind. I just did not find it at all interesting. Not sure what my point is - maybe that a really easy way to learn if a seemingly lucrative and interesting thing is for you or not is to go ahead and learn the very basics of it directly, and see if you are at all motivated. I was not. And after that it was out of my mind.
• uwuemu 4y ago
  > Like bro, you want to do stuff with “diffusion models”? You don’t even know how to add two normal distributions together! You ain’t diffusing shit! Made me LOL. First time I did that while reading a tech post/blog in years. Also neatly describes half the HN audience fawning over the latest AI thing.
• bricss 4y ago
  It is a masterpiece! And it can be applied to modern software engineering as well
• fumeux_fume 4y ago
  I have to agree with the author on all of the points he makes here. My projects with the most impact typically had a stronger data engineering and software engineering contribution from my team than data science. Data scientists today are what surgeons were in the 1700s. Hard for people to tell the difference between the good ones, the bad ones and the charlatans.
• brosciencecode 4y ago
  > 23 year-old data scientists should probably not work in start-ups, frankly; they should be working at companies that have actual capacity to on-board and delegate work to data folks fresh out of college. If a 23 year old manages to get a data science job at a startup, and then actually delivers the results that the start-up expected of them, the learning experience there is infinitely more valuable than going to Google and using a bunch of tools that don't exist in the real world, on unrealistic timelines because you're not on the ads team and don't need to make money. You can go learn "best practices" later, but working in a startup is an exercise in pragmatism. You deliver results, or you die.
  • deleted 4y ago
    [deleted]
• namkt 4y ago
  Damn, this hits so many nails that it's got to be one of the biggest HN posts in a while.
• mikaza 4y ago
  I don't see how this personal take generalizes to a trend specific to data science (as much as the term looks weird to me). In other words, I don't see how creating useless Python notebooks is different from over-engineering or creating useless features. To me the proper context of the story is "cheap money have been flooding the market for _decades_".
• ISL 4y ago
  Can state from personal experience that there exists at least one company where each of the article's leading bullet-points are simultaneously false. If your DS employer isn't making real use of the capabilities of a skilled data-scientist and that makes you sad, consider looking for a company that will.
• anton_ai 4y ago
  Reading this and other similar posts, I feel so lucky to have found a 1) great boss that really knows his stuff thanks to the fact that he started working on this before it was cool, 2) a great company that believes that data will be the future and 3) a good team that follows me without too many frictions.
• zeroCalories 4y ago
  I found the section on "Poor self-directed education" to be interesting. I've had similar feelings as a full stack developer, where in many ways I'm just learning APIs and other narrowly useful trivia. The only things I've had meaningful improvement in are stuff like soft skills, giving a good code review, or managing a complex project. Wondering what kind of skills someone like me should focus on to avoid the "embarrassing" skills/resume gap.
• antipaul 4y ago
  While the tone is a bit too negative (expected of someone leaving one place for “greener grass”), there are terrific points here that completely resonate with my own experience in data science - albeit not at a “median” place, but at a large non-Tech corp. The biggest point I’d emphasize: “there is a general industry-wide need for people who are good at both data science and coding to oversee firms’ data science practices in a technical capacity.” Even more, most of the data tech leaders I hear strongly suggest this is not possible without C-suite representation of a data/engineer expert (I’m talking about non-tech companies) Finally, it’s actually not uncommon to see people shift from data science to data engineering due to similar motivations. It’s actually the technical leadership that is sometimes surprised at the shift. You hear about this in podcasts from DS/DE people.
• kleene_op 4y ago
  This article resonates with me. I love data science, but I hate the bullshit around.
• grepLeigh 4y ago
  I'm reading a lot of sour grapes here, but I'd like to offer a more optimistic future snapshot (if you're a data scientist and happen to be reading). I agree that many companies hire data scientists with only a vague idea about how to utilize them, but the same is true of software people in general. "Software is eating the world" and so is the practice of extracting value from data. The margin on software is high - often more than 95% - so there's a lot of room to screw up and "figure it out" as a business. I think that's why there's a low bar for software and data management compared to, say, an automotive manufacturing line manager. But that's where the opportunity is, if you're a budding data scientist: - The business might not know how to effectively use/manage/train/mentor you. - Upper management might have 20+ years of line of business experience, but will need your help to understand how your team can impact the business. - You're going to need to seek out ways to impact the bottom line of the business. All of the above is a recipe for leaping forward in your career. Since data science is a relatively new field, the demand for senior leadership FAR outstrips the available supply. If you can learn how to effectively manage yourself, your team mates, and your function within the business - you have a ton of negotiating leverage and can name your price. Source: I'm a data person who "retired" in their early 30s. Now I do all the research and hard science I want. ;)
  • disgruntledphd2 4y ago
    > the demand for senior leadership FAR outstrips the available supply. This is so very, very true. Most of the "bad" data science orgs I've spent time with, are bad because leadership is either bluffers or a data engineer/BI type person. It's generally hard for these types to run effective DS orgs as the skills needed are very, very different.
• 3pt14159 4y ago
  > Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. Story time. There was once a junior data scientist at Shopify that had learned Python and SQL and was tasked to figuring out how to fix their "broken app store recommendation engine" but since they didn't know Ruby, they asked for my help in figuring out what was going on. Well somewhere in the soup of math was a fuzz factor at the very top. Think of it like factor = 0.something # Not 100% sure what the decimal portion was. some_complicated_math_that_maxed_out_at_one_pt_zero() + rand(factor) Now the thing about ruby is that rand is basically broken for floats. Negative or floating point values for max are allowed, but may give surprising results. https://ruby-doc.org/core-2.4.0/Kernel.html#method-i-rand https://ruby-doc.org/core-2.4.0/Kernel.html#method-i-rand So basically what they thought they were doing was introducing a bit of randomness that would hinder others from reverse engineering their algorithm. What they actually did was make the recommendation algorithm fifty percent total noise. Yes, it's true. On every load half the recommended app scores were noise. They fixed the bug and I'm sure a ton of balance sheets for businesses around the world are markedly different know because of it, but I never heard of it again. This is one of the core problems with data science. The lack of feedback.
  • mattkrause 4y ago
    Ironically.... Having a subset of totally random recommendations wouldn't be a totally terrible idea---especially if you know which they were! It could help push the system out of local minima and it's the obvious benchmark to beat.
• ok_dad 4y ago
  > The only way to win is to become a stooge. I sometimes feel like that as a software dev, though, when management pushes some changes or "fixes" that aren't useful and won't fix anything.
• civilized 4y ago
  Great post. A few random comments... > The median data scientist is horrible at coding and engineering in general. The few who are remotely decent at coding are often not good at engineering in the sense that they tend to over-engineer solutions, have a sense of self-grandeur, and want to waste time building their own platform stuff (folks, do not do this). > It was obvious that there is a general industry-wide need for people who are good at both data science and coding to oversee firms’ data science practices in a technical capacity. The job of overseeing a crowd of stubborn self-important over-engineerers sounds pretty thankless. > 23 year-old data scientists should probably not work in start-ups, frankly; they should be working at companies that have actual capacity to on-board and delegate work to data folks fresh out of college. So many careers are being ruined before they’ve even started because data science kids went straight from undergrad to being the third data science hire at a series C company where the first two hires either provide no mentorship, or provide shitty mentorship because they too started their careers in the same way. Startups are a low-paid job with a lottery ticket for a little dash of excitement. You get what you pay for. > ...I live in constant anxiety that someone will pop quiz me with questions like “what is the formula for an F-statistic,” and that by failing to get it right I will vanish in a puff of smoke. So my brain tells me that I must always refresh myself on the basics. Focusing on the basics is better than pretending to understand fancy things, but even this level of "continuous professional training" or whatever you want to call it is, to me, a bit off the mark. We can look up formulas whenever we want these days. We need more meaningful ways to test our understanding of things.
  • languageserver 4y ago
    > 23 year-old data scientists should probably not work in start-ups, frankly; they should be working at companies that have actual capacity to on-board and delegate work to data folks fresh out of college. Ageism is disgusting and I cannot believe such blatant discriminatory language is seen as OK for a link posted to hackernews. How would you all say if he wrote that 40+ year old programmers should xx?
    • civilized 4y ago
      Eh, there's some stereotyping going on here that I don't 100% agree with, but I'm not offended by the notion that people fresh out of college are generally inexperienced in the working world and lack skills that are needed in industry more than in academic work.
• hk1337 4y ago
  This is how I feel about software development. The more people that get into it, the more it seems to be more that you're just a code monkey pushing out shitty code.
• jeffrallen 4y ago
  > I have a sense that, if my current place of business needed to chop employees, that it would be a dumb decision to chop me over any data scientist. Yeah, dude, good luck with that. That is emphatically not how layoffs happen. :(
• tommykins 4y ago
  I think is true for >general< data science houses and firms that offer Data Science across any domain - I'm generally wary of anyone who writes about Data Science as a concept, rather than the use of Data Science to solve a particular problem in a particular domain, the rise of the Machine Learning bros is very real.
• SergeAx 4y ago
  Lots of people are citing bad management as a most annoying thihg in tech. I wonder two things: 1) To recognize bad management, there needs to be awareness of what good management is. What is the common source of that awareness? How do one know good manager from bad one? 2) If good management is so crucial for tech company success, why there is no worldwide trend to help engineering managers become better? There's an awful lot of courses, bootcamps, learning videos, tutorials, git repos and so on for those who wants to be software engineers, but all I see for managers is self-help style books and articles, centered around typical situation "oh, shit, they appointed you to a managerial position, how do you cope with that?"
• FridgeSeal 4y ago
  I relate to this so much. I have experienced the same things, and I also find myself moving across to data engineering. In my case, I was in organisations that wanted data science, but had no capability or interest in supporting the role, so a lot of my time and effort has been having to put down the data science tools, and learn devops, software development and data engineering so that I can get back to the point where I do my data science work. I’ve also become frustrated with my data science peers lack of knowledge about the surrounding fields- I get that being a top tier software dev isn’t the primary responsibility of a DS, but it would certainly make their life, and the life of everyone around them a lot easier if they did make an effort. There’s a sense of “learned helplessness” in parts of data science (and parts of data engineering too) in which “if some third-party tool can’t do it for us, we just can’t do it” and imagination is limited to the features the latest framework de jour offers.
• Reebz 4y ago
  >The work is downstream of engineering, product, and office politics, meaning the work was only often as good as the weakest link in that chain. >The work was often very low value-add to the business (often compensating for incompetence up the management chain). The chain is what is important. This is not a data science specific issue. Swap in or rearrange any team names to the original list and the last position or two will find this article true.
• d_burfoot 4y ago
  This is a great rant. A lot of good engineers stand by the rule "use boring technology". Data scientists should adopt the rule "use boring math".
• salty_hash 4y ago
  I've had similar experiences to the author and fellow HNers in this thread. I just wanted to add that my personal transition from DS to DE has allowed me to work with a wider variety of data. DS is mostly tabular; data out in the wild comes in all shapes and sizes, and learning different techniques has been interesting.
• salty_biscuits 4y ago
  See a lot of pessimism here, and heard a lot of the same stuff by grumpy engineers over my career "nobody knows how to do statistics properly", "scientists are bad at coding" and "managers don't care about quality and rigour". I'm old enough to say it all pre-dates data science as a term. It all comes across as a conspiracy like everyone is bad on purpose. There is a null hypothesis here, that the average person in role x is just average at that role. It is an extraordinary person who has high level skills across multiple domains like maths/science and coding, maybe so extraordinary that they wouldn't be working with you... I'll admit that the article rings true, but I think there is an implied intentionality that I don't agree with. We are all just plodding along, doing our best with limited information and skills.
  • arcturus17 4y ago
    > there is an implied intentionality that I don't agree with. We are all just plodding along, doing our best with limited information and skills. Never got the feeling the author blamed the data scientists (he was one himself), but rather management, and not their bad intentions but their incompetence.
• in9 4y ago
  Regarding the business impact of data science related work, as a working ds and team lead a have a few thoughts. There are some sectors and activities that are very data science like, to using a blunt anachronism, have been like that for a long time, and have a lot of teachings on how to develop and measure impact on business by the automated decision systems and/or analytical work that they build. Some examples are: - demand forecasting for supply chain processes - credit scoring for loan approval - portfolio risk analysis in financial settings - some misc optimizations work on operations - maybe even six sigma can be listed here Those have always had a data-science-like feel to it. The problem I see is when companies try to implement a data science team is: 1. push out subject expert knowledge and requirements just for the "freedom"; 2. have stake holders to be out of touch with the solutions; 3. too litle focus on putting stuff "in production", tracking, beeing able to experiment, in whatever sense those have for the company; 4. too much focus on numbers that can come out of the ds's computer, and treating operation related numbers as an after thought; 5. no basic knowledge of simple/common/classic solutions for their problem at hand. So yeah, making business impact is way harder than is sounds, and too out of the skill set for the 23yo STEM graduate to actually make impact. And too buzzwordy and impressive for the typical decision maker. I mean, I've heard countless times things like: "if an AI knows how to tell a cat from a dog by using one of those neural nets logarithms, surely it can know how to partition my marketing budget, optimize my coupon giving logic and determine the strategy we should have in order to achieve a very obscurely constructed OKR". (yes, I have worked for people that used the word logarithm to mean algorithm).
• zmmmmm 4y ago
  Feels like data science outside of deep learning is slowly edging towards the trough of disillusionment. Time to rebrand your CVs folks in line with the new hype, while waiting it out to get to the plateau of productivity ...
  • r-zip 4y ago
    The funny thing is that it shouldn't be. There is plenty of potential, but all sorts of (mostly non-technical) factors are getting in the way.
• baazaa 4y ago
  People who make important decisions get paid a lot and have a lot of power. Why on earth would they then delegate that decision-making to data-peeps and make themselves entirely redundant? Data scientists are always going to be funnelled into make-work projects just so companies can claim they're doing well on the data-side of things. The same is true of automation incidentally, there's lots of big companies doing a lot of easily automatable work, but the guy who manages all those people doing easily automatable-work is hardly going to be scrap his own area by calling in some SWEs. On a completely unrelated note, the author's data eng sounds like nothing I've seen. Hell veto power over code? I'm not even sure all of them can code. They're just glorified sys-admins who now can provision some cloud infra. Somehow data engs are probably even more incompetent on average than data scientists, and the reason you move upstream is because it's an easier job and you don't need to spend your weekends grinding through maths-problems or learning new languages while probably still having higher value-add.
• xiaodai 4y ago
  > Rather the main bottlenecks I’ve faced were always crappy infrastructure and lacking (quality) data, so it has always felt natural to focus my efforts toward learning that stuff to unblock myself. Right. What can you learn to over come crappy infrastructure?
• 29athrowaway 4y ago
  The field of software engineering learned to punish cowboy coding and prima donnas. Now it's the turn of data science.
• roland_nilsson 4y ago
  'Like bro, you want to do stuff with “diffusion models”? You don’t even know how to add two normal distributions together! You ain’t diffusing shit!' This made me laugh :) I'm in academia and this post does remind me of certain students...
• oerpli 4y ago
  I applied as SWE in 2018 an was asked whether I ever thought doing Data Science instead. I said I thought about it but it didn't appeal to me, for reasons similar to the article. Still, I got hired as Data Scientist. The money was good, but I didn't believe in the field and was kinda ashamed of the title (though it did seem to impress people - I think rural Europe is a few year behinds). After 3 years I left for SWE SRE and my only regret is not doing that earlier. Decent grasp of SQL and a bit of handwavy knowledge about stats comes in handier than expected.
• Moissanite 4y ago
  > So many careers are being ruined before they’ve even started because data science kids went straight from undergrad to being the third data science hire at a series C company This is something that bothers me about the industry in general, but thankfully seems rare inside tech giants. My perception is that VC companies in pursuit of the next unicorn idea believe it can only come from a young mind, and so pile money and opportunities onto young people who aren't ready for that responsibility. Either they crash and burn, or they are carried over the finish line by multiple investment rounds then swan off, thinking they did it all themselves, to go through the experience again as an angel investor.
• ngc248 4y ago
  I moved from Systems software to DE/ML Engineer and then back to Systems software. There is no more fun involved in setting up pipelines, writing spark jobs and prductionizing models.
• findalex 4y ago
  > Like bro, you want to do stuff with “diffusion models”? You don’t even know how to add two normal distributions together! You ain’t diffusing shit! This is funny. Great article with lots of truth.
• samuell 4y ago
  > Personally, I’ve benefited a ton from reading the first couple chapters out of advanced textbooks (while ignoring the last 75% of the textbook) Anecdotally, this is something I've thought about many times, that an awful lot of books cover like 80% of the important stuff in the first say 4-5 chapters, and then are just full of filler. I find this extremely frustrating as I'm at the same time fearing to miss out on some important insight in the latter chapters, while reading the full length of most books is simply super hard to do in any sensible quantity. Otherwise, great post. Going through a similar transition, for some of the same reasons :)