24 ms·

Most data work seems fundamentally worthless

• scrollinondubs 4y ago
  I rarely comment on HN but this article struck a nerve as I have worked at a handful of toxic bureaucracies that suffered from this issue back in the day. It's incredibly demoralizing and yet every place like this eventually reaches a weird equilibrium that preserves this mess as this unspoken pact amongst the people who are getting paid for nothing (queue "Office Space" scene - "what would you say you dooooo here Bob?"). If you're not in debt and massively dependent on the job my best advice is run, don't walk, to find a place where you can get paid for value delivered. Startups are generally not afflicted by this cancer because there's simply no place to hide- everyone is doing six jobs and it would be nearly impossible to conceal this level of inefficiency. And if it ever did proliferate in a startup then that's _definitely_ not some place you want to be. It sounds like you've gotten a few bad dice rolls but rest assured there are plenty of good startups out there creating important products, paying their employees and valuing their work. As challenging as startups are, I would way rather take that environment any day over the slow boil (or the slow bake following the lasagna analogy) and gradual erosion of my soul at the Office Space job. One other thought for you: if your circumstances dictate that you _must_ stay in a toxic job environment like this and there's simply zero way to change the environment or leave, partition your work life into a box so that toxicity doesn't permeate the rest of your life. Clock-in, tolerate the nonsense, laugh about it with colleagues then clock-out and find your meaning from other pursuits, whether that's spending time with your family, taking up a hobby, doing volunteer work, whatever. It's great when your job produces a sense of meaning but if meaning is absent from that, you can definitely find or create it elsewhere. I've spent the past year building up https://Problemattic.app https://Problemattic.app for just this scenario. I was fortunate to have finished a 26-yr career in various tech roles and now have the luxury to try and work on solving this issue. I'm convinced now more than ever that it's possible to find a profound sense of meaning by applying those professional skills that are currently squandered and redirecting at least a fraction of them towards the pursuit of solving important societal issues. Anyways good luck. We can always use data science people so feel free to peruse the projects and contribute a few data science cycles to anything that grabs you. Or propose a project and rally a team if you have an idea for something not currently listed. cheers
  • ludicity 4y ago
    As someone figuring life out (and also the OP), just wanted to say that I really appreciate you taking the time to write this. Advice is always great. For the most part, this is largely what I've been doing after a lot of analysis, though I avoided including my post-realization strategy in the article as... well, firstly, I haven't seen results yet, and secondly, I don't have enough experience to guide people through this. Some people are fine with the work as long as they can spend time with their kids, and I respect the hell out of that. I've got a few engineers also looking to find a way out, and we'll definitely check out your platform!
• jasmer 4y ago
  It's very hard to know what creates value without more senior leadership experience. Sometimes it's only '1 thing out of 10' but that '1 thing' is absolutley essential. But honestly, I suggest the entire 'big data' movement is a bit outlandish, there really isn't a need for most of it. The 'saving grace' frankly is AI, because maybe-maybe-maybe we can make use of that data in some future alg because god knows we're really not making that much of use of it. But it's hard.
  • baxtr 4y ago
    This. So many people underestimate the value of experience. Seeing patterns comes with seeing many different things over an extended period of time.
    • jeffreygoesto 4y ago
      I think the author wound have been very grateful if at least some of the seniors would have seen it as part of the leading task to craft some reward structure for their subordinates. Like that barber did. Life is so much nicer when money is for people, not the other way round.
    • spaceprison 4y ago
      The teams that brought value to the company were embedded alongside of the folks producing the data to be 'scienced' and had leadership that was able to understand and communicate the information (and its limits) to their leaders and adjacent teams and use their experience to point the data science folks in the right direction.
  • stuartjohnson12 4y ago
    Anecdotally, it seems like companies advertising data + AI capacities are the worst offenders when it comes to buzzwords and grift. Also anecdotally, I know the founder of a small agency that does rapid prototyping of predictive models for logistics-heavy industries and adds a lot of value. I think the problem is really one of scale - it's way harder to pin down your value add when you're part of a very large collective. Being responsible for 10% of a 10-man output is much more tangible than being responsible for 0.001% of a 10,000-man output, and the difference between 10% and 0% is similarly much easier to discern from a management and accountability perspective.
• rrjjww 4y ago
  The author seems to extrapolate from their experience of having a “six figure data job straight out of college clocking out at 5pm everyday producing no value” (paraphrasing) that all “data work” must be producing no societal value. I don’t like throwing around out the phrase “privileged/out of touch” too often but this post doesn’t seem fit for the top position of HN.
  • jeffreygoesto 4y ago
    Care to elaborate where the value created for millions of people is in an industry that pivoted years ago to focus on ad revenue and psychological tricks to extract money from 'whales'?
    • rrjjww 4y ago
      Sure, advertising is collectively a multi-trillion dollar industry propelled largely by computer/data science, but it is still only a fraction of overall economic activity driven by advancements in data analysis. The past couple decades have seen huge advancements in safety, reduced workplace injuries, auto injuries, etc. all driven in some part by analytics. This sentiment that “all data analysis = advertising = evil” seems very reductive. It reminds me of all the comments I see from my technical peers about how “useless” other departments are such as HR and Sales, when they’re fundamental to a healthy business which pays the salaries of programmers.
  • seydor 4y ago
    Most of jobs in academia are like that. Everyone knows it, some few admit it, most do not want to rock the boat.
  • apwheele 4y ago
    I found it resonated (although agree the title is click-baity, they just talk about their own experiences in multiple orgs). It is hard to do good data work. Offhand it takes some combination of: - business understanding and goals (that don't themselves come directly from data) - using data to effectively orient those goals to the best opportunities - using data to measure whether you are successful or not Part of that can involve meta goals -- our data is not sufficient now to meet bullet 2, so we need to start doing something different to measure it. I find many people in these roles act like they are just human machines producing reports. You really need to take agency in many situations IMO, and direct higher ups to look at data in the right way. If you just wait to be told what report to produce, it will not go well.
• boringds 4y ago
  I feel like the title is a bit clickbaity. This post is more of a rant against poor management than actual data work per se. That being said it's true that unlike other roles such as developers, artists, copywriters, etc. that still can produce tangible elements of the product under poor leadership, data will be the first victim of poor management and will have a higher probability of not producing anything at all. Data buy-in within a company is absolutely essential for it to work.
• Wonnk13 4y ago
  This resonates. I've moved from data engineering into sales engineering. The variety of problems is fun, albeit the context switching with Account Execs that don't respect my calendar is... something. A lot of my previous data work was someone saying "show me what's interesting"... "no, not that- something else"... "huh, this doesn't jive with my gut, start over". Bringing _value_ out of data sciene / engineering is incredibly hard. You have to have the engineering skills as baseline, but I firmly believe everyone undervalues the amount of work to "sell" your analysis to less technical folks in a way that's inline with the needs of the org. It's incredibly difficult.
• specproc 4y ago
  Long-time non-profit guy here. Totally get where the author is coming from and I'm in a very similar position in terms of where my role relates to my organisation, but I still love my job despite having exactly the same issues. Why? Non-profits are a funny old space. In theory, they're not out there to make money, so people gravitate towards them seeking "purpose". Everything is fine once you realise that you're not going to get that, or at least in the way people come to it for. Non-profits have ridiculous, completely unachievable goals, particularly when you put their budgets in context. Some provide truly excellent services and products (e.g., reports), most don't. So what's so great about a non-profit tech job? I've got an interesting problem space, I love trying to work out how to measure things that are inherently difficult to measure. I have a hell of a lot of autonomy, freedom of tooling, people listen to me on tech issues -- even when they probably shouldn't. I bust my arse because I'm interested in what I'm doing, but I could easily coast if I wanted to. The pay is enough for me, and my colleagues are for the most part super nice and interesting. If I want to learn something, I can make an excuse to play with it at work and run with it. Seeking to have "impact" through a data job at an average non-profit is naive, but a lot of these jobs have stuff to recommend them beyond that. ---- Edit: slight addition to my list of joys of non-profit nerding
  • einpoklum 4y ago
    > you're not going to get that If you believe what the organization does furthers your purpose, then you are going to get that, although I agree that the day-to-day feeling may not be that "purposeful". > Non-profits have ridiculous, completely unachievable goals There a gazillion kinds of non-profits; some do, some don't. If your non-profit is the town museum management foundation, you have a perfectly achievable goal (it's right there in the name). If your non-profit is "the coalition to end world hunger", then maybe not so much. Then again, even such a non-profit might have the actual, stated, legally-filed goal of delivering food aid to areas in the world hit with natural disasters like floods or drought - and that's not an achievable goal in the sense of being ever done with it, but it is achievable in the sense of being able to do just that continuously. > So what's so great about a non-profit tech job? That there is almost never the situation of a huge pile of closed-source code developed internally which you have to live with; and no custom expensive hardware. So typically you need to both rely on, and contribute to, free software that basically anybody can use. > Seeking to have "impact" through a data job at an average non-profit is naive On the contrary. There is great potential of technological impact outside the organization due to what I said above; and there is potential for impact on what the organization does, because it's often more fluid, and using data, you can make convincing arguments about steering activity in a different way - appealing to your colleagues desire to serve noble purposes.
• ak_111 4y ago
  I am surprised there are no private equity like entities that are driven by a single thesis which is to buy non-tech public corporations (or as far away from tech as possible: like commodities and real estate) then cancelling all 'digital transformation' or 'data engineering' contracts with consultancies like McKinsey, replicate the important 10% part of the contract in-house by recruiting a solid small tech team. Then take the company public later at 10x the valuation, having slashed the cost by 10x without impacting operations or growth.
  • wodenokoto 4y ago
    If you figure out how to identify those 10% and get the business on board with following that, I’d suggest skip the buying companies part and just sell your consulting hours at triple McKinsey rates.
    • deleted 4y ago
      [deleted]
    • speed_spread 4y ago
      McKinsey doesn't get brought in on the basis of their competence even though it's of course part of the image they project. If you want to go with quality work you'll have to buy your way through because the upper management doesn't have a clue about tech and you will not be able to justify your higher rates if you're a nobody.
  • gitfan86 4y ago
    Executives at those companies understand that risk and set up all kinds of landmines to scare private equity away. A simple example would be, they could put the IT support department under the "VP of Crypto" instead of under the "Director of IT". So private equity comes in and fires the entire crypto department because they assume it is useless and then find out that no one can get their passwords reset or computer replaced. More complex examples involve finance. You could transfer stock in your company to a banker as collateral for a loan with the condition that if the loan is paid off early there is a 1 billion dollar fee. Taking a company private requires that the buyer buy all the public stock.
• Neil44 4y ago
  You could say a lot of the work is speculative because you might be wrong about how much value it ends up producing. But if you say that 1 project in 10 pays off you just take the cost of the other 9 as the cost of the 10th.
  • somecommit 4y ago
    That why this kind of department for sure was called R&D... His organization just didn't reached the "D"...
• deleted 4y ago
  [deleted]
• seydor 4y ago
  Having way too many bullshit jobs backfires , and we are not prepared for that
• dodgerdan 4y ago
  Watching growing startups hire a junior data scientist to analyse their new mountain of data is very entertaining. One of their first findings is their revenue/growth seems to be dropping… “we need someone to focus on growth/revenue ASAP”. They hatch a plan, build a team and magically their growth starts to recover. The issue was spotted in April/May, the building happens during the summer and is launched In September… I’ve watch this cycle happen so many times, I’ve given up trying to point out that the issue is just seasonality.
  • datavirtue 4y ago
    I worked with a guy who loudly pointed this out in executive meetings where marketing and analytics were presenting. Then he wouldn't let it go, loudly proclaiming the sham in the hallways and through various emails. He was laughing at them in their faces, even, and was 100% correct. They excluded him from any further meetings. Every time they brought up a new stat and set off alarm bells he would bring up their massive fuckup and rub it in their face, and then would dig through the data and question every point. He was eventually fired for pointing out how incompetent our new owners were after an acquisition, and would have been a lot happier joining everyone in their sham.
    • draw_down 4y ago
      [dead]
    • ludicity 4y ago
      Yeah, I haven't seen anyone do this firsthand, but all my social intuition tells me that you can't challenge the social reality being presented. You can yell at one manager, call his team incompetent, so on (all deeply inadvisable), but in my heart, I don't think anything would get you strategically blackballed and removed faster than saying "So... what do we all produce, exactly? Oh? You realize that's absolutely worthless, right?"
• deleted 4y ago
  [deleted]
• kinnth 4y ago
  I'm going to take an educated guess that you've just turned 30 or are about to turn 30? I had this exact same problem where I became entirely disillusioned with the output of my work after having become reasonably senior and realising that the majority of my time and effort ammounted to projects which either failed or treaded water. The unfortunate reality in business is that the projects that truly change the world are 1 in 100. They don't look or feel much different from any other at the begining but when traction really does start to take off you know you have something right. I think from your description, you need to work at a startup where you have genuine impact. This is the only way you truly "feel" meaning. You might still build something that's reasonably worthless to humanity, but you will do it your way and take your own risks.
  • ragnarsson 4y ago
    I felt disillusioned 4-5 months in my first job after college. I could see the project I was working would take a long time to create something of value (3-5 years) but we were always put under pressure to show some stupid demos to management every quarter. So at the end we were just doing these demos and not working on actual things that we needed to do. The project was important, if implemented successfully it would cut jobs of ~60 people in the organization. So management kept on adding more people until they realized that there was no real progress and they dumped the project and we got reassigned. I realized that 1 year of my work as well as ~15 other engineers just went to waste.
• ismokedoinks 4y ago
  While I work in data and do appreciate the sentiments in some ways, I'm not sure this feeling is unique to any type of office work?
• steppi 4y ago
  Working in the ML/data space and have been fortunate to have largely steered clear of this problem so far. My heuristic when evaluating potential jobs is only to consider positions where the output of statistical analysis and machine learning models has a clear and immediate impact within the organization, while avoiding adtech for ethical reasons and because of concerns toxic work could enable a toxic environment. Jobs like this exist, though you may have to take a pay cut compared to the bullshit. My first job after finishing my PhD was as a staff scientist in an academic lab using ML engineering + data science to support scientific research. There seems to be a fair number of grant supported jobs like this and pay isn’t terrible. Just under or just at 6 figures. You can make more in industry, but scientific work feels very meaningful. Now I’m working in credit risk modeling, something I never expected to be doing, but so far it’s been a good fit. The models are applied directly in decision making for the business, and there’s a real incentive to get everything right because mistakes could harm real people’s lives. The team I’m on is strong and ethically sound and I feel good about what I’m doing. For anyone in the data space who’s despairing about the state of the industry, non-bullshit jobs do exist, you just have to look for them, and use your judgment when scoping out new roles.
  • pram 4y ago
    I’ve been doing “big data” in the retail space for almost a decade now and it’s pretty clear to me how it affects the bottom line. It really can be as simple as “we need more of this milk” and “the spicy queso is very popular”
    • sebastiansm 4y ago
      I'll really would like that the reatailer prepared for me my montlhy basket based on my previous and recurrent consumption.
      • ryanianian 4y ago
        And if stores had meals pre-shopped in bags you could just remove unwanted items from. Grocery stores seem ripe for simple optimizations even without big data
• j7ake 4y ago
  Why not try to do data work in science? Analysing astronomy data, particle physics data, or DNA sequences sounds high impact, and you have world experts on the data to make sure you’re getting quality curated data.
  • ludicity 4y ago
    Absolutely an option! I've got my own strategy out, but it seemed self-absorbed to go on about it. People don't want to hear about me specifically beyond whatever minimum lived experience is required to make my point. Plus it hasn't panned out yet.
• deleted 4y ago
  [deleted]
• taylorius 4y ago
  I suspect you could remove the word "Data" from that title sentence and it will still ring true to many people.
  • duderific 4y ago
    Maybe replace "Data" with "Information Technology". Most people don't work in information technology, and occupations like teacher, construction, health care worker, lawyer etc. are certainly not worthless.
• vitehozonage 4y ago
  Worthless would be great. I struggle to find tech jobs, especially in data, that aren't worse than worthless - i.e. actively making the world worse.
  • maxlamb 4y ago
    Thanks for the chuckle.
• jawadch93 4y ago
  [dead]
• black_13 4y ago
  [dead]
• KingLancelot 4y ago
  [dead]
• Tade0 4y ago
  > It's the despair that comes from knowing that we spend most of our time producing nothing of value. I feel like this is true for IT in general. Projects often fail or don't deliver on their promises. Years of my work either never reached production or were scrapped and re-written because some key requirement changed. For this reason I appreciate time spent with my family - it's always rewarding one way or another.
  • doubled112 4y ago
    I have learned that sometimes it is better to wait work out and see how it plays out. To some I'm sure it looks like laziness, but I consider it more weaponized procrastination. With experience I've become better at it. Sometimes I get that gut feeling about a project, and just can't bring myself to get started. Some basic research happens, but I spend close to zero actual time on it, sticking to other things that matter. In two weeks, priorities have changed and all of what I was asked for has now changed or been cancelled anyway. Glad I didn't get too invested.
• swyx 4y ago
  > The absolutely fucked up thing is that everyone I've met in this space seems to have totally given up on doing anything meaningful at work. The goal is to get paid, not stress out, have a happy office where everyone can collect their strange handout, and not think too deeply about how unfulfilling is it to produce nothing for forty hours a week. this is why i think its kind of funny that people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices. I do think this is a gross misallocation of resources, but insofar as moral judgment goes it's not obviously less moral than many other inequities in society.
  • jayd16 4y ago
    I agree with you, but it's not like you can wave a wand and trade UBI for bullshit jobs. The bullshit will persist.
  • prottog 4y ago
    > people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices Private money being wasted due to inefficiencies at the office (as you say, a misallocation) is surely entirely different than a government-granted right to not produce anything yet still be supported.
    • eptcyka 4y ago
      Ultimately, these systemic inefficiencies are financially supported by people, the public at large.
      • luckylion 4y ago
        At different rates though. You can absolutely choose not to pay the lazy-tax and buy from a different vendor that doesn't employ an army of people who do nothing, you can even build whatever it is you need, but you can't easily choose to not pay the taxes that the government collects for the same purposes.
        • 4y ago
• cushychicken 4y ago
  To rephrase the title a bit: most data work seems fundamentally like research. You look at a bunch of data seeking a nice, clean X/Y relationship that yields positive ROI for the org. These types of situations are rare - that’s the sort of e-commerce landing page / conversion optimization problem where a bunch of other factors (particularly whether anyone needs or wants the widget) are answered already. That doesn’t sound like the nature of these data engineering type roles. Sounds like there’s an explicit “Get us this spreadsheet!” BI-grunt functionality, with the overhead of doing researchy type work that might pay off for the company later. I’d guess that the “get the business unit their spreadsheets” is the actual valuable part of this role, and any other useful insights about the data are just considered gravy by the org.
• jkdll 4y ago
  "There are usually at least a few tasks required for the organisation to function, such as producing some sort of report for the government, that technically does need to happen, so we can't simply lay everyone off." - At highly regulated companies and companies of sufficient size, reporting to upper management, government and external stakeholders is more than just "a few tasks". This, alongside a continuously developing product ("terrible data"), product pivots ("vision is hard") and growing data volumes IS what "most" data work is, and it is not fundamentally worthless. So the premise of the post seems a bit flawed. The author seems a bit jaded by bureaucracy and lack of vision at a few organizations they have worked at, which is unfortunate, but by no means an accurate representation of data work.
  • trompetenaccoun 4y ago
    That's ignoring that much of that government mandated regulation is worthless in itself, not producing the originally intended results and only leading to useless jobs like the author's. The data is adjusted so that it fits the regulatory demand, not the other way around. In many cases it's still pointless work that produces no value for society.
    • ludicity 4y ago
      Yes, I considered raising this point. My work is sometimes necessary to comply with regulation, but frequently that regulation is inherently pointless. That said, this seems to be a different category of worthless. My regular work can literally not be done. This category of work must be done or you risk terrible consequences, but it probably shouldn't be done.
• mritchie712 4y ago
  Startups are a good place to look for data jobs where you can make an impact. It's pretty easy to use data the "right way" if you start early, but it's incredibly hard to change the way data is used at 30+ year old companies or ones that are already hundreds of people large. If you're the first data hire at a company of 10 to 20 (mostly engineers) and the founders seem sharp, there's a good chance you'll be doing things that actually help the business. There's obviously a risk that the company will go under, but if the answer to "would I use this?" is "yes", that's been enough for me to be happy at work.
• davidthewatson 4y ago
  After a decade of referring to "data engineering" by asking candidates, "So you want to be a plumber?", this resonates. My dad wasn't a plumber though he was a member of the steamfitter's union at one point. These are similar problems though they may not look alike at a casual glance: Plumbers make good money, but I'm not sure there's deep meaning in the pipes because, like metadata, pipes go sideways too. I'm having trouble thinking of a Turing award winner who won because they went sideways. The key is that the data insights of a Turing award winner are durable, unlike the transient nature of most of what we do with data in modern organizations. Ultimately, this sounds more like Grothendieck in math than computer science: https://www.psychologytoday.com/intl/articles/201707/the-mad-genius-mystery https://www.psychologytoday.com/intl/articles/201707/the-mad... Metadata sounds like it's the wrong word as the structure we are looking for isn't the ephemeral structure that most of us discover, but rather, it's closer to special relativity. The Turing award winner who comes to mind is Leslie Lamport: https://en.wikipedia.org/wiki/Logical_clock https://en.wikipedia.org/wiki/Logical_clock There may be a better choice that demonstrates the worthlessness paradox of data engineering or metadata or data science in a modern management context. It's not that metadata are actually worthless as in fitness-for-a-particular-purpose. Rather, it's that the insights that we and our organizations reap from that metadata is increasingly short-term - anything but durable. https://plus.maths.org/content/stuff-happens-ordering-history-computer-science https://plus.maths.org/content/stuff-happens-ordering-histor... The only references I can think of on quality are either Pirsig's MOQ: https://en.wikipedia.org/wiki/Robert_M._Pirsig https://en.wikipedia.org/wiki/Robert_M._Pirsig or Wallace's posthumous Pale King: https://en.wikipedia.org/wiki/The_Pale_King https://en.wikipedia.org/wiki/The_Pale_King Your friend's experience working in government sounds exactly like the Pale King to me. The worthlessness you're referring to seems much more dependent on the fact that management culture has evolved to somewhere between a cargo cult and celebrating short-term things that don't count, both illuminating an illusion of control. I mean, does anybody, anywhere still get a gold Rolex and actually retire?
• timeon 4y ago
  This platform looks refreshing.
  • ludicity 4y ago
    It's the only thing I can write on, because it is snappy and I have nothing to focus on other than writing.
• bjornsing 4y ago
  I’ve spent some time in the data science world and can emphasize. I think one reason it feels like this is that you’re relatively close to the power structures of the organization: the CEO says everything should be “data driven” and you are “data”. That makes everything very political. EDIT: And to state the obvious: corporate politics is hugely inefficient and mostly worthless.
  • ludicity 4y ago
    Incredibly insightful take. I need to think about this. It's absolutely the case that a C-suite Acronym person will have an idea, and suddenly the whole organisation is scrambling to pivot it. The reason that I mentioned vision is important is that at one of the roles, we had an extremely competent Chief Data Officer. While I think all their management were basically lying to them actively, they nonetheless managed to plot a clear enough course that the organisation has improved substantially. Is all the work still worthless? Yes, absolutely. Do code review, CI/CD, and cloud enablement all exist? Also, yes, absolutely. If we do this for another 5-10 years, we might actually get something done. The other places though? Absolute chaos around everything where the word data is mentioned.
  • duderific 4y ago
    Corporate politics may be "worthless" but they are the manifestation of human nature in a large organization. Thus, they can't be avoided entirely, other than not joining a corporation at all.
• vslira 4y ago
  If you’re a data professional in a company where data quality is low or non-existent, then improving the quality of data in the organization is absolutely part of your job, most likely the most important one It’s less cool and flashy than the real time online machine learning model you want to build, or the multi-level bayesian model to determine causality between revenue and an obscure event, but that data dictionary and table/db catalog really needs to be built before everything else Then document the essential reports to the extent that a high schooler should be able to produce/maintain them Then adjust the reports so that the metrics indicate what management actually needs to learn from them (if you don’t know what’s important for them and for the business, then finding out is part of the job, too) All these things are extremely important, and it’s silly to suggest your work is meaningless just because it’s not fun or interesting. There are no tv series about civil engineers building sewage networks but public sanitation saves more lives than medicine. Data work works the same
  • mywittyname 4y ago
    > then improving the quality of data in the organization is absolutely part of your job, most likely the most important one You're absolutely right, but it can be hard to get management to bat for this kind of stuff. The biggest hurdle I've seen with getting "good" data is fixing the issues involves other teams prioritizing the work on their backlogs. Depending on the company and where the teams lie on the org chart, fixing things might require a multi-quarter directive from the CTO. Which isn't happening.
    • pas 4y ago
      so then ... confidence intervals are going to be so use as to be useless. and every second week the data science team can present a 2 slide presentation, where the first slide says bad data = bad result, and the second says good data = good results, and then furiously flip between the two during the Q&A. and everybody clapped!
  • trompetenaccoun 4y ago
    That isn't possible most of the time working for a larger company. Too often the problem with bad data starts not at the collection stage but the data is bunk to begin with.
• datavirtue 4y ago
  The only decent data job I ever had was analyzing time series data from war planes. I say decent because we had the data needed and clear goals about what we needed to achieve. Something real and tangible. Then it turns out I'm directly helping war criminals (Saudi Arabia) bomb mothers and their children. Had to hit "eject" as soon as possible.
  • pilotneko 4y ago
    I mean, you wrote code for a weapon system. Did you think only “bad guys” were going to get killed?
    • datavirtue 4y ago
      Weapon systems are banned at GE. It takes an army of contractors in the private sector to keep fighter planes working (any modern military hardware, really). Operate them in a sandpit and they last about one day. Unexpected failures routinely occur and numerous data scientists, data engineers, and aviation engineers get spun up to figure out what happened ASAP. No, I did not expect to be helping warlords. The US is stuck in a really super-shitty policy with the Saudis right now. Contractors (defense firms) have zero choice in supporting them unless they are sanctioned. Meanwhile the Saudis have been acting unilaterally (bombing whomever without consulting the President) for about fifteen years now.
• apienx 4y ago
  > "The institution produced a fair amount of academic research, but the vast majority of it was essentially fraudulent." Wow!
  • trompetenaccoun 4y ago
    Sadly that seems to be the norm. It stems from a culture where "research" gets abused to support preconceived ideologies. It should be the exact opposite way, organizations forming their ideology based on proper research.
    • p0pcult 4y ago
      This was true at my private, for-profit employer too. I have since changed jobs, as I was getting severely depressed from having to continually fight with my boss (the Chief Research Officer) about being unwilling to "fudge" results.
• hobs 4y ago
  Most software engineering work seems fundamentally worthless (and it is)
  • starbugs 4y ago
    How is that?
    • hobs 4y ago
      A lot of software is just like a lot of white collar work - just bullshit moving stuff around. The key word is most - A good example is many enterprise software companies. They sell a product made to fit a purpose that someone in power thinks it should, but doesn't qualify whether or not that's the case. The customer is sold the software as an appeal to their ego and supposed needs that they might spend millions of dollars on. The people on the ground have the software "purchased" for them and they basically are told "it's this or the highway" quarterly releases often leave people head scratching why anyone would give a shit. Big parties are thrown to celebrate implementations that even in the medium term have no utility and you would been better off spending it on hookers and blow. There's so much of the industry that's just running on a treadmill.
    • woeirua 4y ago
      The way I look at it, is that if your job disappeared overnight would anyone even notice? A great example here is Twitter. Elon fired (by conservative estimates) over 2/3 of their total staff. Yet the service runs just as it did before. What were those people doing? Did their work matter at all?
• staunton 4y ago
  I'm somewhat surprised at the level of cynicism here. "Data work" is basically science: trying to understand things empirically, building models, testing them, using them to achieve goals. That should explain it. Yes, most of the time spent doing science is necessarily a waste of time. You can't just "go and do" something if you don't know how or what it is you want to do in the first place. That's very different though from saying that one shouldn't do science because of that. A cynic may accept this and maintain the position that a lot of "data work" shouldn't be done, that it only plays a political role and isn't aimed at understanding in the first place, etc. However, that's a fully general argument against ever trying to understand or improve anything. It doesn't hold in general, only points at ways in which the system may be improved. You may complain about lawyers, call them goons doing "bullshit jobs", but I strongly suspect you would stop complaining about the legal system if the thugs came to beat you up every weak you didn't pay them protection money.
• OliverJones 4y ago
  I've spent a good chunk of a half-century-long career, now winding down, doing the kind of work our author describes. Most of it was on my own initiative. I'd notice some kind of anomaly in the logs of whatever app I was working on, investigate it, and generate a report program highlighting it. Most of the time people ignored this stuff. That took some getting used to. But occasionally one of these little report programs would show a solvable problem. The usual example was a whole bunch of customers using the app in some way unpredicted by product designers and developers, and so missing out on a valuable feature of the app. Often we were able to add some kind of workflow feature, or add a UI afffordance to help customers take advantage of existing features. (Other times the little reports detected performance trouble. Always consider using time-of-week as an x-axis when working on these little reports.) The hard thing about doing all this is that I, and my colleagues, never knew ahead of time which data anomalies would be actionable and which were just fun facts to know and tell. Sometime along the way, that part of my work was dubbed "data science". That's about the same time a bunch of enterprisey software entrepreneurs discovered that "dashboards" generate sales because they appeal to front-office folks with control over money. Irony: I developed a bunch of integrations for a really expensive data-analysis product using Jetbrains tools I paid for personally. I always thought of that part of my jobs as diagnostic and exploratory. What can we learn from how this system works in the real world? In my case, of course, I also had some responsibility for the systems generating the data I analyzed. My advice to others doing this: 1. Always always assume you'll be called on to generate recurring reports with your little report programs. Make or buy a report-generating tool that can, at a minimum, deliver CSV files by email. 1. Indulge your curiosity. Especially with strange and incomplete data sets. Don't think of your task as "generating a report from bogus data for people who don't give a s**". Think of it as "figuring out how to make sense of the process from the data it captures as it operates". 2. Learn all you can about the processes you're analyzing, be they failed logins to SaaS apps or ambulance-calls to wrong addresses, or whatever. 2. When you have nothing much left to learn at a particular employer, teach somebody else how to do your job and then move on.
• token8791 4y ago
  My data work saved one of my clients in one of the project probably about 2M for the price of <500k - and enabled revenue streams of a few M. Seeing a bunch of opportunities of similar proportions. It's ugly work, but my theory is that this it why it pays.
• now__what 4y ago
  My SO and I both work in data (in healthcare and higher education, respectively). Neither of us has had this experience. Sure, some of our dashboards have died, but many of them have also provided real value. My SO's have literally saved lives! We each benefit from having direct contact with senior leadership in our orgs, and good understanding of the business side of our industries. Other than that, I'm not sure what advice I can give to avoid this problem, but I'd like to assert that the worthlessness of data is not a universal experience.
• Damogran6 4y ago
  I'm 53 and have been in the IT Security space since before there WAS an IT security space. I have installed 6 SIEMs, just put a bullet in SIEM number 5 (oddly the first time I'd ben in an organization long enough to both install and uninstall one.) and the Current generation has an on-prem and off-prem installation due to data sensitivity...we're re-doing the on-prem installation for reasons...so I'm kinda at SIEM 6.5 of the first 5...NONE of them are still in existance. I recognize the circle of life nature of my job and have gradually changed my raison de etre from 'Protect the company!' to 'Advance my career!' to 'Pay off my mortgage.' It was a bitter pill when I realized my job was 'look at magnetic patterns on a spinning disk on a computer half a world away and determine if they were GOOD patterns (company data) or BAD Patterns (bad guys)'.
• fedeb95 4y ago
  with a six figures job and leaving it at 5PM, you still have a lot of resources most people don't to make something impactful. I think you're just trying to justify your inability to do something meaningful with other people bad decisions. Sorry if I seem harsh, but that's how I see it.
  • ludicity 4y ago
    I thought this for a while too. Nothing wrong with calling it like you see it. Suffice it to say that I've seen a lot of people try (and even succeed partially) to improve things, but all of them would be producing stuff ten times better if they struck out on their own or found a better organisation. In that brief government stint, I realized there are very good reasons that a few well-meaning and smart people haven't just fixed the government. It is a horrendously difficult problem.
    • fedeb95 4y ago
      ethics is probably the hardest problem out there
• cfeduke 4y ago
  I worked as the technical half of in a sales organization selling a big data solution so I've caught a whiff of what the blog post is talking about. A handful of organizations interested in/using our solution were doing, what I gather, meaningful work. A larger number - my guess, being an outsider and I could be totally wrong, but just based on signals I caught from the people at these organizations I was working with - were not. The ones which were not were those places where there's some sort of data arm or data scientist team with no clear mission and were more than willing to sink hundreds of thousands to millions of dollars into big data solutions on a short sales cycle. The industries where I felt that data was being used tended to have data as their core competency. The companies I worked with that were probably doing something useful were in the pharmaceutical, gaming, and data provider spaces. (Okay, so gaming space core competency is not big data, but their data team handled everything associated with microtransactions and other sales.) I also worked for a short time at a PAC-adjacent company working with voter and demographic data. Because that company's core competency was data, the work was meaningful - but it is extremely important that if you do that sort of work your politics match.
• penguinvondoom 4y ago
  OMG, pretty much a lot of what I have personally been through. But the problem is deeper and goes beyond a company not knowing what to do with someone that does ML/Data Science ... Most of software that comes out either does not contribute to society at all or is in some form harmful to it, usually due to the need for such things to be somehow profitable. And pretty much all the senior devs I know are burned out on this in one form or another and want to quit and have a farm/bakery or whatever that actually does something
• shaftoe444 4y ago
  This sounds like most of my dev jobs to be quite honest with you.
• ghelmer 4y ago
  "The more things change, the more they stay the same" - Jean-Baptiste Alphonse Karr In the late 1980s, "executive dashboards" were all the rage. Vendors were selling tools to tap data in mainframe databases to provide insights directly to execs to improve business knowledge. It generally felt like a massive waste of resources. (But, in that timeframe, I did build a tool to scrape data from a mainframe DB and turn it into useful statistics on a weekly basis at a big organization. Not sure how long it lasted, though, because my boss required that the reporting interface be implemented in his favorite tool, Lotus 1-2-3.) (edited for spelling)
• photochemsyn 4y ago
  In scientific research, data mining as it's apparently often practiced in the private sector isn't a thing - there's study design, there's controls, there's a whole system of how to go about collecting data that's been designed in advance so that datasets are collected that can be used to answer the questions of interest. Garbage in, garbage out, that's worth remembering.
  • crabbone 4y ago
    Here's an example from my wife: they are working in cooperation with some hospital. They were tasked with extracting breathing information from the EKGs. (This could be helpful in detecting when to take patients off the ventilator). Worked for half a year with some previously acquired data. The hospital part of the team aren't particularly tech-savvy and didn't participate in the research as much. But once they actually had to produce current EKG data, it turned out that the sampling frequency of their equipment is half of the minimum required in order to detect breathing. Now the research is worth nothing. Another similar story where I got tangentially involved: my wife had to reproduce a notebook of some guy who wrote a thesis on detecting something in EEG signals. After a lot of digging and research into that guy's work, I discovered that the format they used to acquire EEG signal in could have either 32 or 16 bits per "event", the event streams were not contiguous, but per sampling per electrode attached to the scalp. I.e. physically, they were written as event1_electorde1, event1_electrode2, ... event1_electrodeN, ... event2_electrode1, ... etc. The original research author didn't understand this aspect, and the data he used was generated by different EEG machines in different format. He used some framework function that tried to guess how many bits per signal were used, and when it failed to guess, it'd print a warning. He simply silenced the warnings. And that's how he ran his ML models, and that's how he submitted his thesis. He has a graduate degree now and had moved to a different country. His work is worthless.
• crabbone 4y ago
  Very early in my programming career I still couldn't decide whether I want to do programming or graphic design. I worked for an online game written in ActionScript (Flash) and was designing armors and some other in-game objects. I also participated in the game's forum on behalf of game's admins where they collected and discussed player's requests and suggestions for in-game new features. My ICQ number was thus available to the wide audience as I was sometimes the focal point of player's complaints or requests. One day a player I didn't know personally contacted me on ICQ, and his avatar featured a piece of headgear I designed. I had tears standing in my eyes knowing that someone valued my work so much they actually decided to use it to represent their online identity. Never since have I been so close to the situation that whatever I did mattered. My wife works for the government (also in IT sector), but she has better job satisfaction than I have in the sense of knowing that whatever she does has any impact at all. In my case, I'm automating testing and deployment of a piece of software that's essentially exclusive in its market niche. So much so that customers have no idea how bad it actually is, since they have nothing to compare it to. The testing of the software I work on is outright worthless, so whether it's more automated or not -- it will have no impact on the outcome. Not to mention that the biggest problems with the software are in its design, in the decisions that went towards the core elements of it, that, at this point, nobody dears even to question, let alone to reverse them. So, it sucks. It's going to suck. And it's going to suck for a very long time in the very same way because no replacement is coming. But, hey, I get a visa! And a permanent contract! Yee-haw!
  • bombcar 4y ago
    Sometimes being able to make something suck just a bit less can be powerful, and in ways you may never see.
    • ludicity 4y ago
      Very true! The times when I've been happy at work have frequently been when I've failed at basically everything in my job description for the week, but managed to write up a script that automates some horrific process that a colleague I like was forced to do. Would the organisation actually keep me hired if they knew that was my main output? Absolutely not. Did it make me feel great? For sure! Taking note of those small moments has been helpful in working out exactly what I (and many others) find distasteful about office work.
• f0ld 4y ago
  Basically all engineering job is kind of... pointless circa 2011 if you belong to working class. The novelty value of engineering is gone. We see where it's all going now. We won't enjoy the fruit we help grow. The protest of 1968 was right about everything. The boomer We're here to repeat the same mistake. Modernism and endless worship for overproduction. Overproduction of goods led humanity into two world wars and now we have overproduced information on top of all that. What's even going to happen? What's this all for if we can't stop the climate change anyway? But don't lose hope. Ai is the last hope sort of... or the worst thing.
• timrobinson33 4y ago
  I'm a developer (age 50+) rather than a data scientist and I've been lucky enough to enjoy all my jobs. Partly I put this down to working at small companies where you have the autonomy to get things done without too much bureaucracy. However if I was put in a situation where I was working in a large organization and felt the job wasn't making the most of my talent, I would do 2 things: (a) take my best guess of what I could do that was in my power that would have the greatest impact on the bottom line, and spend as much time as possible doing that (this might not have much relation to my job description) (b) go as far up the management chain as necessary to try and get someone to understand what I was doing and why. This maybe a naive and optimistic approach but I'd like to think that after you've tried this with a few different employers you'd find somewhere you were properly appreciated.
• alexpetralia 4y ago
  Often, the job of a data team is not directly to produce value. It is to support value creation through observability. Observability gives the business side intuition and knowledge about how the business works. The fact is, no one in the organization can get visibility at scale into their operations without a data team. Now, I have argued before that data teams can and should also focus converting knowledge into action. This moves data teams out of a passive, reactive role into a proactive, value-creating role.
• benjaminwootton 4y ago
  I don’t think as an industry we’ve actually solved the problem and started to deliver on the potential of data yet. Most companies are hacking around with messy stale data, excel sheets, dashboards and doing rudimentary analysis. This is essential level zero on the maturity scale, it’s like rubbing sticks together to try and spark a fire. Nowadays we know what good looks like. Data engineering, online machine learning, collaborative analytics environments, embedded analytics, streaming etc. We just need to get there. My impression however is that the people at end clients are a bit stuck in a local optima. They think ETL, data warehouses and dashboards when what they actually need a total rethink and change in approach.
• jackmott42 4y ago
  I can't speak to "data" jobs in particular, but I have had programming jobs in my career where I felt like the OP, and ones where I didn't. If it is important to you that your work be valuable in some way, I suggest you not give up. Do what you can to find work that is. It is out there, maybe you have to better yourself to raise the odds, so try to do that if you can.
• louwrentius 4y ago
  This is the best thing I read in months. People should read David Graeber’s (in memoriam) Bullshit Jobs. People are great at fooling themselves but on the other hand we often instinctively know when our jobs are bullshit. And I think 30%-50% of IT jobs, it being operations, software development or supporting roles, it doesn’t matter. It’s a combination of incompetence of people and the incentives of capitalism.
  • ludicity 4y ago
    Thank you! And when pressed, almost everyone I've spoken to that hasn't moved into management (if you ask management, you're asking for people who have largely self-selected into delusion, at least at these orgs that are struggling), has admitted their work is useless. I met an engineer considerably more senior than myself recently, who said something along the lines of "A reckoning is coming". I think they have a lot more faith than I do in these ultra-bloated organisations trimming fat. The funny thing is, we both think that we're the people who should get trimmed, not for lack of trying, but because our best efforts have yielded nothing. I keep getting promoted, but if you look at my track record, almost everyone I've ever worked with including the super smart and diligent, are abject failures.
    • louwrentius 4y ago
      Yes, I agree 100%. The reckoning isn’t really coming because too many people in positions of power - who often care about headcount - and so many people don’t have the insight / knowledge that the IT team can be 50% of what it currently is.
• pjdesno 4y ago
  Someone I know works on catastrophe modeling for the insurance industry. It seems like a fascinating area, especially since there's a strong link with climate science - a large part of the business seems to be modeling hurricane risk in Florida and wildfire risk in California. The complexity is because these are fairly rare, highly correlated risks - unlike e.g. auto insurance you can't just take a few years worth of claim data and predict your losses for next year, unless you want to go insolvent the next time a sufficiently big hurricane goes through a sufficiently prosperous part of Florida.
• d23 4y ago
  > Piles of money + unclear outcomes = every grifter under the sun begins to migrate to your organisation. It is very hard to keep them all out, and they naturally begin to let other grifters in because they all run interference for each other. Sure, they might betray each other constantly, but they won't challenge the social fiction that some sort of meaningful work is happening. One of the best summaries I’ve ever read about this phenomenon.
• college_physics 4y ago
  "Data science" related work is absolutely fundamental for fixing many of the world's sustainability problems and also creating conditions for more enjoyable and fulfilled lives. Alas, we have political and economic systems that misallocate human talent at massive scale. This is not new or unusual. Heck, human life itself was never worth much when it came to power struggles between various oligarchies. People with a gift to work productively at the interface between digital machines and society should feel privileged and act on it rather than lament about bullshit jobs.
• angry_moose 4y ago
  The CTO of my previous company was fond of saying something like: > According to [some book he read once], the average company stands to increase their profits by [50%?] by applying data science. Even if [author] is only 10% correct, that's still tens of millions of dollars for us. (some variation on this at every all hands meeting) When I left the company, they were still desperately searching for that first 1% of benefit after nearly a decade of effort. Like most management fads, there is some solid foundational truth there, but it quickly becomes a solution in search of a problem when it gets tossed at every single problem faced.
  • bombcar 4y ago
    The trick is to discover what the current management fad at your work is, and then reword whatever problem you ACTUALLY want to solve as being a case of implementing the fad. I've known of very successful middle managers who had multi-decade projects that were really advancing the actual work of the business but the top-levels thought it was various separate buzzword projects during those years.
  • trilobyte 4y ago
    That's a pretty poor understanding of basic prob & stats. What if you are not an average company? What if you're not an average company and on the low end of that bell curve? If the author is only 10% correct, maybe the author is only 10% right about the type of company that could benefit from "applying data science".
• mywittyname 4y ago
  This is somewhat the reason I switched careers from data science into data engineering. My experience was that most companies weren't mature enough to really make use of any sort of "real" data science. But there was always some data sources sitting around somewhere that could be copied around, cleaned up, then thrown into Looker. After a while, this collection of data sources will turn into a data warehouse, and a network effect will kick in, making looker (or tableau/etc) broadly useful for other people in the company. Most of the teams I work with need simple reports to do their job. There's no need for predictions or self-learning classifications. Mostly things like, is the information in these three systems consistent? If not, can an alert be sent to notify someone? That being said, even my current company has 10x as many DSes as DEs and I'm pretty confident that all of them are like the author of this article. I can't blame them for taking a better paying, far more prestigious job with several times as many openings in the industry.
  • datatrashfire 4y ago
    To be fair, as a DE I feel no different than the post author. I build pipes that might as well be going to /dev/null
• MrPowers 4y ago
  I've been working in the data space for 8 years and find it incredibly fulfilling. Open source community is great (I've built and contributed to a lot of projects), LinkedIn has tons of great data content, meetups are fun. The work is interesting and the problems are important. It's funny cause the parts the author is mentioning are easy are the hard parts for me. The author says "The pressure is non-existent"... That hasn't been the case for me. Deadlines are tight, jobs fail, running jobs overnight, checking if they're still running before going to bed - it's not great for a restful sleep. I've always given talked, created new open source libraries, and blogged whenever my day job gets a little boring. It's such a fast growing, exciting field. I feel like it's hard to keep up, let alone get bored!
  • ludicity 4y ago
    That's the funny thing - as I mentioned, it's easy to feel bad complaining because these jobs were objectively quite cushy. But you're absolutely right, giving these things up and doing something that matters entails actual stress. I'm sure that someone else could write an article talking about how deeply they wish they had work that doesn't matter, and the absurd thing about human experience is that they wouldn't be wrong at all. I just started reading some of the sites linked on your profile and really enjoy them, by the way! Really wish it had turned up on Google the last time I was working with Databricks.
• icapybara 4y ago
  > The institution produced a fair amount of academic research, but the vast majority of it was essentially fraudulent. We had some contracts with industry to produce technology in the space, but these mostly seemed to consist of the organisation fumbling the projects and massaging the truth (read: lying) until they could collect more funding. > Piles of money + unclear outcomes = every grifter under the sun begins to migrate to your organisation. It is very hard to keep them all out, and they naturally begin to let other grifters in because they all run interference for each other. Wow, this is very honest, which is refreshing in today's world. Thanks for the post OP.
• cratermoon 4y ago
  I wonder if OP has considered the possibility that the work produced is not valued because the target audience doesn't want the work to be of value? If the ML/data science work is good, then it will provide guidance for making decisions about the business to management. If it's great, it will make predictions better than the people in management. Pretty soon, someone will realize that if the data model too good, there'll be no need for layers of management. The people actually running the company will just use the reports to steer and dispense with all the supporting analytics people. It's no wonder OP's output is mostly ignored except for signaling. The people can pretend to look at, but still make decisions and recommendations based on their gut instinct, common sense, or whatever squishy justification they want, and make themselves look necessary. If the outcomes are bad, they can just blame The Model. So the moral panic about AI putting people out of jobs isn't coming from the people working on the line, it's coming from middle to upper management realizing that their role is absolutely useless when data science can make data-driven managements become real. Who'll be left? The top 1%, as usual, extracting value from data and the people who create it.
• irrational 4y ago
  I don't really understand the despair. The only thing I can think of is they somehow have their self-worth tied up in their work. For me, work is merely a way to earn money for those things that actually matter to me - family, hobbies, volunteer work, etc. My actual work has no real benefit to society. As long as they keep paying me, I don't really care. That's on them to want me to work on things that fundamentally don't matter.
• flappyeagle 4y ago
  Most science is fundamentally worthless. We're not accustomed to that in a business setting. 99% of science yields no interesting result.
• Mizoguchi 4y ago
  This is typical of the public sector, lots of money from tax payers and over spending with no accountability. No wonder why we can't keep our debt under control. I have many friends and acquaintances working for the federal government in different sectors, from DoD, to education, housing and health and the stories are similar, outrageous budgets that go to waste and practices that sound borderline ilegal. There's no common goal or long term vision and everyone is aiming at their promotion to the next pay bracket. For service providers there's also no incentive to be efficient and deliver the best results because public contracts are not necessarily assigned to the most capable organizations, all you need is the right connections and once you are in you keep getting projects.
• woeirua 4y ago
  I disagree with the assertion that quality and meaning are somehow related. Paradoxically, it is often the worst code that is the most meaningful in terms of its impact to the customer and business.
• taeric 4y ago
  Worse than the ones that are fundamentally worthless, are the ones that seem worthwhile until you turn them off and find that nothing changed. Which is far more of them than you'd expect.
• Awelton 4y ago
  I had a job like that once and I had to bail. It was essentially free money but I could barely make myself get up in the morning. I despise busy work, and if I am spending time making things that nobody is using then busy work is exactly what it is. Either give me something useful to do or give me the day off.
  • ludicity 4y ago
    Amen, it's so rough. I couldn't understand why my health deteriorated over Covid, because surely not having to really do any work from 9-5 was a good thing! Turns out it's like pumping a mild poison straight into your brain over several months.
• lgbalik 4y ago
  Most of data work is just middle management paperwork at the end of the day. The incremental value of the nth dashboard, nth model, nth analysis is decayed very quickly. What's worse is the demand side for "data stuff" is largely driven by ad spend and companies trying to get alpha on beating ad spend. So you have all this venture money getting burned up by ads, then burned up by all the headcount and compute of "data teams" trying to beat ads. Some people in the thread have pointed out that this is like UBI. I think it's actually more socially reckless and far more sinister than UBI. Given that VC funds are largely pension funds, essentially the pooled money of the middle class is what is subsidizing $200k+ salary analytics engineers, data scientists, as well as the $200k+ "influencers" selling stuff to the data engineers and scientists and analytics engineers and whatever other invented titles VC comes up with to get 2% management fees off laundering this pension fund money and university endowments into Snowflake credits so that to make "insights" can be made for middle management. I write about these things here and there. https://medium.com/p/bfd4b4e33ac7 https://medium.com/p/bfd4b4e33ac7 I also did a podcast recently where we covered how pension money goes to Sequoia and how that resulted in a Reverse ETL vendor sending me a panini press bribe and how panini press GTM tactics are unlikely to result in returns back to grandma's pension funds. You might be putting grandma on the street by participating in this whole charade. https://www.youtube.com/watch?v=yInWs4NUs0E&t=233s https://www.youtube.com/watch?v=yInWs4NUs0E&t=233s It's too much nonsense in the data world. It's enough to make you completely insane.
• sage76 4y ago
  I have to ask, is it normal to join a startup that claims AI powers their main product, and then join the company only to realize they aren't using anything remotely resembling AI? Like expecting CNNs and vision algorithms, and finding out they are using off the shelf OCR software? So their core product itself is bought off the shelf?
  • naijaboiler 4y ago
    not uncommon
  • ludicity 4y ago
    One of the first jobs I was offered was for a startup in SEA, which I didn't mention in the article. It was an AI powered trading platform, and they revealed in the interview that they were going to launch without AI and just do all the trades manually. A few friends have seen similar, but less well-funded, ventures do similar things. So yes, absolutely common.
  • dogleash 4y ago
    Yes. It is conventional wisdom to assume that all startups are lying though their teeth. Claims about what they've accomplished are understood to be what they hope to achieve eventually.
• chubot 4y ago
  Reminds me of "Goodbye, Data Science" from a few months ago: https://news.ycombinator.com/item?id=33787270 https://news.ycombinator.com/item?id=33787270 The main reason I soured on data science is that the work felt like it didn’t matter, in multiple senses of the words “didn’t matter”: https://ryxcommar.com/2022/11/27/goodbye-data-science/ https://ryxcommar.com/2022/11/27/goodbye-data-science/ For part of my career over 10 years ago, I worked in data science for big tech, and I can definitely see the hazards ... It is extremely easy to get hoodwinked into joining a useless data science team, one that is mere signaling for executives, ineffective, ignored, or otherwise
• wiercin 4y ago
  I know of no better life purpose than to perish in attempting the great and the impossible.
• thatguymike 4y ago
  Places where data is the product are categorically different experiences to places where data is ancillary. Having a direct link to revenue or cost makes your team and your work self justifying, which is good for meaning, for influencing decisions, and for job security. It doesn't even have to be the product of the whole company - a financial risk org is a good example. If you've been brought in as a Data Whizzkid to do Cool Data Stuff, then your job is Sales. Or you could call it consulting, but that's just Sales too. You have double the work because you need to constantly be identifying good problems to solve; solving the problems; AND convincing people that they should adjust what they're doing to use yoru work, which is always an uphill battle. My guess is that OP enjoys #2 but hasn't done enough of #1 or #3. Which is possibly not their fault: for reasons others have outlined, many businesses are set up to make #1 and #3 impossible. My anecdotal and situational advice: if you don't like doing the sales work, do everything you can to turn your job from Consulting DS to Product DS. Find the place where your insights are most valuable to sales or revenue, and productize it -- make it repeatable, build processes in other teams to operationalize it, automate those processes, measure the outcomes, repeat. If that's not possible then accept that you're in Sales, and that's ok, even if you need to spend way more of your time on sales than "the real work". Organizations are _hard_, arguably the hardest problem we face as humanity right now. We aren't entitled to them being perfect, we're all creators of friction as well as victims ("you are traffic"), and if changing processes and agitating is what's needed to make your work valuable then that's what you actually got hired to do - regardless of what your job description says. (Not trying to say it's possible or worth it in in all or even most situations like the ones OP describes, but it's worth a look before becoming resigned or quitting.)
• justsocrateasin 4y ago
  I think you can say the same about most professions that are "big picture" ("individual picture" professions meaning teachers, doctors, EMS; folks that help individuals rather than larger trends). The reason here is similar to why science oftentimes appears fundamentally worthless - most of the time you uncover absolutely nothing of importance. I think the same is true for data organizations. When you're looking for large patterns in the data you're going to strike out a lot. And that's not on an individual level, but on an org level - a lot of teams within the broader data organization will never produce anything of value. But at least they'll try. With respect to data quality - fix it then! That seems like an immediate way to make your work less "fundamentally worthless". I understand that fixing data quality is hard (I spent >12 months working on that problem at my current company), but it is not a worthless endeavor.
  • ludicity 4y ago
    Depends massively on the company! I am indeed one of the nerds that goes to bat for fixing data quality issues, but it takes long enough that I frequently look at these projects and think: "It might take the majority of my working life to help this one company migrate this terrible enterprise system to a much better enterprise system. All my time will be spent in meetings, and educating hostile people who just want to do the work faster. Is that something I'd be proud of at the end of my career?" And the answer is, to some degree! But I think we'd all like to work with people who get it or our friends, and if we're not there yet, it's really a question of whether we're at a position in our lives where we can search or prioritize other life improvements.
• kanyethegreat 4y ago
  loved this. when you say > They also can't code - the degree to which my teams sucked was basically directly correlated to how good my manager's people skills were _and_ whether they had programming experience do you mean your manager knowing how to code correlated to a good or bad team?
  • ludicity 4y ago
    Good question, I should have been clearer. I've heard that managers who can program can do some horrific micromanagement. To be clearer, when I've had a manager who both had good people skills and could program, life was pretty good overall. Good people skills alone was still okay in some situations, especially if they at least learned the basics of what our tech stack did conceptually. I've never had a manager who could program but had awful people skills, and from the stories, that's for the best.
• ilamont 4y ago
  The second was my barber. He's an immigrant here, and started a very successful business. Unlike the young tech entrepreneurs I typically meet (who glorify money and the wealthy in a way I find grotesque), he's a very quiet man who has incredibly happy employees and remembers every customer by name. Refreshing to hear this. For a long time, there was a persistent view in tech startup land that anything not designed to scale fast and cash out 10x for investors was fundamentally worthless, and was typically disparaged by the VC class as a mere "lifestyle business."
• TrackerFF 4y ago
  I've worked as a data analyst for a regulatory agency, and while the pay was lower than many "bullshit" data jobs, the tasks work itself actually resulted in real-world results. Then again, you can't play fast and loose when your work results in people potentially losing business, getting hefty fines, and criminal persecution. You have to approach it with the utmost respect, and have solid scientific integrity.
• nuancebydefault 4y ago
  Let's take one step back and have a look. The main cause of the output produced being bad is... bad input. Find a data science job where you are part of producing that input by gathering raw data yourself or at least have an impact on how it gets measured. The first datasets will be garbage but at least you have a feedback loop that tells you how bad, in what way the data could be wrong. You can bring in theories, read papers,... to improve (the measurements of) the data. You will see gradually your data starting to make sense. Wouldn't that be fulfilling?
• miguelazo 4y ago
  A lot of the problem has to do with the entire approach of these organizations: tinkering at the margins of neoliberalism. Very little work gets at the real problems of our political economy. But that’s by design.
• thedudeabides5 4y ago
  most data work is fundamentally worthless future knowledge workers are going to look back on the 80s-20s the same way we look at miners using hand pickaxe and shovels
• pphysch 4y ago
  If your product (data, dashboards) has no users, then it has no flaws, and it is perfect. Put in the absolute bare minimum of effort and move on to more interesting things.
• alfalfasprout 4y ago
  Unfortunately this post is basically the author just showing they're young and out of touch. Here's the thing-- some data work is fundamentally worthless. Yes, some companies are a total mess when it comes to their data. Some roles are tasked with projects that are... to put it bluntly, a waste of time. But it's your job to be improving the status quo rather than wringing your hands and declaring it all pointless. This is frankly where the difference lies between someone more junior and someone that can be an effective engineering leader-- they're looking to constantly improve. Yeah, that means making a case to leadership sometimes. Being an effective engineer (software/data/ML) ultimately requires strong people skills.
  • lolinder 4y ago
    > Being an effective engineer (software/data/ML) ultimately requires strong people skills. This is something I wish people had emphasized to me earlier in my career. It doesn't come to me naturally, but it's essential, and I wish I had started building these skills sooner. Writing code (or analyzing data) is what sets your job apart from other jobs, but it's not where you provide the most value. You provide the most value at the intersection between your specialty and what the rest of the business is doing, and in order to provide value at that intersection you have to be able to understand what people need, adapt your work to those needs, and then influence them to adopt your (now actually helpful) work. If you can't do that then your work is all happening in an ivory tower.
    • alfalfasprout 4y ago
      Yep. And this is ultimately, IMO, what separates a "programmer" from an "engineer". Same could be said in the DS space I suppose. At a certain point in your career the easy part becomes the coding. Anyone can learn to code... it's really not that hard. It's understanding best practices, keeping up to date with the state of the art, knowing how to solve complex problems (not in the sense that the problem is necessarily novel but in the sense that you must do so pragmatically with existing data models, infrastructure in place, and legacy crap), and how to work with stakeholders both above and below you that pose the greatest challenges.
  • 4y ago
• ttr2021 4y ago
  I joined a scale up in the BNPL space as head of data engineering. It was one of the most poorly managed IT environments I'd ever seen. There was so much technical debt, that basic systems where literally dead or dying (day 3, Tableau stopped working), and while I did my absolute best to address it, the constant overarching priorities were always more data, and faster data. Reliability, security, performance was my problem to deal with. (As well as literally anything technical, yes even configuration of mail clients). The demands for solutions came thick and fast from the head of data science and where always clear as mud, yet I was pushed to the wall on immediate and unquestioning commitment for data projects and was held accountable for dates regardless of the resources, capacity, dependencies or technical debt we were fighting on daily basis. Attempts to push back on any commitments or seeking clarification or coming back with a different solution were often met with my manager telling me, publicly in meetings, that I was going against company principles. At other times the head of data science would get extremely angry and storm out. I ended up having to track our team's time, and I remember at one point in time in a quarterly planning session (that they had just transitioned to ) we could actually see how under resourced we where because based on our estimates (backed by data, tracked meticulously by time) of projects they expected in that quarter we could justify a 10x increase in our resources. This apparently shocked them and us so much, they ( my manager and some other data representative) penned a pretty crazy letter to the CEO basically saying all the work the team had been doing for the last X years (since I joined) was wrong and bad and etc etc... completely unfounded. I only found out about it because the CTO raised some eyebrows on some of the claims. I ripped it to shreds pretty much questioned every claim with counter examples, and they had to throw it away. At that point I quit.
  • mcmcmc 4y ago
    A BNPL company demanding instant gratification and ignoring the cost of debt? How ironic
• BiteCode_dev 4y ago
  Most software is. I'd say 1% of what I've been paid to code was useful to humanity.
• low_tech_love 4y ago
  Maybe it's good to also notice that the author is mentioning "data" but talking specifically about doing PowerBI dashboards, reports, etc. Which means that they are working with "data" in its least objective and more bullshit-prone use case (I'm a visualization researcher and I can tell you, it is 99% bullshit). One could possibly cut the fat and turn into machine learning, but that'll also require a lot of extra training and a thick skin to survive in what's probably the most competitive IT market right now.
• ChrisMarshallNY 4y ago
  I know that everyone is getting into the nitty-gritty of the Data Science vocation, but I (a non-Data Scientist) see this as someone that is fundamentally dissatisfied by their career, and that is something that many of us can relate to. It's certainly something that happened to me, and getting laid off, and then shunned by the industry, ended up being one of the best things that ever happened to me.
• blakeburch 4y ago
  This is the dirty secret of the industry. Companies hire data roles without knowing what to do with them. It's not that data work is worthless. It's that most people aren't very good at trying to figure out how to tie data to actionable events that drive business value. Bad news - It's not the business' fault - it's your fault. If you're a data practitioner and you feel this way about your job, learn the business and seek to find opportunities. If you just do the work you're told to do, you probably won't magically create value. Good news - There's a lot of opportunity to help educate data practitioners how to think about data through the lens of business problems. I'm personally trying to tackle it through developing an online course.
• jrochkind1 4y ago
  Definitely not just data work either.
• tranchms 4y ago
  I’m supporting a $30B companies ERP migration to S4 as a Client Partner providing project support and this resonates. Two years in, the program is so f’ed, morale is so low, the project plan and execution is so terrible, it’s inconceivable how a company can exist with so much pervasive dysfunction. Go live is May. No chance that will happen. So many defects they can’t keep track. No business processing mapping or desktop procedures beyond L1. No data mapping. No requirements were ever submitted. Still processing change requests in UAT. Controls are laughable and do not meet PCAOBs criteria for completeness and accuracy. Still missing and adding reports. And a dozen other concurrent technology implementations happening in parallel with interdependent integrations that depend on this go-live, with the legacy platforms discontinuing by end of 2023 that will leave the business incapable of performing critical business processes. None of my consultants want to be on the project. They think it’s professional suicide to be apart of a train headed off the cliff. I am still in disbelief and am hoping that somehow I will gain a unique wisdom by sticking it out and watching them pull this off. Or it will be a spectacular apocalyptic disaster that will make for a great story.
• trimethylpurine 4y ago
  In almost every medium or large business there are easily millions to be saved or made annually with the correct data. You're doing something wrong. Read what you've written here. You're doing a lot wrong. If you're on a garbage team, stand up a little taller and take charge. Hire your own team to do things correctly. You're unhappy because you're failing. You need to reflect on that and change or remake yourself, possibly in another field.
• penciltwirler 4y ago
  I switched out of data engineering, and became a software engineer instead. I work on building the data infrastrue tools that I used to use. Maybe this only applies to me, but building the tools that I used to use is somehow incredibly satisfiying. I think it's because as someone who used to use the tools, I know what I want, and then I build it, completing the "cycle of needs". So my advice to some of the data scientist/engineers out there is to go a little deeper into the tooling and try to understand how they were built.