25 ms·
"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is
by fdgsdfogijq 4y ago
"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size."
The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.
- revolvingocelot 4y agoIt's absolutely this. "Decision-based evidence-making" is what I've seen it called.
- capableweb 4y agoIs that what's happening at Amazon as well? As they seemingly is loosing more and more track of the "Customer Obsession" schtick.
- aintgonnatakeit 4y agoThey are encouraging their customers to have a bias toward action. Away from that asshole Bezos.
- fuzzylightbulb 4y ago"customer obsession" was always at the mercy of the real obsession: "making money hand over fist". The former will ALWAYS lose out to the latter given enough cycles.
- pphysch 4y agoYeah, "customer obsession" really just means "market share / growth obsession" which is a means to (eventually) making monopoly profits. Which Amazon seems to have achieved.
- ethbr0 4y agoOne may follow the other, but not vice versus. It's a pretty strong argument to say that Microsoft under Gates was technically obsessed, but that really faltered under Balmer. Microsoft continued to win profits, but they made major strategic missteps that cost the revenue. Amazon feels like it's going down the same path: empowering the tree-gazers without remembering that the forest also matters.
- bombcar 4y agoYou can also “coast upwards” for quite awhile - as an example we now pay Microsoft less than we were for periodic upgrades to Office for the entire Microsoft 365 suite (including email hosting, etc) but all the machines are now Macs. They make more from us in one way, but less in total dollars.
- ethbr0 4y agoExactly. I can't imagine some accountant hasn't come up with a way to quantify this. {Revenue attributable to previous R&D} (aka coast) vs {Revenue attributable to current R&D} (aka acceleration)
- deleted 4y ago[deleted]
- LarryMullins 4y ago
- bluedays 4y agoTheir mission hasn't changed. They're still obsessed with customers, just not the way you think.
- slt2021 4y agoI used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick some flashy metric like a percentage and run around with it. Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc. if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be). Data Scientist's job is to launder management's intuition using quantitative methods :)
- tpoacher 4y agosadly same with academia and funding sources
- karmajunkie 4y ago"Data launderer" would be a good job title...
- ren_engineer 4y ago"data" is usually just twisted to make leadership look good or justify what decision they wanted to make anyway. Analytics data is sliced at arbitrary time periods to make growth in whatever metric look good, certain subsets are just removed, etc. doesn't help that most of this data goes through multiple layers of BS where each person is putting it through filters to make themselves look better. And a good chunk of people don't have enough understanding of stats to understand when they are being tricked
- mritchie712 4y agoSize isn't the real problem, it's time. Are you going to take the time / money to set up a warehouse, get all the data into with an ETL product, set up dbt or some other transformation layer, set up a BI tool and build the reports and dashboards, etc. Regardless the size of your data, you still need to get it in one place and model it in a way it's actually usable.
- didgetmaster 4y agoExactly. It isn't just time to set up all the data in a way that makes the right query possible. It is also having queries fast enough to be able to run a vast number of them in order to find what you are looking for (or even things you were not looking for). https://didgets.substack.com/p/data-science-and-serendipity https://didgets.substack.com/p/data-science-and-serendipity
- stainforth 4y agoIs it queries on live data or data thats been moved usually?
- fijiaarone 4y agoIt’s moving the data around that is slow… and expensive. Getting the data into the data warehouse, then getting it to the processors then moving it around to filter and transform. Getting your data to the cloud is expensive, but then you can’t do anything with it because distributing it to process in multiple stages is too expensive and you’re already paying so much to keep all that useless data.
- olyjohn 4y agoCan't we just give it to that one IT guy down in the basement?
- mritchie712 4y agoHey, I used to be that guy (and still am).
- Consultant32452 4y agoI don't want you to tell me what the data says, I want to tell you what the data says and you go find data that confirms it. kthnxbye
- hn_throwaway_99 4y agoAgree with some of what you've said, but disagree with a lot: > Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation. 100% Individual humans are emotional creatures with their own wants and needs, and it's important to understand how organizational incentives drive decision making. > Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". This has not been my experience, though. The more common thing I've seen is that, sometimes data is boring and doesn't really show much actionable insight, but as everyone wants to justify their job, I've seen data scientists come up with really questionable conclusions that fell apart on further inspection (call it "p-hacking the enterprise"). Plus, a lot of this data in these data wearhouses is messy. Often times data scientists are siloed at the end of the process, but then you get "garbage in/garbage out" results, where there is some bug in data tracking that isn't understood until it's too late. Much better in my opinion to have data engineers and data scientists work much more closely with product engineering teams up front so they can help ensure the data they collect is accurate.
- tracerbulletx 4y agoI find a lot of organizations don't have the discipline to harness whatever power their data may have. Sure collect everything, but god forbid you have any sort of data governance, or spend a single resource minute of time manually tagging or organizing or validating it. Then they try to make shitty ML models or products out of it, but don't care if the models actually work or not, just that they have AI now. Then a year later when the model has provided no value they are like, oh well big data is worthless I guess.
- hef19898 4y agoPalantir, you have to have Palantir. Oh, and a bunch of data scientists with zero domain knowledge for whatever data they are analyzing, preferably with PhDs in maths, but some ML background will do. And agile, because of course all those Palantir dashboards can only be developed using agile. Once all is said and done, zero insight was created but a whole lot of consultants, contractors and project managers have been paid handsomely, while some higher ups can now put "implemented agile and big data at X" on their LinkedIn profiles.
- vjk800 4y agoI'm one of those PhDs with zero domain knowledge analyzing data and I share the sentiment. Most of my analyses provide very little value because they are sort of common sense to people with domain knowledge. When I ask people what could be more useful, one of two things usually happen: 1) it's impossible due to data and/or infrastructure limitations, 2) what they ask turns out to be nonsensical in further analysis (like asking for average of something that follows a very fat tailed distribution with a few observations dominating the phenomenon. Of course it's usually impossible to explain this to people). The more I think about this, the more I think that in truly data powered companies, both the decision making and data analysis have to be carried out by more or less the same people. The organizational hierarchies have to be much flatter. Essentially the employees will have to be some kind of "secret agents" who have both the skills and the mandate to steer the company in the direction they see fit. I sort of see this already happening in the FAANG companies where, or so I hear, it's very difficult to get hired, the staff count is quite small compared to traditional companies and the senior engineers have a lot of power in the company. Using math PhDs or Palantir or whatever as a sort of modularized black box for "insights" while giving them no real skin-in-the-game does not work.
- commandlinefan 4y ago> a huge amount of thought Which itself both takes time and is wildly unpredictable - neither of play well with today's Taylorist managements schemes.
- aicharades 4y agodeleted- length
- LeanderK 4y agoof course it won't and it's really ironic that you reply with the next hype
- snapetom 4y agoAgree on the size not being the issue. I transitioned from a data engineering manager to a data product manager at a new company. You know how much data a typical customer generates? Less than 1TB a year. I told my VP that the engineering foul ups in the current product are easily fixable. Standard tooling and patterns exist to re-architect and solve the bottlenecks. What is much harder is a data architect to make sense of the complex data and make sure there is good value for our customers. Guess what position I don't have on the team, and won't have due to budget issues.
- boh 4y agoThe other thing to consider is that data simply has nothing of value. Part of the marketing of big data is the almost fairy tale belief in "insights" existing in any data set if you just look hard enough.
- ryguyrg 4y agoCorrect, much data has no value. The cost for storing the data, maintaining the data [in the day-and-age of privacy requirements especially], and combing through the data is often much greater than the value obtained from the data itself. The expertise we need in the industry is people who understand applications in-and-out and make great decisions on what data is worth keeping for present and future applications. And what data is needed to be kept, but only in aggregates (or anonymized, which reduces costs of maintenance)
- antisthenes 4y agoWait, are you telling me GIS data of deer poop distribution in my backyard has no practical value? But I have thousands of data points, thousands, I tell ya!
- gymbeaux 4y agoReplace “data scientists” with “software engineers” and you have another accurate insight. They don’t want to listen to us about how to write software or derive value from data.
- sportstuff 4y agoNo data is better than Bad data
- naijaboiler 4y agoOr even worse, bad understanding of data
- bcrl 4y agoIt doesn't matter what the data says if it's telling you to go in a direction that is unethical.
- RA_Fisher 4y agoI think the recent industry layoffs reflect that in part. Though I think there’s a better way— that is executive data science, what I do at Zapier. The key is that I’ve built up a huge amount of econometric and economic / business skills that I apply to affect good change in collaboration with company leaders. It allows me to work with Execs using sensible analysis. I improve growth and output by helping us catch errors of assumption before they go into production and cost growth / bad surprises. I also help the executives gain alignment around good information. This multiplies their departments’ output by allowing them to work better together, more in concert. That helps avoid issues with data being bent to decisions. They typically carry a lot of hard-won valuable domain knowledge (that I combine with my economic-statistic knowledge and skills for rigor). It’s my job to ensure Execs start with good sensible information regarding objectives. They usually ask fantastic questions and share a lot of great analysis of their own. There are times I learn about what I might call controversial implications. This is typical of innovation using technology. It’s in these moments that I feel I create the most value by highlighting the trade offs I believe we face / potential regret.
- bigbluedots 4y agoThat sounds amazing, what does your day to day consist of?
- RA_Fisher 4y agoI get to apply my causal inference and measurement skills to a lot of compelling situations in collaboration with some really intelligent and skilled people. I truly feel like I get to perform technically excellent work, and I feel very privileged in that way. My day to day is mostly economic / econometric analysis, conceptualization and formulation of technology, and writing, but also occasional meetings to gather information, feedback, answer questions, teach and collaborate with others. I try to effectively “sample” the org by interacting with people up and down the org, so I can assist Execs in incorporating directors, managers and ICs knowledge / pool good information / systematize planning, etc.
- agentofoblivion 4y agoWhat “data driven” normally looks like: business people ask questions that are not answerable, but the DS team often goes off and tries to answer them anyway. I’ve been in so many meetings, in multiple organizations, where a senior leader asks a question like, “why did this number go up? We need an owner to deep dive this.” Then, someone disappears for a few hours or days and comes back with some nice narrative that more or less might make some sense, and is directionally consistent. Then, best case, they go “hmm, ok” and worst case, they say it’s an interesting view and we need to add it to the tracking. Then, move on to a new question + action item, rinse and repeat. And then, 6 months or a year later, some brave soul goes, “geeze, there are way too many things to look at here, can we streamline this reporting to focus on the handful of most important metrics, and then assign owners to review the others and surface anything interesting?” And then all those things people asked for over time get stuffed into an appendix or deleted, and the whole thing starts all over again.
- Simon_O_Rourke 4y ago> Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". I worked in a "data-driven" company of 5,000+ employees, with 4 data scientists who were spilt into two separate non-collaborating teams. In effect, they were so under resourced they got nothing done.
- hpen 4y agoI would argue that insights from statistical models “[are] just one parameter in a complex equation.” No doubt PR from political issues can help the bottom line of a company. Can also hurt it. Just to be clear I’m not arguing political choices are inherently good or bad.
- michaelcampbell 4y agoI've found this to be true in the (albeit small) # of samples I've run up against. There's an agenda by some level of management, and they use "data" to forward their agenda, or disregard it due to "unexplainability" (a legitimate concern for BD/ML) if it disagrees with the agenda.