12 ms·
Is BI dead? – On dismantling data's ship of Theseus
- drfuchs 5y agoIf you make it to the 8th paragraph, you may learn that "BI" stands for "business intelligence".
- meepmorp 5y agoIf, like me, you've sustained enough corporate brain damage, you assumed that's what it meant from reading the headline.
- kgwgk 5y agoIt was either that or Boehringer Ingelheim.
- da_chicken 5y agoAnd what it actually means is "visualizations and dashboards". Why make decisions based on analyzing the data from a report when you can get a vague impression from a pretty chart and go with that instead? IMX, the core problem is that all BI software is built around the idea that everything is a financial value that can always be arbitrarily aggregated meaningfully, and that looking pretty is more important than presenting the data in a way that your organization actually finds logical. It's a way to sell very expensive software to decision-makers, while also producing software complex enough that it can't be configured well enough to properly evaluate until well after you've already bought it. Only then do you find that the feature set is very broad, but very shallow. It's only 18 months later that you discover how limited the report writer is, or that you have to do it this one way for everything even if you really could use it formatted slightly differently. It's like using a pivot table in Excel and trying to control the order of the columns, or to make one table aggregate two values distinctly, etc. You end up with 10 seconds to pull your data and create the table, and 4 hours trying to get it to display in the manner you want before giving up.
- otabdeveloper4 5y agoWhere I've worked BI wasn't ever used by decision makers. It's used to make dashboards that are part of a sales pitch to clients, because clients love pretty pictures and "science".
- nemacol 5y agoIf you can pepper in the phrase "single pane of glass" you will be sure to hook em with the sale pitch.
- 58x14 5y agoI laughed, I exclaimed... and then I had a moment of silence.
- horsawlarway 5y agoThis matches my experience entirely. BI was not for making good decisions, it was for making better sales pitches.
- x86_64Ubuntu 5y agoThat largely depends on the culture of the company. I've been at places that used BI to make decisions, and other places that just used it to nitpick over events from years ago. BI, like every other tool doesn't exist in a vacuum. It has to be present in an environment that has peopeople that know how to help themselves use the self-service attributes (i.e not Boomers), it also has to occur in a culture where KPIs and metrics are understood, and used in day-to-day discussions of the business, there should also be a training component to deploying your BI solution too. Without these aspects being present, your BI solution will devolve into either a sales prop with pretty colors, or it will become a web version of Excel.
- BeefWellington 5y ago> IMX, the core problem is that all BI software is built around the idea that everything is a financial value that can always be arbitrarily aggregated meaningfully, and that looking pretty is more important than presenting the data in a way that your organization actually finds logical. I used to work directly modeling data for various sized companies on behalf of two different BI sellers. As I see it there are perpetually two problems: 1. BI is marketed at and sold to anyone needing any kind of data visualization capabilities, and most packages I've worked with can do this but that's not really where they shine. 2. They really shine when you have huge amounts of data stored in different systems and you are trying to build an environment up where you can coalesce that data into one place and then visualize it, and you need to routinely report on these sorts of things. I think most BI solutions fall into the same space as JIRA does -- highly-customizable solutions that are sold as "Turn-key" without any warning ahead of time that it'll require someone (or several people) on your team become ridiculous levels of expert in areas that don't necessarily help the business. It's a specialized skill most places don't really need when what they're looking for is a simple reporting package that can connect to just any database. Similarly, I saw a lot of people with BI insisting they had "Big Data" and thus "needed" things like Hadoop in order to process their stuff, when in reality the MySQL, SQL Server, or Oracle DB they were already invested in would do the same job faster 99.9999% of the time.
- majormajor 5y agoTo be fair - if you aren't already familiar with that, you probably aren't the target market for the article at all. (Though even in the target audience, I'm not taking much away from this. Tools should be better. Yep.)
- Joeboy 5y agoMy first though was bisexuality, then Basic Income, then glanced at the article and found it was something to do with Salesforce, then found your comment. Thank you so much.
- deleted 5y ago[deleted]
- intrasight 5y agoI remember learing decades ago that "In UNIX, everything is a file" I approach BI now as "In BI, everything is in your database". With everything in the database (SQL Server in my case) I can deliver BI to my users with quality tools like Tableau and Power BI - or just with generic web reports.
- jon-wood 5y agoIt really is amazing how far you can get by just dumping everything into a single database and letting people do joins. We’re doing the same thing with Stitch replicating data from production databases and a few SaaS products into a big Postgres instance and it’s so low on maintenance hassle. The most advanced our “BI pipeline” gets is a few stored procedures run from a cron job to aggregate data in a particularly large table.
- intrasight 5y agoExactly. I do BI at a Fortune 500. We have most everything anyone would care about in a database. Our job is to load and curate, create views and write queries.
- CRConrad 5y agoDid you just invent Data Warehousing? If so, I have bad news for you...
- intrasight 5y agoI do know that most project fail, and I suspect that a big part of that is having people focus on shiny new tools instead of just using a database and hiring some developers.
- CRConrad 5y agoYeah, I was mainly just being facetious about how what you described has "been a thing" for decades, and there's a whole industry around it (in which I've earned my living this century). So if you thought your insight was new, you'd be disappointed... :-) But, come to look at it in hindsight, my comment may have been misplaced -- maybe it should have gone as a reply to the same comment you replied to, with its "It really is amazing how far you can get..." At least if they meant "amazing" in the sense of "surprising". To me, what's amazing and surprising is only how many people don't already know this.
- 58x14 5y agoI thoroughly appreciate when an author uses an extensive variety of sources that are particularly relevant to the topic. I wonder how long this took to write.
- oxfordmale 5y agoSalesforce is anything but "No Code", it was just a clever marketing ploy. Just look at job postings for Salesforce Developers. I don't think BI analysts have to get worried quite yet.
- mdoms 5y agoNot sure why the author spent half the article writing about Salesforce which turned out to be entirely irrelevant.
- isoprophlex 5y agoOh, I OTOH really enjoyed the historical perspective they sketched. I appreciated the work put in to give some context.
- slt2021 5y agoit is because Salesforce conference is happening right now in San Francisco
- monkeydust 5y agoAs someone who runs teams deploying BI to internal stakeholders (product, sales) I am pretty fed up with them. Firstly Tableau, QlikView or PowerBI are all pretty much doing the same thing whatever flavour you prefer. We find that maybe 10% of users will actually use them to garner new actionable insights and 90% will moan there's too much data for them to process and no one appreciates the immense amount of data munging that goes on behind the scenes to make those pretty graphs. Where do we go? Personally I see a lot of the insights being automated and turned into actions on the server side without anyone touching a BI tool. This can be achieved with rules based approach and perhaps some correlation, trend analysis over time series data. Where you do need to go beyond that then perhaps AR/VR will provide a novel and more valuable approach.
- ndnichols 5y agoI couldn't agree more! We built Lexio to address exactly that issue. (Our tagline is literally "Stop building dashboards that no one uses".) It does data storytelling (discovering the insights and conveying them in natural language) on the server and gives users a newsfeed experience for consuming them. I'm happy to talk shop if folks are interested in how it works. https://narrativescience.com/lexio https://narrativescience.com/lexio
- TOMDM 5y agoI've been doing a fair amount of BI work myself recently. I reckon of the 90% that don't make any data driven decisions with the dashboards they're provided, there are three causes I run into frequently. 1. They can't describe what info they need 2. That info isn't available. 3. They don't make decisions based on data at all. I feel I'm getting better at addressing 1 and 2 the more I do it, but I figure there's a fair amount from group 3 who demand the reports nonetheless... As an aside, you mention Tableau, QlikView or PowerBI, have you tried Apache Superset at all?
- gumby 5y ago> there's a fair amount from group 3 who demand the reports nonetheless... It’s a very important function: provide data that can be cherry-picked to support a decision already made. If the decision is too complex for that, hire a big consulting firm to write a report suggesting and justifying the action desired. A decison staple surely going back millennia.
- 1-6 5y agoI’ve gotten away with Plotly. Easy enough and looks great.
- alach11 5y agoPlotly exists somewhere between BI tools and full blown internal software on the continuum between Easy/Limited and Hard/Customizable. If you and I sit in a room and race to analyze a few CSVs of data, I bet I can find compelling trends faster using a BI tool.
- chris_wot 5y agoThis seems like a promotion for Salesforce.
- pram 5y agoI’ve spent the last 10 years of my life building “Big Data” platforms for companies (Kafka, Druid, Hadoop, Databricks, Presto, Teradata, Informatica, MicroStrategy, etc etc) and I still have no idea what people need most of this crap for. There’s so much duplicative functionality, and a basically never ending list of OLAP adjacent products emerging every year. I mean it’s good for me personally, it’s pretty much a gravy train, but I always question if a lot of it is a solution searching for a problem lol
- the_af 5y agoInteresting. To me, you seem to be conflating some pieces of data infra plumbing with very sensible and concrete goals (Kafka, Hadoop) with others that are adjacent tools which may or may not make sense. Regardless of any tech details, the idea behind Kafka makes a lot of sense: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... Or did you mean, "I have no idea what business needs all this data for", a much broader question?
- pram 5y agoIt can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config + cluster)
- MangoCoffee 5y ago>BI should include all consumption we use Microsoft PowerBI. it work great within the Microsoft ecosystem. if you try to pull data from NetSuite with PowerBI. it went to crap.
- riskneutral 5y agoWhy? What is special about NetSuite?
- MangoCoffee 5y agoits the ERP software that we use. people needs to do report with data from NetSuite.
- edgarvaldes 5y agoWhy not create a "copy" of NetSuite data to a more friendly format? I mean to sync NetSuite data to another database.
- NortySpock 5y agoBuuuuuuuut Mommmmmmmmmmm, I don't wanna write Another ETL Script! stamps feet
- jl6 5y agoThe ugly truth of BI tools is that they are of most benefit to the organizations which are least capable of using them. The goal of properly exploiting BI comes with many prerequisites which sound superficially reasonable but turn out to be decade-long side-quests. Things like having all your data in one data model. Things like understanding where your data comes from, and exactly what it means. These prerequisites are easy for small orgs, but small orgs benefit least from BI and typically get better bang-for-buck from Excel. Large orgs find themselves mired in the political meta-problems of meeting those prerequisites, like joining up the fiefdoms that own data sources with the cabals that run data governance and the accountants who want a return on the investment of simplifying a sprawling legacy estate. BI tools are generally incredibly poor at dealing with the bag-of-spanners heterogeneous data landscapes that exist in these organizations, and their analytics nirvana remains largely unattainable. The trend that the article describes - towards simpler, composable BI components, each with more modest goals - is progress. It helps move focus away from the relatively-easy problem of visualization, to the rest of the data stack.
- Foobar8568 5y agoOr that most people are totally clueless about their own companies, processes and data.
- mberning 5y agoThis is so true. I worked a BI tooling startup and it was a lot of fun. But the big clients all had the same types of issues which ultimately dooms their quest. At some point the CFO, CTO, CEO notices they are spending more on “BI” than marketing and they start to get very curious about what they are getting for their money. It’s never quite what they expect.
- toyg 5y ago> BI tools are generally incredibly poor at dealing with the bag-of-spanners heterogeneous data landscapes It's not that they are poor, it's that they eventually require some moderately-challenging coding - because there is only so much complexity you can hide behind point&click before things become untractable. Programmers get bored doing BI plumbing, and those commercial ecosystems are quite secretive and often expensive to train in; so the talent pool is small and costs can skyrocket pretty quickly, making the whole proposition unappealing. Maybe if we accepted that "BI Programmer" is a respectable career and something that is fundamentally unavoidable, providing clear career paths inside companies for it, instead of trying so hard (and failing) to get rid of such figures as soon as they progress a bit, there would be more predictability and less angst in the space.
- PhilipA 5y agoI do believe that we have seen solutions trying to be a horizontal solution to problems, and I think the evolution will turn into vertical solutions. I have also invested a lot of time, as we are a couple of people who has been building on a vertical BI tool for the SaaS space for about a year. The main goals are: - Easy accessible for novice users. We want to make the Google for BI to help empower non-technical people (no more salespeople asking for reports from devs) - More advanced editors for the more technical people - Advanced alerts + integrations to 3rd parties - Later on proactive reports Hit me up on @philipanderse if you want to test it out.
- molsongolden 5y agoInterested in learning more but it looks like your DMs aren't open.
- PhilipA 5y agoJust follow me and I can follow you back.
- jpmonteiro 5y agoI entered in a discussion with Benn on Twitter on Friday around this topic. Here is my take on it in case you are interested in a different perspective: https://jpmonteiro.substack.com/p/a-friday-fight-and-the-internet-of https://jpmonteiro.substack.com/p/a-friday-fight-and-the-int... TL;DR: I believe the trend will be to have an ecosystem-like approach to BI with tools complementing each other and communicating with each other through open standards.
- 0xbadcafebee 5y agoData is clay. We are still struggling to make bricks. We are still very far from building a house out of it, much less a large building. And we're going to eventually find out that we can't make skyscrapers out of it at all. Just like bricklayers haven't even remotely been replaced with robots, I doubt the data-mungers will be, either. We are probably on the cusp of a whole new generation of "data people" who will have careers that span a generation, and do nothing but sift through data.
- d--b 5y agoI am building Jig (based on observable): https://www.jigdev.com https://www.jigdev.com Because I believe that BI should be done by people who have a mixed set of skills between stats and programming. A programmer alone can’t make the stats, and a mathematician can’t build a reporting tool. Jig is a shortcut to build reports quickly, but I am somehow convinced by javascript is just not good enough to do that job. I think I might have to try and design a language that has indexed containers as first class objects…
- b9a2cab5 5y agoIn my experience BI has no business being done by technical people. You need business people who are just technical enough to use the system for what they need. You need domain knowledge and deep product understanding to know what data to be looking for, not a stats and CS whiz. In 99% of cases, SQL backed by a fast distributed analytics database and Tableau+Excel is enough for that. Even as a software engineer doing basic analytics charts for the service I own I don't want to be writing JavaScript. Ideally I wouldn't be writing SQL either.
- deshpand 5y agoA little python training in the python stack (pandas/numpy/matplotlib or other visualization libraries) can go a long way to simplifying tech stack and get rid of these mind numbing BI tools. And companies are trying. Ex:https://www.bobsguide.com/articles/barclays-gordon-risk-managers-to-be-trained-in-python-and-r/ https://www.bobsguide.com/articles/barclays-gordon-risk-mana... But I also see spending more on these tools, in the name of innovation, because some bigshot likes tool X and that's what he wants to use. And guess what, now you also need it to be made available in the cloud.
- blakeburch 5y agoThe interesting point here is that the unbundling of BI is resulting in tools for every occasion. What I'm seeing on my end is that the "modern data stack" is starting to become very fragmented as a result. Sure, each of these tools are better suited to accomplishing a very specific task with your data. But the problem is that now these tools aren't talking to each other. No one has a single place to view their data end-to-end. No one can show me every automated process in their organization that relies on a specific table. Everything is tied together with trust. Trust that the Data Engineering team won't deploy bad code. Trust that the API won't change how it returns data. Trust that the data actually loaded completely. This trust-based system means that one problem during ingestion/cleaning ends up spelling disaster for all of the downstream ML models, reports, data extracts, etc. and causes a day worth of headaches to resolve. If BI used to hold everything together under one roof, we need something to make these tools talk to each other. The only solution, as I see it, is to build out better data orchestration that effectively glues these tools together and lets you see the "big picture" with your data.
- CRConrad 5y agoDon't worry, you won't have to invent a whole new thing: There is a whole sub-industry of IT called Data Warehousing that does exactly this.
- dgudkov 5y ago>BI shouldn’t handle data ingestion and storage I've been working in the BI industry for almost 20 years and I don't know why the author believes that the "Original BI" includes ETL/storage. He mentions BusinessObjects and Microstrategy as an example of traditional BI tools popular 20 years ago, but neither of them had ETL/storage capabilities. Both were basically visual SQL generators. (Although, BusinessObjects later acquired an ETL company). Qlik has a bit of everything - ETL, columnar storage, and dataviz. But that's more of an exception than a rule. Nevertheless, the author has a point - BI seems to not have a clear path forward. There are experiments with natural language queries (won't work), automatic text summary generation (useless), AI-assisted automatic insights (might be a nice feature, but barely a "product"), and a few more. The self-service story of BI never really took off (I believe self-service ETL is a more interesting story [1]). Neither did story-telling. Dashboards are oversold and are mainly used to impress CxOs to close big deals. Geospatial dataviz is useful, but has limited application. I would envision two possible directions to advance for the BI industry: 1) Analytical notebooks, something like Jupiter Notebooks but no-code and for BI might be a good idea to explore, but I haven't seen anything like that. 2) No-code/lo-code analytical app building. Reporting and dashboarding tools are just highly specialized app builders. Why not make a step further and generalize it a bit? [1] https://easymorph.com https://easymorph.com (Disclaimer: it's my company)
- CRConrad 5y ago> I've been working in the BI industry for almost 20 years and I don't know why the author believes that the "Original BI" includes ETL/storage. I've done some BI, but mostly DW/ETL, for over twenty years... And I can't remember ever seeing ETL viewed as a sub-category of BI in any project I've been part of. It hasn't even been mentioned as such -- except in weird articles like this -- this side of the turn of the century. Before that, yes, the categories were muddled. But not since then. > 2) No-code/lo-code analytical app building. Reporting and dashboarding tools are just highly specialized app builders. Why not make a step further and generalize it a bit? Sure. I've long thought the best tool, one that could do both ETL and BI -- is an IDE with a full-blown programming language and a good library: Something like D̵e̵l̵p̵h̵i̵ Free Pascal / Lazarus.
- infinite8s 5y ago
- goodlinks 5y agoI appologise in advance for probably using poor terminology.. Maybe I am a naive fool but IMHO the benifits of BI are reconciliation of the business model: - reconciling processes - reconciling data - reconciling governance Its a painful process to impliment and you will hit all of your compliance, governance and change management issues head on. IMHO if you accept that this is what you are doing then its potentially very valuable for your business, but it must be done humbly with humilty as every single measure and many dimensions are each a project on their own. The pretty graphs are largely irrelevant apart from making employees feel valued and to re-enforce abstractions via visual metaphors. I get pissed off when end users are not trained in consuming the database warehouse through tools like excel or access. This project should include general IT education programme for most companies. Also.. if the end users cannot use a pivot table then you need to teach them that and prototype a bunch of stuff in excel before you go anywhere near BI of any flavour.
- nivenkos 5y agoLooker is amazing, I just wish my current company used it. Tableau is a nightmare (no Linux support for a start, nevermind the hassle of editing each dashboard individually).
- zhendlin 5y agoA lot of conventional BI was focused on 'single player' mode -- basically an analyst running analysis at their computer and maybe dropping that graph into a presentation. This fundamentally limited the usefulness of of BI results (they got stuck in time and rarely updated in) and made follow-ups hard since the graph was no longer connected to the data source. All the while things like Google Docs and Slack made collaboration around documents and ideas much easier with @mentions, threads, etc. So BI can be a lot more useful if it is 1\ accessible anywhere (not just desktop or a crappy mobile app) 2\ collaborative -- bringing the mentions, shared questions, and ability to make decisions together 3\ not crazy expensive (looking at your tableau + looker)...because at these high price points the tool ends up not getting offered to everybody, it ends up being more limited in who gets access and hence less useful across a company Full disclosure, I built Zing Data which is an app for mobile first business intelligence and is free for small teams. Works with PostgreSQL, MySQL, and Google BigQuery. https://www.getzingdata.com/ https://www.getzingdata.com/ Would love any feedback folks have -- we're actively improving it and I'm sure this community has a lot of great ideas!