6 ms·
Show HN: Open-Source Business Intelligence for BigQuery – Looker Alternative
- verdverm 8y agoLink to the open source part? I can't seem to find any on the site...
- rahimnathwani 8y agoThere's a github link at the top of the page: https://github.com/mprove-io/mprove https://github.com/mprove-io/mprove
- verdverm 8y agoOh, had to rotate my phone, does not show up in the menu on a smaller screen layout.
- pugworthy 8y agoWow, very different menus on mobile based on orientation. Vertical has “Pricing” and no open source mention. Horizontal gives a GitHub link and no mention of pricing. I assume there was a time when this was meant to be a non-open source project.
- akalitenya 8y agothanks, i just fixed it
- mwexler 8y agoIn mobile, it doesn't seem to appear til you rotate your phone, but it's there in desktop view.
- sandGorgon 8y agothis is super cool and i would pay for this. Bigquery is a cheap alternative to a lot of the mobile analytics tools. Quick point however - why do you need a new database ? You can use a table inside bigquery itself. It seriously reduces the dependencies required.
- akalitenya 8y agoMprove creates permanent derived tables in BigQuery if the user wants it. But you can't use BigQuery for OLTP.
- sandGorgon 8y agoNo - I'm talking about MySQL being needed as a dependency. https://github.com/mprove-io/mprove/blob/master/deploy/docker/ce-prod/docker-compose.yml https://github.com/mprove-io/mprove/blob/master/deploy/docke... Can you not use biquery database itself. Create tables for your internal use instead of MySQL ?
- akalitenya 8y agoThe MySQL image specified in the docker-compose file you mentioned. It is used for the internal data of the Mprove application (users, projects, members, etc.). Each user action in the web client (angular) can initiate several queries to this database through an backend request. Delays here are crucial. network latency - so you need to keep the database as close as possible to your server side, read / write delays - for queries such as finding a user / setting a username / creating a member, etc.
- sandGorgon 8y agoHmm..I would prefer to not have it. In production, managing database persistence is very hard. Especially when you go down the kubernetes road. I would take higher latency, but avoid pulling in a whole database infrastructure. Plus a huge number of us use postgresql..so that becomes another set of a mess. I would strongly urge you to do this on the same bigquery database that you would connect to anyways.
- segah 8y ago@akalitenya are you even in the clear with this? Some of the Old LookML syntax is an exact copy. But more importantly, the challenge for any such tool is to go beyond use by 2-3 people. At 2-3 people anything will work. Where BI tools (open source and close source) struggle is scale: having all the right features for, essentially, a group of users who actually don't know how to work with data (did I just say that aloud?). Chartio caps at 20 people. RJ capped at 50-100 (and later became Stitch for that reason). We haven't seen where Metabase caps, but I bet it is in a similar range. Very few BI products have actually surpassed 100 users at target installations. And beyond 1,000 is a real challenge that only few, and even then with a lot of assistance, can support: Tableau, Looker, Microstrategy, maybe Birst, maybe Domo. Also, a combination of BI with LookML is a complicated product. During my days at Looker, we were handling 50+ bugs / week, and filing 1,000+ tickets. Every day we were filing over a 100 new features. So with all that, the question is, is it really worth the struggle? What's the end vision for supporting this? Why should someone who implements BI for a living bet on this product?
- MechanicalTwerk 8y agoI agree with pretty much everything you've said. However, as someone who has used Chartio extensively, I'd say 20 is wayyy too low. It can definitely handle 100s. But, like you said, 1,000s is a struggle for anyone. Also, if you think this is a rip off of LookML, you should take a look at what GitLab is doing with Meltano. They completely jacked LookML.
- g14i 8y agoPlease, do you mind sharing what exactly makes a BI solution difficult/struggling for 1,000+ users? Is it something technically related or more business/feature related?
- segah 8y agommmm, both! Tech-wise, data stacks are complicated. Hundreds of pre-existing tools, with many owners and diverse interests. BI is where they all meet. With some organizations its easy to build a central data warehouse, but in a large corporate environment that's a dream. Imagine how many data warehouses an organization like Microsoft might have? OK, MS is too large - how about Expedia or Zillow? Thousands? So you connect a tool like Looker--that was designed for 2-10 connections--to a thousand? How do you even administer all the connections? How do you govern access? Access permissions at Looker (and other BI tools) are great, but it would take you 10 years to set them up for 1,000 DBs. And what about tool access? Some want SSH tunneling, others want complete on-prem. Some need Google authentication, while others want one through a pre-existing corporate login. Most large organizations typically end up building--or hiring someone to build--their own custom solutions on top of these tools. Such custom solutions are bigger projects than the implementation of a BI tool to begin with. Oh, and most also fail. The success of a BI in an enterprise environment is partly good sales/marketing/support, but a big part is due to the product achieving some form of maturity with all this side tech--the stuff totally not core to the analytics itself. Business-wise, try getting people with diverse backgrounds agree on common terminology. Let's take marketing for instance. Should be easy, no? But hey, marketing is actually like 10-20 different kinds of people - some know SQL, others can't put 2+2 together and arrive at anything other than the word "magic". Some think in terms of stories--others think in terms of conversion funnels. OK, so you've put together some data dictionary, did some training, segmented users into 1) technical ones (SQL/LookML/BackML...), 2) business (explore), 3) consumers (dashboards/pretty charts). But now it turns out that much of the data they rely on is generated by a different team--say, operations or sales. Again, you've got 10-20 different kinds of personalities there. Somehow all these people have to agree. How do you make them all agree? Short answer: you cannot. No one can. They don't even like speaking to one another - and now you are going to come in and make them agree on what kind of KPI is going to determine their success => bonus. Hell no! There are ways to make progress on both fronts. And, no doubt, an open source project has some chance. But it is not easy. And no one has really done it in 30 years. Many have tried. Full disclosure: I love BigQuery (Google Cloud partner). Love Looker. Rely on Open Source constantly. Just trying to demonstrate a realistic view of how hard this problem is.
- danpalmer 8y agoThe headlining feature seems to be that it has a dark and a light theme. This isn’t very encouraging for the rest of the product. I’m sure there’s a lot of good stuff here, but themes aren’t important enough to be the first feature mentioned.
- morenoh149 8y ago+1 it should not be the first feature listed, bump that down
- akalitenya 8y agoThank you, I plan to add more videos on the page.
- vgt 8y agoBigQuery PM here. Nice work! Here's a recently built Graphite connector as well: https://twitter.com/vadimska/status/1112816503055843330 https://twitter.com/vadimska/status/1112816503055843330
- siculars 8y agoWill this handle (repeatable) “record” (struct/array) data types natively? /disclaimer: work in google cloud
- akalitenya 8y agoYes it should, there is special "unnest" parameter for fields in BlockML reference - https://mprove.io/docs/blockml/fields/dimension https://mprove.io/docs/blockml/fields/dimension
- segah 8y agowow, impressive! how about symmetric aggregates (e.g. being able to do correct summation/aggregation on numeric values despite a one_to_many join)?
- akalitenya 8y agoYes, look at this page - https://mprove.io/docs/blockml/fields/measure https://mprove.io/docs/blockml/fields/measure. You need to specify measure "type" and "sql_key" that will be used to avoid counting duplicates.
- infinite8s 8y agoHow do the underlying queries look for symmetric aggregates? Sadly SQL never supported the ability to compute an aggregate based on the unique values of another column.
- akalitenya 8y agoMprove does it the way the Looker did it before: CREATE TEMPORARY FUNCTION mprove_array_sum(ar ARRAY<STRING>) AS ((SELECT SUM(CAST(REGEXP_EXTRACT(val, '\\|\\|(\\-?\\d+(?:.\\d+)?)$') AS FLOAT64)) FROM UNNEST(ar) as val)); ... SELECT COALESCE(mprove_array_sum(ARRAY_AGG(DISTINCT CONCAT(CONCAT(CAST(a.id AS STRING), '||'), CAST(a.population AS STRING)))), 0) as a_cohort_size Recently, Looker began to do it differently, most likely to improve bigquery performance: COALESCE(ROUND(COALESCE(CAST( ( SUM(DISTINCT (CAST(ROUND(COALESCE(lesson_5_cohorts.population ,0)*(1/1000*1.0), 9) AS NUMERIC) + (cast(cast(concat('0x', substr(to_hex(md5(CAST(lesson_5_cohorts.id AS STRING))), 1, 15)) as int64) as numeric) * 4294967296 + cast(cast(concat('0x', substr(to_hex(md5(CAST(lesson_5_cohorts.id AS STRING))), 16, 8)) as int64) as numeric)) * 0.000000001 )) - SUM(DISTINCT (cast(cast(concat('0x', substr(to_hex(md5(CAST(lesson_5_cohorts.id AS STRING))), 1, 15)) as int64) as numeric) * 4294967296 + cast(cast(concat('0x', substr(to_hex(md5(CAST(lesson_5_cohorts.id AS STRING))), 16, 8)) as int64) as numeric)) * 0.000000001) ) / (1/1000*1.0) AS FLOAT64), 0), 6), 0) AS lesson_5_cohorts_m_sum_distinct
- yantra_ml 8y agoDo you support on-prem?
- akalitenya 8y agoSource code is open. Anyone can deploy Mprove to his server and use it for free.