16 ms·

Yandex open sourced it's BI tool DataLens

268 points by SergeAx 3y ago

RedShift1 3y ago
Any screenshots of how it looks?
- kasperni 3y ago
  https://cloud.yandex.com/en/services/datalens https://cloud.yandex.com/en/services/datalens
- capableweb 3y ago
  The website has screenshots: https://datalens.tech/ https://datalens.tech/
  - RedShift1 3y ago
    Ah thanks, I was browsing through the github repos.
mobileexpert 3y ago
Cool. Rolling your own BI seems fraught with peril at most orgs where I imagine the buy vs build decision is always buy. How many PowerBI or Tableau seats do you need before rolling your own internal BI platform starts to make sense?
- htrp 3y ago
  I don't think it ever makes sense because the large players will always be able to make new innovations (mobile apps, natural language querying, SSO integrations, etc) that, short of large corps hiring BI teams to invest in the open source ecosystem like superset, your open source solution won't have.
  - mritchie712 3y ago
    Yeah, unless your at unicorn scale, I don't see how self-hosting BI would ever make sense relative to other investments you could make.
- totalhack 3y ago
  I must like to live dangerously. In all seriousness though there are low cost alternatives to those mega BI tools that suit many use cases. If I wasn't rolling my own I'd probably start with Metabase or Superset. What I use: https://github.com/totalhack/zillion https://github.com/totalhack/zillion
xnx 3y ago
Such a huge landscape of BI and data visualization tools. Are there any clear open source leaders? Apache Superset?
- noughtme 3y ago
  Also Metabase, which I found easier to deploy and use.
  - dacort 3y ago
    Metabase 47 also has new serialization features in paid editions that allow for git-based workflows. https://www.metabase.com/learn/administration/git-based-workflow https://www.metabase.com/learn/administration/git-based-work...
- RedShift1 3y ago
  Right now I'm heavily into Grafana but when it comes to BI it kinda falls flat, I regularly have to fall back to using the Plotly plugin to create the charts (but it's getting better, at least you can do a normal scatter or bar chart out of the box since version 8. Labeling the axes is still a problem though). Navigation is also problematic, like jumping to a source table takes a lot of effort to make (basically you have to create a new dashboard and do some hyperlinking instead of there being a ready to go "view source data" button). I feel like there's a lot of friction to get Grafana to do BI, but I've also invested so much time in it I'm afraid to jump ship...
- amcaskill 3y ago
  Superset and metabase are definitely the OSS leaders. I work on an open source code-based BI tool called Evidence, which might be of interest to you. It's effectively a static site generator aimed at building automated reports and analysis. https://github.com/evidence-dev/evidence https://github.com/evidence-dev/evidence Previous discussions on HN: https://news.ycombinator.com/item?id=28304781 https://news.ycombinator.com/item?id=28304781 - 91 comments https://news.ycombinator.com/item?id=35645464 https://news.ycombinator.com/item?id=35645464 - 97 comments
lasermike026 3y ago
I'm giving up Power BI and I'm moving to Domo. I look at rolling my own occasionally.
- _boffin_ 3y ago
  We have Domo at work and it just seems overly complicated and insane. I'm wanting to learn it instead of just polling data sources and adding them to a local Opensearch instance, but... too verbose for me.
- RedShift1 3y ago
  If you want to roll your own, maybe have a look at Plotly's Dash?
  - chatmasta 3y ago
    Observable Plot [0] is also nice. AFAIU it's the same library powering the visualizations within Observable itself. [0] https://observablehq.com/plot/ https://observablehq.com/plot/
- mritchie712 3y ago
  (I'm the founder of a competitor to Domo) I really like the concept of Domo. They have ETL, modeling, a warehouse and BI in one app ("data-stack-in-a-box"). I've interviewed 20 of their customers and the general sentiment was pretty bad. There's a long sales process, a longer process to get it set it up, and they've built all the modeling and connectors themselves (vendor lock in, none are best-in-class). Definite (https://www.definite.app/ https://www.definite.app/) is a data-stack-in-a-box. We have a built-in modeling layer for core metrics and an AI assistant to answer any one-off questions. A few ways we're different: Built on open source - We run the data stack for you and give you a single app to manage and analyze your data, but it's all built on open source standards. So if you decide at any point you want to run it all yourself, the code is yours to lift and shift to your own infrastructure. Battle tested connectors - We're using Meltano / Singer (open source library from Stitch) for our connectors, so they've been used heavily in production for years. Self-serve that actually works - A lot of tools promise self-serve, but AI is making this real. We've invested heavily in making it possible to ask questions and get accurate answers. The AI queries a modeled view of your data that can answer questions that depend on well defined metrics (e.g. ARR, DAU, etc.).
londons_explore 3y ago
Yandex makes some pretty cool tech- they clearly have a lot of smart engineers. It's a shame that geopolitics means most of it will have to be reinvented by someone else before it'll see any use.
- efxhoy 3y ago
  Yeah. I’d love to use clickhouse but yandex ties to the russian government makes me not want to.
  - cpursley 3y ago
    Isn't it all open-source?
    - Dah00n 3y ago
      Some projects have refused code from developers because they are Russian or Chinese. Just because it is open source doesn't mean it is unbiased, free from politics, and without racism.
      - deleted 3y ago
        [deleted]
      - slt2021 3y ago
        some projects racially discriminate developers to avoid possible bias, politics, and racism? lolz
        londons_explore 3y ago
        They would argue they are discriminating based on nationality not race. I can't see that argument working in a court of law...
        neoromantique 3y ago
dwheeler 3y ago
Nit on title: it's => its
- pklausler 3y ago
  Just be glad it wasn't " its' " (sic), which has been showing up more and more in my input streams.
cvalka 3y ago
This is awesome! Let's not forget about their second generation SQL database YDB which is an open source alternative to TiDB and YugabyteDB.
- mdekkers 3y ago
  Yandex consistently pumps out great software. Clickhouse is awesome, for starters, as is this.
wutwutwat 3y ago
Yandex open sourced all of its code a few months back when it was all stolen and leaked.
- 6c696e7578 3y ago
  Didn't the CEO flee due to the war? https://www.intellinews.com/russian-tech-titan-yandex-ceo-volozh-slams-putin-s-barbaric-war-in-ukraine-287651/ https://www.intellinews.com/russian-tech-titan-yandex-ceo-vo...
  - wutwutwat 3y ago
    I would have no idea. Why are you asking me and sharing a link to something which tells you. That’s weird, and I’m not interested in clicking the link anyway.
  - bthomas 3y ago
    Not sure why the downvotes - very useful context and phrased correctly as a question.
anonyfox 3y ago
Since the discussion started in the comments already, I have a similar question: any recommendations for a solution (don’t care if OSS or not) that has the best UX for nontechnical people to assemble some data and reports anyhow? I have Salesforce, some mariadb/postgres and (optionally) hubspot as data sources. I can buy or manually provision anything, no technical hurdles or policies from that side. My absolute focus is the raw UX for business people. Suggestions?
- arthurwu 3y ago
  I'm a co-founder of Dataland.io where we're building a powerful dataset viewer + search engine that can work on top of your Postgres or data warehouse. We designed it specifically to provide an excellent UX to business users while reducing BI burden on the data team. We find that most business users often just need to search, filter, and sort instead of looking at charts to make operational decisions. UX-wise, what sets us apart are: - <1s full-text search (even on billions of rows of data), feels like Cmd+F in Google sheets, but faster - Performance: we stream billions of rows into the web browser, seamless scrolling (no paging of 50 records at a tieme) - Rich cells make tables easier to scan/read (enum strings => colored tags, numbers => color-coded based on value => checkboxes, timestamptz => clear date time pills) If that fits what you need, happy to give you a demo. arthur(at)dataland.io Otherwise, I think the simplest BI (if charts are impt) could be something like evidence.dev or Metabase. But I also think it's going to require some curation on your part. Can you reasonably expect business users to navigate the entire schema/table tree across these three sources? That's where I think the bottleneck often lies -- if your BI tool allows engineering to just expose a subset of curated core tables.
- robertlagrant 3y ago
  The problem is that developing the perfect UI for nontechnical people to assemble reports probably requires a bespoke frontend for your business, and one that likely lags behind the reality of its changes. Most businesses instead opt to just hire semitechnical people that can do a bit of data work and give answers to the report-writers, as they can accommodate business changes over time and understand how to construct new queries out of the overall business' data sources. Maybe that'll change one day with AI, and when it does that will be bought by every big company in the world (-:
totalhack 3y ago
Looks pretty neat. Having not fully investigated it yet, I will say the one thing I usually run into with this and other BI tools is a lack of flexibility in the UI for forming queries. It's sometimes limited to one table or view at a time. I wonder why more of them don't use more flexible querying techniques, perhaps just due to the risk of a bad query being formed? My preferred approach is implemented in Zillion, which I use for BI at my company: https://github.com/totalhack/zillion https://github.com/totalhack/zillion
riffic 3y ago
glaring misuse of it's in the title
- Logans_Run 3y ago
  Well played ;-) ps - I spotted the pedant that is technically correct, and I claim my 5 McFun bucks!
- deleted 3y ago
  [deleted]
- culebron21 3y ago
  Add "'s" for genitive case in English, they said.
felixhummel 3y ago
Apache 2.0 licensed (from a cursory glance at the first few repos).
vgt 3y ago
I'm Ukrainian
- macinjosh 3y ago
  [flagged]
- alexlur 3y ago
  Aren’t you American?
esotericimpl 3y ago
[dead]
zxspectrum1982 3y ago
Is there any open source BI tool that can be embedded in some other product so that users (not product developers but final users) can create their own dashboards?
- throw03172019 3y ago
  Metabase
  - zxspectrum1982 3y ago
    IMHO Metabase is too complex for end users: too much "code" (queries) to write and very limited in the presentation layer (widgets users can use to show the results). I was thinking of something closer to Power BI, e. g. something like AppSmith but more BI-oriented (AppSmith is a generic tool to build your own applications, kind of like Visual Basic or Delphi). https://www.appsmith.com/ https://www.appsmith.com/
- moltar 3y ago
  QuickSight has great embedding story Also CubeJS if you want to a bit more flexibility.
unixhero 3y ago
>Where can I find persistent application data storage? >We use the .us-data folder to store PostgreSQL data permanently. You can delete this folder if you want, it will be recreated with the demo data after restarting the datalens-us container Why "us"?
- nijave 3y ago
  Unified Storage. https://github.com/datalens-tech/datalens-us https://github.com/datalens-tech/datalens-us

268 points by SergeAx 3y ago

RedShift1 3y ago

Any screenshots of how it looks?

kasperni 3y ago
https://cloud.yandex.com/en/services/datalens https://cloud.yandex.com/en/services/datalens
capableweb 3y ago
The website has screenshots: https://datalens.tech/ https://datalens.tech/
- RedShift1 3y ago
  Ah thanks, I was browsing through the github repos.

mobileexpert 3y ago

Cool. Rolling your own BI seems fraught with peril at most orgs where I imagine the buy vs build decision is always buy. How many PowerBI or Tableau seats do you need before rolling your own internal BI platform starts to make sense?

htrp 3y ago
I don't think it ever makes sense because the large players will always be able to make new innovations (mobile apps, natural language querying, SSO integrations, etc) that, short of large corps hiring BI teams to invest in the open source ecosystem like superset, your open source solution won't have.
- mritchie712 3y ago
  Yeah, unless your at unicorn scale, I don't see how self-hosting BI would ever make sense relative to other investments you could make.
totalhack 3y ago
I must like to live dangerously. In all seriousness though there are low cost alternatives to those mega BI tools that suit many use cases. If I wasn't rolling my own I'd probably start with Metabase or Superset. What I use: https://github.com/totalhack/zillion https://github.com/totalhack/zillion

xnx 3y ago

Such a huge landscape of BI and data visualization tools. Are there any clear open source leaders? Apache Superset?

noughtme 3y ago
Also Metabase, which I found easier to deploy and use.
- dacort 3y ago
  Metabase 47 also has new serialization features in paid editions that allow for git-based workflows. https://www.metabase.com/learn/administration/git-based-workflow https://www.metabase.com/learn/administration/git-based-work...
RedShift1 3y ago
Right now I'm heavily into Grafana but when it comes to BI it kinda falls flat, I regularly have to fall back to using the Plotly plugin to create the charts (but it's getting better, at least you can do a normal scatter or bar chart out of the box since version 8. Labeling the axes is still a problem though). Navigation is also problematic, like jumping to a source table takes a lot of effort to make (basically you have to create a new dashboard and do some hyperlinking instead of there being a ready to go "view source data" button). I feel like there's a lot of friction to get Grafana to do BI, but I've also invested so much time in it I'm afraid to jump ship...
amcaskill 3y ago
Superset and metabase are definitely the OSS leaders. I work on an open source code-based BI tool called Evidence, which might be of interest to you. It's effectively a static site generator aimed at building automated reports and analysis. https://github.com/evidence-dev/evidence https://github.com/evidence-dev/evidence Previous discussions on HN: https://news.ycombinator.com/item?id=28304781 https://news.ycombinator.com/item?id=28304781 - 91 comments https://news.ycombinator.com/item?id=35645464 https://news.ycombinator.com/item?id=35645464 - 97 comments

lasermike026 3y ago

I'm giving up Power BI and I'm moving to Domo. I look at rolling my own occasionally.

_boffin_ 3y ago
We have Domo at work and it just seems overly complicated and insane. I'm wanting to learn it instead of just polling data sources and adding them to a local Opensearch instance, but... too verbose for me.
RedShift1 3y ago
If you want to roll your own, maybe have a look at Plotly's Dash?
- chatmasta 3y ago
  Observable Plot [0] is also nice. AFAIU it's the same library powering the visualizations within Observable itself. [0] https://observablehq.com/plot/ https://observablehq.com/plot/
mritchie712 3y ago
(I'm the founder of a competitor to Domo) I really like the concept of Domo. They have ETL, modeling, a warehouse and BI in one app ("data-stack-in-a-box"). I've interviewed 20 of their customers and the general sentiment was pretty bad. There's a long sales process, a longer process to get it set it up, and they've built all the modeling and connectors themselves (vendor lock in, none are best-in-class). Definite (https://www.definite.app/ https://www.definite.app/) is a data-stack-in-a-box. We have a built-in modeling layer for core metrics and an AI assistant to answer any one-off questions. A few ways we're different: Built on open source - We run the data stack for you and give you a single app to manage and analyze your data, but it's all built on open source standards. So if you decide at any point you want to run it all yourself, the code is yours to lift and shift to your own infrastructure. Battle tested connectors - We're using Meltano / Singer (open source library from Stitch) for our connectors, so they've been used heavily in production for years. Self-serve that actually works - A lot of tools promise self-serve, but AI is making this real. We've invested heavily in making it possible to ask questions and get accurate answers. The AI queries a modeled view of your data that can answer questions that depend on well defined metrics (e.g. ARR, DAU, etc.).

londons_explore 3y ago

Yandex makes some pretty cool tech- they clearly have a lot of smart engineers. It's a shame that geopolitics means most of it will have to be reinvented by someone else before it'll see any use.

efxhoy 3y ago
Yeah. I’d love to use clickhouse but yandex ties to the russian government makes me not want to.
- cpursley 3y ago
  Isn't it all open-source?
  - Dah00n 3y ago
    Some projects have refused code from developers because they are Russian or Chinese. Just because it is open source doesn't mean it is unbiased, free from politics, and without racism.
    - deleted 3y ago
      [deleted]
    - slt2021 3y ago
      some projects racially discriminate developers to avoid possible bias, politics, and racism? lolz
      - londons_explore 3y ago
        They would argue they are discriminating based on nationality not race. I can't see that argument working in a court of law...
        neoromantique 3y ago

dwheeler 3y ago

Nit on title: it's => its

pklausler 3y ago
Just be glad it wasn't " its' " (sic), which has been showing up more and more in my input streams.

cvalka 3y ago

This is awesome! Let's not forget about their second generation SQL database YDB which is an open source alternative to TiDB and YugabyteDB.

mdekkers 3y ago
Yandex consistently pumps out great software. Clickhouse is awesome, for starters, as is this.

wutwutwat 3y ago

Yandex open sourced all of its code a few months back when it was all stolen and leaked.

6c696e7578 3y ago
Didn't the CEO flee due to the war? https://www.intellinews.com/russian-tech-titan-yandex-ceo-volozh-slams-putin-s-barbaric-war-in-ukraine-287651/ https://www.intellinews.com/russian-tech-titan-yandex-ceo-vo...
- wutwutwat 3y ago
  I would have no idea. Why are you asking me and sharing a link to something which tells you. That’s weird, and I’m not interested in clicking the link anyway.
- bthomas 3y ago
  Not sure why the downvotes - very useful context and phrased correctly as a question.

anonyfox 3y ago

Since the discussion started in the comments already, I have a similar question: any recommendations for a solution (don’t care if OSS or not) that has the best UX for nontechnical people to assemble some data and reports anyhow? I have Salesforce, some mariadb/postgres and (optionally) hubspot as data sources. I can buy or manually provision anything, no technical hurdles or policies from that side. My absolute focus is the raw UX for business people. Suggestions?

arthurwu 3y ago
I'm a co-founder of Dataland.io where we're building a powerful dataset viewer + search engine that can work on top of your Postgres or data warehouse. We designed it specifically to provide an excellent UX to business users while reducing BI burden on the data team. We find that most business users often just need to search, filter, and sort instead of looking at charts to make operational decisions. UX-wise, what sets us apart are: - <1s full-text search (even on billions of rows of data), feels like Cmd+F in Google sheets, but faster - Performance: we stream billions of rows into the web browser, seamless scrolling (no paging of 50 records at a tieme) - Rich cells make tables easier to scan/read (enum strings => colored tags, numbers => color-coded based on value => checkboxes, timestamptz => clear date time pills) If that fits what you need, happy to give you a demo. arthur(at)dataland.io Otherwise, I think the simplest BI (if charts are impt) could be something like evidence.dev or Metabase. But I also think it's going to require some curation on your part. Can you reasonably expect business users to navigate the entire schema/table tree across these three sources? That's where I think the bottleneck often lies -- if your BI tool allows engineering to just expose a subset of curated core tables.
robertlagrant 3y ago
The problem is that developing the perfect UI for nontechnical people to assemble reports probably requires a bespoke frontend for your business, and one that likely lags behind the reality of its changes. Most businesses instead opt to just hire semitechnical people that can do a bit of data work and give answers to the report-writers, as they can accommodate business changes over time and understand how to construct new queries out of the overall business' data sources. Maybe that'll change one day with AI, and when it does that will be bought by every big company in the world (-:

totalhack 3y ago

Looks pretty neat. Having not fully investigated it yet, I will say the one thing I usually run into with this and other BI tools is a lack of flexibility in the UI for forming queries. It's sometimes limited to one table or view at a time. I wonder why more of them don't use more flexible querying techniques, perhaps just due to the risk of a bad query being formed? My preferred approach is implemented in Zillion, which I use for BI at my company: https://github.com/totalhack/zillion https://github.com/totalhack/zillion

riffic 3y ago

glaring misuse of it's in the title

Logans_Run 3y ago
Well played ;-) ps - I spotted the pedant that is technically correct, and I claim my 5 McFun bucks!
deleted 3y ago
[deleted]
culebron21 3y ago
Add "'s" for genitive case in English, they said.

felixhummel 3y ago

Apache 2.0 licensed (from a cursory glance at the first few repos).

vgt 3y ago

I'm Ukrainian

macinjosh 3y ago
[flagged]
alexlur 3y ago
Aren’t you American?

esotericimpl 3y ago

[dead]

zxspectrum1982 3y ago

Is there any open source BI tool that can be embedded in some other product so that users (not product developers but final users) can create their own dashboards?

throw03172019 3y ago
Metabase
- zxspectrum1982 3y ago
  IMHO Metabase is too complex for end users: too much "code" (queries) to write and very limited in the presentation layer (widgets users can use to show the results). I was thinking of something closer to Power BI, e. g. something like AppSmith but more BI-oriented (AppSmith is a generic tool to build your own applications, kind of like Visual Basic or Delphi). https://www.appsmith.com/ https://www.appsmith.com/
moltar 3y ago
QuickSight has great embedding story Also CubeJS if you want to a bit more flexibility.

unixhero 3y ago

>Where can I find persistent application data storage? >We use the .us-data folder to store PostgreSQL data permanently. You can delete this folder if you want, it will be recreated with the demo data after restarting the datalens-us container Why "us"?

nijave 3y ago
Unified Storage. https://github.com/datalens-tech/datalens-us https://github.com/datalens-tech/datalens-us