7 ms·

Cube.js: Headless Semantic Layer

113 points by klaussilveira 4y ago

rektide 4y ago
I saw a project in similar space- Malloy- go by recently, in a post, What Happened to the Semantic Layer, which I thought nicely setup the space & problem. https://carlineng.com/?postid=semantic-layer#blog https://carlineng.com/?postid=semantic-layer#blog https://news.ycombinator.com/item?id=35715410 https://news.ycombinator.com/item?id=35715410 They indeed mentioned Cube.js. Bunch of Malloy links over time: https://hn.algolia.com/?dateRange=all&query=Malloy&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&query=Malloy&sort=byDa... And cube.js, https://hn.algolia.com/?dateRange=all&query=cube.js&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&query=cube.js&sort=byD... It's all very interesting to me because I get the sense these folks feel like we haven't figured out to to make meaning from the data we have. As a developer, that rings true; the developee is the agent responsible for understanding what we have in sql now & how to eek out meaning, how to structure more info in. The information architectures are so occluded & concealed, so stoggily low level. I love these attempts to try to help us think more about our data.
- adolph 4y ago
  My operating theory is that in app centric development the data store is a component of the application. As a result, much of the metadata that changes data into information (data with meaning) is stored in app code and config. Upon ETL to reporting, warehouse, lake, or the like, the app’s semantic layer is lost.
  - rektide 4y ago
    It's a crying shame how little of the data-core makes it's way out of applications. For both the app's owning entity & especially for the users. I paid respects[1] recently yet again to Window Manager Improved Improved (wmii), which kept it's state in a 9p filesystem any user could easily browse & modify. The state is still a component of the app/window manager, but it's at least malleable to all. [1] https://news.ycombinator.com/item?id=35768686 https://news.ycombinator.com/item?id=35768686
- igorlukanin 4y ago
  So good you brought Malloy here. I like it quite a bit because the folks really try to innovate (heck, they even have their own data querying syntax to replace SQL). But what I like even more — being part of the Cube team — that the "cons" of existing solutions that Carlin mentions in his blog are actually already solved by Cube. With Cube, Data exploration, ideation on the data model, querying, and bringing the insights all the way down to BI tools or data apps takes minutes rather than hours or days. Done, case closed :-)
  - carlineng 4y ago
    My understanding of Cube is that iterating on the data model requires the user to (1) write SQL to develop a metric (2) edit YAML or JS config to incorporate the new metric (3) issue API request to Cube server and (4) compare results to raw SQL. Am I mistaken? Does Cube offer a smoother way to do this exploration/iteration?
    - igorlukanin 4y ago
      Hey, Carlin! Nice to see you here in comments! (Waving "hi" to the Malloy team.) Usually, the experience would look like this: one directly develops the data model in YAML (with only bits of SQL, if needed) and instantly explores metrics. No need to start with SQL in a separate tool/place (1), no need to use the API to check metrics (2) (for that, we have Playground, an interactive UI tool), and, thus, no need to compare results to raw SQL (4). You iterate but changing the data model and seeing the metrics in an instant, quite similar to how you work with Malloy, if I may.
rendall 4y ago
It looks very important, popular and well established, but what is it? I looked at the README hoping to understand what a semantic layer for building data applications means but no love. It helps data engineers and application developers access data from modern data stores, organize it into consistent definitions, and deliver it to every application. Like an ORM? Or a middleware? > Cube was designed to work with all SQL-enabled data sources, including cloud data warehouses like Snowflake or Google BigQuery, query engines like Presto or Amazon Athena, and application databases like Postgres. Still not getting it. Is it that it can perform a single query across multiple databases?
- jrvarela56 4y ago
  From what I read it’s a way to expose SQL via APIs (along with stuff needed to do it like auth, perf, query reuse, etc) Instead of starting from a general purpose web framework+orm you have your data/schema and can query it over http conveniently to build BI/dashboards.
- pastacacioepepe 4y ago
  Say you want to build a dashboard with charts and custom timerange selection using data you already have in Postgres/other DB, without killing your DB under the pressure of queries AND without having to write an additional API? Cube.js is the tool for that. Handles data modeling (you can define a schema on top of your SQL schema), caching, access control and API for you.
  - sails 4y ago
    Data modelling is important to highlight, and if OP not familiar with the concept and need then likely won’t see the obvious value of Cube.
- danieka 4y ago
  Cube has saved me hundred of hours. I use it as backend for reporting and dashboard inside our SaaS. In our frontend I've build a light-version of PowerBI and I use Cube for a backend. Instead of manipulating SQL directly I use Cube's JSON query format. Kind of difficult to explain, but Cube might be the best piece of software I have ever used. Maybe a good tagline would be "self-hostable Backend as a Service for data analysis"?
totalhack 4y ago
Does cube support dynamically figuring out joins or is each cube a hard coded set of joins? My similar, much less polished project doesn't require you to specify joins ahead of time outside of optionally defining a tree-like table lineage: https://github.com/totalhack/zillion https://github.com/totalhack/zillion
- pavel_tiunov 4y ago
  Yep. Cube would automatically figure out the join path for you on top of the defined join graph in the data model using the Dijkstra algorithm. The best practice however is to use views: https://cube.dev/docs/schema/reference/view/#views https://cube.dev/docs/schema/reference/view/#views. Those can be used to explicitly control join paths and get an effect similar to what Looker Explore can provide.
  - totalhack 4y ago
    Cool, do you have an example showing a config/setup that can do dynamic joins? All the examples I see have joins explicitly defined at the cube level in the yaml config. Zillion uses networkx to create a graph of tables and relationships so it sounds like it's doing something similar. Views are a nice guardrail but can also get in the way! I've experienced this frustration first hand (and seen it with my business users) when using tools that focus on such a premise. Guess it depends how complex your data model is and how quickly your business is evolving / adding to that model. In the move-fast-break-stuff phase of a company they can get annoying at least.
- mdaniel 4y ago
  What's happening with this license? I am always worried when the GH license detector says ":shrug:" and I don't know of any easy way to diff the license file against what I presume is a GPLv3 base layer(?) https://github.com/totalhack/zillion/blob/v0.9.14/LICENSE https://github.com/totalhack/zillion/blob/v0.9.14/LICENSE
  - josephcsible 4y ago
    I just did a diff of it against <https://www.gnu.org/licenses/lgpl-3.0.txt https://www.gnu.org/licenses/lgpl-3.0.txt> and the only differences are the lines above "GNU LESSER GENERAL PUBLIC LICENSE" at the top and a blank line at the bottom.
revskill 4y ago
Is this tool kind of useless ? I tried to find all documentation on the website but find no way to embed the dashboard inside my Nextjs application. Or atleast, exposing a metrics as API to be consumed directly from frontend.
- pavel_tiunov 4y ago
  https://cube.dev/docs/config/downstream https://cube.dev/docs/config/downstream might be a great starting point for that. Here you can get a grasp of how dashboard apps can be built: https://cube.dev/docs/examples#tutorials-front-end-integrations https://cube.dev/docs/examples#tutorials-front-end-integrati.... Here's the post about embedding into Next.js app in particular: https://cube.dev/blog/building-nextjs-dashboard-with-dynamic-charts-and-ssr https://cube.dev/blog/building-nextjs-dashboard-with-dynamic....
- fisf 4y ago
  Come on. The three relevant ways to access the system via API (Rest, graphql, SQL) + available frontend integrations are top level entries in the documentation (https://cube.dev/docs https://cube.dev/docs).
  - revskill 4y ago
    Look like the documentation is updated.
jspash 4y ago
There used to be a product called Statsbot which was a friendly UI built on top of Cube.js. They shuttered the service a few years back and we're still struggling to find an alternative that is as simple to use by anyone in the company and easy to set up. We've gone through all the usual BI suspects and nothing comes close. I'd love to find the time to rebuild this.
- bbickell 4y ago
  You might take a look at Delphi (https://www.delphihq.com/ https://www.delphihq.com/) which is built on top of semantic layers and recently integrated with Cube.
- igorlukanin 4y ago
  Oh, it's interesting to meet a Statsbot user in the wild! Indeed, Cube was spun off Statsbot and became the foundation on which others can build products like the one mentioned in the sibling comment: Delphi. Cube acts as the semantic layer, providing the access to data sources and centralizing the data model. Delphi acts as the UI for the end user, enabling them to ask questions in natural language. I've blogged about Cube and Delphi here: https://cube.dev/blog/conversational-interface-for-semantic-layer https://cube.dev/blog/conversational-interface-for-semantic-.... Also, here's a demo video on YouTube I've recorded recently: https://www.youtube.com/watch?v=FotEaaf20gY https://www.youtube.com/watch?v=FotEaaf20gY
- pastacacioepepe 4y ago
  I've never used Statsbot but you can easily integrate cube with retool. They even have a guide for it, it's the first result on google for cube + retool. Also worth mentioning retool it's really cheap compared to "delphi" that I'm seeing shared in other comments.
  - igorlukanin 4y ago
    Here's the Retool guide (https://cube.dev/blog/building-an-internal-dashboard-with-retool-and-cube https://cube.dev/blog/building-an-internal-dashboard-with-re...) but I don't think Retool should be compared to Delphi; one is a low-code tool builder; the other one is a conversational interface for the semantic layer. Both are great for their purposes, both can be used with Cube, even at the same time :-)
krono 4y ago
Beware of their quite well hidden opt-out telemetry collection: https://cube.dev/docs/config#options-reference-telemetry https://cube.dev/docs/config#options-reference-telemetry It's really quite improper to not have this clearly mentioned anywhere obvious.
- charcircuit 4y ago
  Both the configuration option and environment variable for its anonymous telemetry are documented. It's not hidden.
  - krono 4y ago
    Fair enough, although it's not as clearly advertised as I'm of the opinion that it should be, either.
- metadat 4y ago
  A hot topic. Related recent discussions on active-by-default telemetry: Dropbox telemetry can't be disabled (4 days ago, 241 comments) https://news.ycombinator.com/item?id=35724939 https://news.ycombinator.com/item?id=35724939 1Password to Add Telemetry (7 days ago, 342 comments) https://news.ycombinator.com/item?id=35691383 https://news.ycombinator.com/item?id=35691383 Telemetry in Front-End Tools (26 days ago, 141 comments) https://news.ycombinator.com/item?id=35458974 https://news.ycombinator.com/item?id=35458974 Go claims telemetry objectors arguing in bad faith and violating Code of Conduct (77 days ago, 337 comments) https://news.ycombinator.com/item?id=34771472 https://news.ycombinator.com/item?id=34771472 Transparent telemetry for open-source projects (82 days ago, 305 comments) https://news.ycombinator.com/item?id=34707583 https://news.ycombinator.com/item?id=34707583 p.s. The story I first thought of when I saw your reply was the Golang discussion, aka "Transparent telemetry" above.
igorlukanin 4y ago
(Igor from the Cube team here.) Whoa! Great to see Cube here. Would love to take questions about all things Cube, use cases, our docs, developer experience, etc.
domlebo70 4y ago
Cube.js originally allowed embedding the app within an express server as just another route. They took this away and insist on using their Docker deployment approach. I'm still using the original versions that support this.
moltar 4y ago
I’ve toyed around with Cube and it’s certainly a solid and mature product! My only gripe was the cube definition. It uses a weird JS-like DSL, but not real JS. You can’t use any packages or anything. Feels like a strange limitation. Also lacks types, which are table stakes these days. We’ll probably adapt it in the coming SaaS build out!
- pavel_tiunov 4y ago
  You actually can use any npm packages. Here's an example on how you can use `node-fetch`: https://cube.dev/docs/schema/advanced/dynamic-schema-creation#generation https://cube.dev/docs/schema/advanced/dynamic-schema-creatio.... Same capabilities are coming soon for Python and YAML as well.
smrtinsert 4y ago
This is a frontend for data is that right? That's a phenomenal concept if performance is a major priority. Does it work with Salesforce?
- igorlukanin 4y ago
  Frontend as in "access layer" or "API"? Yes, kind of (more than that). Frontend as in "HTML/SVG/JavaScript in the browser"? Certainly not :) Re: Salesforce. The best way to integrate two is to move (ETL) the data from Salesforce to a data warehouse and have Cube connect to that data warehouse. Works really well in the real world!

113 points by klaussilveira 4y ago

I saw a project in similar space- Malloy- go by recently, in a post, What Happened to the Semantic Layer, which I thought nicely setup the space & problem. https://carlineng.com/?postid=semantic-layer#blog https://carlineng.com/?postid=semantic-layer#blog https://news.ycombinator.com/item?id=35715410 https://news.ycombinator.com/item?id=35715410 They indeed mentioned Cube.js. Bunch of Malloy links over time: https://hn.algolia.com/?dateRange=all&query=Malloy&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&query=Malloy&sort=byDa... And cube.js, https://hn.algolia.com/?dateRange=all&query=cube.js&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&query=cube.js&sort=byD... It's all very interesting to me because I get the sense these folks feel like we haven't figured out to to make meaning from the data we have. As a developer, that rings true; the developee is the agent responsible for understanding what we have in sql now & how to eek out meaning, how to structure more info in. The information architectures are so occluded & concealed, so stoggily low level. I love these attempts to try to help us think more about our data.

adolph 4y ago
My operating theory is that in app centric development the data store is a component of the application. As a result, much of the metadata that changes data into information (data with meaning) is stored in app code and config. Upon ETL to reporting, warehouse, lake, or the like, the app’s semantic layer is lost.
- rektide 4y ago
  It's a crying shame how little of the data-core makes it's way out of applications. For both the app's owning entity & especially for the users. I paid respects[1] recently yet again to Window Manager Improved Improved (wmii), which kept it's state in a 9p filesystem any user could easily browse & modify. The state is still a component of the app/window manager, but it's at least malleable to all. [1] https://news.ycombinator.com/item?id=35768686 https://news.ycombinator.com/item?id=35768686
igorlukanin 4y ago
So good you brought Malloy here. I like it quite a bit because the folks really try to innovate (heck, they even have their own data querying syntax to replace SQL). But what I like even more — being part of the Cube team — that the "cons" of existing solutions that Carlin mentions in his blog are actually already solved by Cube. With Cube, Data exploration, ideation on the data model, querying, and bringing the insights all the way down to BI tools or data apps takes minutes rather than hours or days. Done, case closed :-)
- carlineng 4y ago
  My understanding of Cube is that iterating on the data model requires the user to (1) write SQL to develop a metric (2) edit YAML or JS config to incorporate the new metric (3) issue API request to Cube server and (4) compare results to raw SQL. Am I mistaken? Does Cube offer a smoother way to do this exploration/iteration?
  - igorlukanin 4y ago
    Hey, Carlin! Nice to see you here in comments! (Waving "hi" to the Malloy team.) Usually, the experience would look like this: one directly develops the data model in YAML (with only bits of SQL, if needed) and instantly explores metrics. No need to start with SQL in a separate tool/place (1), no need to use the API to check metrics (2) (for that, we have Playground, an interactive UI tool), and, thus, no need to compare results to raw SQL (4). You iterate but changing the data model and seeing the metrics in an instant, quite similar to how you work with Malloy, if I may.

rendall 4y ago

It looks very important, popular and well established, but what is it? I looked at the README hoping to understand what a semantic layer for building data applications means but no love. It helps data engineers and application developers access data from modern data stores, organize it into consistent definitions, and deliver it to every application. Like an ORM? Or a middleware? > Cube was designed to work with all SQL-enabled data sources, including cloud data warehouses like Snowflake or Google BigQuery, query engines like Presto or Amazon Athena, and application databases like Postgres. Still not getting it. Is it that it can perform a single query across multiple databases?

jrvarela56 4y ago
From what I read it’s a way to expose SQL via APIs (along with stuff needed to do it like auth, perf, query reuse, etc) Instead of starting from a general purpose web framework+orm you have your data/schema and can query it over http conveniently to build BI/dashboards.
pastacacioepepe 4y ago
Say you want to build a dashboard with charts and custom timerange selection using data you already have in Postgres/other DB, without killing your DB under the pressure of queries AND without having to write an additional API? Cube.js is the tool for that. Handles data modeling (you can define a schema on top of your SQL schema), caching, access control and API for you.
- sails 4y ago
  Data modelling is important to highlight, and if OP not familiar with the concept and need then likely won’t see the obvious value of Cube.
danieka 4y ago
Cube has saved me hundred of hours. I use it as backend for reporting and dashboard inside our SaaS. In our frontend I've build a light-version of PowerBI and I use Cube for a backend. Instead of manipulating SQL directly I use Cube's JSON query format. Kind of difficult to explain, but Cube might be the best piece of software I have ever used. Maybe a good tagline would be "self-hostable Backend as a Service for data analysis"?

totalhack 4y ago

Does cube support dynamically figuring out joins or is each cube a hard coded set of joins? My similar, much less polished project doesn't require you to specify joins ahead of time outside of optionally defining a tree-like table lineage: https://github.com/totalhack/zillion https://github.com/totalhack/zillion

pavel_tiunov 4y ago
Yep. Cube would automatically figure out the join path for you on top of the defined join graph in the data model using the Dijkstra algorithm. The best practice however is to use views: https://cube.dev/docs/schema/reference/view/#views https://cube.dev/docs/schema/reference/view/#views. Those can be used to explicitly control join paths and get an effect similar to what Looker Explore can provide.
- totalhack 4y ago
  Cool, do you have an example showing a config/setup that can do dynamic joins? All the examples I see have joins explicitly defined at the cube level in the yaml config. Zillion uses networkx to create a graph of tables and relationships so it sounds like it's doing something similar. Views are a nice guardrail but can also get in the way! I've experienced this frustration first hand (and seen it with my business users) when using tools that focus on such a premise. Guess it depends how complex your data model is and how quickly your business is evolving / adding to that model. In the move-fast-break-stuff phase of a company they can get annoying at least.
mdaniel 4y ago
What's happening with this license? I am always worried when the GH license detector says ":shrug:" and I don't know of any easy way to diff the license file against what I presume is a GPLv3 base layer(?) https://github.com/totalhack/zillion/blob/v0.9.14/LICENSE https://github.com/totalhack/zillion/blob/v0.9.14/LICENSE
- josephcsible 4y ago
  I just did a diff of it against <https://www.gnu.org/licenses/lgpl-3.0.txt https://www.gnu.org/licenses/lgpl-3.0.txt> and the only differences are the lines above "GNU LESSER GENERAL PUBLIC LICENSE" at the top and a blank line at the bottom.

revskill 4y ago

Is this tool kind of useless ? I tried to find all documentation on the website but find no way to embed the dashboard inside my Nextjs application. Or atleast, exposing a metrics as API to be consumed directly from frontend.

pavel_tiunov 4y ago
https://cube.dev/docs/config/downstream https://cube.dev/docs/config/downstream might be a great starting point for that. Here you can get a grasp of how dashboard apps can be built: https://cube.dev/docs/examples#tutorials-front-end-integrations https://cube.dev/docs/examples#tutorials-front-end-integrati.... Here's the post about embedding into Next.js app in particular: https://cube.dev/blog/building-nextjs-dashboard-with-dynamic-charts-and-ssr https://cube.dev/blog/building-nextjs-dashboard-with-dynamic....
fisf 4y ago
Come on. The three relevant ways to access the system via API (Rest, graphql, SQL) + available frontend integrations are top level entries in the documentation (https://cube.dev/docs https://cube.dev/docs).
- revskill 4y ago
  Look like the documentation is updated.

jspash 4y ago

There used to be a product called Statsbot which was a friendly UI built on top of Cube.js. They shuttered the service a few years back and we're still struggling to find an alternative that is as simple to use by anyone in the company and easy to set up. We've gone through all the usual BI suspects and nothing comes close. I'd love to find the time to rebuild this.

bbickell 4y ago
You might take a look at Delphi (https://www.delphihq.com/ https://www.delphihq.com/) which is built on top of semantic layers and recently integrated with Cube.
igorlukanin 4y ago
Oh, it's interesting to meet a Statsbot user in the wild! Indeed, Cube was spun off Statsbot and became the foundation on which others can build products like the one mentioned in the sibling comment: Delphi. Cube acts as the semantic layer, providing the access to data sources and centralizing the data model. Delphi acts as the UI for the end user, enabling them to ask questions in natural language. I've blogged about Cube and Delphi here: https://cube.dev/blog/conversational-interface-for-semantic-layer https://cube.dev/blog/conversational-interface-for-semantic-.... Also, here's a demo video on YouTube I've recorded recently: https://www.youtube.com/watch?v=FotEaaf20gY https://www.youtube.com/watch?v=FotEaaf20gY
pastacacioepepe 4y ago
I've never used Statsbot but you can easily integrate cube with retool. They even have a guide for it, it's the first result on google for cube + retool. Also worth mentioning retool it's really cheap compared to "delphi" that I'm seeing shared in other comments.
- igorlukanin 4y ago
  Here's the Retool guide (https://cube.dev/blog/building-an-internal-dashboard-with-retool-and-cube https://cube.dev/blog/building-an-internal-dashboard-with-re...) but I don't think Retool should be compared to Delphi; one is a low-code tool builder; the other one is a conversational interface for the semantic layer. Both are great for their purposes, both can be used with Cube, even at the same time :-)

krono 4y ago

Beware of their quite well hidden opt-out telemetry collection: https://cube.dev/docs/config#options-reference-telemetry https://cube.dev/docs/config#options-reference-telemetry It's really quite improper to not have this clearly mentioned anywhere obvious.

charcircuit 4y ago
Both the configuration option and environment variable for its anonymous telemetry are documented. It's not hidden.
- krono 4y ago
  Fair enough, although it's not as clearly advertised as I'm of the opinion that it should be, either.
metadat 4y ago
A hot topic. Related recent discussions on active-by-default telemetry: Dropbox telemetry can't be disabled (4 days ago, 241 comments) https://news.ycombinator.com/item?id=35724939 https://news.ycombinator.com/item?id=35724939 1Password to Add Telemetry (7 days ago, 342 comments) https://news.ycombinator.com/item?id=35691383 https://news.ycombinator.com/item?id=35691383 Telemetry in Front-End Tools (26 days ago, 141 comments) https://news.ycombinator.com/item?id=35458974 https://news.ycombinator.com/item?id=35458974 Go claims telemetry objectors arguing in bad faith and violating Code of Conduct (77 days ago, 337 comments) https://news.ycombinator.com/item?id=34771472 https://news.ycombinator.com/item?id=34771472 Transparent telemetry for open-source projects (82 days ago, 305 comments) https://news.ycombinator.com/item?id=34707583 https://news.ycombinator.com/item?id=34707583 p.s. The story I first thought of when I saw your reply was the Golang discussion, aka "Transparent telemetry" above.

igorlukanin 4y ago

(Igor from the Cube team here.) Whoa! Great to see Cube here. Would love to take questions about all things Cube, use cases, our docs, developer experience, etc.

domlebo70 4y ago

Cube.js originally allowed embedding the app within an express server as just another route. They took this away and insist on using their Docker deployment approach. I'm still using the original versions that support this.

moltar 4y ago

I’ve toyed around with Cube and it’s certainly a solid and mature product! My only gripe was the cube definition. It uses a weird JS-like DSL, but not real JS. You can’t use any packages or anything. Feels like a strange limitation. Also lacks types, which are table stakes these days. We’ll probably adapt it in the coming SaaS build out!

pavel_tiunov 4y ago
You actually can use any npm packages. Here's an example on how you can use `node-fetch`: https://cube.dev/docs/schema/advanced/dynamic-schema-creation#generation https://cube.dev/docs/schema/advanced/dynamic-schema-creatio.... Same capabilities are coming soon for Python and YAML as well.

smrtinsert 4y ago

This is a frontend for data is that right? That's a phenomenal concept if performance is a major priority. Does it work with Salesforce?

igorlukanin 4y ago
Frontend as in "access layer" or "API"? Yes, kind of (more than that). Frontend as in "HTML/SVG/JavaScript in the browser"? Certainly not :) Re: Salesforce. The best way to integrate two is to move (ETL) the data from Salesforce to a data warehouse and have Cube connect to that data warehouse. Works really well in the real world!