3 ms·

Show HN: OpenMetadata – OSS platform for data discovery observability governance

19 points by jpnguyen 2y ago

Hi HN, OpenMetadata is an open-source metadata platform built to solve complex data management challenges and deliver high quality data assets. It’s based on the experiences of solving the data problems encountered at companies like Uber and Hortonworks. Our team includes the creators of Hadoop, Atlas, and Uber Databook, as well as committers/PMC of projects like Apache Kafka and Storm. Unified Metadata Platform * Unified: A single platform for data discovery, governance, and data observability. * API-first: Standardized metadata specifications using well-designed REST APIs, crucial for anyone consuming or pushing metadata for modern data teams. * Metadata platform: The data catalog is just one of the many applications of metadata built on the platform, vs different siloed data catalog tools, quality tools, governance tools, etc. that each have to be set up and maintained. Please check out more details about our project at https://open-metadata.org https://open-metadata.org and join our community in Slack https://slack.open-metadata.org https://slack.open-metadata.org. You can also try out our live product sandbox with demo data at https://sandbox.open-metadata.org/ https://sandbox.open-metadata.org/.

shailesh_parmar 2y ago
Thanks for sharing the details!
purpleblue 2y ago
What is "data discovery, governance, and data observability"?
- harsha_ch 2y ago
  We explained the need for a centralized metadata repository here https://blog.open-metadata.org/announcing-openmetadata-20399b816e60 https://blog.open-metadata.org/announcing-openmetadata-20399... . Please give it a read.
DavidMankin 2y ago
How does this compare to DataHub? I have been comparing the two and I don't have any pros/cons that really help me pick. What I've noticed: * It seems like DataHub has an async Kafka ingestion approach while OpenMetadata is API * It seems that the data model of OpenMetadata can be extended/modified by API calls while DataHub only allows for changes to be made in code (in a forked version, no less) and deployed. * It seems that DataHub comes with more data types out-of-the-box such as JSON Schema
- harsha_ch 2y ago
  * It seems like DataHub has an async Kafka ingestion approach while OpenMetadata is API We do not use Kafka by default. If someone needs kafka they can add it. However for Metadata APIs, we do not feel like Kafka is needed. Lot of projects are getting dependent on Kafka and calling it as real-time. Its unnecessary burden on users who are going to operate in production for 99% of use-cases Kafka is not needed, coming from a Kafka committer :) 2. Yes all of our APIs and Entity definitions are generated using JsonSchema. For us, Json Schema has been awesome, all of our backend / ingestion and UI is generated from JsonSchema and its easy to extend and add new models when needed 3. IMO, we have much more coverage , you can look at the types available here https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema https://github.com/open-metadata/OpenMetadata/tree/main/open... and we are support JsonSchema as a type from a long time
chirag-madlani 2y ago
Indeed fastest growing OSS project. I love the support and active slack community.
deleted 2y ago
[deleted]
shah_sank3t 2y ago
It is one of the fastest-growing OSS projects focusing on a great user experience. Not all open-source projects focus on UI, as the OpenMetadata team does.
bloody_user 2y ago
Fastest growing OSS project in data cataloging space! Love the Slack community support and the business challenges the project aims to solve for Organisations!
sushi30 2y ago
wish I knew about this project sooner. so much easier to manage our data ecosystem with OpenMetadata. We used to have everything scattered all over the place. catalog here, docs on a siloed website, conversations in slack and for data quality we had a legacy process that was only accessible to engineers. With OpenMetadata everything is one place which means better collaboration and faster deliveries!
aballiet 2y ago
Fast growing project with a supportive community. A lot of connectors available and an UI that is smooth. Deployment is easy with a lot of possibilities to bundle with managed services (elastic, Postgres, external orchestrate like Kestra in my case). Easy to build on top of it thanks a large range of APIs endpoints making it easy to query what's needed.
- pmbrull 2y ago
  + a lot of great contributors such as yourself :) Glad to have you in the community!

19 points by jpnguyen 2y ago

shailesh_parmar 2y ago

Thanks for sharing the details!

purpleblue 2y ago

What is "data discovery, governance, and data observability"?

harsha_ch 2y ago
We explained the need for a centralized metadata repository here https://blog.open-metadata.org/announcing-openmetadata-20399b816e60 https://blog.open-metadata.org/announcing-openmetadata-20399... . Please give it a read.

DavidMankin 2y ago

How does this compare to DataHub? I have been comparing the two and I don't have any pros/cons that really help me pick. What I've noticed: * It seems like DataHub has an async Kafka ingestion approach while OpenMetadata is API * It seems that the data model of OpenMetadata can be extended/modified by API calls while DataHub only allows for changes to be made in code (in a forked version, no less) and deployed. * It seems that DataHub comes with more data types out-of-the-box such as JSON Schema

harsha_ch 2y ago
* It seems like DataHub has an async Kafka ingestion approach while OpenMetadata is API We do not use Kafka by default. If someone needs kafka they can add it. However for Metadata APIs, we do not feel like Kafka is needed. Lot of projects are getting dependent on Kafka and calling it as real-time. Its unnecessary burden on users who are going to operate in production for 99% of use-cases Kafka is not needed, coming from a Kafka committer :) 2. Yes all of our APIs and Entity definitions are generated using JsonSchema. For us, Json Schema has been awesome, all of our backend / ingestion and UI is generated from JsonSchema and its easy to extend and add new models when needed 3. IMO, we have much more coverage , you can look at the types available here https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema https://github.com/open-metadata/OpenMetadata/tree/main/open... and we are support JsonSchema as a type from a long time

chirag-madlani 2y ago

Indeed fastest growing OSS project. I love the support and active slack community.

deleted 2y ago

[deleted]

shah_sank3t 2y ago

It is one of the fastest-growing OSS projects focusing on a great user experience. Not all open-source projects focus on UI, as the OpenMetadata team does.

bloody_user 2y ago

Fastest growing OSS project in data cataloging space! Love the Slack community support and the business challenges the project aims to solve for Organisations!

sushi30 2y ago

wish I knew about this project sooner. so much easier to manage our data ecosystem with OpenMetadata. We used to have everything scattered all over the place. catalog here, docs on a siloed website, conversations in slack and for data quality we had a legacy process that was only accessible to engineers. With OpenMetadata everything is one place which means better collaboration and faster deliveries!

aballiet 2y ago

Fast growing project with a supportive community. A lot of connectors available and an UI that is smooth. Deployment is easy with a lot of possibilities to bundle with managed services (elastic, Postgres, external orchestrate like Kestra in my case). Easy to build on top of it thanks a large range of APIs endpoints making it easy to query what's needed.

pmbrull 2y ago
+ a lot of great contributors such as yourself :) Glad to have you in the community!