7 ms·

The Modern Metadata Platform: What, Why, and How?

35 points by marslan2000 5y ago

virgilp 5y ago
What is the product? It's really hard to figure from the home page, so I looked up on the web and found this[1] > Today, we are excited to announce that Mars, Seyi, and I have teamed up to start Metaphor Data, a company dedicated to building out the DataHub ecosystem. Our mission at Metaphor Data is to help all organizations better understand and manage their data through the power of the metadata knowledge graph. So, it's DataHub (hosted I guess) and ... something else? But I'm still not sure what this "ecosystem" is. [1] https://pardhugunnam.medium.com/announcing-metaphor-data-c86170c81202 https://pardhugunnam.medium.com/announcing-metaphor-data-c86...
- tedmiston 5y ago
  Yeah, the post didn't do a good job of explaining what their product actually is / does. Their About page is similarly vague: > We are on a mission to empower individuals and companies to change the world through data. This mission started when we created DataHub at LinkedIn. We’re leveraging the learnings from that experience to build a modern, scalable, and cloud-first data catalog and discovery service. It sure sounds like vaporware from that description. ¯\_(ツ)_/¯ All I could gather from their website is that they catalog schemas across various databases / data warehouses. It looks like the team is all engineers — they could really use a marketing & comms person…
- marslan2000 5y ago
  Thank you for your interest. We launched our product about 2 months ago. You can find out more about the problems Metaphor is solving here: https://metaphor.io/blog/metaphor-product-launch https://metaphor.io/blog/metaphor-product-launch
  - virgilp 5y ago
    I still don't fully get what the product is. In the launch, you state that "Metaphor has its root in DataHub [...] However, Metaphor is not a managed version of DataHub". My observation: - perhaps intentionally, but you really go to great lengths to obfuscate what the product actually does. How is it NOT DataHub? Is it a new implementation of DataHub? Is it extending DataHub somehow? (I would expect the latter, but it's unclear exactly how it extends it, and you never mention exactly how it's different). - it's very weird to place that important description in "who we are" (I almost skipped it), instead of e.g. "How does Metaphor work". - lastly, I kinda' feel that this needs to be in the homepage somewhere, not in a blog post somewhere. Not sure who you're targeting with your marketing copy - but if you're targeting engineers who'd deploy a data catalog, this can't be all fluff. Give me something concrete that I can work with: I'm interested in the domain, I want to implement this in my company, but I need to understand what you're offering. Saying that you have a "platform" tells me nothing. Tell me why I should use Metaphor instead of e.g. deploying DataHub myself.
    - marslan2000 5y ago
      Appreciate the feedback. We'll work on improving the clarity in our messaging.
lloydatkinson 5y ago
Is this just the new way of saying "big data"?
- edmundsauto 5y ago
  No, it’s saying how important metadata is, going over the components and key features for a platform to manage metadata, and includes some of the history of that category of software at LI. If you’re at a company with more than 10-20 people doing data science type work, this is an important service.
  - blueblob 5y ago
    Perhaps, but it's also a service you are unlikely to want to outsource if your data is proprietary.
    - marslan2000 5y ago
      You're absolutely right. That's why we treat our customer's data with utmost care, even if it's just metadata. Here's our security white paper if you're interested in learning more: https://www.notion.so/metaphordata/Security-Whitepaper-371ac3692dba4d2da998bc1b0d80db6c https://www.notion.so/metaphordata/Security-Whitepaper-371ac...
    - rising-sky 5y ago
      Why not? I don't think that's an issue if the actual data is already sitting on a third-party cloud in some middle-of-nowhere undisclosed location or being transited through various other cloud services, right?
      - edmundsauto 5y ago
        Depends on the data, but everyone should be treating metadata at the same classification level as the underlying data. Perhaps even moreso, as metadata leaks information about the company, versus leaking data about the user. (Rightly or wrongly, things that harm the company are more likely to be prioritized.)
iblaine 5y ago
I'm losing count of all the metadata management systems being open sourced. First there was Wherehows out of LinkedIn [1]. Then there was Amundsen out of Lyft, which is now stemma.io [2]. Then there was Marquez out of WeWork [3]. Now we have DataHub out of LI. It's a little crazy how underserved this problem was several years ago and how over served it's becoming. I use Amundsen and am pretty happy with it. [1] https://engineering.linkedin.com/blog/2016/03/open-sourcing-wherehows--a-data-discovery-and-lineage-portal https://engineering.linkedin.com/blog/2016/03/open-sourcing-... [2] https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9?gi=2f233e51dc79 https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-... [3] https://marquezproject.github.io/marquez/ https://marquezproject.github.io/marquez/
- marslan2000 5y ago
  Thanks for pointing it out. You may find this post interesting and many of these OSS projects (including DataHub) failed to deliver the promises: https://metaphor.io/blog/why-cant-i-find-the-right-data https://metaphor.io/blog/why-cant-i-find-the-right-data
- adamfeldman 5y ago
  I think it’s https://www.stemma.ai https://www.stemma.ai. Thank you for this broad list!
beckingz 5y ago
One day we will be able to automagically add an alert to a dashboard saying that the database four systems upstream is having a temporary quality issue. Any tool that gets us in that direction is good in my book.
- marslan2000 5y ago
  Thank you for the suggestion. That's exactly what we have with our recent integration with Soda: https://metaphor.io/blog/metaphor-and-soda-partner-to-unify-the-modern-data-stack-with-trusted-data https://metaphor.io/blog/metaphor-and-soda-partner-to-unify-...

35 points by marslan2000 5y ago

What is the product? It's really hard to figure from the home page, so I looked up on the web and found this[1] > Today, we are excited to announce that Mars, Seyi, and I have teamed up to start Metaphor Data, a company dedicated to building out the DataHub ecosystem. Our mission at Metaphor Data is to help all organizations better understand and manage their data through the power of the metadata knowledge graph. So, it's DataHub (hosted I guess) and ... something else? But I'm still not sure what this "ecosystem" is. [1] https://pardhugunnam.medium.com/announcing-metaphor-data-c86170c81202 https://pardhugunnam.medium.com/announcing-metaphor-data-c86...

tedmiston 5y ago
Yeah, the post didn't do a good job of explaining what their product actually is / does. Their About page is similarly vague: > We are on a mission to empower individuals and companies to change the world through data. This mission started when we created DataHub at LinkedIn. We’re leveraging the learnings from that experience to build a modern, scalable, and cloud-first data catalog and discovery service. It sure sounds like vaporware from that description. ¯\_(ツ)_/¯ All I could gather from their website is that they catalog schemas across various databases / data warehouses. It looks like the team is all engineers — they could really use a marketing & comms person…
marslan2000 5y ago
Thank you for your interest. We launched our product about 2 months ago. You can find out more about the problems Metaphor is solving here: https://metaphor.io/blog/metaphor-product-launch https://metaphor.io/blog/metaphor-product-launch
- virgilp 5y ago
  I still don't fully get what the product is. In the launch, you state that "Metaphor has its root in DataHub [...] However, Metaphor is not a managed version of DataHub". My observation: - perhaps intentionally, but you really go to great lengths to obfuscate what the product actually does. How is it NOT DataHub? Is it a new implementation of DataHub? Is it extending DataHub somehow? (I would expect the latter, but it's unclear exactly how it extends it, and you never mention exactly how it's different). - it's very weird to place that important description in "who we are" (I almost skipped it), instead of e.g. "How does Metaphor work". - lastly, I kinda' feel that this needs to be in the homepage somewhere, not in a blog post somewhere. Not sure who you're targeting with your marketing copy - but if you're targeting engineers who'd deploy a data catalog, this can't be all fluff. Give me something concrete that I can work with: I'm interested in the domain, I want to implement this in my company, but I need to understand what you're offering. Saying that you have a "platform" tells me nothing. Tell me why I should use Metaphor instead of e.g. deploying DataHub myself.
  - marslan2000 5y ago
    Appreciate the feedback. We'll work on improving the clarity in our messaging.

lloydatkinson 5y ago

Is this just the new way of saying "big data"?

edmundsauto 5y ago
No, it’s saying how important metadata is, going over the components and key features for a platform to manage metadata, and includes some of the history of that category of software at LI. If you’re at a company with more than 10-20 people doing data science type work, this is an important service.
- blueblob 5y ago
  Perhaps, but it's also a service you are unlikely to want to outsource if your data is proprietary.
  - marslan2000 5y ago
    You're absolutely right. That's why we treat our customer's data with utmost care, even if it's just metadata. Here's our security white paper if you're interested in learning more: https://www.notion.so/metaphordata/Security-Whitepaper-371ac3692dba4d2da998bc1b0d80db6c https://www.notion.so/metaphordata/Security-Whitepaper-371ac...
  - rising-sky 5y ago
    Why not? I don't think that's an issue if the actual data is already sitting on a third-party cloud in some middle-of-nowhere undisclosed location or being transited through various other cloud services, right?
    - edmundsauto 5y ago
      Depends on the data, but everyone should be treating metadata at the same classification level as the underlying data. Perhaps even moreso, as metadata leaks information about the company, versus leaking data about the user. (Rightly or wrongly, things that harm the company are more likely to be prioritized.)

iblaine 5y ago

I'm losing count of all the metadata management systems being open sourced. First there was Wherehows out of LinkedIn [1]. Then there was Amundsen out of Lyft, which is now stemma.io [2]. Then there was Marquez out of WeWork [3]. Now we have DataHub out of LI. It's a little crazy how underserved this problem was several years ago and how over served it's becoming. I use Amundsen and am pretty happy with it. [1] https://engineering.linkedin.com/blog/2016/03/open-sourcing-wherehows--a-data-discovery-and-lineage-portal https://engineering.linkedin.com/blog/2016/03/open-sourcing-... [2] https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9?gi=2f233e51dc79 https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-... [3] https://marquezproject.github.io/marquez/ https://marquezproject.github.io/marquez/

marslan2000 5y ago
Thanks for pointing it out. You may find this post interesting and many of these OSS projects (including DataHub) failed to deliver the promises: https://metaphor.io/blog/why-cant-i-find-the-right-data https://metaphor.io/blog/why-cant-i-find-the-right-data
adamfeldman 5y ago
I think it’s https://www.stemma.ai https://www.stemma.ai. Thank you for this broad list!

beckingz 5y ago

One day we will be able to automagically add an alert to a dashboard saying that the database four systems upstream is having a temporary quality issue. Any tool that gets us in that direction is good in my book.

marslan2000 5y ago
Thank you for the suggestion. That's exactly what we have with our recent integration with Soda: https://metaphor.io/blog/metaphor-and-soda-partner-to-unify-the-modern-data-stack-with-trusted-data https://metaphor.io/blog/metaphor-and-soda-partner-to-unify-...