3 ms·

Custom pipes mostly coordinated by Luigi (heavily customized) -> Redshift -> DBT -> Looker. Some spark mixed in at the ingestion and transformation phases. L

by sixo 7y ago

Custom pipes mostly coordinated by Luigi (heavily customized) -> Redshift -> DBT -> Looker. Some spark mixed in at the ingestion and transformation phases. Like someone else said in this thread, we're currently battling Redshift scaling problems and are looking to offload more of the raw data to S3 using Spark to create read views. No data catalog right now but the Looker setup is extremely well-maintained. Hoping to deploy Lyft's Amundsen to fix this more generally.

buremba 7y ago
Since your Looker setup is well-maintained, did you consider adopting the data catalog in Looker, what would be the challenge doing it?
- sixo 7y ago
  The main reason is that only a subset of all our datasets are actually exposed in Looker, and Amundsen would have applicability to other data sources - certain production semi-OLAP DBs, the future hive-on-S3 deployment... that we have no plans to expose in Looker at all. That said I haven't looked closely at how far we could go with Looker, I'll check it out.
  - buremba 7y ago
    I see. We actually develop a BI tool similar to Looker but also want to implement a data catalog feature embedded into it. I'm familiar with Amundsen but I believe that this kind of tool needs to have tight integration with the BI tool and Amundsen still has a way in terms of integrations. (searching among saved reports, dashboards and collaboration with teams, etc.). Otherwise, using spreadsheets is still easier and maintainable. I would love to hear the features that you would need in a data catalog system.
    - mgrover 7y ago
      One of the Amundsen people here - happy to answer questions or hear feedback, if there's anything. Agreed there's more to be done in terms of integrations. Our integrations are pretty focussed on databases and people (https://github.com/lyft/amundsen#supported-connectors https://github.com/lyft/amundsen#supported-connectors). Work on indexing dashboards is being scoped, you can see a rough product spec with a rough mock here: https://docs.google.com/document/d/16cSKgM2sCYvhKq54yfwaHKwslJEGtdS2g5dcPV4p5qo/edit# https://docs.google.com/document/d/16cSKgM2sCYvhKq54yfwaHKws... Feedback welcome!

by sixo 7y ago

buremba 7y ago

Since your Looker setup is well-maintained, did you consider adopting the data catalog in Looker, what would be the challenge doing it?

sixo 7y ago
The main reason is that only a subset of all our datasets are actually exposed in Looker, and Amundsen would have applicability to other data sources - certain production semi-OLAP DBs, the future hive-on-S3 deployment... that we have no plans to expose in Looker at all. That said I haven't looked closely at how far we could go with Looker, I'll check it out.
- buremba 7y ago
  I see. We actually develop a BI tool similar to Looker but also want to implement a data catalog feature embedded into it. I'm familiar with Amundsen but I believe that this kind of tool needs to have tight integration with the BI tool and Amundsen still has a way in terms of integrations. (searching among saved reports, dashboards and collaboration with teams, etc.). Otherwise, using spreadsheets is still easier and maintainable. I would love to hear the features that you would need in a data catalog system.
  - mgrover 7y ago
    One of the Amundsen people here - happy to answer questions or hear feedback, if there's anything. Agreed there's more to be done in terms of integrations. Our integrations are pretty focussed on databases and people (https://github.com/lyft/amundsen#supported-connectors https://github.com/lyft/amundsen#supported-connectors). Work on indexing dashboards is being scoped, you can see a rough product spec with a rough mock here: https://docs.google.com/document/d/16cSKgM2sCYvhKq54yfwaHKwslJEGtdS2g5dcPV4p5qo/edit# https://docs.google.com/document/d/16cSKgM2sCYvhKq54yfwaHKws... Feedback welcome!