4 ms·

could anyone tldr what meltano does and where the name comes from? is it mELTano of ELT?

rconti 5y ago
anagram of Menalto? (Menlo/Palo Alto)?
tayloramurphy 5y ago
I can't find the issue, but originally it stood for different parts of the data lifecycle - Model, Extract, Load, Transform, Analyze, Notebook, Orchestrate. We don't really highlight that much anymore, but there's no denying the roots!
sytse 5y ago
Meltano is a open source platform for automating ELT pipelines.
- rad_gruchalski 5y ago
  Is there any significant difference from the traditional ETL?
  - Guest42 5y ago
    I could imagine a scenario where the target server has more capacity and thereby a better place to do the transforming steps.
  - visch 5y ago
    The idea is lean towards doing no transformation before loading into your Target Database/Datawarehouse. Do all transformations after the data hits your target database. DEV time is expensive, storage and CPU is cheap now. DBs and DWs are blazing fast now. You can still only pull the data that you need using Meltano, but you select things on a "Stream" / "Attributes" level. For source DBs this means something like Table and Field level is as granular as you get.
aliswe 5y ago
not clear at all what it is
- DouweM 5y ago
  Meltano is an open source platform for ELT (Extract, Load Transform) pipelines that leverages Singer for data integration and dbt for transformation. It may not be clear from the blog post (since it's not focused on introducing the product), but I hope the homepage would get that across: https://meltano.com/ https://meltano.com/ Please let me (Meltano CEO) know how we could make that more obvious!
  - simonw 5y ago
    https://meltano.com/ https://meltano.com/ right now says: "ELT for the DataOps era Meltano is open source, self-hosted, CLI-first, debuggable, and extensible." That doesn't tell me what Meltano is. I happen to recognise "ELT" as "Extract, load, transform" but I imagine a lot of people would have to look up the acronym. I hadn't seen the term DataOps before, despite working in this area. DevOps for Data? But the definition of DevOps itself is still pretty vague! I know how hard it is to explain what a product like this does - I'm definitely not there with my own https://datasette.io/ https://datasette.io/ project yet - but what you've got right now could definitely do with some improvement. Here's my attempt for Meltano, based on a very loose understanding of what it is: "Meltano is an open-source tool for building automated pipelines for extracting and converting data. Pipelines are defined using version-controlled code, and the framework can be extended using plugins written in Python." This really isn't very good, but it does fill in the questions that I had when I first visited https://meltano.com/ https://meltano.com/
    - Perihelion 5y ago
      Thanks! This is really useful feedback. To date, Meltano has been geared towards data folks who are likely to understand these concepts and acronyms, but there's definitely a need to explain it for folks who aren't as familiar. We're going to be working on the website copy in the coming weeks to make it easier to understand with as little digging as possible. If you think of anything else please feel free to reach out to us.

anagram of Menalto? (Menlo/Palo Alto)?

I can't find the issue, but originally it stood for different parts of the data lifecycle - Model, Extract, Load, Transform, Analyze, Notebook, Orchestrate. We don't really highlight that much anymore, but there's no denying the roots!

sytse 5y ago

Meltano is a open source platform for automating ELT pipelines.

rad_gruchalski 5y ago
Is there any significant difference from the traditional ETL?
- Guest42 5y ago
  I could imagine a scenario where the target server has more capacity and thereby a better place to do the transforming steps.
- visch 5y ago
  The idea is lean towards doing no transformation before loading into your Target Database/Datawarehouse. Do all transformations after the data hits your target database. DEV time is expensive, storage and CPU is cheap now. DBs and DWs are blazing fast now. You can still only pull the data that you need using Meltano, but you select things on a "Stream" / "Attributes" level. For source DBs this means something like Table and Field level is as granular as you get.

aliswe 5y ago

not clear at all what it is

DouweM 5y ago
Meltano is an open source platform for ELT (Extract, Load Transform) pipelines that leverages Singer for data integration and dbt for transformation. It may not be clear from the blog post (since it's not focused on introducing the product), but I hope the homepage would get that across: https://meltano.com/ https://meltano.com/ Please let me (Meltano CEO) know how we could make that more obvious!
- simonw 5y ago
  https://meltano.com/ https://meltano.com/ right now says: "ELT for the DataOps era Meltano is open source, self-hosted, CLI-first, debuggable, and extensible." That doesn't tell me what Meltano is. I happen to recognise "ELT" as "Extract, load, transform" but I imagine a lot of people would have to look up the acronym. I hadn't seen the term DataOps before, despite working in this area. DevOps for Data? But the definition of DevOps itself is still pretty vague! I know how hard it is to explain what a product like this does - I'm definitely not there with my own https://datasette.io/ https://datasette.io/ project yet - but what you've got right now could definitely do with some improvement. Here's my attempt for Meltano, based on a very loose understanding of what it is: "Meltano is an open-source tool for building automated pipelines for extracting and converting data. Pipelines are defined using version-controlled code, and the framework can be extended using plugins written in Python." This really isn't very good, but it does fill in the questions that I had when I first visited https://meltano.com/ https://meltano.com/
  - Perihelion 5y ago
    Thanks! This is really useful feedback. To date, Meltano has been geared towards data folks who are likely to understand these concepts and acronyms, but there's definitely a need to explain it for folks who aren't as familiar. We're going to be working on the website copy in the coming weeks to make it easier to understand with as little digging as possible. If you think of anything else please feel free to reach out to us.