3 ms·

http://Singer.io http://Singer.io is an open source ETL project written in Python. The components are small, composable programs that you can run independently

http://Singer.io http://Singer.io is an open source ETL project written in Python. The components are small, composable programs that you can run independently, so you should be able to walk before your run. A good beginner project is to build an adapter to a new data source (known as a "tap" in Singer). Most taps pull data out of business tools like Salesforce or Marketo, but people also build them to pull characters from the Marvel API (https://www.stitchdata.com/blog/tapping-marvel-api/ https://www.stitchdata.com/blog/tapping-marvel-api/) Check out the getting started guide (https://github.com/singer-io/getting-started https://github.com/singer-io/getting-started), or jump into the Singer Slack and ask for help (linked from the guide)

zabana 8y ago
I will look into this, thanks for sharing !
arafalov 8y ago
An Apache Solr (Search Engine) Target could be quite nice here.
vladsanchez 8y ago
Fantastic reference! Thanks for sharing. As an ETL specialist I've looked for a composable solution but only found Luigi. I wanted something simpler, more integrated, like IFTTT. This has the feel of Yahoo Pipes. I think Singer is the answer. ;-)
- dttos 8y ago
  Composableanalytics.com might be the tool you are looking for then....
tomnipotent 8y ago
And after you use Singer to ingest data, add in dbt [0] to transform the raw data into something more useful - I cannot speak highly enough of this library and how much work it will save you. Also keep an eye on Gitlab's new Meltano [1] project which IMHO has huge potential. [0] https://www.getdbt.com/ https://www.getdbt.com/ [1] https://gitlab.com/meltano/meltano/tree/master https://gitlab.com/meltano/meltano/tree/master
- veritas3241 8y ago
  Thanks for the shout-out on Meltano! And we love dbt! Great project and great people behind the code. Their next release is going to be awesome - the new docs feature is beautiful.

zabana 8y ago

I will look into this, thanks for sharing !

arafalov 8y ago

An Apache Solr (Search Engine) Target could be quite nice here.

vladsanchez 8y ago

Fantastic reference! Thanks for sharing. As an ETL specialist I've looked for a composable solution but only found Luigi. I wanted something simpler, more integrated, like IFTTT. This has the feel of Yahoo Pipes. I think Singer is the answer. ;-)

dttos 8y ago
Composableanalytics.com might be the tool you are looking for then....

tomnipotent 8y ago

And after you use Singer to ingest data, add in dbt [0] to transform the raw data into something more useful - I cannot speak highly enough of this library and how much work it will save you. Also keep an eye on Gitlab's new Meltano [1] project which IMHO has huge potential. [0] https://www.getdbt.com/ https://www.getdbt.com/ [1] https://gitlab.com/meltano/meltano/tree/master https://gitlab.com/meltano/meltano/tree/master

veritas3241 8y ago
Thanks for the shout-out on Meltano! And we love dbt! Great project and great people behind the code. Their next release is going to be awesome - the new docs feature is beautiful.