4 ms·

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is n

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

ben_jones 8y ago
No tool but I've had success writing code generators to better interface between various data systems, autogeneratoring various accessors and utility functions.
fraserharris 8y ago
I would suggest checking out my employer, Fivetran (fivetran.com). Many tech firms use us to centralize their data for analysis. We have startup pricing for sub-50 employees.
- NovaX 8y ago
  Do you know when DB/2 support will be added (Windows, AS/400)? I've been wanting to switch over from AWS DMS and our in-house support.
  - fraserharris 8y ago
    We are planning on adding support for DB/2 in Q1/Q2 2019. We are getting SAP HANA out first
  - jakestein 8y ago
    We (Stitch) have DB2 in production today https://www.stitchdata.com/integrations/db2/ https://www.stitchdata.com/integrations/db2/
    - NovaX 8y ago
      Is it log-based cdc? The documentation says only 2 dbs are supported that way. We want to replicate customer databases which have schemas outside of our control, with the goal of migrating them to our products. The other vendor schemas might not have your required columns. There are also many flavors of DB2. So very interested but the details are sparse at best.
      - jakestein 8y ago
        Our DB2 integration is SELECT replication. Our MySQL, Posgtgres, and Oracle integrations are log base. MySQL and Postgres actually have the option of either logs or SELECT. You're totally right about all of the flavors DB2. Our support team would be the ones to figure out for sure whether or not we can work with your setup, and you reach them at support@stitchdata.com
mycelium 8y ago
I would highly highly recommend ETL as service, after adopting it recently. It substantially changes your relationship with your data sources in a really positive way. And frankly, ETL for common data sources is code that you just don't need to write. I would say that you should pilot with a few ETL vendors. We currently use Fivetran, they're fine but we've had enough burps that I cannot cold recommend them over other vendors. I cannot for the life of me remember the details, but I think we went with them over Stitch for pricing reasons.
- georgewfraser 8y ago
  I'm Fivetran's CEO and I just want you to know, whatever "burps" you experienced, these things keep me up at night and the whole team is always striving to make the pipeline "just work". The whole vision of our product is that you should be able to plug in and get a perfect mirror image of all your data sources in your data warehouse. Anytime we fall short of that it drives us crazy.
  - Petefine 8y ago
    Do you have a forum or suggestions tool at all? Fivetran has been amazing for our new datawarehouse and we're very pleased with the service, but there are a few little (non-bug) things that would have made it even easier.
    - fraserharris 8y ago
      You can email me: fraser@fivetran.com
veritas3241 8y ago
We used them for a single data source at GitLab (Zendesk). Worked pretty well! But we quickly hit a road block where what you could get via the UI wasn't all the tables available. We wound up forking it and adding to the tap. Basically all of our extractors and loaders are going to follow the Singer spec for taps and targets - it's a pretty nice model. Internally, we're using a tool called Meltano which is aimed at solving just your problem. Most of our data warehouse is coming from external business ops tools (Salesforce, Zuora, Zendesk, Marketo, etc.) and we're using dbt for transformations w/ Looker as the BI layer. Definitely check our primary analytics repo [0] as all of our code is out in the open. Feel free to ping me if you have more questions - tmurphy at gitlab. [0] https://gitlab.com/meltano/analytics/ https://gitlab.com/meltano/analytics/
Zaheer 8y ago
I work on AWS Glue (https://aws.amazon.com/glue/ https://aws.amazon.com/glue/). Depending on how complicated the pipeline you'd like to setup, you may not need to write any code.
jakecodes 8y ago
Check out Meltano, a GitLab startup. It’s new and we’re iterating but it sounds like it could solve your exact situation. Feel free to leave issues where we could improve. https://gitlab.com/meltano/meltano https://gitlab.com/meltano/meltano
drunkpotato 8y ago
At the company I work for, we have integrations with Stitch Data and Fivetran. Both are good and have been responsive to my needs. Neither have been perfect, so when I've noticed a problem I've had to keep on top of them to fix it. I also maintain a few of our own ETL jobs for data sources that aren't supported. I will say that I recommend using an ETL vendor without reservation. The nominal cost is more than made up for in the headaches you'll save yourself in creating and maintaining a homegrown ETL.
jpulec 8y ago
I've used StitchData at a startup with AWS Redshift. Pair it with something like dbt for transforming your data, and you have a great match. A little pricey, but totally worth it, IMO.
deleted 8y ago
[deleted]

ben_jones 8y ago

No tool but I've had success writing code generators to better interface between various data systems, autogeneratoring various accessors and utility functions.

fraserharris 8y ago

I would suggest checking out my employer, Fivetran (fivetran.com). Many tech firms use us to centralize their data for analysis. We have startup pricing for sub-50 employees.

NovaX 8y ago
Do you know when DB/2 support will be added (Windows, AS/400)? I've been wanting to switch over from AWS DMS and our in-house support.
- fraserharris 8y ago
  We are planning on adding support for DB/2 in Q1/Q2 2019. We are getting SAP HANA out first
- jakestein 8y ago
  We (Stitch) have DB2 in production today https://www.stitchdata.com/integrations/db2/ https://www.stitchdata.com/integrations/db2/
  - NovaX 8y ago
    Is it log-based cdc? The documentation says only 2 dbs are supported that way. We want to replicate customer databases which have schemas outside of our control, with the goal of migrating them to our products. The other vendor schemas might not have your required columns. There are also many flavors of DB2. So very interested but the details are sparse at best.
    - jakestein 8y ago
      Our DB2 integration is SELECT replication. Our MySQL, Posgtgres, and Oracle integrations are log base. MySQL and Postgres actually have the option of either logs or SELECT. You're totally right about all of the flavors DB2. Our support team would be the ones to figure out for sure whether or not we can work with your setup, and you reach them at support@stitchdata.com

mycelium 8y ago

I would highly highly recommend ETL as service, after adopting it recently. It substantially changes your relationship with your data sources in a really positive way. And frankly, ETL for common data sources is code that you just don't need to write. I would say that you should pilot with a few ETL vendors. We currently use Fivetran, they're fine but we've had enough burps that I cannot cold recommend them over other vendors. I cannot for the life of me remember the details, but I think we went with them over Stitch for pricing reasons.

georgewfraser 8y ago
I'm Fivetran's CEO and I just want you to know, whatever "burps" you experienced, these things keep me up at night and the whole team is always striving to make the pipeline "just work". The whole vision of our product is that you should be able to plug in and get a perfect mirror image of all your data sources in your data warehouse. Anytime we fall short of that it drives us crazy.
- Petefine 8y ago
  Do you have a forum or suggestions tool at all? Fivetran has been amazing for our new datawarehouse and we're very pleased with the service, but there are a few little (non-bug) things that would have made it even easier.
  - fraserharris 8y ago
    You can email me: fraser@fivetran.com

veritas3241 8y ago

We used them for a single data source at GitLab (Zendesk). Worked pretty well! But we quickly hit a road block where what you could get via the UI wasn't all the tables available. We wound up forking it and adding to the tap. Basically all of our extractors and loaders are going to follow the Singer spec for taps and targets - it's a pretty nice model. Internally, we're using a tool called Meltano which is aimed at solving just your problem. Most of our data warehouse is coming from external business ops tools (Salesforce, Zuora, Zendesk, Marketo, etc.) and we're using dbt for transformations w/ Looker as the BI layer. Definitely check our primary analytics repo [0] as all of our code is out in the open. Feel free to ping me if you have more questions - tmurphy at gitlab. [0] https://gitlab.com/meltano/analytics/ https://gitlab.com/meltano/analytics/

Zaheer 8y ago

I work on AWS Glue (https://aws.amazon.com/glue/ https://aws.amazon.com/glue/). Depending on how complicated the pipeline you'd like to setup, you may not need to write any code.

jakecodes 8y ago

Check out Meltano, a GitLab startup. It’s new and we’re iterating but it sounds like it could solve your exact situation. Feel free to leave issues where we could improve. https://gitlab.com/meltano/meltano https://gitlab.com/meltano/meltano

drunkpotato 8y ago

At the company I work for, we have integrations with Stitch Data and Fivetran. Both are good and have been responsive to my needs. Neither have been perfect, so when I've noticed a problem I've had to keep on top of them to fix it. I also maintain a few of our own ETL jobs for data sources that aren't supported. I will say that I recommend using an ETL vendor without reservation. The nominal cost is more than made up for in the headaches you'll save yourself in creating and maintaining a homegrown ETL.

jpulec 8y ago

I've used StitchData at a startup with AWS Redshift. Pair it with something like dbt for transforming your data, and you have a great match. A little pricey, but totally worth it, IMO.

deleted 8y ago

[deleted]