4 ms·

Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.

• Joeri 10y ago
  We've been trying out goldengate to get streaming replication, but it has proven rather unreliable. Stops replicating if you sneeze in its general vicinity. I wonder whether the alternatives like shareplex and tungsten are more reliable.
  • capkutay 10y ago
    You can try out Striim for streaming data integration (full disclosure, I work there): http://www.striim.com/download-striim/ http://www.striim.com/download-striim/
• technofiend 10y ago
  I feel like I've recommended it enough times I'm turning into a shill but Pentaho is an open-source and commercially supported ETL tool that will natively do what you want, or call sqoop when you discover that's kinda slow. :-) And no I definitely don't work there.
• rathboma 10y ago
  Using Sqoop from something like Luigi as the ETL manager is a pretty great workflow - https://github.com/spotify/luigi https://github.com/spotify/luigi You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline
  • natekupp 10y ago
    +1 to this, we kick off our Sqoop jobs using Airflow - http://airbnb.io/projects/airflow/ http://airbnb.io/projects/airflow/ Airflow is very similar to Luigi; we've been using in in production to schedule all of our workflows for ~4 months now and it's worked out really well for us.
  • machbio 10y ago
    Thats a great idea - but could you elaborate on the scheduling of jobs on Luigi - it does not have a scheduler like AirFlow - how do you schedule Luigi tasks ?
    • rathboma 10y ago
      Check out this Foursquare talk that goes through how we used to do scheduling -- basically you make jobs dependent on a date - http://www.slideshare.net/OpenAnayticsMeetup/luigi-presentation-17-23199897 http://www.slideshare.net/OpenAnayticsMeetup/luigi-presentat...
    • allengeorge 10y ago
      You have to use an external scheduler. We built one on top of AP Scheduler: https://apscheduler.readthedocs.io/en/latest/ https://apscheduler.readthedocs.io/en/latest/
• falaki 10y ago
  Ever since using Apache Spark's Data Sources API was released, I have been relying no different Spark Data Source packages for my ETL jobs.
• dn5 10y ago
  Have a look at Kafka Connect (http://docs.confluent.io/2.0.0/connect http://docs.confluent.io/2.0.0/connect). The JDBC Connector will poll for database changes changes and push them to a Kafka topic. Means you should see all the changes in the database rather than a snapshot say once a day