5 ms·
We have been using Luigi in production for a year now at AdRoll, to manage a graph of tens of data processing tasks. We have been really happy with it. You can
by vtuulos 11y ago
We have been using Luigi in production for a year now at AdRoll, to manage a graph of tens of data processing tasks.
We have been really happy with it.
You can read more about our setup in these two blog posts:
http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d...
http://tech.adroll.com/blog/data/2015/10/15/luigi.html http://tech.adroll.com/blog/data/2015/10/15/luigi.html
- suresk 11y agoJust curious - have you guys used the hadoop contrib stuff with Luigi? We use it almost exclusively to kick off Hadoop jobs and when I went to refactor some of my predecessors stuff that just kicked off a raw process that called 'hadoop jar' to the hadoop contrib stuff, I ran into a lot of weird issues (largely, the way arguments get passed to the hadoop job). Just was curious if many other people were using the hadoop contrib stuff successfully, or if I am trying to use something that isn't very well supported.
- vtuulos 11y agoNo, we haven't used Luigi with Hadoop. For batch processing we use containerized jobs with a simple job queue, like described in the blog article.