3 ms·
I don't understand why Cascading is missing. It's by far one the easiest batch flow controllers on the platform. You can test it is memory locally. When you dep
by virmundi 11y ago
I don't understand why Cascading is missing. It's by far one the easiest batch flow controllers on the platform. You can test it is memory locally. When you deploy to a real cluster, you know it will just work.
- mtanski 11y agoI don't agree with the last statement. Based on experience with Hadoop (over 5+ years now) running locally is a poor indicator of running on the cluster. Many sleepless night have been spent trying to figure out why the job that runs locally doesn't want to run on Hadoop. I do like cascading and scalding tho. Only so many times you want to implement job flow, filters and joins by hand in lifez
- virmundi 11y agoMaybe the statement could be a bit hyperbolic, but it could be tested. I tested large complex flows locally, 59 steps, within JUnit. These test ran with every build. So the whole build took 6 minutes for 75 fraud models, but I cold easily focus on just my unit in Eclipse. I haven't tried PigUnit for a while, but last time I did, it didn't support macros and took minutes for how has would take seconds in Cascading. It's this difference that's cemented in my mind that Cascading is for repeatable processes while Pig is for probe ring and experimentation. This is not to say you can't reuse Pig scripts. I mean that I have greater confidence in the things I can create repeated tests for.