2 ms·
Yeah that right there was our biggest error I think. We should have had a "compilation pass" that used JRuby's ruby parser to do program transformation on the o
by gfodor 13y ago
Yeah that right there was our biggest error I think. We should have had a "compilation pass" that used JRuby's ruby parser to do program transformation on the original source, extracting the inline operator code and doing "something" with it to generate fast JVM bytecode (probably using something like Mirah) and then transforming the call site into something that would use this. It would have been a few months of nightmarish debugging but would have also provided the leverage we needed to do data flow-level type checking/annotation as well, in addition to not having to write our UDFs in another language.
I think the thing to remember with the c.j stuff is at the time Cascading's Java API was pretty much state of the art, and we had to shore up a lot of what was in c.j in order to get to a viable DSL. And at that point, it was a huge leap forward from a readability standpoint. (Cascading's API had the same shortcomings as our c.j API wrt types etc, fwiw)
- StefanKarpinski 13y agoAt that point you're just implementing a crappy statically type language inside of a Ruby DSL. Using a real statically typed JVM language that's more expressive than Java – i.e. Scala – would be much better. So, that's basically Scalding. Of course, Scalding wasn't even close to existing when we chose Cascading.JRuby, so yeah.
- gfodor 13y agoYeah I'm just talking counterfactually. At the time there were no solid cascading DSLs whatsoever, cascalog wasn't even around for another 2 years or so iirc. Cascading itself was the new hotness vs writing raw map reduce, so c.j was kind of on the edge of what was out there. The decision to keep UDFs outside of Ruby was made at the outset (out of concern for performance, and being icing on the cake, etc.) and not really revisited.