3 ms·

One other unforced error with c.j was having almost every function take a dictionary of args instead of a sane argument list. That left "grep the codebase" and

One other unforced error with c.j was having almost every function take a dictionary of args instead of a sane argument list. That left "grep the codebase" and "read the entire function definition and god help you if it passes on the argument dict" as the two horrible options for figuring out how to even call most of the functions. But we probably could have erected a blast shield around that mess if only we could have written functions and aggregators in ruby.

StefanKarpinski 13y ago
Yeah, that's an unfortunate part of Ruby's design and culture though and not really specific to c.j. Since Ruby doesn't have real keyword arguments, you don't get any checking (even at run time) that the things you passed in aren't completely wrong (and thus silently ignored). We could have fixed that bit with some API redesign, so it's not nearly as dire a problem. The other two things couldn't really be fixed without changing languages entirely.
gfodor 13y ago
Yeah that right there was our biggest error I think. We should have had a "compilation pass" that used JRuby's ruby parser to do program transformation on the original source, extracting the inline operator code and doing "something" with it to generate fast JVM bytecode (probably using something like Mirah) and then transforming the call site into something that would use this. It would have been a few months of nightmarish debugging but would have also provided the leverage we needed to do data flow-level type checking/annotation as well, in addition to not having to write our UDFs in another language. I think the thing to remember with the c.j stuff is at the time Cascading's Java API was pretty much state of the art, and we had to shore up a lot of what was in c.j in order to get to a viable DSL. And at that point, it was a huge leap forward from a readability standpoint. (Cascading's API had the same shortcomings as our c.j API wrt types etc, fwiw)
- StefanKarpinski 13y ago
  At that point you're just implementing a crappy statically type language inside of a Ruby DSL. Using a real statically typed JVM language that's more expressive than Java – i.e. Scala – would be much better. So, that's basically Scalding. Of course, Scalding wasn't even close to existing when we chose Cascading.JRuby, so yeah.
  - gfodor 13y ago
    Yeah I'm just talking counterfactually. At the time there were no solid cascading DSLs whatsoever, cascalog wasn't even around for another 2 years or so iirc. Cascading itself was the new hotness vs writing raw map reduce, so c.j was kind of on the edge of what was out there. The decision to keep UDFs outside of Ruby was made at the outset (out of concern for performance, and being icing on the cake, etc.) and not really revisited.

StefanKarpinski 13y ago

Yeah, that's an unfortunate part of Ruby's design and culture though and not really specific to c.j. Since Ruby doesn't have real keyword arguments, you don't get any checking (even at run time) that the things you passed in aren't completely wrong (and thus silently ignored). We could have fixed that bit with some API redesign, so it's not nearly as dire a problem. The other two things couldn't really be fixed without changing languages entirely.

gfodor 13y ago

Yeah that right there was our biggest error I think. We should have had a "compilation pass" that used JRuby's ruby parser to do program transformation on the original source, extracting the inline operator code and doing "something" with it to generate fast JVM bytecode (probably using something like Mirah) and then transforming the call site into something that would use this. It would have been a few months of nightmarish debugging but would have also provided the leverage we needed to do data flow-level type checking/annotation as well, in addition to not having to write our UDFs in another language. I think the thing to remember with the c.j stuff is at the time Cascading's Java API was pretty much state of the art, and we had to shore up a lot of what was in c.j in order to get to a viable DSL. And at that point, it was a huge leap forward from a readability standpoint. (Cascading's API had the same shortcomings as our c.j API wrt types etc, fwiw)

StefanKarpinski 13y ago
At that point you're just implementing a crappy statically type language inside of a Ruby DSL. Using a real statically typed JVM language that's more expressive than Java – i.e. Scala – would be much better. So, that's basically Scalding. Of course, Scalding wasn't even close to existing when we chose Cascading.JRuby, so yeah.
- gfodor 13y ago
  Yeah I'm just talking counterfactually. At the time there were no solid cascading DSLs whatsoever, cascalog wasn't even around for another 2 years or so iirc. Cascading itself was the new hotness vs writing raw map reduce, so c.j was kind of on the edge of what was out there. The decision to keep UDFs outside of Ruby was made at the outset (out of concern for performance, and being icing on the cake, etc.) and not really revisited.