haha my job makes me rewrite sql that Data analysts come up with into spark jobs in scala. Their main argument is that sql isn't unit testable like scala. Thi

haha my job makes me rewrite sql that Data analysts come up with into spark jobs in scala. Their main argument is that sql isn't unit testable like scala. This grunt work of sql =>spark translation is easily the most boring part of my job. Other than that, I do kind of agree that sql isn't really good at large scale programming. There is no ide support while you are working, hard to extract common logic , hard to reuse , hard to compose, hard to unit test. All these things have been mainstays of conventional programming.