5 ms·
i pity the developer who has to maintain tagless final plumbing code after the “functional programming enthusiast” moves on… in a Go first org no less.
by valzam 2y ago
i pity the developer who has to maintain tagless final plumbing code after the “functional programming enthusiast” moves on… in a Go first org no less.
- epgui 2y agoI would much rather inherit an FP data pipeline than anything else. You do realize data pipelines (and distributed computing) are an ideal use case for FP?
- pjmlp 2y agoI guess the issue being point out is the choice in a Go culture shop, and we all know their common point of view regarding "fancy" languages.
- epgui 2y agoIt’s not clear to me why having two different sets of tooling for solving two different kinds of problems, is an issue. In most well-resourced companies, you’re probably not going to have to ask your Go engineers to fix data pipelines in Scala.
- pjmlp 2y agoThat is why they pointed out the leaving the company, as possible scenario. As for well resourced I guess it depends, that variable usually doesn't mean much, as we can see by companies firing whole departments, while swimming in profits.
- otter-in-a-suit 2y agoAuthor here. This decision went through all proper architecture channels, including talks with our engineers, proof of concepts and the like. I’ve been doing this too long to shoehorn in my pet languages if I didn’t think they’re a good fit. And I think that scala/FP + Flink _is_ a good fit for this use case. We did also explore the go ecosystem fwiw - the options there are limited (especially around the data tooling like iceberg) and go is simply not a language that’s popular enough in the data world. Python’s typing system (or lack thereof) is a huge hinderance in this space in general (imo), and Java didn’t cause many happy faces on the Eng team either, but it’s certainly an option. I just find FP semantics a better fit for data / streaming work (lots of map and flat map anyways), and Scala makes that easy. Also no cats/zio - just some tangles final _inspired_ composition and type classes. Not too difficult to reason about, not using any obscure patterns. I even mutate references sometimes. :-)
- boltzmann-brain 2y agoscala? why not haskell instead?
- otter-in-a-suit 2y agoNot assuming you’re serious, but in any case: the reason is the JVM (+ Scala) ecosystem in the data space.
- epgui 2y agoFWIW, I do believe there is a serious case to be made for Haskell… But it’s probably beyond the scope of this context / would require changing many other decisions. If integrating with java tools was important then personally I’d ask “why not Clojure”. :)
- atomicnumber3 2y agoSpark is written in scala and Scala is its first-class language - other languages suffer from either second-class APIs (Java) or suffer from codec/serde overhead (pyspark) (though pyspark actually also is missing a few APIs that scala has, as well).
- atomicnumber3 2y agoI'm assuming the parent commenter hasn't worked in data/spark before either. The functional rabbit hole goes WAY deeper than even just cats et al, and Scala and spark themselves both encourage a fair amount of functional-style code on their own.
- bfors 2y agoCould you speak to how you're interfacing scala with flink? I looked into using scala with flink a while back, and stopped when I found out that the scala API was deprecated.
- moandcompany 2y agoThere was a prior effort to create a Golang SDK for Apache Beam https://beam.apache.org/documentation/sdks/go/ https://beam.apache.org/documentation/sdks/go/ The BEAM Golang SDK work came from Googlers working on Beam that were Golang fans, and internally there were Golang-oriented tools for batch data processing that needed a migration path forward. Historical Note: Apache Beam also originated from Google as "Dataflow"