4 ms·

I think Go is pretty OK as a language for building data pipelines (I’m assuming you meant statistical ones, but the same argument applies to more data transform

by ratorx 2y ago

I think Go is pretty OK as a language for building data pipelines (I’m assuming you meant statistical ones, but the same argument applies to more data transform-y ones). What it is not good for is doing exploratory analysis (which is where Python shines). Manual loops are pretty annoying when the focus is on figuring out which loops to write (exploratory phase). However, they are pretty nice once you’ve figured it out and need write a durable bit of code where your prioritise readability over conciseness (productionisation). Going from Python to <any language> between the exploratory phase and the productionised pipeline is going to be a pain, I don’t think Go is particularly worse than others. At that point it’s all about the classic software tradeoffs (performance vs velocity vs maintainability) and I didn’t think Go is a good choice in many situations.

sanderjd 2y ago
Well I totally disagree that writing manual loops is ever "pretty nice", but I agree that it's not as big an issue in final-version code as it is in exploration. And I'm also in strong agreement that making any language transition between exploration and implementation is problematic. I do think go is worse than most, because I just think it has a mostly cultural allergy to manipulating collections of data as collections rather than element-by-element, but I agree that this is mostly lost in the noise of doing any re-write into a new language. But this is why Python is best in this space. It simply has the best promotion path from experimentation to production. It is better than other "real" languages like go, because it thrives in the exploratory phase, and it is better than purpose-specific languages, like R, because it is also a great general-purpose language. The other contender I see is Julia, which comes more from the experimentation-focused side, while trying to become a good general purpose language, but unfortunately I think it still needs to mature a lot on that side, and it's not clear that it has the community to push it far enough fast enough in that direction (IMO). Even very performance-critical use cases work with python, because the iteration process can follow experimentation -> productionization -> performance analysis -> fixing low-hanging bottlenecks by offloading to existing native extensions -> writing custom native extensions for the real bottlenecks.

by ratorx 2y ago

sanderjd 2y ago

Well I totally disagree that writing manual loops is ever "pretty nice", but I agree that it's not as big an issue in final-version code as it is in exploration. And I'm also in strong agreement that making any language transition between exploration and implementation is problematic. I do think go is worse than most, because I just think it has a mostly cultural allergy to manipulating collections of data as collections rather than element-by-element, but I agree that this is mostly lost in the noise of doing any re-write into a new language. But this is why Python is best in this space. It simply has the best promotion path from experimentation to production. It is better than other "real" languages like go, because it thrives in the exploratory phase, and it is better than purpose-specific languages, like R, because it is also a great general-purpose language. The other contender I see is Julia, which comes more from the experimentation-focused side, while trying to become a good general purpose language, but unfortunately I think it still needs to mature a lot on that side, and it's not clear that it has the community to push it far enough fast enough in that direction (IMO). Even very performance-critical use cases work with python, because the iteration process can follow experimentation -> productionization -> performance analysis -> fixing low-hanging bottlenecks by offloading to existing native extensions -> writing custom native extensions for the real bottlenecks.