3 ms·
I will never understand python API developers obsession with overloading the operator magic methods. Apache Beam is one of the worst offenders to me. The recom
by red_hare 6y ago
I will never understand python API developers obsession with overloading the operator magic methods.
Apache Beam is one of the worst offenders to me. The recommended method is to connect your functional transforms when building your pipeline as if it were a series Bash commands piped together (overloading bitwise OR). Who thought “oh, you know what language and API is synonymous with ‘write a large complicated amount of business logic in me’, Bash.”
- pyuser583 6y agoYour comment made me smile because Apache Beam is one of my favorite tools, but it seems nobody outside Spotify has had heard of it. The bitwise operation dunder methods have no definition outside of integers. So they’re basically freebies to do what you want with. Honestly I think the Apache Beam API makes a lot of sense.
- pushingice 6y agoThis is because it started out as a Google project. There is an internal time-series framework there called Monarch that does the same thing for composing operations.
- heavyset_go 6y agoI have to agree with this sentiment.
- red_hare 6y agoTo be clear, I love Beam. It runs all of my startups stream and batch processing. I just think the Java API's version... pipeline = pipeline .apply("FirstTransform, MyFirstParDo()) .apply("SecondTransform, MySecondParDo()) is much clearer than the Python version... pipeline = ( pipeline | "FirstTransform" >> MyFirstParDo() | "SecondTransform" >> MySecondParDo() )
- pyuser583 6y agoAgreed. I prefer the Java version, for a number of reasons. Scio is even better. But the bitwise operations make sense python wise.
- patrickserrano 6y agoOh man, I ran into that when I made my first contribution to our internal Airflow project and was just like wtf is happening.
- recov 6y agoTo be fair it's pretty nice when you get used to it as I doubt you're going to be using bit shifts much.
- red_hare 6y agoIt's fine. But there are some unintuitive edge cases. Like, for example, if you have tasks A, B, C, and D you can do... A >> (B, C) >> D but not... (B, C) >> A >> D you have to write... D << A << (B, C)