5 ms·

Dear god, so much antipatterns on a single page. Interleaving configuration management in the business logic code is a massive technical debt waiting to engulf

Dear god, so much antipatterns on a single page. Interleaving configuration management in the business logic code is a massive technical debt waiting to engulf future generations in a world of maintainance pain.

Maro 4y ago
It's example code in a blog post.
- joelschw 4y ago
  I'm not sure it's framed like that, it's specifically talking about production patterns
  - pvg 4y ago
    'production' for many kinds of data analysis is not the same kind of production as for, say, a web app or service. You can have a production environment for running ad hoc jobs.
  - deleted 4y ago
    [deleted]
- bigDinosaur 4y ago
  Unironically that means it will end up in hundreds if not thousands of production codebases.
roomey 4y ago
I'm so glad I'm not the only one. I was reading the code thinking this seems like a different language to the one I use (albeit I'm not too experienced). That seems obtuse to me in the extreme. Maybe I'm just not used to that style however.
deleted 4y ago
[deleted]
lpmiss 4y ago
Scientific code in the Python ecosystem is horrible in general. Architecture astronauts, stack overflows from recursion, bloat, truncating int64 in casts to double, version incompatible mess due to the latest features at all costs. I have seen it all. They treat Python as if it were a sound language with the guarantees of Haskell.
- gregsadetsky 4y ago
  Are there good examples (established libraries or projects) of scientific code in Python? And more broadly, would you have examples of what good scientific code could/does look like? Not being facetious! I’m genuinely curious and would love to learn more. Thanks
  - Zeratoss 4y ago
    https://github.com/scverse/scanpy https://github.com/scverse/scanpy This is incredibly popular in single cell analysis
  - geysersam 4y ago
    The Scipy project is a good example, with code from many scientific domains.
rmnclmnt 4y ago
The whole Data Science / Jupyter ecosystem reminds me so much of the old days of "MATLAB to C++": the Python ecosystem was supposed to be the best of both worlds and ease the transition from prototyping to production. Laziness won once again. Data scientists need to be trained with software engineering skills, software engineers need to be trained with data science skills. This is the only way we do not end-up with non-sense like this.
- gregsadetsky 4y ago
  Would you have reccomendations / pointers for this? I straddle both worlds - I’m much more on the tech side, but sometimes interact with scientists / MATLAB codebases. What data science skills / methdologies would be useful for me to learn? P.S. And what did you mean by MATLAB to C++? That was a specific time frame (I suppose in the early 2000s ish) when C++ was taught to scientists in the hope they’d be able to productionize their MATLAB code? With not great results (i.e. C++ learning curve + lack of software engineering skills…?) Thanks!
  - rmnclmnt 4y ago
    From my experience in the fields of DSP / Data / AI during the last 10 years, issues arise when product teams are segregated into jobs (one guy for initial prototype, then one guy to prep the integration, then one guy for the ops side of things, etc.): people need to be interested and involved in the product they are building end-to-end! Yes this is more demanding, yes this requires perpetual training, but gosh it is rewarding! My take (non-exhaustive) with the current ecosystem is to apply Agile and DevOps methodologies: - Use Git everywhere all the time, always - Use Jupyter early one: great for quick prototypes & demos, keynotes, training material, articles - Once the initial prototype is approved, archive Jupyter notebooks as snapshots - Write functional tests (ideally in a TDD fashion) - Build and/or integrate the work into a real software product, be it in Python, C++, Java, etc. - Use tools for deterministic behavior (package manager, Docker, etc.) - Use CI/CD and GitOps methodologies - Deliver iteratively, fast and reliably And by "MATLAB to C++" was a reference to a time (2010's) when corporation were deeply involved with MATLAB licenses, could not afford to switch easily to Python and lots of SWE with applied math background had to deal with MATLAB code written by pure math guys without any consideration for SWE best practices and target product constraints. Nowadays, if the target product is also in Python, there is way less friction, hopefully :)
saeranv 4y ago
What other antipatterns are there? I can understand the criticism of having configuration code with business logic, and think there should be better commenting/docstrings. But other then that, thought it was fine. The use of type hints is pretty awesome, in particular. Do you guys just not like higher-order functions?
- gjulianm 4y ago
  Type hints in that code are erasing the actual signature of the function, so you won't get type checks on the arguments. For @redirect, you'd be better served by using the logging module. Higher order functions are ok, but the problem with this what they're using it for. Code that behaves different based on environment without any explicit warning for the caller? That's fairly dangerous.
  - mrtranscendence 4y ago
    > Type hints in that code are erasing the actual signature of the function True enough. For the reader of these comments, it's possible (but non-obvious) to properly respect type hints in a decorator. See here: https://mypy.readthedocs.io/en/stable/generics.html#declaring-decorators https://mypy.readthedocs.io/en/stable/generics.html#declarin...
pvg 4y ago
That seems a little unlikely since implementing business logic is not typically what a data scientist does.
- s_Hogg 4y ago
  Not as far as the data scientist in question is concerned, anyway. I'm reminded of that joke: Business logic (n): your stuff, which is shit, as opposed to my code, which is beautiful I say this as a machine learning engineer

Maro 4y ago

It's example code in a blog post.

joelschw 4y ago
I'm not sure it's framed like that, it's specifically talking about production patterns
- pvg 4y ago
  'production' for many kinds of data analysis is not the same kind of production as for, say, a web app or service. You can have a production environment for running ad hoc jobs.
- deleted 4y ago
  [deleted]
bigDinosaur 4y ago
Unironically that means it will end up in hundreds if not thousands of production codebases.

roomey 4y ago

I'm so glad I'm not the only one. I was reading the code thinking this seems like a different language to the one I use (albeit I'm not too experienced). That seems obtuse to me in the extreme. Maybe I'm just not used to that style however.

deleted 4y ago

[deleted]

lpmiss 4y ago

Scientific code in the Python ecosystem is horrible in general. Architecture astronauts, stack overflows from recursion, bloat, truncating int64 in casts to double, version incompatible mess due to the latest features at all costs. I have seen it all. They treat Python as if it were a sound language with the guarantees of Haskell.

gregsadetsky 4y ago
Are there good examples (established libraries or projects) of scientific code in Python? And more broadly, would you have examples of what good scientific code could/does look like? Not being facetious! I’m genuinely curious and would love to learn more. Thanks
- Zeratoss 4y ago
  https://github.com/scverse/scanpy https://github.com/scverse/scanpy This is incredibly popular in single cell analysis
- geysersam 4y ago
  The Scipy project is a good example, with code from many scientific domains.

rmnclmnt 4y ago

The whole Data Science / Jupyter ecosystem reminds me so much of the old days of "MATLAB to C++": the Python ecosystem was supposed to be the best of both worlds and ease the transition from prototyping to production. Laziness won once again. Data scientists need to be trained with software engineering skills, software engineers need to be trained with data science skills. This is the only way we do not end-up with non-sense like this.

gregsadetsky 4y ago
Would you have reccomendations / pointers for this? I straddle both worlds - I’m much more on the tech side, but sometimes interact with scientists / MATLAB codebases. What data science skills / methdologies would be useful for me to learn? P.S. And what did you mean by MATLAB to C++? That was a specific time frame (I suppose in the early 2000s ish) when C++ was taught to scientists in the hope they’d be able to productionize their MATLAB code? With not great results (i.e. C++ learning curve + lack of software engineering skills…?) Thanks!
- rmnclmnt 4y ago
  From my experience in the fields of DSP / Data / AI during the last 10 years, issues arise when product teams are segregated into jobs (one guy for initial prototype, then one guy to prep the integration, then one guy for the ops side of things, etc.): people need to be interested and involved in the product they are building end-to-end! Yes this is more demanding, yes this requires perpetual training, but gosh it is rewarding! My take (non-exhaustive) with the current ecosystem is to apply Agile and DevOps methodologies: - Use Git everywhere all the time, always - Use Jupyter early one: great for quick prototypes & demos, keynotes, training material, articles - Once the initial prototype is approved, archive Jupyter notebooks as snapshots - Write functional tests (ideally in a TDD fashion) - Build and/or integrate the work into a real software product, be it in Python, C++, Java, etc. - Use tools for deterministic behavior (package manager, Docker, etc.) - Use CI/CD and GitOps methodologies - Deliver iteratively, fast and reliably And by "MATLAB to C++" was a reference to a time (2010's) when corporation were deeply involved with MATLAB licenses, could not afford to switch easily to Python and lots of SWE with applied math background had to deal with MATLAB code written by pure math guys without any consideration for SWE best practices and target product constraints. Nowadays, if the target product is also in Python, there is way less friction, hopefully :)

saeranv 4y ago

What other antipatterns are there? I can understand the criticism of having configuration code with business logic, and think there should be better commenting/docstrings. But other then that, thought it was fine. The use of type hints is pretty awesome, in particular. Do you guys just not like higher-order functions?

gjulianm 4y ago
Type hints in that code are erasing the actual signature of the function, so you won't get type checks on the arguments. For @redirect, you'd be better served by using the logging module. Higher order functions are ok, but the problem with this what they're using it for. Code that behaves different based on environment without any explicit warning for the caller? That's fairly dangerous.
- mrtranscendence 4y ago
  > Type hints in that code are erasing the actual signature of the function True enough. For the reader of these comments, it's possible (but non-obvious) to properly respect type hints in a decorator. See here: https://mypy.readthedocs.io/en/stable/generics.html#declaring-decorators https://mypy.readthedocs.io/en/stable/generics.html#declarin...

pvg 4y ago

That seems a little unlikely since implementing business logic is not typically what a data scientist does.

s_Hogg 4y ago
Not as far as the data scientist in question is concerned, anyway. I'm reminded of that joke: Business logic (n): your stuff, which is shit, as opposed to my code, which is beautiful I say this as a machine learning engineer