4 ms·

There are benefits to standardizing on a single language and codebase, so it may be worth having some code that is less readable or performant in order to optim

There are benefits to standardizing on a single language and codebase, so it may be worth having some code that is less readable or performant in order to optimize for the global maximum and not prematurely introduce too many languages.

appplication 3y ago
My least favorite thing about performance optimizations is how hard they are to guard against regressions with standard CI/CD. Unit tests, integration tests, etc. are rarely the right tools. And in some systems (e.g. distributed data processing), local performance often does not indicate true performance at scale.
- sgarland 3y ago
  If you want truly accurate results, you have to have a dedicated machine – not a VM (unless you’re controlling the hypervisor, and can guarantee no overcommit), not a container, but an entire machine – and tightly control every aspect of it. Updated packages? All results are invalid until you can prove the baseline hasn’t changed. I don’t think most people need this level of accuracy, but it is something to consider. Database queries are even worse, because not only do you need representative data at a similar scale (or you’ve scaled both hardware and working set accurately), but you have to hope that the table/column stats in prod and test are kept at a similar level of accuracy, which is more art than science. Did the query regress because of the change, or because writes to an unrelated table have increased load and the analysis daemon is now struggling? Did it regress in test because someone else pushed a change to a rarely-used query, causing buffer churn when CI ran your job? Etc.
  - appplication 3y ago
    I agree with all of this. Because of this, even at massive scale it’s hard to justify spending the time and effort needed to get this level of testing.
FridgeSeal 3y ago
Conversely, freeing data science/numerical analysis tooling from the tire-fire of the Python ecosystem might actually be a net benefit for the community at large. Get a bit more variety into the options, explore some alternative approaches. Who knows, maybe some ideas from other programming languages will yield some useful advantages.
- sgarland 3y ago
  Python’s ecosystem is great IFF you use 3rd party tooling carefully, and understand how and when to use it. The biggest one is to use venvs liberally, and to let as few things as possible interact with system Python. If there’s a CLI tool written in Python, ideally install it with pipx. Use Poetry for every project. Unfortunately these aren’t inherently obvious until you have a raging tire fire, and even then, you have to know the source of it and that better options exist. But hey, at least it isn’t Node.
  - pjmlp 3y ago
    Node has a JIT by default, while Python is still fighting to get one that is widely accepted. I still have hopes that Julia and Ruby competition makes it change.
    - sgarland 3y ago
      In theory, Node is faster than Python due to this, among other things. In practice, I have seen Node apps blocking or crashing far more often than Python apps, which drags the performance down just a bit. In the meantime, @njit decorator on Python functions, and boom, done.
      - pjmlp 3y ago
        I am yet to see such thing, to each its own I guess. @njit doesn't work in regular Python, requires installing Numba and only works in very specific use cases.
PredictorX1 3y ago
Language or platform standardization for data science is over-rated. Most often, analysis is performed remotely, and resulting models and other work product are deployed, "frozen" on production platforms. There is seldom need to match the data scientist's tools to match those of the organization.

appplication 3y ago

My least favorite thing about performance optimizations is how hard they are to guard against regressions with standard CI/CD. Unit tests, integration tests, etc. are rarely the right tools. And in some systems (e.g. distributed data processing), local performance often does not indicate true performance at scale.

sgarland 3y ago
If you want truly accurate results, you have to have a dedicated machine – not a VM (unless you’re controlling the hypervisor, and can guarantee no overcommit), not a container, but an entire machine – and tightly control every aspect of it. Updated packages? All results are invalid until you can prove the baseline hasn’t changed. I don’t think most people need this level of accuracy, but it is something to consider. Database queries are even worse, because not only do you need representative data at a similar scale (or you’ve scaled both hardware and working set accurately), but you have to hope that the table/column stats in prod and test are kept at a similar level of accuracy, which is more art than science. Did the query regress because of the change, or because writes to an unrelated table have increased load and the analysis daemon is now struggling? Did it regress in test because someone else pushed a change to a rarely-used query, causing buffer churn when CI ran your job? Etc.
- appplication 3y ago
  I agree with all of this. Because of this, even at massive scale it’s hard to justify spending the time and effort needed to get this level of testing.

FridgeSeal 3y ago

Conversely, freeing data science/numerical analysis tooling from the tire-fire of the Python ecosystem might actually be a net benefit for the community at large. Get a bit more variety into the options, explore some alternative approaches. Who knows, maybe some ideas from other programming languages will yield some useful advantages.

sgarland 3y ago
Python’s ecosystem is great IFF you use 3rd party tooling carefully, and understand how and when to use it. The biggest one is to use venvs liberally, and to let as few things as possible interact with system Python. If there’s a CLI tool written in Python, ideally install it with pipx. Use Poetry for every project. Unfortunately these aren’t inherently obvious until you have a raging tire fire, and even then, you have to know the source of it and that better options exist. But hey, at least it isn’t Node.
- pjmlp 3y ago
  Node has a JIT by default, while Python is still fighting to get one that is widely accepted. I still have hopes that Julia and Ruby competition makes it change.
  - sgarland 3y ago
    In theory, Node is faster than Python due to this, among other things. In practice, I have seen Node apps blocking or crashing far more often than Python apps, which drags the performance down just a bit. In the meantime, @njit decorator on Python functions, and boom, done.
    - pjmlp 3y ago
      I am yet to see such thing, to each its own I guess. @njit doesn't work in regular Python, requires installing Numba and only works in very specific use cases.

PredictorX1 3y ago

Language or platform standardization for data science is over-rated. Most often, analysis is performed remotely, and resulting models and other work product are deployed, "frozen" on production platforms. There is seldom need to match the data scientist's tools to match those of the organization.