4 ms·

Isn't Numpy about making things go fast? People who expect things to go fast want it all cached up front, for very obvious reasons. Seems straightforward.

by urthor 4y ago

eesmith 4y ago
"Go fast" has multiple dimensions. For one example, I had a command-line tool which needed to compute something related to hypergeometric distribution. (It's been a few years; I forget the details.) This available in scipy, which need numpy. Most of my program's run-time was spent importing numpy. (The following timings are best-of-3.) % time python -c pass 0.027u 0.011s 0:00.04 75.0% 0+0k 0+0io 0pf+0w % time python -c "import numpy" 0.212u 0.059s 0:00.17 152.9% 0+0k 0+0io 14pf+0w % time python -c "import scipy" 0.252u 0.077s 0:00.32 100.0% 0+0k 0+0io 14pf+0w While 0.2 seconds doesn't seem like much to people used to spending hours developing a notebook, or running some large matrix computation, I could make my program 8x faster by writing the dozen or so lines I needed to evaluate that function myself. Numpy is not about making short-lived programs fast. In any case, this specific choice of importing all submodules is not to cache things up "for obvious [performance] reasons", because there is no machine performance improvements. Instead, it's to make the API easier/faster to use. When you're in a notebook and you need numpy.foo.bar() you can just use it, and not have to go up and use an import statement first.
- urthor 4y ago
  Yeah exactly. However, what I'd say is that a short launch from rest is an even more obscure use case than the one numpy supports. numpy has made the correct compromise IMO, optimizing for long lived programs and notebooks.
  - eesmith 4y ago
    While my observation is that most other Python packages don't import all subpackages. "import urllib" does not also import urllib.parse, so you can't do: import urllib urllib.parse.quote("A&W") but instead must explicitly import urllib.parse. scikit-learn supports essentially the same use cases as NumPy and it doesn't import all its subpackages: >>> import sklearn >>> sklearn.linear_model Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'sklearn' has no attribute 'linear_model' >>> import sklearn.linear_model >>> sklearn.linear_model <module 'sklearn.linear_model' from '[...]sklearn/linear_model/__init__.py'> Thus leading to my earlier comment: > The NumPy developers think "practicality beats purity" is more important, and their main use-case is long-running programs where startup costs are slow, which I interpret as meaning their special cases is special enough to break the rules.

Isn't Numpy about making things go fast? People who expect things to go fast want it all cached up front, for very obvious reasons. Seems straightforward.

by urthor 4y ago

Isn't Numpy about making things go fast? People who expect things to go fast want it all cached up front, for very obvious reasons. Seems straightforward.

eesmith 4y ago

"Go fast" has multiple dimensions. For one example, I had a command-line tool which needed to compute something related to hypergeometric distribution. (It's been a few years; I forget the details.) This available in scipy, which need numpy. Most of my program's run-time was spent importing numpy. (The following timings are best-of-3.) % time python -c pass 0.027u 0.011s 0:00.04 75.0% 0+0k 0+0io 0pf+0w % time python -c "import numpy" 0.212u 0.059s 0:00.17 152.9% 0+0k 0+0io 14pf+0w % time python -c "import scipy" 0.252u 0.077s 0:00.32 100.0% 0+0k 0+0io 14pf+0w While 0.2 seconds doesn't seem like much to people used to spending hours developing a notebook, or running some large matrix computation, I could make my program 8x faster by writing the dozen or so lines I needed to evaluate that function myself. Numpy is not about making short-lived programs fast. In any case, this specific choice of importing all submodules is not to cache things up "for obvious [performance] reasons", because there is no machine performance improvements. Instead, it's to make the API easier/faster to use. When you're in a notebook and you need numpy.foo.bar() you can just use it, and not have to go up and use an import statement first.

urthor 4y ago
Yeah exactly. However, what I'd say is that a short launch from rest is an even more obscure use case than the one numpy supports. numpy has made the correct compromise IMO, optimizing for long lived programs and notebooks.
- eesmith 4y ago
  While my observation is that most other Python packages don't import all subpackages. "import urllib" does not also import urllib.parse, so you can't do: import urllib urllib.parse.quote("A&W") but instead must explicitly import urllib.parse. scikit-learn supports essentially the same use cases as NumPy and it doesn't import all its subpackages: >>> import sklearn >>> sklearn.linear_model Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'sklearn' has no attribute 'linear_model' >>> import sklearn.linear_model >>> sklearn.linear_model <module 'sklearn.linear_model' from '[...]sklearn/linear_model/__init__.py'> Thus leading to my earlier comment: > The NumPy developers think "practicality beats purity" is more important, and their main use-case is long-running programs where startup costs are slow, which I interpret as meaning their special cases is special enough to break the rules.