4 ms·

The size of python packages is becoming an increasingly worrying issue. A lot of maintainers don't pay attention to excluding test cases and artifacts from the

The size of python packages is becoming an increasingly worrying issue. A lot of maintainers don't pay attention to excluding test cases and artifacts from their packages, leading to ridiculous package size growth. Last I checked numpy was close to 100MB, a huge chunk of that being test case artifacts. Django packages all translations for all languages to ever exist. Pandas bundles pre-built dynamic libraries with debug symbols. Etc. Most of my "production" virtualenvs are close to a GB nowadays, which is insane.

tedmiston 4y ago
All of those cases sound like good reasons for optionals / extras.
kzrdude 4y ago
Sounds like a bug should be filed towards numpy?
- kzrdude 4y ago
  Apparently it was wontfixed last year, but maybe it could still be suggested to be an extra, not in the standard package? https://github.com/numpy/numpy/issues/19591 https://github.com/numpy/numpy/issues/19591
  - LosWochosWeek 4y ago
    That was implied in the thread and subsequently wonfixed. > Sure, but it's straightforward to install those [0]. A separate test suite would not be. 0: Those = Other packages you would have to download to make use of the testing components, i.e. pytest
    - kzrdude 4y ago
      What about pip install 'numpy[tests]' i.e using the extras feature?
houzi 4y ago
This! Especially with pandas. I did have a look at numpy and on my machine the tests did not bloat it as much as you made me believe. The `core/tests/` modules are 3MB and the `__pycache__` doubled it to 6MB. What do you refer to as "test case artifacts"? The modules, pycache, or both? Also, wrt you statement on pandas; is it the debug symbols that account for the bloat, or the libs themselves?
- Galanwe 4y ago
  I cannot seem to remember which version exactly that was. I remember there were some test input/output files that weighted multiple MBs. Just installed a quick "data science" like virtualenv, here is the example things I talk about: $ du -sh scipy 110M scipy $ find scipy -type f -name '.so' -exec strip -s {} \; $ du -sh scipy 76M scipy $ du -sh django/contrib/admin/locale/ 5.4M django/contrib/admin/locale/
boxed 4y ago
I've had people report missing tests from my python packages as bugs. You can't please all the people all of the time.

tedmiston 4y ago

All of those cases sound like good reasons for optionals / extras.

kzrdude 4y ago

Sounds like a bug should be filed towards numpy?

kzrdude 4y ago
Apparently it was wontfixed last year, but maybe it could still be suggested to be an extra, not in the standard package? https://github.com/numpy/numpy/issues/19591 https://github.com/numpy/numpy/issues/19591
- LosWochosWeek 4y ago
  That was implied in the thread and subsequently wonfixed. > Sure, but it's straightforward to install those [0]. A separate test suite would not be. 0: Those = Other packages you would have to download to make use of the testing components, i.e. pytest
  - kzrdude 4y ago
    What about pip install 'numpy[tests]' i.e using the extras feature?

houzi 4y ago

This! Especially with pandas. I did have a look at numpy and on my machine the tests did not bloat it as much as you made me believe. The `core/tests/` modules are 3MB and the `__pycache__` doubled it to 6MB. What do you refer to as "test case artifacts"? The modules, pycache, or both? Also, wrt you statement on pandas; is it the debug symbols that account for the bloat, or the libs themselves?

Galanwe 4y ago
I cannot seem to remember which version exactly that was. I remember there were some test input/output files that weighted multiple MBs. Just installed a quick "data science" like virtualenv, here is the example things I talk about: $ du -sh scipy 110M scipy $ find scipy -type f -name '.so' -exec strip -s {} \; $ du -sh scipy 76M scipy $ du -sh django/contrib/admin/locale/ 5.4M django/contrib/admin/locale/

boxed 4y ago

I've had people report missing tests from my python packages as bugs. You can't please all the people all of the time.