3 ms·

I came to say say that's not a language problem, that's a packaging problem. But that would be misleading. The reason packaging is still a problem in Python ar

I came to say say that's not a language problem, that's a packaging problem. But that would be misleading. The reason packaging is still a problem in Python are linked to the language: - to get perfs, you need compiled extensions, which java doesn't need. So a Jar can be fast and portable, while in Python it's an "or" proposition. Wheels help, but they still are OS dependent, so you need a build per OS. - PYZ arrived very late in Python, while WAR were a very early thing in Java. Few people even know you can distribute a whole Python program in a single zip file with dependencies using something like shiv (shiv.readthedocs.io). So few people do it, and it's a shame. We do it for 0bin though: https://github.com/Tygs/0bin/releases https://github.com/Tygs/0bin/releases. And it doesn't solve the wheel problem, although nuitka does but it a higher prices to pay. - packaging tooling have been a mess for 2 decades. Setuptools/distutils/setuptools2, easy_install/pip, and setup.cfg/setup.py/pyproject.toml. We failed as a community on this one. It's getting better, but it's a slow process. - the Python community is at least half composed of amateurs. It's a good thing for the dynamism of the community, but it also means the software are not very polished. Hard packaging + amateurs = packages that are releases when they reach "just good enough" stage, not one inch more. It also means you'll get things like leaking stack traces, config files in src and the likes. So yeah, compared to Java, Python packaging and distribution sucks. But compared to rust all packaging and distribution for all languages do, so there is that.

mnw21cam 6y ago
It certainly doesn't help that the software I'm interested in is written for science, so there is a tendency to make it work just well enough on the author's machine to allow a paper to be published, and then abandon it.
- ta988 6y ago
  There is a clear training problem in science. But I believe part of it is that the knowledge on how to write good software is stuck in companies. You can't learn it in school either. I keep hearong about companies that have to make their new hires unlearn what they learned in class... So where would scientists learn good practices? (in my experience books arent the answer either)
  - mnw21cam 6y ago
    I don't think it is primarily a problem in training. It is a problem in incentive. For the most part, scientific software is written to fulfil a goal, and usually that goal is in order to publish a paper. Paper publication is what scientists are judged on, not the quality of the software they write. The software only needs to work well enough to get the paper published. Once the paper is published, all incentive to maintain or improve the software disappears, especially if the message of the paper is primarily the results obtained using the software, because then all of the citations will be referring to those results, not the software. There are two main exceptions to this rule. One is when an actual software engineer ends up working as a scientist and having to write papers (like me). Then, the paper is more likely to be about the software itself, and the citations of the paper are from people who have also used the same software. This creates an incentive to make the software high quality, in order to get those citations. The other exception is when an institute decides that the world needs a decent piece of software to fulfil a vital role, and actually funds its development. Oh, I should probably mention - I just helped out with a programming course for undergraduate medical science students. Most of the time I was tearing my hair out about the terrible programming practices being taught by the module leader, who admittedly had absolutely no computer science qualifications whatsoever (and in fact didn't know the difference between a forward slash and a backslash). I think a solution to this is to recognise software quality as a common benefit, and divert funding towards it, allowing real software engineers to have an actual meaningful career in university. Universities are usually all over common benefit stuff, so you would have thought this would be easy enough. The Research Software Engineer programme in the UK is a start.
newbie2020 6y ago
Is shiv better than pyinstaller? When turned my script into a binary with this, it took 10 seconds extra on every execution just to start running the script (Eg to just print the help message). I found that to be unbearable since this was a command line script that was invoked anew for every execution
- nonameiguess 6y ago
  If you're not already using the --noupx flag for pyinstaller, you can probably cut that startup latency at least in half. With UPX enabled, the executable has to be decompressed every time it is called. The space saving tends to be minimal and not worth it, definitely not for a CLI where startup latency much more directly impacts the user experience.
- BiteCode_dev 6y ago
  They don't serve the same purpose. Pyinstaller builds bundle the python binary with the project. It works without a Python installed on the system, is not portable and the resulting binary is fat. It also can fail in numerous way and be slow, but end users will find it the easier to use when they work. Pyz files just contains python code. Shiv produce them by bundling your code and the dependencies, but you still need a Python installed on the system to run that. You pay the cost of zipping only on the first run though, after that, it's unzipped. And their are portable if you use no binary wheel. It's light and easy to get right. In that sense, they look more like War files in Java. Pyinstaller, like cx_freeze and nuikta are for end users. Pyz are for devs.
KptMarchewa 6y ago
>the Python community is at least half composed of amateurs. or, even worse, scientists.
- michaelcampbell 6y ago
  And most recently, tons of self styled "Data scientists". This has, IMO, diluted the rabid rhetoric of the 90's/oughties python community which is good, but replaced it with people who don't know and don't care what good code even looks like, so it can be difficult to work among them.
  - KptMarchewa 6y ago
    I feel sometimes that they pride in writing shitty, unmaintainable code; that they do it on purpose. One of the ways they enjoy life is to disregard all "best practices". And I'm not talking about some high-level OOP principles, but the basics - like keeping consistent code style, or even writing some tests. The previous crowd with love for monkeypatching everything and hate for abstraction other than dictionaries wasn't too pleasant, too.
    - salixrosa 6y ago
      lol I was just sitting here thinking I miss my Python job and not having trouble with IntelliJ detecting changes in dependencies and how frustrating it is to invalidate caches and reindex the whole project multiple times before I can run tests for the dependency change. I regularly miss the testing libraries we used, too -- it was so absurdly clean and easy to build robust mocks in our DJango stack (and I set up that tooling, largely on my own iniatiative, based on my team's feedback, so it's not like I'm just like wishing other people had already done the work). Now I work with a bunch of Java devs who use to work for Amazon and somehow use that as an excuse for not caring about tests, our ci-cd tools, and all these other things I'd gotten use to thinking of as standard in my past life as a Python dev. Code style! Hah. First day on the job I asked about that and was told nobody cared and everybody had their own preferences. ... I'm very bitter today. I'm working on it.
JAlexoid 6y ago
So... Java is bad, but let's ignore Python's problems. I'm forced to use Python at work, because data scientists demand it, and I can rant about your "good for data" language for at least 24 hours straight. Python isn't great at anything specifically, but it's crap in a few specific places. No multithreading? Fine! But then Python core libraries are crap at IPC, when multiprocess is the preferred way of parallel computing!
- BiteCode_dev 6y ago
  > So... Java is bad, but let's ignore Python's problems. I don't know where you read either of your propositions in any of my comment.
  - JAlexoid 6y ago
    And I quote "But it's way better for high level stuff, data mangling, scripting, gluing, and for most web dev stuff." You've listed things that rely on full stack of libraries and deployment options.(except scripting) As someone who has the pleasure to work with Python - Python is horrible at a lot of these things, in spite of the beauty of the language itself.
- pbourke 6y ago
  > Python isn't great at anything specifically Python is the best language in the world for small tools under 500 lines to solve glue and data munging problems.
  - JAlexoid 6y ago
    Python is too explicit to be the greatest, but it's popular nontheless.

mnw21cam 6y ago

It certainly doesn't help that the software I'm interested in is written for science, so there is a tendency to make it work just well enough on the author's machine to allow a paper to be published, and then abandon it.

ta988 6y ago
There is a clear training problem in science. But I believe part of it is that the knowledge on how to write good software is stuck in companies. You can't learn it in school either. I keep hearong about companies that have to make their new hires unlearn what they learned in class... So where would scientists learn good practices? (in my experience books arent the answer either)
- mnw21cam 6y ago
  I don't think it is primarily a problem in training. It is a problem in incentive. For the most part, scientific software is written to fulfil a goal, and usually that goal is in order to publish a paper. Paper publication is what scientists are judged on, not the quality of the software they write. The software only needs to work well enough to get the paper published. Once the paper is published, all incentive to maintain or improve the software disappears, especially if the message of the paper is primarily the results obtained using the software, because then all of the citations will be referring to those results, not the software. There are two main exceptions to this rule. One is when an actual software engineer ends up working as a scientist and having to write papers (like me). Then, the paper is more likely to be about the software itself, and the citations of the paper are from people who have also used the same software. This creates an incentive to make the software high quality, in order to get those citations. The other exception is when an institute decides that the world needs a decent piece of software to fulfil a vital role, and actually funds its development. Oh, I should probably mention - I just helped out with a programming course for undergraduate medical science students. Most of the time I was tearing my hair out about the terrible programming practices being taught by the module leader, who admittedly had absolutely no computer science qualifications whatsoever (and in fact didn't know the difference between a forward slash and a backslash). I think a solution to this is to recognise software quality as a common benefit, and divert funding towards it, allowing real software engineers to have an actual meaningful career in university. Universities are usually all over common benefit stuff, so you would have thought this would be easy enough. The Research Software Engineer programme in the UK is a start.

newbie2020 6y ago

Is shiv better than pyinstaller? When turned my script into a binary with this, it took 10 seconds extra on every execution just to start running the script (Eg to just print the help message). I found that to be unbearable since this was a command line script that was invoked anew for every execution

nonameiguess 6y ago
If you're not already using the --noupx flag for pyinstaller, you can probably cut that startup latency at least in half. With UPX enabled, the executable has to be decompressed every time it is called. The space saving tends to be minimal and not worth it, definitely not for a CLI where startup latency much more directly impacts the user experience.
BiteCode_dev 6y ago
They don't serve the same purpose. Pyinstaller builds bundle the python binary with the project. It works without a Python installed on the system, is not portable and the resulting binary is fat. It also can fail in numerous way and be slow, but end users will find it the easier to use when they work. Pyz files just contains python code. Shiv produce them by bundling your code and the dependencies, but you still need a Python installed on the system to run that. You pay the cost of zipping only on the first run though, after that, it's unzipped. And their are portable if you use no binary wheel. It's light and easy to get right. In that sense, they look more like War files in Java. Pyinstaller, like cx_freeze and nuikta are for end users. Pyz are for devs.

KptMarchewa 6y ago

>the Python community is at least half composed of amateurs. or, even worse, scientists.

michaelcampbell 6y ago
And most recently, tons of self styled "Data scientists". This has, IMO, diluted the rabid rhetoric of the 90's/oughties python community which is good, but replaced it with people who don't know and don't care what good code even looks like, so it can be difficult to work among them.
- KptMarchewa 6y ago
  I feel sometimes that they pride in writing shitty, unmaintainable code; that they do it on purpose. One of the ways they enjoy life is to disregard all "best practices". And I'm not talking about some high-level OOP principles, but the basics - like keeping consistent code style, or even writing some tests. The previous crowd with love for monkeypatching everything and hate for abstraction other than dictionaries wasn't too pleasant, too.
  - salixrosa 6y ago
    lol I was just sitting here thinking I miss my Python job and not having trouble with IntelliJ detecting changes in dependencies and how frustrating it is to invalidate caches and reindex the whole project multiple times before I can run tests for the dependency change. I regularly miss the testing libraries we used, too -- it was so absurdly clean and easy to build robust mocks in our DJango stack (and I set up that tooling, largely on my own iniatiative, based on my team's feedback, so it's not like I'm just like wishing other people had already done the work). Now I work with a bunch of Java devs who use to work for Amazon and somehow use that as an excuse for not caring about tests, our ci-cd tools, and all these other things I'd gotten use to thinking of as standard in my past life as a Python dev. Code style! Hah. First day on the job I asked about that and was told nobody cared and everybody had their own preferences. ... I'm very bitter today. I'm working on it.

JAlexoid 6y ago

So... Java is bad, but let's ignore Python's problems. I'm forced to use Python at work, because data scientists demand it, and I can rant about your "good for data" language for at least 24 hours straight. Python isn't great at anything specifically, but it's crap in a few specific places. No multithreading? Fine! But then Python core libraries are crap at IPC, when multiprocess is the preferred way of parallel computing!

BiteCode_dev 6y ago
> So... Java is bad, but let's ignore Python's problems. I don't know where you read either of your propositions in any of my comment.
- JAlexoid 6y ago
  And I quote "But it's way better for high level stuff, data mangling, scripting, gluing, and for most web dev stuff." You've listed things that rely on full stack of libraries and deployment options.(except scripting) As someone who has the pleasure to work with Python - Python is horrible at a lot of these things, in spite of the beauty of the language itself.
pbourke 6y ago
> Python isn't great at anything specifically Python is the best language in the world for small tools under 500 lines to solve glue and data munging problems.
- JAlexoid 6y ago
  Python is too explicit to be the greatest, but it's popular nontheless.