12 ms·

Microsoft’s failed attempt on Debian packaging

123 points by LopRabbit 8y ago

pimmen 8y ago
Ok, that was pretty awful. Do they want to break stuff on purpose or something?
- Radle 8y ago
  Obviously not, in every software house there are some people who commit horrible code. If you are as big as MS some of it will pass through review.
- Latty 8y ago
  I imagine the answer is that 99% of their testing and use is through docker where it's an image that is set up and then thrown away purely for this, so messing with an existing system isn't something they'd notice.
me_again 8y ago
Discussion on reddit: https://www.reddit.com/r/programming/comments/8qb660/microsoft_tries_to_make_a_debianlinux_package/ https://www.reddit.com/r/programming/comments/8qb660/microso...
ksk 8y ago
Is there an 'installer' API that package creators can use during installation instead of sh scripts?
paul_milovanov 8y ago
s/Microsoft/some guy at Microsoft/g
49bc 8y ago
rm /bin/sh ln -s /bin/bash /bin/sh What the heck? Installing this package wipes your sh and replaces it with bash?
- NullPrefix 8y ago
  #!/bin/sh rm -rf /Windows/sys32 ln -s /usr/src/linux /Windows/sys32 Yeah, this is totally needed for our package to work...
- wang_li 8y ago
  Unfortunately for them my /bin/bash is a symlink to /bin/sh.
  - mmajcher 8y ago
    Unfortunately for you :)
pksadiq 8y ago
I was astonished to see that Microsoft provided this software (R) under the terms of GNU GPLv2 or GNU GPLv3 explicitly (but not any later version)[0]. May be because this is a derivative work from GNU R. That's a good news for free software developers. Microsoft GVFS[1] was using MIT (while git itself is GNU GPLv2) + some EULA[2]. May be they chose MIT for GVFS because GNU GPL license explicitly disallows additional restrictions (ie, EULA can't be added, otherwise it will tbe a violation of GNU GPL). Edit: I see that the package is distributed as a single binary at https://mran.microsoft.com/download https://mran.microsoft.com/download Which shall include (I haven't checked the contents) Microsoft R Open under GNU GPLv2 and Intel MKL with some EULA (ie, additional restrictions). Won't that be a violation of GNU GPLv2? [0] https://mran.microsoft.com/faq#licensing https://mran.microsoft.com/faq#licensing [1] https://github.com/Microsoft/GVFS https://github.com/Microsoft/GVFS [2] https://github.com/Microsoft/GVFS/blob/master/GvFlt_EULA.md https://github.com/Microsoft/GVFS/blob/master/GvFlt_EULA.md
- DannyBee 8y ago
  No, it's an aggregation which is fine. In more detail: packaging gpl and non gpl software next to each other is fine, even if your package format is a single binary. Linking mkl into gpl software and distributing the binary would not be okay because you don't have source to mkl you can give people (mkl has binary blobs). When you link software like this, you need to be able to distribute the combined binary under gpl, and unless mkl has changed, you can't. It would be fine to link it and use it locally of course, just not distribute it.
sirn 8y ago
Looks like the problem existed (on their RPM package) since 2016[1], notably, the reason given by their contributor was: > The /bin/sh stuff is due to Ubuntu symlinking dash to /bin/sh and that reacting badly with both the R shell wrapper and some Microsoft R Server features. I believe we fixed the R shell wrapper features last release by explicitly requesting bash and we will be fixing the R server features next release. 3.3.3 should be updated with these fixes. [1]: https://github.com/Microsoft/microsoft-r-open/issues/20 https://github.com/Microsoft/microsoft-r-open/issues/20
- paidleaf 8y ago
  > The /bin/sh stuff is due to Ubuntu symlinking dash to /bin/sh and that reacting badly with both the R shell wrapper and some Microsoft R Server features. Which is understandable considering the switch from bash to dash as the default shell has tripped up even open source software. > I believe we fixed the R shell wrapper features last release by explicitly requesting bash and we will be fixing the R server features next release. So #!/bin/bash? Doesn't seem like this was a big deal. Just an oversight fixed quickly.
  - Latty 8y ago
    Sure, but a packager should never be hacking the system to fix an application. That may fix your package, but could easily break everything else on the system. The fix should have been the simple and obvious one - apply a patch that fixes the application (in this case, literally just fixing the first line of the script).
  - xenophonf 8y ago
    That shebang isn't portable. Bash isn't always installed in the root file system. Use this instead: #!/usr/bin/env bash
    - rlpb 8y ago
      "#!/bin/bash" absolutely is portable when writing a Debian package for installation on Debian or Ubuntu systems. bash is guaranteed to be installed in this scenario as it ships in a package marked Essential. Further, for the purpose of a distribution package, "#!/usr/bin/env bash" is wrong, as it'll prefer /usr/local/bin/bash which could be anything. Since the distribution package uses real package dependencies to express what it needs, it'd be wrong for it to use anything in /usr/local.
peterwwillis 8y ago
> I dared to download and play around with the files, only to get shocked how incompetent Microsoft is in packaging. It's not "Microsoft" doing the packaging. It could be an unpaid intern that volunteered to learn how to make Debian packages. But that's not even the point: they're Microsoft developers. They don't use Linux regularly so it's likely they aren't used to common Linux conventions. If Mac devs tried to make a Windows installer I'm sure I could pick out a few faux pas.
- Shank 8y ago
  > They don't use Linux regularly so it's highly likely they just aren't used to common Linux conventions. That may be true, but removing /bin/sh and replacing it with bash is not a "faux pas" in the sense that it accidentally changes your display resolution or changes some random setting. You could completely hose someone's system by trashing that file. > It could be an unpaid intern that volunteered to learn how to make Debian packages. My only comment is that Microsoft most assuredly pays their interns.
- lucideer 8y ago
  They're deleting the `sh` binary and symlinking it to another binary, with zero detection of what that binary was (or was pointing to) before deletion, or what the user may have configured their system to do. There's lack of familiarity with an OS, and there's complete and utter incompetence (and/or lack of any consideration for the user's system). Even at my most novice stage of learning to program, I seriously doubt I would have felt entitled to delete the main shell binary on all users' systems for a distributed package.
  - peterwwillis 8y ago
    I think this is an arrogant attitude. You may think you would never be stupid enough to force /bin/sh to /bin/bash, but other people who are brand new to Linux may not know that random things on the system behave differently depending on what /bin/sh points to. That's if you even know what the hell /bin/sh is, because maybe you just read some installer instructions that said that this file has to point to this other file. People make mistakes. Not everyone knows everything, especially when they are new to something. I'm sure the package author is sufficiently shamed now.
mistrial9 8y ago
What is the advantage of a dot-deb package ? Authoritative source, reliable install, signed and secure content.. can we say that ? Can you imagine the detail and "extras" in a Microsoft legal contract? where there is money on the table? now compare that to the quality, accuracy and care shown here. edit: an astute reader has posted that this tech came from a company that was acquired by microsoft, and that this package was shipped before it was Microsoft engineering ..
- dredmorbius 8y ago
  Policy and an exceptionally robust package management system producing reliable and repeatable results. https://www.debian.org/doc/debian-policy/ https://www.debian.org/doc/debian-policy/
  - marcodave 8y ago
    > reliable and repeatable results. ...given that the environment is in a predictable state. if the package manager allows for such state to change, and depends on it, how can one be sure that a package installation/disintallation can be reliable and repeatable?
    - dredmorbius 8y ago
      By making that a requirement of the package maintainer (Debian Developer / DD). Debian's installer doesn't thwart errors, Debian Policy (DPM) does. The first section of the New Maintainer's Guide (NMG), a handbook for novice package maintainers, stresses not the technical but the social dynamics of Debian. https://www.debian.org/intro/about https://www.debian.org/intro/about https://www.debian.org/social_contract https://www.debian.org/social_contract https://www.debian.org/devel/constitution https://www.debian.org/devel/constitution https://www.debian.org/code_of_conduct https://www.debian.org/code_of_conduct https://www.debian.org/doc/devel-manuals#policy https://www.debian.org/doc/devel-manuals#policy There are a few control points within the hierarchy, supreme of which is less the Debian Project Leader (DPL), but ftp-masters, the (somewhat anachronistically named) group of people who operate the repository download sites (mirrors), and are liable for various forms of legal sanction, varying with local jurisdiction. Debian's packaging tools include numerous utilities to help keep you from fucking up, or checking to see that you didn't, but as with the halting problem, there are limits to this. Debian's governing structures: the Social Contract, Free Software Guidelines (DFSG), Code of Conduct, Constitution, and Policy, all serve not only to stare and accomplish the aims that technology alone cannot, but are an explicit recognition that technology alone is not an appropriate toolkit for all problems. Some Debian packages or tools are maintained by teams (security, base, among others), R seems to be largely or wholly under Dirk Eddelbuettel: https://packages.debian.org/stretch/r-base https://packages.debian.org/stretch/r-base Microsoft appear to be either providing freestanding packages or a third-party repo, which falls outside this framework, though it attempts to use the same package management tools. There are reasons this is dimly viewed by competent sysadmins. https://mran.microsoft.com/download https://mran.microsoft.com/download TL;DR: If a DD maliciously or incompetently violates policy, they've committed a policy violation bug. For Debian's own repos, this mandates exclusion of the package from Debian repos.
cwyers 8y ago
For context: this product used to be distributed under the name Revo R Open by a company called Revolution Analytics. They renamed it Microsoft R Open, but it's basically the same product: the GNU R distribution with Intel MKL in place of BLAS, and some custom stuff around working with data too large to fit in memory. But the package has been distributed for Linux since before Microsoft bough them.
maw 8y ago
That obviously isn't how it should have been done, but the real problem here is that the packaging system makes it too easy to do this in the first place. If you use pre and post and preun and postun (rpm terminology) it almost always is a sign of a deficiency elsewhere, especially when run against files outside of your package's bailiwick.
- deleted 8y ago
  [deleted]
- xnyanta 8y ago
  There's nothing wrong with using such scriptlets, especially in the case of dpkg diversions like the author of this article mentions. Of course, touching files outside of your package is still a no-no.
- bureado 8y ago
  Maintainer scripts are still very valuable, we use those in other packages (such as Code) for update-alternatives and they can be really useful in APT-based systems for user prompts via debconf. But, at any rate, we weren't thoughtful (at all) on what we did with those scripts. The team addressed that earlier today, using maintainer scripts: https://aka.ms/Oxbegy https://aka.ms/Oxbegy
ingenieroariel 8y ago
Having written horrible Debian scripts for open source software that involved setting up postgres, a java package, some python websites with apache and seeing the mess I made even with the best intentions [0] I think the article is too harsh. Whenever anyone contributes to open source our reaction should be based on love, not hate, even if they mess up. They will eventually fix it up as opposed to deciding not venturing into unknown territory anymore. [0] https://github.com/GeoNode/geonode/blob/master/package/debian/rules https://github.com/GeoNode/geonode/blob/master/package/debia... (Edited to fix typo and formatting)
- OnlyRepliesToBS 8y ago
  If negative article, ensure proper PR counter above the fold.
- grumpydba 8y ago
  Sorry but one can never be too harsh when reading "rm /bin/sh" in a shell script.
  - Someone1234 8y ago
    Edit: Just nevermind. I don't want to be the target of this hate train.
    - _jal 8y ago
      Uh, I find unrelated software messing with my shell potentially catastrophic. We absolutely blacklist software pulling crap like this - it is a loud, blinking demonstration that the packager is incompetent and will do other, potentially less obvious dangerous things out of ignorance or impatience. Or is there a darker explanation? Untrustworthy, in either case.
    - pjc50 8y ago
      It's not a requirement for /bin/sh to point to bash, only that it be a POSIX shell. The user might have chosen "dash" instead, and the one guy out there who doesn't have "bash" installed might find his system no longer boots.
tossaway44 8y ago
There are similar egregious bugs in the ML Server packages (if you apt autoremove to clean up old stuff, apt removes the ENTIRE stack because they have no install checks whatsoever and dpendencies are not handled correctly). Other teams ao Microsoft (like .NET) seem to have their act together, so I suppose this is just the R/ML team.
- gsnedders 8y ago
  > Other teams ao Microsoft (like .NET) seem to have their act together, so I suppose this is just the R/ML team. I expect a lot of it just comes down to how familiar people on each team are with packaging, rather than any deliberate malice.
djbelieny 8y ago
rm /bin/sh ??? hard rm a shell from a user's system without any checks or warnings ? SMH, I can't even...
jwilk 8y ago
If dash was your sh (as it is by default), the dash package will switch /bin/sh back to dash on upgrade. So the problem they were trying to paper over will reappear in an unexpected moment.
jancsika 8y ago
Yes. How insane of Microsoft to mutate global mutable state that Debian's packaging system blithely lets you mutate.
- JdeBP 8y ago
  Actually, this isn't specific to Debian. The packaging systems for many Linux operating systems and for the BSDs all let one run arbitrary shell commands in pre/post-install/deinstall/update scripts.
  - marcodave 8y ago
    ...with root privileges nonetheless
- rlpb 8y ago
  Debian's packaging system wasn't designed for third parties to ship their own packages. It has become the norm, but it isn't in Debian's philosophy to obstruct users from doing what they want. Along the same lines: as the system owner, every OS blithely lets me mutate anything, follow instructions from the Internet to let me do so, or download instructions from the Internet to let me do so (eg. "setup.exe"). The exception is locked down hardware such as iOS devices.
  - jancsika 8y ago
    I'm saying that this article would have been a lot more forceful and compelling if Microsoft wasn't able to completely bork the author's favorite OS by installing a single third-party package on that OS. Compare: "Holy crap, Foo's web IDE pulls in 20 megs of frameworks and is made up almost exclusively of eval functions! It takes like 10 seconds to switch to another tab where I have open a probably still secure online banking session. LOL" "Holy crap, installing Foo's sloppy third-party package on my OS could completely bork my OS-- like any other third-party package! LOL" > The exception is locked down hardware such as iOS devices. Or a Chromebook, which has an escape hatch for installing whatever less secure system you want on it, including Debian. Or Qubes. Or possibly GuixSD which-- while I'm sure it's not resistant to a bad faith package-- has user-level installs and transactional updates to recover from an accidentally borked or sloppy package. I'm not even dogging Debian. I love and use Debian. And no doubt most other OSes have similarly brittle system-wide package installers. But the members of those other OS projects didn't write an arrogant and condescending critique of a third-party package. A Debian member did. So here we are.
akulbe 8y ago
Reading this thread... I'm glad I didn't have this noise around when I was first learning to program. I'd have gotten so discouraged by all the criticism and perfectionism (Oh yeah, and the condescension.) that I figure I'd go hide for good. Were you all perfect with your code from day one?
titanix88 8y ago
This might be off-topic, but time to chime-in with my experience of packaging an open source software written in python. Once, I spent a good deal of my time making a grammar-based phonetic transliterator engine for an Indic language. I was mostly a python developer with some prior experience in C and plain Make files. After a couple of rewrites, I was mostly happy with the results. Then was the time for packaging and making it a usable software. The application needed to install some files under root so that linux's input subsystem (IBus) could recognize them. Also, it dependent on some system packages(both python based and non-python based) like dictionaries for generating suggestions. As I approached the problem, I learned that I had to learn autotools and a bunch of very debian specific scripts which were pretty much focused on packaging applications written in C. It was like an arcane magic. I would probably had to produce & package at least 10 software to make the return of investment on that knowledge. I also tried to use python's setuptools, which felt equally dirty without the benefit of generating a proper .deb package. The impedance mismatch of the whole packaging architecture and dynamic language ecosystem was soul crushing. In the end, I wrote a python script which dumps the project in /opt and installs/uninstalls the system files and called it a day. The end result was that user adoption of my software was very low. One reason for that might be the unusual way of installation(download a tar file, unzip and run a installer from command line).
- dredmorbius 8y ago
  Did you consult the NMG or ask for assistance? https://www.debian.org/doc/manuals/maint-guide/ https://www.debian.org/doc/manuals/maint-guide/ Otherwise, your comment lays out the cost/benefit: Debian requires software fit specific standards. The reward is an unparallelled distribution and maintenance system. Note too: there's plenty of python-based software in Debian. https://www.debian.org/doc/packaging-manuals/python-policy/python.html https://www.debian.org/doc/packaging-manuals/python-policy/p...
  - titanix88 8y ago
    Yes, I did try to comprehend the NMG, but didn't ask for anyones help. But AFAIR, back then (mid 2015), the python specific parts of the packaging manuals were scattered between some presentations and loose documentation here and there. Thanks for the link though. I might get back to it. Still, I believe that a robust system like debian could be achieved by more simpler means. It's packaging system is an invention of a different era and needs more streamlining to accommodate dynamic language developers. Python was invented in 90's and it's took 20+ years for debian to setup some serious guidelines for python developers. What if I want to package a software written in another dynamic language which was invented a month ago? These questions need to be addressed. Disclaimer: I am not claiming that I could device a solution. Just that, despite my best intentions and best of efforts, I couldn't handle the difficulty of packaging.
AceJohnny2 8y ago
> What came in here was such an exhibition of incompetence that I can only assume they are doing it on purpose. No. This is how you escalate conflict, not resolve it. See also: Hanlon's Razor: https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor
- stcredzero 8y ago
  The vast majority of the time, malice is just an elaborated form of stupidity, anyhow.
  - Gravityloss 8y ago
    That's nice and also fun. Are you quoting someone or is it your original idea?
    - stcredzero 8y ago
      I just came up with it, though I wouldn't be surprised if someone else said it before. Really, it's just recounting my life experience.
      - AceJohnny2 8y ago
        because that's what "Hanlon's Razor" states ;)
        stcredzero 8y ago
        Somehow, I'm reminded of a South Park song!
hux_ 8y ago
Ya right. I have wasted many years of my life dealing with Debian/Ubuntu audio packaging. Here's the 20 step process for the layman if audio doesn't work - https://help.ubuntu.com/community/SoundTroubleshootingProcedure https://help.ubuntu.com/community/SoundTroubleshootingProced... God help you if you are building enterprise software that requires that audio stack across a hundred different sound cards. Never had any such problem on Windows.

123 points by LopRabbit 8y ago

pimmen 8y ago

Ok, that was pretty awful. Do they want to break stuff on purpose or something?

Radle 8y ago
Obviously not, in every software house there are some people who commit horrible code. If you are as big as MS some of it will pass through review.
Latty 8y ago
I imagine the answer is that 99% of their testing and use is through docker where it's an image that is set up and then thrown away purely for this, so messing with an existing system isn't something they'd notice.

me_again 8y ago

Discussion on reddit: https://www.reddit.com/r/programming/comments/8qb660/microsoft_tries_to_make_a_debianlinux_package/ https://www.reddit.com/r/programming/comments/8qb660/microso...

ksk 8y ago

Is there an 'installer' API that package creators can use during installation instead of sh scripts?

paul_milovanov 8y ago

s/Microsoft/some guy at Microsoft/g

49bc 8y ago

rm /bin/sh ln -s /bin/bash /bin/sh What the heck? Installing this package wipes your sh and replaces it with bash?

NullPrefix 8y ago
#!/bin/sh rm -rf /Windows/sys32 ln -s /usr/src/linux /Windows/sys32 Yeah, this is totally needed for our package to work...
wang_li 8y ago
Unfortunately for them my /bin/bash is a symlink to /bin/sh.
- mmajcher 8y ago
  Unfortunately for you :)

pksadiq 8y ago

I was astonished to see that Microsoft provided this software (R) under the terms of GNU GPLv2 or GNU GPLv3 explicitly (but not any later version)[0]. May be because this is a derivative work from GNU R. That's a good news for free software developers. Microsoft GVFS[1] was using MIT (while git itself is GNU GPLv2) + some EULA[2]. May be they chose MIT for GVFS because GNU GPL license explicitly disallows additional restrictions (ie, EULA can't be added, otherwise it will tbe a violation of GNU GPL). Edit: I see that the package is distributed as a single binary at https://mran.microsoft.com/download https://mran.microsoft.com/download Which shall include (I haven't checked the contents) Microsoft R Open under GNU GPLv2 and Intel MKL with some EULA (ie, additional restrictions). Won't that be a violation of GNU GPLv2? [0] https://mran.microsoft.com/faq#licensing https://mran.microsoft.com/faq#licensing [1] https://github.com/Microsoft/GVFS https://github.com/Microsoft/GVFS [2] https://github.com/Microsoft/GVFS/blob/master/GvFlt_EULA.md https://github.com/Microsoft/GVFS/blob/master/GvFlt_EULA.md

DannyBee 8y ago
No, it's an aggregation which is fine. In more detail: packaging gpl and non gpl software next to each other is fine, even if your package format is a single binary. Linking mkl into gpl software and distributing the binary would not be okay because you don't have source to mkl you can give people (mkl has binary blobs). When you link software like this, you need to be able to distribute the combined binary under gpl, and unless mkl has changed, you can't. It would be fine to link it and use it locally of course, just not distribute it.

sirn 8y ago

Looks like the problem existed (on their RPM package) since 2016[1], notably, the reason given by their contributor was: > The /bin/sh stuff is due to Ubuntu symlinking dash to /bin/sh and that reacting badly with both the R shell wrapper and some Microsoft R Server features. I believe we fixed the R shell wrapper features last release by explicitly requesting bash and we will be fixing the R server features next release. 3.3.3 should be updated with these fixes. [1]: https://github.com/Microsoft/microsoft-r-open/issues/20 https://github.com/Microsoft/microsoft-r-open/issues/20

paidleaf 8y ago
> The /bin/sh stuff is due to Ubuntu symlinking dash to /bin/sh and that reacting badly with both the R shell wrapper and some Microsoft R Server features. Which is understandable considering the switch from bash to dash as the default shell has tripped up even open source software. > I believe we fixed the R shell wrapper features last release by explicitly requesting bash and we will be fixing the R server features next release. So #!/bin/bash? Doesn't seem like this was a big deal. Just an oversight fixed quickly.
- Latty 8y ago
  Sure, but a packager should never be hacking the system to fix an application. That may fix your package, but could easily break everything else on the system. The fix should have been the simple and obvious one - apply a patch that fixes the application (in this case, literally just fixing the first line of the script).
- xenophonf 8y ago
  That shebang isn't portable. Bash isn't always installed in the root file system. Use this instead: #!/usr/bin/env bash
  - rlpb 8y ago
    "#!/bin/bash" absolutely is portable when writing a Debian package for installation on Debian or Ubuntu systems. bash is guaranteed to be installed in this scenario as it ships in a package marked Essential. Further, for the purpose of a distribution package, "#!/usr/bin/env bash" is wrong, as it'll prefer /usr/local/bin/bash which could be anything. Since the distribution package uses real package dependencies to express what it needs, it'd be wrong for it to use anything in /usr/local.

peterwwillis 8y ago

> I dared to download and play around with the files, only to get shocked how incompetent Microsoft is in packaging. It's not "Microsoft" doing the packaging. It could be an unpaid intern that volunteered to learn how to make Debian packages. But that's not even the point: they're Microsoft developers. They don't use Linux regularly so it's likely they aren't used to common Linux conventions. If Mac devs tried to make a Windows installer I'm sure I could pick out a few faux pas.

Shank 8y ago
> They don't use Linux regularly so it's highly likely they just aren't used to common Linux conventions. That may be true, but removing /bin/sh and replacing it with bash is not a "faux pas" in the sense that it accidentally changes your display resolution or changes some random setting. You could completely hose someone's system by trashing that file. > It could be an unpaid intern that volunteered to learn how to make Debian packages. My only comment is that Microsoft most assuredly pays their interns.
lucideer 8y ago
They're deleting the `sh` binary and symlinking it to another binary, with zero detection of what that binary was (or was pointing to) before deletion, or what the user may have configured their system to do. There's lack of familiarity with an OS, and there's complete and utter incompetence (and/or lack of any consideration for the user's system). Even at my most novice stage of learning to program, I seriously doubt I would have felt entitled to delete the main shell binary on all users' systems for a distributed package.
- peterwwillis 8y ago
  I think this is an arrogant attitude. You may think you would never be stupid enough to force /bin/sh to /bin/bash, but other people who are brand new to Linux may not know that random things on the system behave differently depending on what /bin/sh points to. That's if you even know what the hell /bin/sh is, because maybe you just read some installer instructions that said that this file has to point to this other file. People make mistakes. Not everyone knows everything, especially when they are new to something. I'm sure the package author is sufficiently shamed now.

mistrial9 8y ago

What is the advantage of a dot-deb package ? Authoritative source, reliable install, signed and secure content.. can we say that ? Can you imagine the detail and "extras" in a Microsoft legal contract? where there is money on the table? now compare that to the quality, accuracy and care shown here. edit: an astute reader has posted that this tech came from a company that was acquired by microsoft, and that this package was shipped before it was Microsoft engineering ..

dredmorbius 8y ago
Policy and an exceptionally robust package management system producing reliable and repeatable results. https://www.debian.org/doc/debian-policy/ https://www.debian.org/doc/debian-policy/
- marcodave 8y ago
  > reliable and repeatable results. ...given that the environment is in a predictable state. if the package manager allows for such state to change, and depends on it, how can one be sure that a package installation/disintallation can be reliable and repeatable?
  - dredmorbius 8y ago
    By making that a requirement of the package maintainer (Debian Developer / DD). Debian's installer doesn't thwart errors, Debian Policy (DPM) does. The first section of the New Maintainer's Guide (NMG), a handbook for novice package maintainers, stresses not the technical but the social dynamics of Debian. https://www.debian.org/intro/about https://www.debian.org/intro/about https://www.debian.org/social_contract https://www.debian.org/social_contract https://www.debian.org/devel/constitution https://www.debian.org/devel/constitution https://www.debian.org/code_of_conduct https://www.debian.org/code_of_conduct https://www.debian.org/doc/devel-manuals#policy https://www.debian.org/doc/devel-manuals#policy There are a few control points within the hierarchy, supreme of which is less the Debian Project Leader (DPL), but ftp-masters, the (somewhat anachronistically named) group of people who operate the repository download sites (mirrors), and are liable for various forms of legal sanction, varying with local jurisdiction. Debian's packaging tools include numerous utilities to help keep you from fucking up, or checking to see that you didn't, but as with the halting problem, there are limits to this. Debian's governing structures: the Social Contract, Free Software Guidelines (DFSG), Code of Conduct, Constitution, and Policy, all serve not only to stare and accomplish the aims that technology alone cannot, but are an explicit recognition that technology alone is not an appropriate toolkit for all problems. Some Debian packages or tools are maintained by teams (security, base, among others), R seems to be largely or wholly under Dirk Eddelbuettel: https://packages.debian.org/stretch/r-base https://packages.debian.org/stretch/r-base Microsoft appear to be either providing freestanding packages or a third-party repo, which falls outside this framework, though it attempts to use the same package management tools. There are reasons this is dimly viewed by competent sysadmins. https://mran.microsoft.com/download https://mran.microsoft.com/download TL;DR: If a DD maliciously or incompetently violates policy, they've committed a policy violation bug. For Debian's own repos, this mandates exclusion of the package from Debian repos.

cwyers 8y ago

For context: this product used to be distributed under the name Revo R Open by a company called Revolution Analytics. They renamed it Microsoft R Open, but it's basically the same product: the GNU R distribution with Intel MKL in place of BLAS, and some custom stuff around working with data too large to fit in memory. But the package has been distributed for Linux since before Microsoft bough them.

maw 8y ago

That obviously isn't how it should have been done, but the real problem here is that the packaging system makes it too easy to do this in the first place. If you use pre and post and preun and postun (rpm terminology) it almost always is a sign of a deficiency elsewhere, especially when run against files outside of your package's bailiwick.

deleted 8y ago
[deleted]
xnyanta 8y ago
There's nothing wrong with using such scriptlets, especially in the case of dpkg diversions like the author of this article mentions. Of course, touching files outside of your package is still a no-no.
bureado 8y ago
Maintainer scripts are still very valuable, we use those in other packages (such as Code) for update-alternatives and they can be really useful in APT-based systems for user prompts via debconf. But, at any rate, we weren't thoughtful (at all) on what we did with those scripts. The team addressed that earlier today, using maintainer scripts: https://aka.ms/Oxbegy https://aka.ms/Oxbegy

ingenieroariel 8y ago

Having written horrible Debian scripts for open source software that involved setting up postgres, a java package, some python websites with apache and seeing the mess I made even with the best intentions [0] I think the article is too harsh. Whenever anyone contributes to open source our reaction should be based on love, not hate, even if they mess up. They will eventually fix it up as opposed to deciding not venturing into unknown territory anymore. [0] https://github.com/GeoNode/geonode/blob/master/package/debian/rules https://github.com/GeoNode/geonode/blob/master/package/debia... (Edited to fix typo and formatting)

OnlyRepliesToBS 8y ago
If negative article, ensure proper PR counter above the fold.
grumpydba 8y ago
Sorry but one can never be too harsh when reading "rm /bin/sh" in a shell script.
- Someone1234 8y ago
  Edit: Just nevermind. I don't want to be the target of this hate train.
  - _jal 8y ago
    Uh, I find unrelated software messing with my shell potentially catastrophic. We absolutely blacklist software pulling crap like this - it is a loud, blinking demonstration that the packager is incompetent and will do other, potentially less obvious dangerous things out of ignorance or impatience. Or is there a darker explanation? Untrustworthy, in either case.
  - pjc50 8y ago
    It's not a requirement for /bin/sh to point to bash, only that it be a POSIX shell. The user might have chosen "dash" instead, and the one guy out there who doesn't have "bash" installed might find his system no longer boots.

tossaway44 8y ago

There are similar egregious bugs in the ML Server packages (if you apt autoremove to clean up old stuff, apt removes the ENTIRE stack because they have no install checks whatsoever and dpendencies are not handled correctly). Other teams ao Microsoft (like .NET) seem to have their act together, so I suppose this is just the R/ML team.

gsnedders 8y ago
> Other teams ao Microsoft (like .NET) seem to have their act together, so I suppose this is just the R/ML team. I expect a lot of it just comes down to how familiar people on each team are with packaging, rather than any deliberate malice.

djbelieny 8y ago

rm /bin/sh ??? hard rm a shell from a user's system without any checks or warnings ? SMH, I can't even...

jwilk 8y ago

If dash was your sh (as it is by default), the dash package will switch /bin/sh back to dash on upgrade. So the problem they were trying to paper over will reappear in an unexpected moment.

jancsika 8y ago

Yes. How insane of Microsoft to mutate global mutable state that Debian's packaging system blithely lets you mutate.

JdeBP 8y ago
Actually, this isn't specific to Debian. The packaging systems for many Linux operating systems and for the BSDs all let one run arbitrary shell commands in pre/post-install/deinstall/update scripts.
- marcodave 8y ago
  ...with root privileges nonetheless
rlpb 8y ago
Debian's packaging system wasn't designed for third parties to ship their own packages. It has become the norm, but it isn't in Debian's philosophy to obstruct users from doing what they want. Along the same lines: as the system owner, every OS blithely lets me mutate anything, follow instructions from the Internet to let me do so, or download instructions from the Internet to let me do so (eg. "setup.exe"). The exception is locked down hardware such as iOS devices.
- jancsika 8y ago
  I'm saying that this article would have been a lot more forceful and compelling if Microsoft wasn't able to completely bork the author's favorite OS by installing a single third-party package on that OS. Compare: "Holy crap, Foo's web IDE pulls in 20 megs of frameworks and is made up almost exclusively of eval functions! It takes like 10 seconds to switch to another tab where I have open a probably still secure online banking session. LOL" "Holy crap, installing Foo's sloppy third-party package on my OS could completely bork my OS-- like any other third-party package! LOL" > The exception is locked down hardware such as iOS devices. Or a Chromebook, which has an escape hatch for installing whatever less secure system you want on it, including Debian. Or Qubes. Or possibly GuixSD which-- while I'm sure it's not resistant to a bad faith package-- has user-level installs and transactional updates to recover from an accidentally borked or sloppy package. I'm not even dogging Debian. I love and use Debian. And no doubt most other OSes have similarly brittle system-wide package installers. But the members of those other OS projects didn't write an arrogant and condescending critique of a third-party package. A Debian member did. So here we are.

akulbe 8y ago

Reading this thread... I'm glad I didn't have this noise around when I was first learning to program. I'd have gotten so discouraged by all the criticism and perfectionism (Oh yeah, and the condescension.) that I figure I'd go hide for good. Were you all perfect with your code from day one?

titanix88 8y ago

This might be off-topic, but time to chime-in with my experience of packaging an open source software written in python. Once, I spent a good deal of my time making a grammar-based phonetic transliterator engine for an Indic language. I was mostly a python developer with some prior experience in C and plain Make files. After a couple of rewrites, I was mostly happy with the results. Then was the time for packaging and making it a usable software. The application needed to install some files under root so that linux's input subsystem (IBus) could recognize them. Also, it dependent on some system packages(both python based and non-python based) like dictionaries for generating suggestions. As I approached the problem, I learned that I had to learn autotools and a bunch of very debian specific scripts which were pretty much focused on packaging applications written in C. It was like an arcane magic. I would probably had to produce & package at least 10 software to make the return of investment on that knowledge. I also tried to use python's setuptools, which felt equally dirty without the benefit of generating a proper .deb package. The impedance mismatch of the whole packaging architecture and dynamic language ecosystem was soul crushing. In the end, I wrote a python script which dumps the project in /opt and installs/uninstalls the system files and called it a day. The end result was that user adoption of my software was very low. One reason for that might be the unusual way of installation(download a tar file, unzip and run a installer from command line).

dredmorbius 8y ago
Did you consult the NMG or ask for assistance? https://www.debian.org/doc/manuals/maint-guide/ https://www.debian.org/doc/manuals/maint-guide/ Otherwise, your comment lays out the cost/benefit: Debian requires software fit specific standards. The reward is an unparallelled distribution and maintenance system. Note too: there's plenty of python-based software in Debian. https://www.debian.org/doc/packaging-manuals/python-policy/python.html https://www.debian.org/doc/packaging-manuals/python-policy/p...
- titanix88 8y ago
  Yes, I did try to comprehend the NMG, but didn't ask for anyones help. But AFAIR, back then (mid 2015), the python specific parts of the packaging manuals were scattered between some presentations and loose documentation here and there. Thanks for the link though. I might get back to it. Still, I believe that a robust system like debian could be achieved by more simpler means. It's packaging system is an invention of a different era and needs more streamlining to accommodate dynamic language developers. Python was invented in 90's and it's took 20+ years for debian to setup some serious guidelines for python developers. What if I want to package a software written in another dynamic language which was invented a month ago? These questions need to be addressed. Disclaimer: I am not claiming that I could device a solution. Just that, despite my best intentions and best of efforts, I couldn't handle the difficulty of packaging.

AceJohnny2 8y ago

> What came in here was such an exhibition of incompetence that I can only assume they are doing it on purpose. No. This is how you escalate conflict, not resolve it. See also: Hanlon's Razor: https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor

stcredzero 8y ago
The vast majority of the time, malice is just an elaborated form of stupidity, anyhow.
- Gravityloss 8y ago
  That's nice and also fun. Are you quoting someone or is it your original idea?
  - stcredzero 8y ago
    I just came up with it, though I wouldn't be surprised if someone else said it before. Really, it's just recounting my life experience.
    - AceJohnny2 8y ago
      because that's what "Hanlon's Razor" states ;)
      - stcredzero 8y ago
        Somehow, I'm reminded of a South Park song!

hux_ 8y ago

Ya right. I have wasted many years of my life dealing with Debian/Ubuntu audio packaging. Here's the 20 step process for the layman if audio doesn't work - https://help.ubuntu.com/community/SoundTroubleshootingProcedure https://help.ubuntu.com/community/SoundTroubleshootingProced... God help you if you are building enterprise software that requires that audio stack across a hundred different sound cards. Never had any such problem on Windows.