9 ms·

This is definitely the most reasonable case I've heard made from this argument, and it's changed my stance on the issue. That said, I feel the cryptography pro

by macksd 6y ago

This is definitely the most reasonable case I've heard made from this argument, and it's changed my stance on the issue. That said, I feel the cryptography project handled this poorly. I encountered the issue when one day, installing a Python Ansible module the same way I did the day before required a compiler for a different language that I've never used before and that is hardly used, if at all, in the rest of my stack. The Ansible project seemed to be taken by surprise, and eventually I found the root upstream issue, where the initial attitude was "too bad". Some people making those comments worked for Red Hat / IBM, Ansible is heavily backed by that. What company cares more about Linux on S390 than Red Hat / IBM? I would suggest none. So the fact that they had this attitude and community reaction to me suggests the problem is not one of expecting free work for a corporation on a non-free architecture. It was IMHO a combination of a lack of forethought, communication, and yes, perhaps a change that is overdue and will just be a little painful. The suggestion to maintain an LTS release while people figure out how to proceed is the right move.

steveklabnik 6y ago
> Some people making those comments worked for Red Hat / IBM, Ansible is heavily backed by that. Okay, so, I don't mean to pick on you here, but I've seen this sentiment cropping up a few times. Not everyone can be familiar with everything, but I would caution you against immediately assuming corporate politics are at the root here. Anyone familiar with Alex Gaynor and his work over the last few years would know that he cares about Python, and memory safety. That has remained consistent regardless of his employer. Immediately assuming that this has something to do with company politics, rather than just a tireless open source contributor working to improve the things that he cares about, for years, is making a bit of a category error, in my opinion.
- macksd 6y ago
  I agree - I don't feel picked on :D I guess my point is that it's overly dismissive to say "no free work for companies on niche architectures" when projects backed by the same company were a bit blind-sided by this and a bunch of people lost time chasing down what happened and why. To me that's a sign that this isn't just a mismatch of ideology or someone wanting free work: it was a failure in communication and mismatched expectations. If it had just been on a major version bump, I wonder if we'd even be having this discussion.
  - steveklabnik 6y ago
    Glad to hear it :) I think it gets really hard the larger the company you talk about. Before I worked at bigger places, I assumed a lot more coherence than is actually the case.
  - rodgerd 6y ago
    Sure, but the message - that if it's a problem for Red Hat's Ansible team or Red Hat's mainframe Linux team, they should be doing the work needed to make it not a problem. Ansible Tower subscriptions aren't exactly cheap, and neither is RHEL s390x. If there's not the fat to be uplifting the core infrastructure needed to run RHEL or Ansible on Red Hat's products, that's most likely a choice.
- stonogo 6y ago
  I think corporate politics are at the root here, even if they're not Alex Gaynor's corporate politics. This blog post amounts to blessing amd64 and aarch64 as the only sustainable instruction set architectures. Rust is currently the apex of a tall stack of hundreds of millions of lines of code, once you account for LLVM etc. Using it as the basis for other software means only processors with sufficient market penetration are 'worthy' candidates. In the long run, if Rust is as successful as lots of us hope it will be, this will kill what innovation is left in the hardware space. If a company is sufficiently motivated to care, it will almost certainly be cheaper to fork the code back to using C than it would be to forklift LLVM to a new architecture. Is anyone aware of a direct Rust compiler project? Even discounting GCC, Go was bootstrapped from relatively simple (and naive) C compilers until it became self-hosting. I think a basic non-optimizing Rust compiler would go a long way toward leaving the door open for onboarding older -- and more importantly, novel -- architectures into the ecosystem.
  - steveklabnik 6y ago
    I didn’t compare HEADs because I’m on my phone, but found reports that in 2019, LLVM was 7 million LOC, and gcc was 15. Both are C++ compiler projects. Why the double standard? I also don’t know why re-writing an entire project and creating a compiler for it is somehow easier than writing an LLVM backend.
    - stonogo 6y ago
      What double standard? I wouldn't recommend a language tie itself to GCC any more than I would recommend LLVM. The point is that the Rust compiler catalog is unhealthily small, and that introduces problems like this one. To your other point, writing an LLVM backend is one thing. Getting it upstreamed is another, and maintaining it is another still. Then you have to navigate the politics of two foundations, both of whose boards of directors are basically the Who's Who of competing interests. I've watched more than one project fail to navigate those waters. Anyway, the cryptography package going from python + c to python + C + C++ + Rust is a cambrian explosion of build time complexity, and in my work we found it simpler to just get rid of the python and the cryptography package, so it's mostly academic to me.
pornel 6y ago
The switch hasn't been sudden. It's just that many levels of disconnect between the project authors and downstream users meant it was basically impossible to communicate to all the affected users — nobody looks at their deps-of-deps until they break. And they've released Rust as optional component you can disable, precisely because nobody paid attention until it actually shipped.
- masklinn 6y ago
  > The switch hasn't been sudden. It's just that many levels of disconnect between the project authors and downstream users meant it was basically impossible to communicate to all the affected users — nobody looks at their deps-of-deps until they break. Exactly, from the original shitstorm issue: > Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings. > Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels? At one point there's sadly not much the project can do and still make progress.
  - its-summertime 6y ago
    > > Do you have constructive suggestions how to communicate changes That one kinda got me, when python intentionally has a runtime developer-to-developer communication system, per say: https://docs.python.org/3/library/warnings.html https://docs.python.org/3/library/warnings.html
    - ploxiln 6y ago
      Warnings are a funny thing, lots of projects like to turn all warnings into failing errors in CI, sometimes even at package build time, they think it's a best-practice ... but it means that nobody else can use warnings to communicate things, or else everything breaks, nullifying the utility of warnings. see also https://lwn.net/Articles/740804/ https://lwn.net/Articles/740804/
laserharvest 6y ago
Why is it necessarily Cryptography’s fault that Python Ansible was taken by surprise? Or any of the other affected parties along the chain? That Cryptography was starting to include Rust in the project was announced on the mailing list by Gaynor last summer. And that email said exactly what future (at that time) release would require a Rust toolchain if the project was to be built from source. The reply to that by the Gentoo guy (who started the Github issue) was that package maintainers cannot follow every single mailing list of every dependency. That is debatable (e.g. maybe you should make an exception for security applications), but let’s take that as a given for now. In that case, what is Cryptography to do? Where should they announce such things in a way that orgs like Gentoo will see it? And also notice it and not just mentally gloss over it as some kind of “spam”? If the Gentoo guy didn’t see it half a year ago, would he have seen an announcement (or the reminders) if it was made five years ago?
- klyrs 6y ago
  Folks with an eye towards backwards compatibility typically don't implement breaking changes without a year of deprecation warnings emitted by the software. Contrast that to a notice posted in a sub-basement 6 months before instituting a breaking change. The shock and alarm do seem warranted IMO.
  - Twirrim 6y ago
    > project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels It's hardly a notice posted in a sub-basement 6 months before instituting a breaking change. They communicated via numerous mechanisms.
    - klyrs 6y ago
      It's a sub-basement from the perspective of folks for whom this is a second-order dependency. And it sounds like there are many more of them than there are folks that caught wind of this. Regardless. The common practice is a year of emitting warnings from the software, that the end-user will see. That's prominent and allows package maintainers enough time to work around the upcoming breakage. Six months notice on a mailing list is simply not prominent enough, and it's half the standard year which puts undue strain downstream.
ploxiln 6y ago
There was advocacy and pressure to adopt pyca/cryptography in other popular python packages. Don't roll your own crypto, don't use old unmaintained crypto libraries (pycrypto ... but these days there's pycryptodome) etc. If pyca/cryptography was instead advertised as "python crypto library for rust lovers" and other python developers knew that a _gigantic_ and _obnoxious_ rust toolchain dependency was involved, I think many would have avoided cryptography, or made it optional. This was a bait-and-switch. This is obnoxious for people who want to start with a tiny standard distro image with just a C toolchain, and build their app and all dependencies from source. It is also obnoxious for distros that want to build all packages from source. rust depends on llvm, itself a huge complicated finicky package ... but not just that, it depends on a custom-patched llvm! ...but not just that, it also depends on an extremely recent version of rust! this problem is recursive! And there are many other niche cases ... and "niche cases" of "unauthorized" porting/modification/substitution is how much of the open source ecosystem we love got its start. This is really about issues of paranoia and control in the "security" realm. They consider it irresponsible to enable users to do anything they can't guarantee "safe". "Your organization should be paying for a $$$ corporate support contract if you want to do anything we don't officialy support." Maybe you could have fixed a portability bug in some python or C code (I've fixed a couple when bringing up a little-endian mips64 system a decade ago), but no you'll need rust+llvm experts to port that whole crazy toolchain. It's for your own good.
- tptacek 6y ago
  Here you seem to be demanding that the authors and maintainers of their own cryptography library lighten up about security, because you find the way they manage their own package to be controlling. You see some irony in that, right?
  - will4274 6y ago
    I don't read /u/ploxiln as demanding anything. They seem frustrated with the changing dependency profile of the dependency, and a little frustrated with the way some people like to shout "security" as if it's a trump card, ignoring that security is a spectrum and that the only perfectly secure software does nothing (perfectly).
brundolf 6y ago
There's a phenomenon in lots of domains where "a change that is itself neutral or good can be bad if it happens too suddenly". I think that was the case here; we can say that this is a shift that reasonably can or should happen, but can still have caused needless disruption by catching a bunch of people off-guard and not giving them time to adapt to it
zamalek 6y ago
> installing a Python Ansible module It's funny that you mentioned having to install Rust to use some Python because, for us non-Python users, everything needs Python :). I don't use Python at all, but I need both versions installed. Just pointing out that many people are already wearing that shoe. Python is the Lingua Franca of getting shit done. Rust is seemingly becoming the Lingua Franca of [more] secure code.
bonzini 6y ago
Red Hat doesn't care at all about Linux on 32-bit s390, and neither does IBM as far as I know (except possibly the consulting group, which is interested in anything that makes them money). s390x has no problem with switching to Rust/LLVM. Red Hat and IBM both employ engineers working on LLVM, specifically for s390x in IBM's case.

by macksd 6y ago

steveklabnik 6y ago

> Some people making those comments worked for Red Hat / IBM, Ansible is heavily backed by that. Okay, so, I don't mean to pick on you here, but I've seen this sentiment cropping up a few times. Not everyone can be familiar with everything, but I would caution you against immediately assuming corporate politics are at the root here. Anyone familiar with Alex Gaynor and his work over the last few years would know that he cares about Python, and memory safety. That has remained consistent regardless of his employer. Immediately assuming that this has something to do with company politics, rather than just a tireless open source contributor working to improve the things that he cares about, for years, is making a bit of a category error, in my opinion.

macksd 6y ago
I agree - I don't feel picked on :D I guess my point is that it's overly dismissive to say "no free work for companies on niche architectures" when projects backed by the same company were a bit blind-sided by this and a bunch of people lost time chasing down what happened and why. To me that's a sign that this isn't just a mismatch of ideology or someone wanting free work: it was a failure in communication and mismatched expectations. If it had just been on a major version bump, I wonder if we'd even be having this discussion.
- steveklabnik 6y ago
  Glad to hear it :) I think it gets really hard the larger the company you talk about. Before I worked at bigger places, I assumed a lot more coherence than is actually the case.
- rodgerd 6y ago
  Sure, but the message - that if it's a problem for Red Hat's Ansible team or Red Hat's mainframe Linux team, they should be doing the work needed to make it not a problem. Ansible Tower subscriptions aren't exactly cheap, and neither is RHEL s390x. If there's not the fat to be uplifting the core infrastructure needed to run RHEL or Ansible on Red Hat's products, that's most likely a choice.
stonogo 6y ago
I think corporate politics are at the root here, even if they're not Alex Gaynor's corporate politics. This blog post amounts to blessing amd64 and aarch64 as the only sustainable instruction set architectures. Rust is currently the apex of a tall stack of hundreds of millions of lines of code, once you account for LLVM etc. Using it as the basis for other software means only processors with sufficient market penetration are 'worthy' candidates. In the long run, if Rust is as successful as lots of us hope it will be, this will kill what innovation is left in the hardware space. If a company is sufficiently motivated to care, it will almost certainly be cheaper to fork the code back to using C than it would be to forklift LLVM to a new architecture. Is anyone aware of a direct Rust compiler project? Even discounting GCC, Go was bootstrapped from relatively simple (and naive) C compilers until it became self-hosting. I think a basic non-optimizing Rust compiler would go a long way toward leaving the door open for onboarding older -- and more importantly, novel -- architectures into the ecosystem.
- steveklabnik 6y ago
  I didn’t compare HEADs because I’m on my phone, but found reports that in 2019, LLVM was 7 million LOC, and gcc was 15. Both are C++ compiler projects. Why the double standard? I also don’t know why re-writing an entire project and creating a compiler for it is somehow easier than writing an LLVM backend.
  - stonogo 6y ago
    What double standard? I wouldn't recommend a language tie itself to GCC any more than I would recommend LLVM. The point is that the Rust compiler catalog is unhealthily small, and that introduces problems like this one. To your other point, writing an LLVM backend is one thing. Getting it upstreamed is another, and maintaining it is another still. Then you have to navigate the politics of two foundations, both of whose boards of directors are basically the Who's Who of competing interests. I've watched more than one project fail to navigate those waters. Anyway, the cryptography package going from python + c to python + C + C++ + Rust is a cambrian explosion of build time complexity, and in my work we found it simpler to just get rid of the python and the cryptography package, so it's mostly academic to me.

pornel 6y ago

The switch hasn't been sudden. It's just that many levels of disconnect between the project authors and downstream users meant it was basically impossible to communicate to all the affected users — nobody looks at their deps-of-deps until they break. And they've released Rust as optional component you can disable, precisely because nobody paid attention until it actually shipped.

masklinn 6y ago
> The switch hasn't been sudden. It's just that many levels of disconnect between the project authors and downstream users meant it was basically impossible to communicate to all the affected users — nobody looks at their deps-of-deps until they break. Exactly, from the original shitstorm issue: > Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings. > Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels? At one point there's sadly not much the project can do and still make progress.
- its-summertime 6y ago
  > > Do you have constructive suggestions how to communicate changes That one kinda got me, when python intentionally has a runtime developer-to-developer communication system, per say: https://docs.python.org/3/library/warnings.html https://docs.python.org/3/library/warnings.html
  - ploxiln 6y ago
    Warnings are a funny thing, lots of projects like to turn all warnings into failing errors in CI, sometimes even at package build time, they think it's a best-practice ... but it means that nobody else can use warnings to communicate things, or else everything breaks, nullifying the utility of warnings. see also https://lwn.net/Articles/740804/ https://lwn.net/Articles/740804/

laserharvest 6y ago

Why is it necessarily Cryptography’s fault that Python Ansible was taken by surprise? Or any of the other affected parties along the chain? That Cryptography was starting to include Rust in the project was announced on the mailing list by Gaynor last summer. And that email said exactly what future (at that time) release would require a Rust toolchain if the project was to be built from source. The reply to that by the Gentoo guy (who started the Github issue) was that package maintainers cannot follow every single mailing list of every dependency. That is debatable (e.g. maybe you should make an exception for security applications), but let’s take that as a given for now. In that case, what is Cryptography to do? Where should they announce such things in a way that orgs like Gentoo will see it? And also notice it and not just mentally gloss over it as some kind of “spam”? If the Gentoo guy didn’t see it half a year ago, would he have seen an announcement (or the reminders) if it was made five years ago?

klyrs 6y ago
Folks with an eye towards backwards compatibility typically don't implement breaking changes without a year of deprecation warnings emitted by the software. Contrast that to a notice posted in a sub-basement 6 months before instituting a breaking change. The shock and alarm do seem warranted IMO.
- Twirrim 6y ago
  > project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels It's hardly a notice posted in a sub-basement 6 months before instituting a breaking change. They communicated via numerous mechanisms.
  - klyrs 6y ago
    It's a sub-basement from the perspective of folks for whom this is a second-order dependency. And it sounds like there are many more of them than there are folks that caught wind of this. Regardless. The common practice is a year of emitting warnings from the software, that the end-user will see. That's prominent and allows package maintainers enough time to work around the upcoming breakage. Six months notice on a mailing list is simply not prominent enough, and it's half the standard year which puts undue strain downstream.

ploxiln 6y ago

There was advocacy and pressure to adopt pyca/cryptography in other popular python packages. Don't roll your own crypto, don't use old unmaintained crypto libraries (pycrypto ... but these days there's pycryptodome) etc. If pyca/cryptography was instead advertised as "python crypto library for rust lovers" and other python developers knew that a _gigantic_ and _obnoxious_ rust toolchain dependency was involved, I think many would have avoided cryptography, or made it optional. This was a bait-and-switch. This is obnoxious for people who want to start with a tiny standard distro image with just a C toolchain, and build their app and all dependencies from source. It is also obnoxious for distros that want to build all packages from source. rust depends on llvm, itself a huge complicated finicky package ... but not just that, it depends on a custom-patched llvm! ...but not just that, it also depends on an extremely recent version of rust! this problem is recursive! And there are many other niche cases ... and "niche cases" of "unauthorized" porting/modification/substitution is how much of the open source ecosystem we love got its start. This is really about issues of paranoia and control in the "security" realm. They consider it irresponsible to enable users to do anything they can't guarantee "safe". "Your organization should be paying for a $$$ corporate support contract if you want to do anything we don't officialy support." Maybe you could have fixed a portability bug in some python or C code (I've fixed a couple when bringing up a little-endian mips64 system a decade ago), but no you'll need rust+llvm experts to port that whole crazy toolchain. It's for your own good.

tptacek 6y ago
Here you seem to be demanding that the authors and maintainers of their own cryptography library lighten up about security, because you find the way they manage their own package to be controlling. You see some irony in that, right?
- will4274 6y ago
  I don't read /u/ploxiln as demanding anything. They seem frustrated with the changing dependency profile of the dependency, and a little frustrated with the way some people like to shout "security" as if it's a trump card, ignoring that security is a spectrum and that the only perfectly secure software does nothing (perfectly).

brundolf 6y ago

There's a phenomenon in lots of domains where "a change that is itself neutral or good can be bad if it happens too suddenly". I think that was the case here; we can say that this is a shift that reasonably can or should happen, but can still have caused needless disruption by catching a bunch of people off-guard and not giving them time to adapt to it

zamalek 6y ago

> installing a Python Ansible module It's funny that you mentioned having to install Rust to use some Python because, for us non-Python users, everything needs Python :). I don't use Python at all, but I need both versions installed. Just pointing out that many people are already wearing that shoe. Python is the Lingua Franca of getting shit done. Rust is seemingly becoming the Lingua Franca of [more] secure code.

bonzini 6y ago

Red Hat doesn't care at all about Linux on 32-bit s390, and neither does IBM as far as I know (except possibly the consulting group, which is interested in anything that makes them money). s390x has no problem with switching to Rust/LLVM. Red Hat and IBM both employ engineers working on LLVM, specifically for s390x in IBM's case.