8 ms·

Clang emits memcpy for std::swap, which can introduce undefined behavior

110 points by luu 11y ago

boulos 11y ago
IIRC, there are passes that also detect memcpy-"equivalent" loops and replace them with the builtin to memcpy (mostly to get the vectorization benefit). And of course there's SimplifyLibCalls (or whatever it's named) that undoes this or any other "you called a standard library function we know how to inline in this case".
- majewsky 11y ago
  The second one sounds like big fun when you LD_PRELOAD a different memcpy (say, for debugging purposes) and spend hours and hours trying to figure out why it is not being called.
  - boulos 11y ago
    Yes, it's a nightmare, particularly on OS X with its funny loader semantics that make replacing malloc and free correctly very painful. For example, you can easily LD_PRELOAD your way out of malloc() but did you remember that asprintf also calls an allocation routine and won't use your new malloc? Enjoy.
    - ktRolster 11y ago
      For debugging purposes, it can sometimes be easier to include a header file "mem_debug.h" or whatever, and redefine malloc: >#define malloc(a) debug_malloc(a, __FILE__, __LINE__) >#define free(a) debug_free(a, __FILE__, __LINE__) >. >. >. Then in your debug_malloc() function you can track where every allocation took place. Not always applicable but nice when it works.
    - comex 11y ago
      Hmm, why won't it? Maybe in the past when asprintf and malloc were located in the same dylib, since OS X dylibs generate direct calls to other public functions in the same image (rather than going through the PLT as GNU does by default), but for many years /usr/lib/libSystem.B.dylib has been split into several reexported libraries in /usr/lib/system, so there shouldn't be an issue there. Probably better to use this technique though (should still work in the latest version but needs mprotect): http://eatmyrandom.blogspot.com/2010/03/mallocfree-interception-on-mac-os-x.html http://eatmyrandom.blogspot.com/2010/03/mallocfree-intercept...
    - astrange 11y ago
      malloc is set up to be specifically overridable so that guardmalloc(3)* will work. Are you saying asprintf doesn't work with guardmalloc? * check it out, I used a man reference, am I a professional UNIX developer yet?
  - achamayou 11y ago
    GCC does the exact same thing, and it can be turned off with -fno-builtin.
hendzen 11y ago
Specifically in the case of a self-swap, which is a somewhat odd thing to do in the first place.
- boulos 11y ago
  Before shrink to fit, you could easily have produced this with a few references and a "use std::swap unconditionally". The std::swap thing is only one pitfall, the reliance on "our memcpy will do the right thing with any overlap" is a bit risky (and doesn't hold with LD_PRELOAD, but I think it may have been switched to using the LLVM based librt project anyway, that is I'm not sure these calls are supposed to become naked memcpy).
- AnthonyMouse 11y ago
  It's quite common actually. Usually in the context of something like "swap element 'n' with the last element in this vector" without any check to see if 'n' is already the last element, because checking for that condition just to avoid a harmless no-op in the uncommon case is ugly and slower in the common case.
- dkopi 11y ago
  Imagine reversing a string: while(pStart <= pEnd) swap(*pStart++, *pEnd--); If swap doesn't work for a self swap, this code would break in the case of an odd length string. It's an easy fix in this case, but being able to swap an element with itself is convenient.
  - Matheus28 11y ago
    A more common case, would be swapping a vector element with the back of the vector and popping it off, which is kinda common.
    - plorkyeran 11y ago
      It was common pre-C++11, but these days you should just be doing a move and pop rather than a swap and pop.
steplee 11y ago
Aside: am i crazy for pronouncing clang see-lang in my head?
- sgift 11y ago
  How else should you pronounce it? (i.e. I do this too, never thought about it before)
  - masklinn 11y ago
    > How else should you pronounce it? The official pronunciation is "klang" with a hard C: http://lists.llvm.org/pipermail/llvm-dev/2008-July/015629.html http://lists.llvm.org/pipermail/llvm-dev/2008-July/015629.ht... Exactly like the metallic "clang" http://www.thefreedictionary.com/clang http://www.thefreedictionary.com/clang which originates from the latin clangere
- boulos 11y ago
  No, you're not crazy. Some people definitely pronounce it C-lang, while others produce a banging sound (klang). I prefer the banging sound ;)
  - masklinn 11y ago
    The banging sound is also the one used by the developer team: http://lists.llvm.org/pipermail/llvm-dev/2008-July/015629.html http://lists.llvm.org/pipermail/llvm-dev/2008-July/015629.ht...
  - jahnu 11y ago
    Klang is also the German for "Sound"
- Jerry2 11y ago
  I've attended 2014 LLVM Developers' Meeting. Everyone there pronounced it as "klang" so that's how I pronounce it as well.
haberman 11y ago
Given how aggressively compilers (including Clang) are exploiting undefined behavior these days, it is surprising to see Clang developers (I presume) on the bug being so blasé about introducing undefined behavior themselves.
- electrograv 11y ago
  > it is surprising to see Clang developers (I presume) on the bug being so blasé about introducing undefined behavior themselves. I have suddenly lost a great deal of respect and confidence for LLVM/Clang after reading this bug thread; I value software reliability above all else, and being able to write reliable software without the compiler negating that effort. It seems I can no longer be assured this is even a priority of my compiler any more, with Clang. All the more reason to switch to Rust I suppose. That the Clang people are trying to deflect this bug as "fine, no fix necessary" is kind of disgusting software engineering behavior. Compilers spontaneously introducing undefined behavior in another layer (to the point of causing codegen warnings from perfectly correct C code, is as clearly cut a BUG as any, independent of current API implementations outside of the ISO spec. If you want to make undefined API calls the norm, then make a new API where the behavior you want is defined. Invoking undefined behavior of a decoupled library that users can swap out is flat out indefensible. That they are reluctant to fix this makes me shy away from ever using Clang/LLVM, which is sad because before I had always held them with very high respect.
  - chandlerc1024 11y ago
    Understand that some of the folks who don't think this is a bug own one of the libc implementations being used. So for their platform, it really is 100% correct and valid. The folks working on other platforms seem universally unhappy with that bug.
    - electrograv 11y ago
      If it's not 100% valid on 100% platforms and 100% of all future Libc revisions (which is impossible if it relies on undefined behavior and can link with different Libc versions), then it's not 100% correct and valid.
nikic 11y ago
The bug report contains a quote from the C++ standard: "An rvalue or lvalue t is swappable if and only if t is swappable with any rvalue or lvalue, respectively, of type T" What does this statement mean and why is it not tautological?
- wfunction 11y ago
  It defines swappability of a single value in terms of swappability of a pair of values.
- yelnatz 11y ago
  If you extract it for only one case, it's easier to read. >An rvalue t is swappable if and only if t is swappable with any rvalue of type T
raverbashing 11y ago
I wonder if it's possible to identify that both objects are the same (or might just be easier to use memmove instead)
mehrdada 11y ago
The compiler causing undefined behavior on its own reminds me of one of the fun bugs we found[1] with 32-bit x86 clang where one compiler pass fools another into thinking there's an undefined behavior by combining three struct field initializations into a single 4-byte load, but the struct itself was 3-bytes long. Another pass looked at this and thought there's an undefined behavior writing past the struct boundary, silently eliminating the initialization altogether, resulting in bad codegen. [1]: https://mehrdad.afshari.me/pubs/compiler-validation-via-equivalence-modulo-inputs.pdf https://mehrdad.afshari.me/pubs/compiler-validation-via-equi...
- slededit 11y ago
  Compilers are way too overzealous eliminating code because of undefined behavior. Legally it is allowed and I'm sure it helps in synthetic benchmarks. But writers of real programs won't be impressed of the perf gains when you stop running half their program.
  - kazinator 11y ago
    Rather, they are getting zealous in eliminating code because of the assumption that there is no undefined behavior. The scope of what undefined behaviors do this is increasing and that breaks programs that make some assumptions on traditional compiler behavior, or that have hidden bugs. I believe that C should have safer semantics, plus standard-defined modes of operation which restore the unsafe semantics selectively. The optimizer then must obey whichever semantics in effect (safer or less safe). Then, in addition to telling your compiler you want a certain optimization level, you could tell it which standard-defined modes of operation you would like. Currently, this is already de facto the case outside of the standard. For instance, users of GCC who want to do certain pointer aliasing without unpredictable behavior tell the compiler to be in a mode in which that is allowed (effectively an altered C dialect) using -fno-strict-aliasing. This kind of thing should be a standard, portable feature. Look, everywhere in this program, I want nice left-to-right evaluation within all expressions and among initializers and function arguments. Except in super-fast.c; when compiling super-fast.c to super-fast.o, please do it in a mode where unspecified-evaluation-order semantics applies and optimize accordingly. (I might even want this on a more finer-grained scale than translation units.) We need safety, and we need to sometimes throw away safe semantics in exchange for better speed (whereby we still ensure that the code we are writing is correct with regard to the weakened semantics; i.e. we promise to the compiler that even if it relaxes the evaluation order, or whatever, we know what we are doing and everything is cool). It would be good to do this without compiler-specific guesswork. Optimization control alone over a language with a single set of semantics (which is largely unsafe) is a bad way to achieve this.
ben0x539 11y ago
You don't even need swap. t = t also emits a memcpy.
kazinator 11y ago
The compiler is the implementation; it cannot cause "undefined behavior". What it can be is "nonconforming". A correct program which hits that situation continues to have well-defined behavior (which we know from the text of the program and the standard). Just the implementation isn't handling the requirements correctly; it is not conforming. An implementation is not bound by the standard in its own use of the standard library functions. The implementation can use a function like memcpy in ways that would be formally undefined if they occured in user code; its implementation of memcpy just has to harmonize with that use, so that the intended behavior is ensured. This is because implementations can add their own requirements to areas that the standard leaves undefined. For instance, an implementation can add the requirement to its memcpy that the copy may overlap, if the destination has a lower address. Then, the implementation can generate code which uses memcpy that way, or make internal uses of memcpy from other library functions which use it that way. It's just using its own (possibly not publicly documented) extension.
- Dylan16807 11y ago
  The compiler has different layers. Layer X is nonconforming, which causes undefined behavior in layer M. Layer M's memcpy doesn't have to have the same semantics as the C standard's memcpy, but the bug report implies that it does.
  - kazinator 11y ago
    Yes, if "undefined behavior" doesn't just mean "a situation where the ISO language standard doesn't impose a requirement", but has a broader meaning like "behavior not defined by anyone at all".
    - Dylan16807 11y ago
      The compiler is using the ISO language standard to define the behavior of an internal piece. It is then using that piece in the wrong way. It's not generic "not defined" behavior, it is specific C "undefined behavior". Think of that part of the transformed code as still C for relevant purposes.
SFJulie 11y ago
Reading most thread of llvm bugs so far about there "undefined behaviours" I see a pattern. Llvm goal is performance over correction. stop. If old code breaks, it is you played with fire, correct your code to eliminate these undefined behaviours. Stop. We made wonderful coloured warning to help you. Full Stop. Well on one hand I get the point, people played with fire. Code should be "correct" by standards that are emerging, but I can predict one thing : before the new "correctness pattern" that enables lots of awesome robust high performing code spreads, there will be chaos. My first self interrogation is in how much time llvm team can build a completly correct stable compiler without any surprises? It it takes 2 decades, it will be a lot of chaos involved... My second interrogation is can the code that "is incorrect" per LLVM standard be obtained without rewriting the source code (I guess not), and how much will it costs before all the code is being "corrected". Do we even have the man power to do it? You see, some C code may be used everyday but are so complex hardly any developers even well paid like to look at them. (openssl, gnutls, libpng, zlib, dhcpd, bind...). And with platform specific ASM, we may discover that code has lost portability, and it may take a lot of time to discover that it is already happening. Don't we risk another strike of bad news one day like discovering old software/libraries are behaving differently and need to be rewritten and that no one want to do the same effort as libressl especially since some software have a LOT of dependencies on C code (I do not think to loud of PHP or python, or Perl). Of course there are unit test, but there are so much of them, some tests (like for python on freeBSD with curses) have been disabled, because "no one has the time to fix them". It is ofc a bad example in this precise case, but in the context, it feels like a growing ghettoisation between the CPU/OS/langage with lots of eyeballs/investments, and the old/poor market. Especially non company backed open source software. A smell of obsolescence by attrition of resource to report bugs/rewrite code. As a normal former experimental physicist I can feel a transition coming, it smells of chaos and instability, and the question is always, how long, how smooth before the new equilibrium if ever we reach it, and what will break in the process? And well since transitions are transitions, we may reach a new equilibrium, but nothing guarantees it will be reversible, hence some stuff may disappear in the process forever. I don't know why, but I guess the most vulnerable ecosystem to the changes to come is the non commercial open source.
- lmm 11y ago

110 points by luu 11y ago

IIRC, there are passes that also detect memcpy-"equivalent" loops and replace them with the builtin to memcpy (mostly to get the vectorization benefit). And of course there's SimplifyLibCalls (or whatever it's named) that undoes this or any other "you called a standard library function we know how to inline in this case".

majewsky 11y ago
The second one sounds like big fun when you LD_PRELOAD a different memcpy (say, for debugging purposes) and spend hours and hours trying to figure out why it is not being called.
- boulos 11y ago
  Yes, it's a nightmare, particularly on OS X with its funny loader semantics that make replacing malloc and free correctly very painful. For example, you can easily LD_PRELOAD your way out of malloc() but did you remember that asprintf also calls an allocation routine and won't use your new malloc? Enjoy.
  - ktRolster 11y ago
    For debugging purposes, it can sometimes be easier to include a header file "mem_debug.h" or whatever, and redefine malloc: >#define malloc(a) debug_malloc(a, __FILE__, __LINE__) >#define free(a) debug_free(a, __FILE__, __LINE__) >. >. >. Then in your debug_malloc() function you can track where every allocation took place. Not always applicable but nice when it works.
  - comex 11y ago
    Hmm, why won't it? Maybe in the past when asprintf and malloc were located in the same dylib, since OS X dylibs generate direct calls to other public functions in the same image (rather than going through the PLT as GNU does by default), but for many years /usr/lib/libSystem.B.dylib has been split into several reexported libraries in /usr/lib/system, so there shouldn't be an issue there. Probably better to use this technique though (should still work in the latest version but needs mprotect): http://eatmyrandom.blogspot.com/2010/03/mallocfree-interception-on-mac-os-x.html http://eatmyrandom.blogspot.com/2010/03/mallocfree-intercept...
  - astrange 11y ago
    malloc is set up to be specifically overridable so that guardmalloc(3)* will work. Are you saying asprintf doesn't work with guardmalloc? * check it out, I used a man reference, am I a professional UNIX developer yet?
- achamayou 11y ago
  GCC does the exact same thing, and it can be turned off with -fno-builtin.

hendzen 11y ago

Specifically in the case of a self-swap, which is a somewhat odd thing to do in the first place.

boulos 11y ago
Before shrink to fit, you could easily have produced this with a few references and a "use std::swap unconditionally". The std::swap thing is only one pitfall, the reliance on "our memcpy will do the right thing with any overlap" is a bit risky (and doesn't hold with LD_PRELOAD, but I think it may have been switched to using the LLVM based librt project anyway, that is I'm not sure these calls are supposed to become naked memcpy).
AnthonyMouse 11y ago
It's quite common actually. Usually in the context of something like "swap element 'n' with the last element in this vector" without any check to see if 'n' is already the last element, because checking for that condition just to avoid a harmless no-op in the uncommon case is ugly and slower in the common case.
dkopi 11y ago
Imagine reversing a string: while(pStart <= pEnd) swap(*pStart++, *pEnd--); If swap doesn't work for a self swap, this code would break in the case of an odd length string. It's an easy fix in this case, but being able to swap an element with itself is convenient.
- Matheus28 11y ago
  A more common case, would be swapping a vector element with the back of the vector and popping it off, which is kinda common.
  - plorkyeran 11y ago
    It was common pre-C++11, but these days you should just be doing a move and pop rather than a swap and pop.

steplee 11y ago

Aside: am i crazy for pronouncing clang see-lang in my head?

sgift 11y ago
How else should you pronounce it? (i.e. I do this too, never thought about it before)
- masklinn 11y ago
  > How else should you pronounce it? The official pronunciation is "klang" with a hard C: http://lists.llvm.org/pipermail/llvm-dev/2008-July/015629.html http://lists.llvm.org/pipermail/llvm-dev/2008-July/015629.ht... Exactly like the metallic "clang" http://www.thefreedictionary.com/clang http://www.thefreedictionary.com/clang which originates from the latin clangere
boulos 11y ago
No, you're not crazy. Some people definitely pronounce it C-lang, while others produce a banging sound (klang). I prefer the banging sound ;)
- masklinn 11y ago
  The banging sound is also the one used by the developer team: http://lists.llvm.org/pipermail/llvm-dev/2008-July/015629.html http://lists.llvm.org/pipermail/llvm-dev/2008-July/015629.ht...
- jahnu 11y ago
  Klang is also the German for "Sound"
Jerry2 11y ago
I've attended 2014 LLVM Developers' Meeting. Everyone there pronounced it as "klang" so that's how I pronounce it as well.

haberman 11y ago

Given how aggressively compilers (including Clang) are exploiting undefined behavior these days, it is surprising to see Clang developers (I presume) on the bug being so blasé about introducing undefined behavior themselves.

electrograv 11y ago
> it is surprising to see Clang developers (I presume) on the bug being so blasé about introducing undefined behavior themselves. I have suddenly lost a great deal of respect and confidence for LLVM/Clang after reading this bug thread; I value software reliability above all else, and being able to write reliable software without the compiler negating that effort. It seems I can no longer be assured this is even a priority of my compiler any more, with Clang. All the more reason to switch to Rust I suppose. That the Clang people are trying to deflect this bug as "fine, no fix necessary" is kind of disgusting software engineering behavior. Compilers spontaneously introducing undefined behavior in another layer (to the point of causing codegen warnings from perfectly correct C code, is as clearly cut a BUG as any, independent of current API implementations outside of the ISO spec. If you want to make undefined API calls the norm, then make a new API where the behavior you want is defined. Invoking undefined behavior of a decoupled library that users can swap out is flat out indefensible. That they are reluctant to fix this makes me shy away from ever using Clang/LLVM, which is sad because before I had always held them with very high respect.
- chandlerc1024 11y ago
  Understand that some of the folks who don't think this is a bug own one of the libc implementations being used. So for their platform, it really is 100% correct and valid. The folks working on other platforms seem universally unhappy with that bug.
  - electrograv 11y ago
    If it's not 100% valid on 100% platforms and 100% of all future Libc revisions (which is impossible if it relies on undefined behavior and can link with different Libc versions), then it's not 100% correct and valid.

nikic 11y ago

The bug report contains a quote from the C++ standard: "An rvalue or lvalue t is swappable if and only if t is swappable with any rvalue or lvalue, respectively, of type T" What does this statement mean and why is it not tautological?

wfunction 11y ago
It defines swappability of a single value in terms of swappability of a pair of values.
yelnatz 11y ago
If you extract it for only one case, it's easier to read. >An rvalue t is swappable if and only if t is swappable with any rvalue of type T

raverbashing 11y ago

I wonder if it's possible to identify that both objects are the same (or might just be easier to use memmove instead)

mehrdada 11y ago

The compiler causing undefined behavior on its own reminds me of one of the fun bugs we found[1] with 32-bit x86 clang where one compiler pass fools another into thinking there's an undefined behavior by combining three struct field initializations into a single 4-byte load, but the struct itself was 3-bytes long. Another pass looked at this and thought there's an undefined behavior writing past the struct boundary, silently eliminating the initialization altogether, resulting in bad codegen. [1]: https://mehrdad.afshari.me/pubs/compiler-validation-via-equivalence-modulo-inputs.pdf https://mehrdad.afshari.me/pubs/compiler-validation-via-equi...

slededit 11y ago
Compilers are way too overzealous eliminating code because of undefined behavior. Legally it is allowed and I'm sure it helps in synthetic benchmarks. But writers of real programs won't be impressed of the perf gains when you stop running half their program.
- kazinator 11y ago
  Rather, they are getting zealous in eliminating code because of the assumption that there is no undefined behavior. The scope of what undefined behaviors do this is increasing and that breaks programs that make some assumptions on traditional compiler behavior, or that have hidden bugs. I believe that C should have safer semantics, plus standard-defined modes of operation which restore the unsafe semantics selectively. The optimizer then must obey whichever semantics in effect (safer or less safe). Then, in addition to telling your compiler you want a certain optimization level, you could tell it which standard-defined modes of operation you would like. Currently, this is already de facto the case outside of the standard. For instance, users of GCC who want to do certain pointer aliasing without unpredictable behavior tell the compiler to be in a mode in which that is allowed (effectively an altered C dialect) using -fno-strict-aliasing. This kind of thing should be a standard, portable feature. Look, everywhere in this program, I want nice left-to-right evaluation within all expressions and among initializers and function arguments. Except in super-fast.c; when compiling super-fast.c to super-fast.o, please do it in a mode where unspecified-evaluation-order semantics applies and optimize accordingly. (I might even want this on a more finer-grained scale than translation units.) We need safety, and we need to sometimes throw away safe semantics in exchange for better speed (whereby we still ensure that the code we are writing is correct with regard to the weakened semantics; i.e. we promise to the compiler that even if it relaxes the evaluation order, or whatever, we know what we are doing and everything is cool). It would be good to do this without compiler-specific guesswork. Optimization control alone over a language with a single set of semantics (which is largely unsafe) is a bad way to achieve this.

ben0x539 11y ago

You don't even need swap. t = t also emits a memcpy.

kazinator 11y ago

The compiler is the implementation; it cannot cause "undefined behavior". What it can be is "nonconforming". A correct program which hits that situation continues to have well-defined behavior (which we know from the text of the program and the standard). Just the implementation isn't handling the requirements correctly; it is not conforming. An implementation is not bound by the standard in its own use of the standard library functions. The implementation can use a function like memcpy in ways that would be formally undefined if they occured in user code; its implementation of memcpy just has to harmonize with that use, so that the intended behavior is ensured. This is because implementations can add their own requirements to areas that the standard leaves undefined. For instance, an implementation can add the requirement to its memcpy that the copy may overlap, if the destination has a lower address. Then, the implementation can generate code which uses memcpy that way, or make internal uses of memcpy from other library functions which use it that way. It's just using its own (possibly not publicly documented) extension.

Dylan16807 11y ago
The compiler has different layers. Layer X is nonconforming, which causes undefined behavior in layer M. Layer M's memcpy doesn't have to have the same semantics as the C standard's memcpy, but the bug report implies that it does.
- kazinator 11y ago
  Yes, if "undefined behavior" doesn't just mean "a situation where the ISO language standard doesn't impose a requirement", but has a broader meaning like "behavior not defined by anyone at all".
  - Dylan16807 11y ago
    The compiler is using the ISO language standard to define the behavior of an internal piece. It is then using that piece in the wrong way. It's not generic "not defined" behavior, it is specific C "undefined behavior". Think of that part of the transformed code as still C for relevant purposes.

SFJulie 11y ago

Reading most thread of llvm bugs so far about there "undefined behaviours" I see a pattern. Llvm goal is performance over correction. stop. If old code breaks, it is you played with fire, correct your code to eliminate these undefined behaviours. Stop. We made wonderful coloured warning to help you. Full Stop. Well on one hand I get the point, people played with fire. Code should be "correct" by standards that are emerging, but I can predict one thing : before the new "correctness pattern" that enables lots of awesome robust high performing code spreads, there will be chaos. My first self interrogation is in how much time llvm team can build a completly correct stable compiler without any surprises? It it takes 2 decades, it will be a lot of chaos involved... My second interrogation is can the code that "is incorrect" per LLVM standard be obtained without rewriting the source code (I guess not), and how much will it costs before all the code is being "corrected". Do we even have the man power to do it? You see, some C code may be used everyday but are so complex hardly any developers even well paid like to look at them. (openssl, gnutls, libpng, zlib, dhcpd, bind...). And with platform specific ASM, we may discover that code has lost portability, and it may take a lot of time to discover that it is already happening. Don't we risk another strike of bad news one day like discovering old software/libraries are behaving differently and need to be rewritten and that no one want to do the same effort as libressl especially since some software have a LOT of dependencies on C code (I do not think to loud of PHP or python, or Perl). Of course there are unit test, but there are so much of them, some tests (like for python on freeBSD with curses) have been disabled, because "no one has the time to fix them". It is ofc a bad example in this precise case, but in the context, it feels like a growing ghettoisation between the CPU/OS/langage with lots of eyeballs/investments, and the old/poor market. Especially non company backed open source software. A smell of obsolescence by attrition of resource to report bugs/rewrite code. As a normal former experimental physicist I can feel a transition coming, it smells of chaos and instability, and the question is always, how long, how smooth before the new equilibrium if ever we reach it, and what will break in the process? And well since transitions are transitions, we may reach a new equilibrium, but nothing guarantees it will be reversible, hence some stuff may disappear in the process forever. I don't know why, but I guess the most vulnerable ecosystem to the changes to come is the non commercial open source.

lmm 11y ago