7 ms·
The Musl Preprocessor Debate
- rektide 5y agoThere's very few debates where I feel good about upstream moderating downstream's access to capabilities because they're afraid of uses. Postgres notably refuses to implement query hints because people might do wrong => forks like OrioleDB emerge. EcmaScript promises refuse to allow synchronous access because ___ => we end up with endless userland dual sync/async constructs such as AbortSignal. In general, I think we need more people to take inspiration from perl. "Perl, the first post-modern programming langauge"[1] is an old old old write-up, but I think it still holds very true that we generally ought enable. Even when we do set down guidelines, I feel like there should usually be ways to go outside the boundaries. To me, it's a huge feature & logical that there be capabilities in Java to peek into the private fields of a class- maybe not all code should have access to this, but this kind of deep access to a system is still, to me, highly desireable. It makes me sad to see languages like JS, once fore-runners in first-class everything, dynamic programming, accrue the trappings of a constrained, restricted, denying language: private fields (with no workarounds), annotations/decorators but load-time compiled away/invisible. Systems that expose what they are & let us monkey around are just better, cooler, have more potential. Imagining nice pleasant happy well groomed langauges & systems is, to me, the ultimate premature optimziation: favoring some internal idea of a plan & binding everyone to it is a horrible mistake, one that often prevents criticism by simply denying the other party's their option, the chance to be explored. [1] http://www.wall.org/~larry/pm.html http://www.wall.org/~larry/pm.html
- password4321 5y agoInteresting re:OrioleDB, but the project seems to be targeting a number of additional major features vs. just query hints which would "in theory" seem to be do-able with relatively minor changes (eg. mostly parser tweaks). The project might win a few more stars if they mentioned query hints in the readme!
- viraptor 5y agoThere's also an extension if you want just the hints: https://pghintplan.osdn.jp/pg_hint_plan.html https://pghintplan.osdn.jp/pg_hint_plan.html
- AstralStorm 5y agoOld adage is "Make right thing simple, make complex thing possible." Thing is, we're taking about C library here, a piece of code that is potentially being built for a system that is not running yet and has no emulator. The complex case is one of the defaults they must consider, but they made it impossible. On the upside, that makes a strong case to just not use or support Musl. Glibc and uclibc exist.
- pxeger1 5y agoJust as a counterexample, I'd like to mention Python. I think Python is very well known for matching your description: > Systems that expose what they are & let us monkey around are just better, cooler, have more potential. But Python really suffers performance-wise because of this. There are a lot of optimisations that could be applied to the interpreter if it weren't for the fact that <edge case> is allowed in Python.
- shp0ngle 5y agoI was writing in perl in the 2000s and early 2010s. I don’t want other languages to be more like perl.
- remexre 5y ago> On my POWER9 Talos workstation, typical ./configure runs take longer than the builds themselves. This is because fork+exec is still a slow path on POWER. It is similar on ARM, MIPS, and many other RISC architectures. Anyone know how this (the RISC-related slowness) can be true? Larger register file? Even that seems unlikely...
- userbinator 5y agoPage table and TLB handling comes to mind, although I'm not sure why ARM was included there since it does TLB automatically in hardware.
- alexjurkiewicz 5y agoThe result of musl's attitude is build systems assuming "__linux__ && !__glibc__ means musl". I pity the compatibility nightmares any new libc pushing for mass adoption will have to deal with. Musl's approach adds cost to everybody but themselves.
- hvdijk 5y agoDo you have examples of this? What __linux__ && !__glibc__ should mean, in my opinion, Linux but without glibc extensions. This does not imply musl, but does happen to also be what musl aims for, so there is going to be users where either would look the same.
- dahfizz 5y agoThere are more libc's out there than just glibc and musl. The `backtrace` call is a great example. It is available in glibc and uClibc, but not musl. Instead of checking if using musl, and removing the backtrace call, the lack a musl define forces you to check for all the other possible libc's and assume musl is the default. This is dumb.
- hvdijk 5y agoThat is probably a bad example. That is a GNU extension, so if handling this via the preprocessor, the right guard should just be #ifdef __GLIBC__, I think. That should cover both glibc and glibc-compatible libcs such as uClibc. (Yes, uClibc does define __GLIBC__ as well.)
- saagarjha 5y agoThe problem with library authors in general is that they, on average, have a better perspective how client programs should be written. Writing a good library is generally far harder than it is to use a good library, and musl is pretty decent so this remains true: most users of musl have far less technical skill and a more myopic view than Rich Felker does. But, as is true of libraries, and programming languages, and life in general, there is always someone smarter than you are, who genuinely needs to do the thing you say people should generally not want to do. This is why taking a moral stance of "not" allowing the thing is almost always the wrong approach: what you want to do is make the non-recommended path unattractive to a casual newcomer, or have sufficiently good education and guardrails in place that people don't stumble upon it, but there has to be a way to do the non-recommended thing. If you don't, the person who really needs it will find a way to do it anyways, and probably curse your name the entire time. Anyways, bringing this back to musl, I think it takes an agreeable stance that it follows POSIX (plus some stuff that it couldn't really leave out, because it breaks too many programs). Cool, it does a good job at that. But sometimes people, who are aware of the user-agent problem that musl is trying to avoid, would just like to make a special codepath that lets them use some more optimized, nonstandard thing that musl doesn't support. If you need your static initializers to get argv, then there isn't really much you can do but provide a macro for detecting musl. So provide that macro, name it THIS_IS_MUSL_BUT_MAKE_SURE_YOU_KNOW_WHAT_YOU_ARE_DOING or whatever, but make it available for things like these. Sometimes practicality has to win out over idealism, and musl contains a bunch of those tradeoffs already: this should be another one of them.
- kitd 5y agoAgreed. Plus: Sometimes practicality has to win out over idealism We're engineering software, not writing constitutions. Practicality always comes first IMHO.
- einpoklum 5y ago> Practicality always comes first ... famous first words of many people who engineer yet another piece of software which later needs to be rewritten, since they / their company could not be bothered to write a decent library. :-(
- AstralStorm 5y agoThis is extra funny coming from musl which is expected to be used in cross compiling, embedded scenarios. If you want feature tests, add more versioned defines not fewer in this case. You can actually version each public API if you want. Plus, inevitably you will hit a case that cannot be detected using reasonable tests (e.g. non-flaky).
- SamReidHughes 5y agoI just encountered this issue with Musl today, trying to make some software that uses backtrace() and backtrace_symbols() work with it. I wound up making the use #if !defined(NO_BACKTRACE) in the code and passing CXXFLAGS=-DNO_BACKTRACE in the build script for Alpine Linux. (The symbol name is slightly genericized.) In my case, whether or not Musl offered a macro, NO_BACKTRACE is a cleaner, more proper solution than adding to the #ifndef _WIN32 #elif defined(__MACH__) /* or __APPLE__? */ #elif defined(__glibc__) chain.
- MereInterest 5y ago> It’s a bug to assume a certain implementation has particular properties rather than testing. Adding further weight to the article, I think this quote from Rich Felker is incredibly language-dependent, and categorically incorrect for languages with undefined behavior, such as C and C++. It's a bug to assume that properties of an implementation can be determined solely through testing. For example, suppose you want to determine what happens when signed integers overflow. You can write a test program that checks the behavior of signed integer overflow in a variety of cases. That only shows that signed integer overflow has the property you want in the cases you've checked for. Even if you were to somehow check every single result of "a + b" for all 2^128 possible combinations of (a,b), that still would only tell you that it works in that particular program, at that particular execution time. It could be that you just haven't hit the particular set of register usage or of allowed optimizations to run into an edge case. There is no possible test that can distinguish between an implementation that guarantees to have a particular property, and an implementation whose undefined behavior happens to match the property in the cases that have been tested. The only way to know is to have some additional piece of information, some additional communication channel to establish intent, rather than inferring intent from the observable behavior. In C, this is typically done with a preprocessor macro.
- kevincox 5y agoA middle ground could be more specific feature macros. For example instead of checking for __MUSL__ and assuming that it supports some function check __MUSL_SOME_FUNCTION or similar. This is basically what your configure script ends up doing (likely poorly) anyways.
- skissane 5y agoDo any C or C++ compilers behave like that in practice – have signed integer overflow which doesn't simply depend on data types and compilation options, but varies more dynamically, depending on the values, runtime execution mode, register allocations, optimisations, etc? In practice, the space of actual undefined behaviours is a lot smaller than the space of legally permitted undefined behaviours – it may be legal per the standard for a compiler in a case of undefined behaviour to make demons fly out of your nose, but I don't think any compiler has ever actually done that, and it would be a mighty strange occurrence if any ever did. Less jocularly, even if some particular type of undefined behaviour is technically permitted by the standard, if all major compilers don't do it, it can be a reasonable engineering decision to assume none ever will, and deal with that unlikely risk if and when it actually occurs.
- humanrebar 5y ago> If header-only libraries start requiring you to use build-time tests, you lose the main reason to use them in the first place. Header-only libraries don't really work as advertised anyway. The need for various parts of a codebase (including copy-pasted third party headers) to coordinate on preprocessor settings is a good example of this. Header only is great for casual research but pretty much immediately require extra work on the part of consumers. The C community would do well to acknowledge that build and packaging system convergence is overdue. Messing around with header-only and preprocessor tricks is leaving the community in an awkward cul-de-sac.
- acuozzo 5y ago> The C community would do well to acknowledge that build and packaging system convergence is overdue. This would require an enormous amount of planning and forward thinking to pull off successfully. I mean, hell, there isn't even a way to trust most libraries. So many of them are littered with type punning that e.g. the moment you move them off of x86 you're greeted with bus errors in stateless functions. Undefined behavior everywhere. And then you have the "Oh, sizeof (long) will always be the same as sizeof (void *) and I can definitely do this cast!" libraries from 1999. Those are fun. Even better are the "CHAR_BIT==8 everywhere in the world!" libraries that somehow forget to #error when that's not the case, so you have to manually review each and every assumption in each dependency before building for that fancy DSP. It's a mess. Even with the recent popularity of ARM, many C programmers either forget platforms other than GNU/Linux on x86 exist or forget to document what they're targeting.
- mlyle 5y ago> And then you have the "Oh, sizeof (long) will always be the same as sizeof (void *) and I can definitely do this cast!" libraries from 1999. Those are fun. Just out of curiosity-- where does this break down in common use? I know of some microcontrollers where e.g. long is 32 and pointers are 16... but most things really are ILP32 or LP64. P.S. Long, long ago you wrote you found Mel Kaye, and I'm always curious about the actual story.
- mc4ndr3 5y agoWhy is bash getting easily confused about POSIX support? Aren't there POSIX macros?
- m3047 5y agoThere's a thread with Felker on the postfix-users mailing list whingeing that water is wet and Wietse or Viktor threatening to do conditional compilation if MUSL is present from April 2020, that sounds sort of like this: Wietse: > Congratulations! You just gave a new definition of security theatre: > using an unauthenticated channel to distribute trust anchors. Felker: It's not security theater because nobody's claiming it's secure. Rather it's a fairly weak form of hardening that increases the required capabilities an attacker needs to exploit a known-insecure system. It's brain rot caused by too much systemd!
- m3047 5y agoIt occurs to me that someone may want to read the background material. https://marc.info/?l=musl&m=158680293802966&w=2 https://marc.info/?l=musl&m=158680293802966&w=2 I can haz DNS! https://marc.info/?l=postfix-users&m=158698506811448&w=2 https://marc.info/?l=postfix-users&m=158698506811448&w=2 "toy" stub resolver https://marc.info/?l=postfix-users&m=158986711614688&w=2 https://marc.info/?l=postfix-users&m=158986711614688&w=2 I'm really disappointed that you're detecting MUSL... https://marc.info/?l=postfix-users&m=158716200010400&w=2 https://marc.info/?l=postfix-users&m=158716200010400&w=2 Harmful to limit the use of (broken) DANE lookups
- m3047 5y agoThanks for the downvotes. This guy literally advocates that we don't follow "secure", so fuck everything over.
- m3047 5y agoWhat started the original thread was "postfix" "failing" security tests because it was installed in a Docker image running Alpine. Insecure! Why? OMG all these people, all this effort. Felker?