7 ms·

My job involves a lot of packaging/cross-compilation, and djb's libraries always seem consistently hostile to the lowly packaging engineer. Would it really be

• geocar 8y ago
  > Why do I need ... assumptions about system paths? > what is the rationale for a build flow that involves: It solves problems. https://cr.yp.to/compatibility.html https://cr.yp.to/compatibility.html https://cr.yp.to/slashpackage/studies.html https://cr.yp.to/slashpackage/studies.html https://cr.yp.to/slashpackage/finding.html https://cr.yp.to/slashpackage/finding.html https://cr.yp.to/slashpackage/sharability.html https://cr.yp.to/slashpackage/sharability.html > Would it really be that much harder to give us a git repo and a ./configure or a CMakeLists.txt? Yes.
  • nrclark 8y ago
    FWIW, I read through each of your links. They don't address cross-compilation at all, or the needs of software packagers. I can 100% promise you that somebody packaging this library for any Linux distro (or for a Yocto/Buildroot system) would grind their teeth in frustration at everything in the those links. The solution to having inconsistent packaging paths isn't to introduce _yet another_ packaging path system, but this one specific to djb stuff. It's to use a standard build system with overrideable paths, and not to assume the author knows better than the packager.
    • geocar 8y ago
      > or the needs of software packagers. They address the needs of users. I don't care about software packagers at all, except insofar as they can address the needs of users. > The solution to having inconsistent packaging paths isn't to introduce _yet another_ packaging path system, but this one specific to djb stuff. It's to use a standard build system with overrideable paths, and not to assume the author knows better than the packager. Having "inconsistent packaging paths" isn't the problem. The problem is I have a program called "foo" and I don't know if it should begin with: #!/usr/local/bin/python or: #!/usr/bin/python Somebody who doesn't have a real job made the joke that it should "obviously" use "overridable paths" like this: #!/usr/bin/env python but that person hasn't used very many unix systems so they're unaware that this doesn't work on some systems. Eventually the "package manager" compromised with: #!/bin/sh ''':' if type python2 >/dev/null 2>/dev/null; then exec python2 "$0" "$@" else exec python "$0" "$@" fi ''' And users are like "what the fuck!?" All we needed to agree was that python "lives" in /usr/local/bin/python and python3 (an incompatible alternative to python) lives someplace else, we would have been fine. And for what? Why did we bother with this? What did we actually gain for all this extra shit? Some package manager felt like he was doing the job of a tarball? > They don't address cross-compilation at all, or the needs of software packagers. Cross-compilation is tricky for programs that need to run programs to figure out how they need to be built; neither cmake or GNU autoconf do anything to address it. The best solution package managers seem to suggest is to "not do that" but those programs are slow.
      • numbsafari 8y ago
        Also, is cross-compilation even a goal for a software library written with the clear caveat that it makes use of specific CPU features?
        • nrclark 8y ago
          Let's say I'm writing firmware for a TV. Why wouldn't I want to use the chip's CPU to the fullest extent at runtime? That doesn't mean I want to compile my TV firmware _on_ the TV.
  • JdeBP 8y ago
    Bear in mind that steps 1, 3, 4, 6, and 7 are not related to slashpackage. nrclark has got step #2 wrong, as all that it yields is the current source package version number not a URL and there's no real parsing going on (except for the shell stripping the trailing newline); and xyr step #8 is xyr own invention and not actually in the build instructions as given. That leaves step #5. What that does is both fairly obvious and documented in the instructions. It tries out a whole bunch of compilers, compiler options, and implementations in turn to see which results in the best code. Ironically, it is probably the sort of thing that autotools might be persuaded to do. But in practice no-one would, or it would be buried under reams of copied-and-pasted cargo cult tests for things that the actual source at hand cannot in fact do without in the first place. This is of course a long-recognized fly in any let's-just-use-autotools ointment, discussed over many years. (-: * https://news.ycombinator.com/item?id=5276876 https://news.ycombinator.com/item?id=5276876 * https://queue.acm.org/detail.cfm?id=2349257 https://queue.acm.org/detail.cfm?id=2349257 * http://freshmeat.sourceforge.net/articles/stop-the-autoconf-insanity-why-we-need-a-new-build-system http://freshmeat.sourceforge.net/articles/stop-the-autoconf-... * https://news.ycombinator.com/item?id=1499738 https://news.ycombinator.com/item?id=1499738 * https://news.ycombinator.com/item?id=4407486 https://news.ycombinator.com/item?id=4407486
  • mcguire 8y ago
    Reading those complaints cold, I would suspect DJB to be between 16 and 22 years old; old enough to have seen more than his own system, but young enough to be absolutely certain his off-the-cuff solution is better than any of those other idiots can come up with. It helps to be the smartest person in any room, I guess. "When oaf was integrated into ``the system,'' it was moved to /usr/bin/oaf, and its files moved to /usr/share/oaf, because FHS doesn't let ``the system'' touch /usr/local. What happened in this case is that oaf didn't find files that were put into /usr/local/share/oaf by another package." Once upon a time, I worked as a sysadmin for a major university computer science department. We supported a half-dozen or so different architectures. We built a long list of software, including all of the X and Gnu programs, so that our users had the same environment essentially everywhere. Oh, and we all built it to live under /lusr, because our convention started before "/usr/local" was a thing. (The weird looks when you started babbling about "slash-loser" were just a fringe benefit.) imake was a giant pain in the rear. Autotools were also a giant pain in the rear. But everything else, except in very, very simple cases, was much worse. DJB's software would come under the header of "sorry, not supportable without unreasonable effort."
    • gowld 8y ago
      Sometimes the smartest person in the room really is the smartest person in the room. If Unix was perfect 40 years ago, nothing would have ever changed.
• ben509 8y ago
  Do you know of any guidelines for making libraries easy to package? Or examples of projects that do it right?
  • wahern 8y ago
    The most important thing is to put all the compiler flags behind standard variables. This is 80% of what autoconf/automake does. The standard variables for C programs are CPPFLAGS = # e.g. -I and -D preprocess options CFLAGS = # various ad hoc compiler switches LDFLAGS = # e.g. -L options LDLIBS = # e.g. -l options SOFLAGS = # -shared or -bundle -undefined dynamic_lookup (NOTE: SOFLAGS is less standard--i.e. not in GNU style guide or GNU make manual--but perfectly fits the pattern and extremely useful to specify independently. You'll often find a variable like this used, if not the same name. IME SOFLAGS it the most common name.) Library dependencies should follow a similar pattern: LIBFOO_CPPFLAGS = # e.g. -I/usr/local/foo/include LIBFOO_CFLAGS = LIBFOO_LDFLAGS = # e.g. -L/usr/local/foo/lib LIBFOO_LDLIBS = -lfoo To make it easier to control feature-specific options without overriding all other defaults, you should do likewise for feature groups: PIC_CFLAGS = -fPIC ... PTHREAD_LDLIBS = -lpthread ... WARN_CFLAGS = -Wall -Wextra If you want to get really fancy, you can use top-level MY* variables so people can easily append flags without overriding existing variables. Then for most basic target recipes you can pre-compose all the flags to make your rules simpler: ALL_CPPFLAGS = $(PIC_CPPFLAGS) $(LIBFOO_CPPFLAGS) $(WARN_CPPFLAGS) $(CPPFLAGS) $(MYCPPFLAGS) ALL_CFLAGS = $(PIC_CFLAGS) $(LIBFOO_CFLAGS) $(WARN_CFLAGS) $(CFLAGS) $(MYCFLAGS) ALL_LDFLAGS = $(PIC_LDFLAGS) $(LIBFOO_LDFLAGS) $(WARN_LDFLAGS) $(LDFLAGS) $(MYLDFLAGS) ALL_LDLIBS = $(MYLDLIBS) $(LDLIBS) $(LIBFOO_LDLIBS) $(WARN_LDLIBS) $(PTHREAD_LDLIBS) .o.c: $(CC) $(ALL_CFLAGS) $(ALL_CPPFLAGS) -c -o $@ $< Because of the ubiquity of GCC and clang, and because of the commonality across most modern Unix platforms (e.g. SOFLAGS will be "-shared" on all BSD- and SysV-derived systems, including Linux, and "-bundle -undefined dynamic_lookup" on macOS), the above will allow people to build your software without modifying the actual Makefile in almost all scenarios. It will certainly make packagers' jobs easier. For installation you should always use the standard prefixes DESTDIR = # temporary root to assist in packaging prefix = /usr/local bindir = $(prefix)/bin datadir = $(prefix)/share includedir = $(prefix)/include libdir = $(prefix)/lib sysconfdir = $(prefix)/etc and install targets should be written like $(DESTDIR)$(libdir)/libacme.so: libacme.so ... This may seem like alot of boilerplate, but the important thing is that it's very simple and time-tested. Autogenerating just this basic boilerplate is overkill and adds unnecessary indirection and complexity. You can get fancier than this scheme, but avoid the temptation as much as you can unless it's an obvious win--remedies a regular, concrete issue that is actually resulting in significant wasted effort. For complex builds (e.g. multiple targets where flags might not have the same semantics across target rules), consider splitting the build into multiple Makefiles. You can create a convenience wrapper, but at least the sub Makefiles will be reusable by packagers or contributors without having to modify anything directly. It's not ideal but there is no ideal solution. (I once heavily advocated non-recursive make, but these days I'm much more averse to complexity.) There's no easy answer for Windows, particularly because NMake uses significantly different syntax for some constructs. For really basic builds the above might actually work as long as you stick to NMake-compatible syntax. With the advent of Windows Subsystem for Linux (WSL) you could consider requiring WSL for the build (so you have access to POSIX-compatible shell and Make) and perhaps providing a wrapper Makefile with more convenient defaults for compiling with Visual Studio (which is very much still a command-line driven compiler). Requiring clang-cl might be even easier. For simple stuff IME autotools and CMake are just overkill.[1] For complex stuff they often won't be sufficient, anyhow.[2] So the important thing is to layer your build so the maximum amount of build code can be reused by somebody with unforeseen needs; and reused in a way that is obvious and transparent so their tweaks can be simple. Stick to convention as much as possible. Even if you don't use autotools, autotools has created de facto conventions wrt build variables that will be intuitive to all packagers. Please don't invent your own naming scheme! (It's annoying when someone uses 'PREFIX' instead of 'prefix'. There's logic to the autotools scheme. Uppercase variable are for [build] flags used in rule recipes, lowercase for [installation] paths used to define rule targets and dependencies. Exceptions like DESTDIR are well-known and few.) Always be mindful of the needs of people doing embedded work--cross-compiling, complex build flags, etc. But don't bend over backwards. All you really need to do is remove obstructions and the necessity to patch. They're used to writing build wrappers, but patching is a maintenance nightmare and impedes tracking upstream. Ideally you would support out-of-tree builds, but in reality few packages support this properly (even autotools or CMake projects), and if you're not regularly doing out-of-tree builds your build probably won't work out-of-the-box anyhow. That's why I didn't mention the use of macros like VPATH, top_srcdir, etc, in your rules. Do that at your discretion, but beware of descending down the rabbit hole. The same thing applies to automatic feature detection--unless you're regularly building on esoteric targets your feature detection is unlikely to work out of the box, and bit rot happens quickly as platforms evolve.[3] Don't put too much effort into this; just make sure it's easy to manually toggle feature availability and that the defaults work on the most common platforms and environments. Maximize benefits, minimize costs, but don't think you can automate everything away. All those fancy builds people tout are ridiculously brittle in reality. People just don't realize it because they're only using the builds on what are in fact relatively homogenous environments. Most of the effort is wasted, and build complexity (including bespoke build environments) is a big reason why people don't reuse software as much as they could. [1] Indeed, for simple builds it's not much of a burden to provide and maintain a POSIX build and a Visual Studio project file build. [2] For CMake this is especially true. For feature detection CMake often requires actually updating your version of CMake. That's a gigantic PITA. Anyhow, most feature detection can be done inline using the preprocessor, or by simply defaulting to being available (this is 2018, not 1988--there's much more commonality today.) And as long as your feature-detection macros are controllable from build-time flags, people can easily work around issues by explicitly toggling availability. Thus, the autotools convention of using "#ifdef FEATURE_FOO" for feature flags is bad advice. Always use boolean flags like "#if FEATURE_FOO", and only define FEATURE_FOO in config.h-like files if not already defined. That allows people to do, e.g., "MYCPPFLAGS=-DFEATURE_FOO=0" to override feature detection as necessary. [3] Autoconf's feature checking (check the feature, not the version) sounds robust in practice, but you'd be surprised how many ways it can break, or how difficult writing a properly robust feature check is. Plus, especially on Linux people expect a feature to work at run-time if the kernel supports it, regardless of whether glibc had a wrapper at build time or has it at run time. These issues are where you'll spend much of your time, and this can't be automated. You can't foresee all these issues! Focus on making it easy to supplement or work around your build!
• numbsafari 8y ago
  Is this software that he presently wants to see disseminated randomly, or is it provided here for academic purposes at this time? I don't know if it's his intention to have this library become part of a bunch of linux or BSD distros or not. But, it seems kind of presumptuous on your part to assume that he does.
  • lorenzhs 8y ago
    I'm a PhD student in theoretical computer science and have an academic interest in this sorter. But his build system is just too much of a pain to deal with voluntarily. Not to mention that his cpu cycle counting doesn't actually seem to compile on Linux, which doesn't have sysctlbyname? Or maybe build errors are expected behaviour?
• dilap 8y ago
  as a user, i'd be pretty fucking happy if ./configure disappeared off the face of the earth.
  • jedisct1 8y ago
    As a user, I'm super happy with `./configure`, that works the same way across all applications using it, and has been for the past 20 years. You don't want to look at the actual script, but as a user, it is very convenient. I don't have to read installation guides to know how to change compilation options, compilers, cross-compile, installation paths, run the test suite, etc.
    • dilap 8y ago
      (i guess i should say i'm a dev as well as a user. but my exposure to configure is as a user.) so here's why i'm not a fan of configure: (1) it's slow (2) if something goes wrong, it's a fucking nightmare to try to diagnose -- it's just an endless barage of crap that doesn't actually have anything to do with the app at hand, and the actual build process is like 2 or 3 layers away from running the configure script. (or for extra fun, maybe you didn't have a configure script, you had the other script that generates the configure script.) i'd much rather see just a simple makefile with well documented dependencies, e.g., like redis does it. (slowly this problem is going away as building becomes an integrated part of the language -- e.g., rust, go, ...) (and to be fair, a large part of it is just an aesthetic objection -- the sheer mass of unnecessary code and checks that configure does & generates is really ugly to me. i want beautiful software!)
      • rkeene2 8y ago
        Simple Makefiles almost never work for portable software that needs to interface with a system. There is a reason GNU autoconf was created -- it solves a real problem. Different kinds of systems are different in a LOT of ways. Almost all of these ways need to be determined at compile-time and without testing for those ways, there is no good way to do this. The older way (before GNU autoconf) that tools like "xmkmf" used was to have a massive database of features for each system and then you could just query that database for the features*platform you were compiling for to get the information -- this did not scale at all, platforms changed, features were added faster than the database could be distributed, and generally it was terrible. Things not covered by a simple Makefile include almost all tasks related to building software: 1. How do you build a shared object (.so, .sl, .dylib, .dll, .a on AIX) ? Can you even do that (MINIX) ? 2. For the platform being compiled for (i.e., nothing to do with the platform you are compiling on) how large is an "int" ? Is there a type that exists for uint_least_32_t ? If not (because the compiler doesn't support stdint.h), is there an unsigned integer type that is atleast 32-bits wide ? 3. How do you link against an entire archive, not just the symbols you currently reference ? Is it -Wl,--whole-archive or -Wl,-z,allextract ? 4. For symbol versioning for your library, how do you set the soname ? How do you specify versions of functions ? Can you ? How do you filter out symbols that are not part of your ABI ? You could start over and mandate a consistent toolset and exclude the majority of platforms (Rust, Go, etc)... or... you could write your own, inferior, version of GNU autoconf, or you could use GNU autoconf... Good luck !
• mwcampbell 8y ago
  There's also a world beyond Unix. Prof. Bernstein may not be interested in targeting Windows. But for library developers in general, I'd recommend CMake. Microsoft has adopted CMake as its standard cross-platform C/C++ build system, and has built a cross-platform library package manager [1] on top of it. [1]: https://github.com/Microsoft/vcpkg https://github.com/Microsoft/vcpkg
  • rurban 8y ago
    As you see in the code this is linux only: CLOCK_MONOTONIC, HW_CPUSPEED, -soname, -rpath, linux/perf_events.h, Even on linux it would fail: sort.c:386:21: error: always_inline function '_mm256_loadu_si256' requires target feature 'sse4.2', but would be inlined into function 'djbsort_int32' that is compiled without support for 'sse4.2' -march=native or -msse4.2 would be needed.
• ur-whale 8y ago
  1. You're assuming the software is intended to be deployed 2. Writing anything sane with autotools or CMakes is an exercise in frustration that borders on torture. Not to mention how either of these systems chafes on one's sense of aesthetics. 3. I've installed a few of djb's package over the years. While indeed non standard, unlike either of the solutions you've mentioned, djb's stuff: a) works b) is usually very simple to understand and change when needed (unlike the autotools steaming monstrosity which in 20 years of using opensource software I've never dared to touch)
  • nrclark 8y ago
    Why would somebody release software that they didn't want to see used? To your other point, Autotools is little crusty for sure. It's Yet Another Language to learn (and so is CMake), but it's really not as bad as people make it out to be. There are many thousands of examples in the wild, good and bad. Yes, Autoconf tests for a bunch of stupid things that don't matter. But so what? You can add the tests you _do_ care about, and take advantage of the directory control and compiler control aspects. If you've never actually worked with Autotools (which you just said you haven't), I'd encourage you to actually try it out. It's not as bad as you think.
    • ur-whale 8y ago
      I have tried working with autotools. Many times. I guess it's down to mindset: some people don't mind working on top of a pile of manure and don't really see what the big deal is. Others just cant.
      • throwanem 8y ago
        > the autotools steaming monstrosity which in 20 years of using opensource software I've never dared to touch > I have tried working with autotools. Many times. Well - which is it?
        • ur-whale 8y ago
          :) I guess "touched" is the wrong word. I did enough "touching" of it unfortunately: using it to compile packages, trying to fix it, wasting precious time wading through pages of horrible shell code to understand why it fails when it does. I even tried a couple of time to use it in my own projects , naively thinking that it's so widespread there's got to be something to it I'm not seeing. I gave up on it in disgust everytime. The list of reasons is long, but I guess if I have to pick the one thing that really drive me nuts: most packages out there compile way faster than it takes to configure them. If that specific fact doesn't make you walk away from it immediately, our brains just aren't wired the same.
• Panino 8y ago
  I've been using DJB's stuff for ~20 years and I don't like his recent build systems either. Without defending it, I'd just like to offer a theory on why he packages things the way he does. Back in the day, his build processes were atypical but still much more "normal" than now. He also released his software without licenses. During this time of heavy software development, DJB was concerned about people screwing around with the internals of his software, hurting security or reliability, and then blaming the software rather than the modifications. So his build systems were, I think, designed to lead to the result he wanted, where software behaved and was administered in the same way on various platforms. In the mid-2000s he re-licensed existing software as public domain and began publishing all new code as public domain as well. Around this time, build systems began to get more wonky. Also, his public work that garnered the most attention shifted away from software toward cryptography. He did some attacks on existing crypto and authored Curve25519, Salsa20, etc. He's also been putting out a tremendous volume of work in multiple categories. I bet he'd rather work on this stuff than on user-friendly build systems. So given these points, I think the explanation for his unfriendly build systems is A) a very strong aversion to people modifying his stuff where he gets blamed if modifications do harm; B) a shift away from software development, where people generally care more about build systems anyway; C) a huge level of productivity which results in very atypical Pareto principle choices/tradeoffs; D) his public-domain licensing. Given these 4 points, I think DJB is unwilling to take time away from crypto and other work and put it into build systems he doesn't enjoy that will take more time upfront and more babysitting down the road. Fewer people will package it, but the software is public domain and competent people can just add their own build system. This squares with his available time and interests. So, I don't like his build systems either, but I think I understand where they come from.
  • nrclark 8y ago
    Appreciate the response! Interesting to read. The Libsodium guys wound up doing exactly what you're suggesting, because of the impossibility of trying to package NaCl as-is. So they essentially had to re-do/duplicate all of his build work just to make it packageable. And now there are two competing implementations (three if you count tweetnacl). And a bit of a confusing mess in the documentation department. It seems a little selfish for djb to take the "works on my machine" attitude, because it means that a bunch of other people have to reverse-engineer all that stuff just to make it portable. But I guess OTOH it's his software, so it's his choice. And maybe he doesn't care whether people choose to use his stuff or not, as long as he's publishing.
• tbirdz 8y ago
  I really disagree with you on autotools, that has been a big problem for me when cross-compiling. Configure often detects features by compiling a binary and executing it, which can be a problem when you're cross compiling, since the generated target can't be executed on your native cpu. Most software build systems don't even take cross building into account at all, since they just copied someone elses' broken autoconf and automake source that only works on native builds so you have to work around a lot of issues. Even "good" software packages written by people who understand autotools like glib have issues. Just look at the official glib cross compiling instructions: https://developer.gnome.org/glib/stable/glib-cross-compiling.html https://developer.gnome.org/glib/stable/glib-cross-compiling... The "correct" way to cross compile it is to figure out all the autoconfigured values yourself, and write them into the configure cache. Personally, I have had the least trouble with plain Makefile based packages. Yes you often have to reach into the Makefile's guts and modify it to make a working cross build, and you'll have to set the rpath manually, and install the files yourself, but at least it's easier than having to modify an autotools based build. Autotools "just works", and when it doesn't it's very hard to fix. Makefiles don't "just work", but it's much easier to fix them yourself.
  • R0b0t1 8y ago
    To clarify for anyone who doubts autotools has problems - it is true that the documentation mentions cross compiling options, but it appears that the vast majority of tests in use today do not actually use these values (host/target arch, etc). Some users recommend compiling with qemu-user to avoid fixing the tests, but besides being very slow, that methodology will still generate garbage. What is the point of running feature detection code on a desktop that will be used to configure code compiled to run on e.g. a small ARM system? The fix is to explicitly set configuration parameters or, even better, to detect features at runtime. However if you try to contact any given project using autotools to help solve these configuration problems you will likely receive "that's not our problem" and no help from the maintainers. Autotools is in practice less work for me than alternatives, at least while using a FOSS operating system. On Windows CMake is very well developed but handling dependencies is always a chore.
  • altfredd 8y ago
• deleted 8y ago
  [deleted]