10 ms·
Getaddrinfo() on glibc calls getenv(), oh boy
- magicalhippo 3y agoWhy would you ever call setenv after spawning threads though? Or are there other sneaky calls which will do that behind you back?
- numbsafari 3y agoThat’s literally explained in the article. It’s worth reading more than the headline. Ed: actually, that’s even spelled out in the headline.
- silisili 3y agoIt's not. OP was asking about setenv, not getenv...
- emmelaich 3y ago? setenv not getenv. You'd rarely use setenv, and even then you'd do it at startup.
- magicalhippo 3y agoRight. That's been my experience so far, hence my question.
- raggi 3y agoIt’s neither in the headline or in the article. The question was about setenv, not getenv. It is best to avoid calling setenv in a threaded program. Some programs do it to make space for rewriting argv with large strings (freeing space from *environ which tends to be right after the tail of argv). Some programs or libraries use *environ directly to stage variables for exec before forking. Some want to pass variable changes to forks. There are alternatives possible, but in the context of something like go calling libc setenv, it’s to make interop easier- sadly it may make other interop harder, such as this case.
- marcosdumay 3y agoWell, that's the gotcha isn't it? It's easy to think about some complex interactive software where the need to call setenv appears only after you have worker threads doing some other thing. Without a warning, you won't know it's a bad thing to do, and the manpage only says that it and unsetenv are not thread safe, as if this was remotely enough information. What nobody is telling is that the environment is so big that you need it to compress data or open an IPv6 connection. It's not obvious at all that you can't do those things while editing a variable.
- hinkley 3y agoThere’s always a lot of weird emergent behavior in bootstrapping an app, and on an app of any serious size, I can’t entirely control if someone decides to spool up a thread pool on startup so that everything is hot before listen() happens. I may think I have control, I may believe that a handful of us are entitled to have that say, but all it takes is someone adding a cross dependency that forces an existing piece of code to jump from 20th position in the load order to 6th and all hell can break loose. Or just as often, set a ticking time bomb that nobody notices until there’s a scaling or peak traffic event or someone adds one more small mistake to the code and foomp! up it goes.
- Joker_vD 3y ago...on Windows, single-threaded programs don't really exist; any DLL can, and most of them do, spawn worker threads as an implementation detail. Some of them do it the moment their initializer is being run, so if you link your program against something else than kernel32 and its friends (the basic Windows system libraries don't spawn worker threads on being loaded), then when a thread finally starts executing your executable's entry point there is no guarantee that this is the only thread that exists in your process. And in fact, finding a non-toy, real-world Windows application that has only one thread is almost impossible (for example, IIRC all .NET runtimes have worker-thread pool from the get go so that rules out any .NET executables). Which is why on Windows there is almost no system APIs (well, almost: there were some weird technical decisions around single-threaded apartments for COM...) that can be safely used only in single-threaded applications. Maybe in several more decades Linux community will also accept the fact that multi-threaded applications are an entirely normal and inevitable thing, not an aberration of nature that we all best pretend don't exist until we're absolutely forced to deal with their reality.
- 38 3y agoI always thought getenv() was code smell, and this confirms it. I guess a better option would be to add a parameter to Getaddrinfo(), and deal with it further up the stack?
- raggi 3y agogetaddrinfo is far too ossified to be changed
- ForkMeOnTinder 3y agoStuff like this is why I'm supportive of newer languages like Go and Zig that sidestep libc entirely (when not using cgo as in TFA of course). libc is a great achievement and has served us well but, boy, it sure is a product of its time. `errno` is another relic that needs to die yesterday.
- eatonphil 3y agoDepending on the operating system, you can't skip libc even in Go. I think it's required on openbsd and illumos/solaris for example. https://utcc.utoronto.ca/~cks/space/blog/programming/Go116OpenBSDUsesLibc https://utcc.utoronto.ca/~cks/space/blog/programming/Go116Op...
- favorited 3y agogolang used to break all the time on macOS, because it was using the syscall ABI, which isn't stable, instead of libSystem, which is.
- dottedmag 3y agoIt has been fixed recently-ish.
- littlestymaar 3y agoA long time ago, but AFAIK the fix was “use the system's libc”.
- bananapub 3y agoit was fixed by using-the-system-libs
- eatonphil 3y agoCould you link to the fix or their docs on it? I.e. what do they do today?
- kentonv 3y agoDoesn't almost every C library function potentially call getenv()? printf()ing a number requires checking the locale which can be configured via the environment, after all. I'd say you shouldn't be calling setenv() at all once you've spawned threads.
- bb88 3y agoBut the question is: Who knows what and what not to do with libc? "But Go compiled it just fine..."
- rollcat 3y agoI build most of my Go binaries with cgo disabled for this, and many other reasons. In case you don't know, cross-building with GOOS/GOARCH will imply CGO_ENABLED=0 unless you also specify CC_FOR_${GOOS}_${GOARCH}; I cross-build most of my code for (and test it on) amd64, arm64, linux, openbsd, and darwin. Go will sometimes link to the local libc for network-related functionality if you don't disable cgo.
- thewakalix 3y agoThat's the entire point of the post and the earlier posts it links to.
- kentonv 3y agoThe post is about getaddrinfo() specifically. It just struck me as odd to call that one out when there are far more common C library calls that use getenv().
- j16sdiz 3y agoThey did a post on mktime() But, yes, in general for libc, if the manpage didn't say it's thread-safe, it is unsafe.
- defrost 3y agoThere's a strong tendency to think of network calls as entirely universal and not tied in any way to to locale settings in the environment. Time, date, physical spellings, ... many things are locale dependant, but socket stuff?. It comes as a Surprise!!, and not the good kind, to many a network programmer with just a few years under their belt to discover threaded networking can segfault because of this. Once you know, you know and don't forget (until next time), but I suspect this was the motivation behind the blog posting, the principal of potentially most surprise.
- jrmg 3y agoI think I must be missing something here, but I’ll ask anyway: Why don’t the OS libraries have some sort of lock around setenv/getenv, so that only one thread can be inside them at a time? I can’t see how it could deadlock. And surely no-one is so dependent on the performance of these calls that the time to lock/unlock would be problematic?
- asveikau 3y agoI agree. I'm shocked this isn't there in 2023. Or even better, a rwlock allowing concurrent reads while serializing writes. Or some lock free algorithm for writes. I'm pretty sure the Windows version of the environment calls has locking. Historically you can access the environment via a global variable too, which would side-step locking schemes. But probably hardly anybody does that anymore.
- jeroenhd 3y agoSetEnvironmentVariable is thread safe on Windows, but their POSIX wrappers aren't. Windows has a better API design in this instance, and the guarantees by POSIX make it impossible to make a compliant implementation that can be used with threads.
- asveikau 3y ago> and the guarantees by POSIX make it impossible to make a compliant implementation that can be used with threads. Can you be more specific than this? I kind of doubt it. For example, seems to me you could write a setenv() that uses a lock-free algorithm or writes in a strategic way that won't result in a fault if getenv() or reading environ(7) runs concurrently, then say all bets are off for thread safety if you write via environ(7). That's safer than the status quo and I don't foresee it breaking POSIX.
- jeroenhd 3y agoReading the spec again, I suppose it's possible to keep copies of environment variables around in memory without violating the spec, basically creating a copy for every getenv call that was modified since the last setenv/putenv. I thought the spec also specified that writes to the returned pointer would update the environment variables, but no such guarantee is given, that's just an implementation detail (and is disallowed by the API spec but good luck enforcing that). The XSI spec does state that the result of getenv() may be overwritten by putenv() but that's not a strict requirement either. You still risk programs and libraries expecting getenv to always return a pointer to *environ failing (I believe Go has an issue like that on MUSL). On the other hand, the POSIX standard explicitly states that getenv() does not need to be reentrant (and therefore doesn't need to be thread safe) so any program relying on thread-safe getenv is already violating the API contract. The rationale also seems to assume you can't make this stuff thread safe because of the standard implementation: > The getenv() function is inherently not reentrant because it returns a value pointing to static data.
- LoganDark 3y agogetaddrinfo is terrible. Did you know it also opens and then connects to a socket? Any process that uses getaddrinfo needs a blanket exception in my firewall in order to work properly, because otherwise it will fail to connect to some randomly-generated port that it just made up.
- tedunangst 3y agoWhat should I use instead that doesn't open a socket?
- hknapp 3y agoThey probably could have given a more apt name
- Sprocklem 3y agoHow do you propose that DNS resolution precede without opening a socket?
- nomadluap 3y agoFor one thing, it could delegate to a local service. Granted, the communication to this service is probably still be over a socket interface, but at least as a purely-local connection you would hopefully have some better worst-case performance characteristics. This is basically what dnsmasq does when you use it as a local DNS cache.
- jxy 3y agoEither Linux's fault or Glibc's fault. POSIX mandates getaddrinfo to be thread safe.
- raggi 3y agoglibc, I’m too lazy to lookup if it removes the getenv lookup with posix build flags.
- o11c 3y agoYes, but `setenv` is documented as not thread-safe. Once you do unsafe things, the nasal demons can spread to safe code elsewhere. Glibc's documentation is more explicit about the propogation of the nasal demons: ┌─────────────────────┬───────────────┬─────────────────────┐ │Interface │ Attribute │ Value │ ├─────────────────────┼───────────────┼─────────────────────┤ │setenv(), unsetenv() │ Thread safety │ MT-Unsafe const:env │ └─────────────────────┴───────────────┴─────────────────────┘ ┌────────────────┬───────────────┬────────────────────┐ │Interface │ Attribute │ Value │ ├────────────────┼───────────────┼────────────────────┤ │getaddrinfo() │ Thread safety │ MT-Safe env locale │ ├────────────────┼───────────────┼────────────────────┤ │freeaddrinfo(), │ Thread safety │ MT-Safe │ │gai_strerror() │ │ │ └────────────────┴───────────────┴────────────────────┘
- jxy 3y agoIt's great that Glibc documents its POSIX violating behavior very well, but it doesn't change the fact that it violates POSIX.
- o11c 3y agoPOSIX does document it, it just requires carefully picking through pages and carefully thinking about the wording, unlike the simplicity of GLIBC documentation. For example, the best information is on the page for `exec`.
- uxp8u61q 3y ago
- woodruffw 3y agoThe value of `setenv(3)` has always been pretty murky to me -- the only time I've ever really needed it is when performing a fork-exec, and even then it's been the wrong tool for the job (the dedicated exec*e variants are the right way). Would there be any significant downsides (besides breakage) to mapping `environ(7)` as read-only? That seems like the kind of thing that a Linux distribution (or more realistically OpenBSD) could do as a way to kill off a persistent family of bugs.
- fsociety 3y agoIf they did it, it would break userspace for a probably surprising amount of processes.
- Brian_K_White 3y agoWhy not find those? Especially in a dedicated arena whose stated goals are things just like that like openbsd.
- j16sdiz 3y agoIt is quite common a pattern setting the timezones with setenv("TZ",...) There are alternatives, but rewriting every program is not always an option. These kind of rewriting sounds exactly like what OpenBSD would do ..
- woodruffw 3y agoThis is perhaps just my ignorance, but when do you find yourself needing to set the timezone like that? Not in 10+ years of C programming have I ever had to do that.
- masklinn 3y agoI assume doing something on behalf of a remote user and still convinced locales can do a thing other than hurt you.
- 3y ago
- jchw 3y agoI wonder if this only applies to the dns resolver, or also other NSS modules, like the systemd resolver. And don't forget about nscd: if nscd is running, then all of the nsswitch stuff will be done out of process. That might mean a viable workaround is enabling nscd, oddly enough. And frankly, maybe libpthread should just overlay thread-safe getenv/setenv like I believe it does for a couple of other libc symbols.
- cyberax 3y agoGAI just needs to go. It needs to be moved to its own daemon with a simple RPC. libc is kinda schizophrenic in this regard. It has mostly obviously low-level functions like string manipulation and memory management, and then unexpectedly a DNS client implementation and a support for arbitrary runtime plugins (for PAM).
- deleted 3y ago[deleted]
- tadfisher 3y agoThis, along with the general inability to handle per-interface DNS resolution, is the primary reason systemd-resolved exists.
- xerxes901 3y agoThis is essentially what systemd-resolved with the nss-resolve NSS module is right? It’s possible to use /etc/nsswitch.conf to entirely disable the built-in DNS resolution in glibc if you want.
- alexgartrell 3y agoIME, familiarity with /etc/nsswitch.conf and the rest of the nss stuff is very highly correlated with people who have seen some shit.
- cyberax 3y agoYep, it's close. But you still depend on the unholy mess that is nsswitch.conf It's still a bit worrying, with manual JSON parsing: https://github.com/systemd/systemd/blob/79f487038444646f5bce1f5a18d51cda5f20b268/src/nss-resolve/nss-resolve.c#L598 https://github.com/systemd/systemd/blob/79f487038444646f5bce... But at least it's just ~600 lines of fairly straightforward code. You also can get most of it with nscd. (Sigh, I wish BUS1 guys pushed their project to completion)
- userbinator 3y agoKeep in mind: Don't use multiple threads unless you really, really need to, and have thought long and hard about concurrency issues. In a way, I think the fact that many library functions are not thread-safe should be viewed as an encouragement to not use threads, or use them only for the bare minimum necessary. I say this from a few decades of experience fighting with race conditions and the like, and whereupon several times I rewrote an existing multithreaded process into a single-threaded one and greatly improved performance and reduced memory usage. The architecture astronauts may have moved on to stuff like microservices now, but in the 90s/2000s threads were overused just as much.
- Dwedit 3y agoThere are many ways to do multiple threads wrong. Seems that the "right way" is to wake up a sleeping but already-created thread, and take elements out of a work queue in a threadsafe way. Your main thread can even be processing elements during the 10000 clock cycles it takes to wake up a thread.
- favorited 3y agoThat's basically impossible in many modern programming environments - even if you never spawn a thread, something else in your executable probably has. By the time your iOS or macOS app has finished launching, it has multiple threads. The Windows loader uses threads to load DLLs.
- gpderetta 3y agoFor one, if you are using getaddrinfo you are often forced to use threads as gai is a slow blocking call.
- throwawaaarrgh 3y agoYou know the source code and manual pages are open source. You can look for these things.
- saagarjha 3y agoShe does in fact know these things, which is why she’s writing about it. For the people who may not have thought to do that yet.
- fsckboy 3y agothis is not a unix beartrap, this is a bug in Go if that's where it was found. If your code is multithreaded, it's up to you to make it threadsafe. You can't declare you're creating the most threadsafe memorysafe newbie safe system, and then go home. You have to write the code. Your program gets its own copy of the environment when the program was launched. Nobody is changing it on you, any contention for that resource is you contending with yourself. You don't expect an operating system to change things out from under you. Unix doesn't. If there is contention for this resource, it's all you (or whoever wrote the library you are using) The environment is name-value pairs, as strings. That's it. That's what makes it accessible and useful. You can swallow it up, the whole thing, into whatever data structure you prefer in your language in a few lines of code, and a millisecond (if that much) of runtime. Just learn how things work and you won't feel helpless.
- nvy 3y agoI'm confused, aren't we talking about the getaddrinfo() syscall? It definitely seems like that syscall is a (multithreaded) footgun that Go just happened to hit.
- fsckboy 3y agoa "syscall" is not going to call back into your code to get the environment. If a "syscall" is using your environment, it's part of your code, probably better described as a library call.
- saagarjha 3y agoFew people make raw syscalls.
- dundarious 3y agoIt's not a syscall, it's a C standard library function that may read files, talk to dbus, read env vars, etc. The problem is people quite understandably expect its re-entrancy to mean more than it really means.
- r2vcap 3y agoI think the problem is in setenv(3), not getenv(3). Reading shared global state is okay as long as it is not mutable. If someone relies on modifying environment variables, one should use execve(3), not setenv.
- metadat 3y agoFor those who, like me, are a bit rusty on the man memorization: execve(3): https://linux.die.net/man/3/execve https://linux.die.net/man/3/execve
- Too 3y agoExactly. Setting environment variables while a program is running is a terrible idea. Thread safe or not. A lot of code, for good reasons, assume envvars are constants set before the program started and caches computations based on them, read config files and so on. The fact that they are essentially global variables should be enough to deter usage of them.
- chriswarbo 3y ago> The fact that they are essentially global variables should be enough to deter usage of them Env var behaviour is much closer to dynamic variables, rather than global variables (which I argue at http://www.chriswarbo.net/blog/2021-04-08-env_vars.html http://www.chriswarbo.net/blog/2021-04-08-env_vars.html ) Either way, I agree that mutating them is usually a bad idea; though I find them very good for constant (or dynamically-bound) config.
- cryptonector 3y agoIt can be made safe: https://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/gen/getenv.c?r=60b81b86#48 https://src.illumos.org/source/xref/illumos-gate/usr/src/lib...
- deleted 3y ago[deleted]
- jakeogh 3y agomusl: https://git.musl-libc.org/cgit/musl/tree/src/network/getaddrinfo.c https://git.musl-libc.org/cgit/musl/tree/src/network/getaddr...
- saagarjha 3y agoYes, that’s why nobody likes musl’s DNS.
- heinrich5991 3y agoPermalink: https://git.musl-libc.org/cgit/musl/tree/src/network/getaddrinfo.c?id=79bdacff83a6bd5b70ff5ae5eb8b6de82c2f7c30 https://git.musl-libc.org/cgit/musl/tree/src/network/getaddr....
- X-Istence 3y agoThis is something that has bit Rust as well: https://github.com/rust-lang/rust/issues/90308 https://github.com/rust-lang/rust/issues/90308
- gumby 3y agoIt’s easy to sterilize your code in this regard (although as this article points out you need to know to do it). POSIX implements a three-argument `main` function (just look at `exec()`) where the third argument is `char* envp[]`. You can call `setenv` to manipulate it. But easiest is to just null out the POSIX extern `char* environ` (save a copy if you want to consult it yourself later). Just `man 7 environ`
- naruhodo 3y agoIt's not clear from the man pages, but `setenv()` mutates `environ`. In particular, setting a previously-undefined variable causes `environ` to be reallocated. Whereas `setenv()` of an extant variable changes just that value in the current `environ` pointer array.
- garaetjjte 3y ago>But easiest is to just null out the POSIX And then wonder why some shared library dependency few layers deep blew up.
- thayne 3y agoMany posix (and c standard library for that matter) functions were not designed with multithreaded programs in mind and don't work well in multithreaded programs. I really think it would be worth creating a new standard API that is built with threading in mind, where functions like mktime, getaddrinfo, localtime, etc. take arguments instead of reading from the environment, that avoid global state as much as possible, and are thread safe if there is global state.
- cookiengineer 3y agohttps://musl.libc.org/ https://musl.libc.org/
- krackers 3y agoDoesn't musl have the same issue? https://github.com/JuliaLang/julia/issues/34726#issuecomment-584608976 https://github.com/JuliaLang/julia/issues/34726#issuecomment... I also wonder about OSX's libc. Newer versions seem to have some sort of locking https://github.com/apple-open-source-mirror/Libc/blob/master/stdlib/FreeBSD/setenv.c https://github.com/apple-open-source-mirror/Libc/blob/master... but they free pointers so it's not safe from a potential use-after-free client side. but older versions (from 10.9) don't even have any locking: https://github.com/apple-oss-distributions/Libc/blob/Libc-997.90.3/stdlib/FreeBSD/setenv.c https://github.com/apple-oss-distributions/Libc/blob/Libc-99...
- cryptonector 3y agoSolaris/Illumos also has a very nice C library with lots more thread-safety than others, including `getenv()` being thread-safe.
- habibur 3y agoWhich is why I have gotten rid of getaddrinfo() calls in my server code, and rather resolve DNS directly reading the DNS server setting from the system. Other issues I faced : - Not epoll() friendly. Always forks a process while resolving domain name. - Valgrind complains of uninitialized memory touches when the function is called and I can't get rid of it.
- AceJohnny2 3y agohow confident are you that the DNS settings you are honoring are the ones the user intended? For example, in a split-VPN situation?
- habibur 3y agoThose redirections happen further in the stack. You are ok reading settings from /etc/resolve.conf, which frequently points to a localhost daemon that redirects your calls to whatever DNS setting you have in your connection. But parsing /etc/resolve.conf and using it is all that you need in your code.
- guappa 3y agoAnd your code correctly parses all the options that can be in that file?
- habibur 3y agoYou don't need all the options. Search google. Parsing resolve.conf is an old technique and the file was written assuming individual apps will be parsing it. You will find instructions on how to do it in say 4 lines. Explicitly for this file. Not any random conf file from the system.
- guappa 3y agoThen your application won't be portable. Which is fine if you have no plans of distributing it. But otherwise I can guarantee it will break on some machine.
- fulafel 3y agogetenv() could be made thread-safe by leaking the memory returned.
- dur-randir 3y agoWe've once spent 3 months debugging crash caused by this.
- cryptonector 3y agoThere is no reason that `getenv()` should ever lock -- it should always be lock-free. `putenv()`/`setenv()`/`unsetenv()` can lock, of course, since there's no point allowing more than one writer at a time. Don't believe me? Look at this work of art (the entire file!): https://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/gen/getenv.c?r=60b81b86#48 https://src.illumos.org/source/xref/illumos-gate/usr/src/lib...
- kimixa 3y agoWhat are you trying to say using that example? The getenv() function shown calls initenv(), which can (clearly) take a lock?
- cryptonector 3y agoOne time. Also, I think that could be removed with some care.
- kimixa 3y agoBut it unconditionally calls a memory barrier, which is most of the cost of an un-contended futex-style lock already. And I'd be interested in your ideas on removing the lock - as I can't see any paths that don't change semantics (e.g. unconditionally doing the init work at process start time when you know there's not multiple threads, for example)
- cryptonector 3y ago> a memory barrier, which is most of the cost of an un-contended futex-style lock already Yes, but it's not a lock. > unconditionally doing the init work at process start time when you know there's not multiple threads That's the most obvious fix, yes. I was thinking (but I've not checked yet) that when `my_environ` is not set up yet then `getenv()` can use `_environ` directly.
- amelius 3y agoYou shouldn't call setenv in the first place. Instead, use a new environment when you call execle/execvpe. This avoids problems with multiple threads.
- pdw 3y agoSome APIs require the use of setenv. For example, you need to setenv("TZ", ...) if you want to use multiple timezones.
- amelius 3y agoYou mean linked libraries you don't have the source of? Yes, in that case set the environment vars as early as possible (if possible before starting any thread).
- rurban 3y agoAlmost everybody calls getenv(). malloc for tunings, checks, tracing and such, half of the string library for localization specifics, all of the locale and time and timezone functions, many math functions need fegetenv(). Also most set errno, ha!
- jonhohle 3y agoerrno is thread safe (and thread local).
- WhyNotHugo 3y agoOn Linux/musl, the man page also specifies that LOCALDOMAIN and RES_OPTIONS are inspected. The man page for OpenBSD indicates the same thing: https://man.openbsd.org/resolv.conf#ENVIRONMENT https://man.openbsd.org/resolv.conf#ENVIRONMENT Apparently it's not a gnu-specific behaviour.
- cratermoon 3y agoWhy threads are a bad idea (for most purposes) https://blog.acolyer.org/2014/12/09/why-threads-are-a-bad-idea/ https://blog.acolyer.org/2014/12/09/why-threads-are-a-bad-id...
- _nalply 3y agoAfter reading some comments, to avoid the problems - immediately copy the contents of the buffer the pointer from getenv() points to - don't use getenv after threads have been started A library could be written which makes an immutable copy of the whole environment before starting main(). This library then hands out pointers to the environment copy. Or to be even more secure make another copy of the environment variable. This trades some efficiency for security. In effect, ignore the mutable accessors like setenv from libc. Or did I miss something? I am not an expert in these things. And of course it won't solve the problem of two other libraries fighting with each other...
- deleted 3y ago[deleted]
- kazinator 3y agoEverything would be cool if it weren't for there being situations in which you need to set an environment variable in the same process in order to get a function to do something. E.g the TZ variable in order to coax a behavior out of the time functions. If environment variables are for child processes only, there is no need to use setenv because you can pass a specified environment array through exec.