10 ms·

Improving Steam Client Stability on Linux

477 points by Venn1 2y ago

bhaney 2y ago
> If this can be addressed in glibc, it may involve a tradeoff on features, maybe an opt-in mechanism with a slight departure from the "impossible" POSIX spec. That's something we may pursue in the long term if we can propose something sensible. Yes please
jeffbee 2y ago
Maybe among the best decisions Java ever made was hiding setenv. You simply cannot set env vars in Java.
- mschuster91 2y ago
  Java in general tends to massively dislike environment variables. Which is a Good Thing, but unfortunately it makes dealing with proxy support so much harder - especially as there are just so damn many HTTP libraries that people use...
- aidenn0 2y ago
  So does that also mean you can't write e.g. a POSIX shell in Java?
  - shakna 2y ago
    System.getenv exists, so getting is easy... But setting env vars generally involves a convoluted process. You can modify the JVM's buffer, or spawn a new ProcessBuilder, or a few other things. But it's a nasty place to be.
    - MBCook 2y ago
      Or make a little JNI class.
      - shakna 2y ago
        That doesn't work for preserving to child threads, unfortunately. Which is something people expect.
  - bhawks 2y ago
    1) java shell starts 2) read getenv into a java hashmap 3) evaluate shell input - any sets update the hashmap, any gets read from the hashmap 4) when you get to a point where you need to exec a process you use the hashmap to pass envvars into process builder - which is going to call execvpe for you. So yeah you definitely could (there'd be other reasons why java wouldn't be a good choice). TBH reading that folks were calling setenv before an exec to propagate env vars to a child made me sad. I would guess that other use cases were leveraging environ as a poor man's global variable - which is also unfortunate.
  - asveikau 2y ago
    execve(2) and friends allow you to specify a new processes's environment variables. How Java may choose to expose that I don't know. Assuming they do, that is all a shell needs. You do not strictly need to call setenv(3) to change a variable that gets evaluated with $, etc. You do need to pass the modified environment to a child process.
  - lmm 2y ago
    You can pass a new environment to a process you spawn, which I think is adequate? You just can't modify the environment of your existing process.
  - sunshowers 2y ago
    There are tons of reasons you can't write a POSIX shell in Java, starting from the lack of support for non-UTF-8 paths :)
- NotACracker 2y ago
  I admire your courage to say something positive about java. On another note, I believe Zig does the same thing.
o11c 2y ago
The real question is: is there any case where a program calls `setenv` in one thread and actually wants it to take effect in other already-existing threads? That said, GLIBC is pretty good at documenting all the dangerous functions, so it is possible to add locking/copying yourself.
- fweimer 2y ago
  Interesting idea. I strongly suspect that there are programs out there that expect that setenv changes the environ array (and they do not treat it as an opaque pointer passed to posix_spawn/execve). With a per-thread setenv, we would need a per-thread environ variable as well. Unfortunately, that's not really compatible with POSIX because environ is not declared in a header. Instead, programmers are expected to write a declaration extern char **environ; into their sources, and that declaration is incompatible with environ being a thread-local variable.
  - o11c 2y ago
    Hm, in the end most of the problems do come down to stuff not coming from blessed headers. Regardless of anything else, how about: * deprecate direct access to `environ` and add functions to replace it. Have a macro that indicates this and provide a canonical compatibility shim for people to copy if they might use old libcs. * using linker magic, change the behavior of programs depending on whether they attempt to access `environ` or not, so old-API programs are still thread-unsafe but new ones are thread-safe. It's amazing how much you can do with the conditionally-linked object files from a static "library". Much of C's cross-TU "UB, no diagnostic required" is inexcusable since we can detect it quite easily with zero overhead (at least, for static linking) using today's linkers by deliberately causing multiple definition errors. Compatibility with old-ABI programs probably means fixing `environ` is not that simple, but you are libc and libc is in control of dynamic linking ...
    - ptsneves 2y ago
      All the inconsistencies you suggest sound like a trap, especially when you suggest ABI and API behaviour divergences. From what I understood of your macro idea this would lead to API changes that would lead to ifdefs for different (g)libcurl versions. Doesn’t feel good especially for software distributors. Also the changes you mention require changes in the linker scripts distributed by a variety of toolchains and would need to check if the libc target was a blessed one. In effect the deprecation would never move to obsolete and kept around forever. To clarify the above post libc is in control of dynamic linking through the dl*(dlopen) family of functions.
fweimer 2y ago
We've got patches under review: https://inbox.sourceware.org/libc-alpha/cover.1722193092.git.fweimer@redhat.com/ https://inbox.sourceware.org/libc-alpha/cover.1722193092.git... (triggered by https://issues.redhat.com/browse/RHEL-42410 https://issues.redhat.com/browse/RHEL-42410, a graphics stack stability issue that wasn't as visible in RHEL 9 for some reason) At least the first one (the getenv thread safety fix) will hopefully make it into glibc 2.41 and it should be quite safe to backport. It turns out that setenv is easier to handle because glibc already never frees environment strings. It's concurrent unsetenv that is rather tricky. Without some snapshot approach, getenv would return null pointers instead of environment variables values that are actually set. I don't want to introduce locking into getenv because getenv without setenv has been async-signal-safe for so long that it would likely break applications. The environ handling fixes are a bit more controversial because vfork+execve make it complicated to avoid memory leaks, but these further fixes are less important to the stability of the graphics stack.
- Asooka 2y ago
  What about using a linked list for variables added after the start of the process, which can be implemented atomically? Then once it gets "too long", a thread executing setenv could construct a new hashmap and replace the pointer to the old one with the pointer to the new one atomically, all without locking? To prevent two threads from rehashing at the same time, use an atomic flag for whether a thread is rehashing right now. That means if other threads call setenv in the meantime, the extra added variables would be appended to the list to be processed by a later setenv call (if one happens). That list could grow unbounded if enough threads call setenv quickly enough, but I think the simplicity of atomically swapping pointers might be worth it and setenv isn't called very frequently.
  - jeffparsons 2y ago
    I like this approach, because it lets everything Just Work without anyone outside of the implementation having to think about it at all, and only incurs meaningful overhead if you're doing something really silly, but crucially still won't break — it'll just be slightly slow. That feels like the right trade-off.
Pannoniae 2y ago
To raise awareness: there's been a bug with the Linux Steam client which has been persistent for a long time. TL;DR: if you have Steam running for more than a ~day or so, you will run out of window handles so you won't be able to open any new graphical application/window until you restart Steam. Using Steam Chat appears to make the issue worse (it happens earlier). This has been documented under https://github.com/ValveSoftware/steam-for-linux/issues/9094 https://github.com/ValveSoftware/steam-for-linux/issues/9094 but for some reason that issue has been closed. I personally just restart Steam every day but if someone else encounters this issue and doesn't know why their windows are not opening, this is why :) I am using KDE/Wayland but I've observed this under X11 too.
- LorenDB 2y ago
  KDE Wayland user here. I have never observed this issue myself, and I leave Steam running all the time (although I never open Steam Chat).
  - snvzz 2y ago
    Although, not albeit. You have a slave sentence there. These are incompatible with albeit.
    - LorenDB 2y ago
      Thanks, fixed! "Albeit" seemed like it would work, but "although" indeed fits much better.
      - soulofmischief 2y ago
        To put it another way, albeit shouldn't be used with independent clauses. 1. I leave Steam running all the time (although I never open Steam Chat). 2. I leave Steam running all the time (albeit never with Steam Chat). "I never open Steam Chat" can stand on its own as a sentence, but "never with Steam Chat" does not and thus can be appropriately modified with albeit.
snvzz 2y ago
Considering glibc's effort, I have to wonder what the other libc do, and whether they already implement something like this.
electromech 2y ago
Thank you! I deeply appreciate that Steam works so well on Linux these days. I don't take for granted the hard work happening behind the scenes to make that a reality for us.
- Forgeties79 2y ago
  I really hope they drop a full steam OS release again soon. I’d love to build a gaming tower around it. Might settle for bazzite but I’d like to do steam OS tbh
  - lordofgibbons 2y ago
    What would that get you over any other distro with the steam software installed?
    - shortrounddev2 2y ago
      1st party support
    - zamadatix 2y ago
      There's a lot more to SteamOS than "it ran `apt install steam -y` for you". Even just having things like Gamescope running out of the box (which is more than just `apt install gamescope`) is a huge amount of headache set aside. Plus the guarantee when it doesn't work it's because of a bug rather than something you did or some incompatibility with your exact distro setup.
      - Forgeties79 2y ago
        Yeah you get it lol I just want to play my damn games and not tinker all the damn time!
      - akimbostrawman 2y ago
        >guarantee when it doesn't work it's because of a bug rather than something you did or some incompatibility with your exact distro setup. Flatpak fixes this problem. You don't need gamescope unless for stuff like scaling. The only few times you need to fiddle with anything on desktop is changing proton version or adding a launch variable from protondb.com (just like on steamdeck)
josephcsible 2y ago
> One of my colleagues rightly dubbed setenv "the worst Linux API". Is setenv really a Linux API, since it's neither defined by Linux (it's in POSIX) nor implemented by the Linux kernel (it's entirely in userspace)?
- deleted 2y ago
  [deleted]
- amiga386 2y ago
  Yes it is, if you program application software for Linux (and your other two supported platforms are Windows and macOS, as is the case for OP and their colleague who appear to work for Valve, supporting Steam) And the reason the colleague dubs it the worst has nothing to do with its specification in POSIX (which doesn't require the function to be thread-safe but also doesn't prevent it from being made thread-safe), but rather its specific implementation in glibc, which is the C library in use on all the Linux distributions that Valve support Steam on and all are equally dubbed as "Linux"
  - deleted 2y ago
    [deleted]
- WhyNotHugo 2y ago
  Technically, this is a GNU libc limitation. A libc _could_ make this thread-safe. Languages like go just implement this feature internally and don't have this limitation. The limitation is contemplated in POSIX itself; from the posix documentation: > The setenv() function need not be thread-safe.
  - deleted 2y ago
    [deleted]
  - deathanatos 2y ago
    I think you get thread-safety and memory leaks (the article alludes to macOS taking this approach), or it's unsafe but doesn't leak. But that is inherent in the API, as specified by POSIX: setenv must necessarily either invalidate the pointer returned by getenv (which is cannot do safely) or leak it. Perhaps the leak is "better", in at least there won't be non-OOM crashes, but it still leaves a bad taste in one's mouth. For a long running program like Steam (that is for some odd reason calling setenv…?) … I'm not sure which is better. Better would be not calling setenv, which it sounds like they've worked on.
nineteen999 2y ago
The stability is fine for me, but the rendering performance in the steam client when the mouse is in the window is abysmal.
- Vilian 2y ago
  Nvidia?
  - yxhuvud 2y ago
    Could just be really old gnome installation and higher poll rate than normal. There used to be issues there..
    - nineteen999 2y ago
      It's AlmaLinux 9.4. Gnome 40.10.
      - yxhuvud 2y ago
        Yeah, that is old enough that those issues may not have been fixed yet. If you have a high poll rate mouse I'd strongly advise upgrading.
  - dpkirchner 2y ago
    Nvidia in my case, and I agree, the app performance is very poor. The easiest way to see the lag, IMO, is to open up the achievements list for a game and try to scroll around. It takes almost a second for mouse scroll wheel movements to have an effect. (Up to date mint, cinnamon)
  - nineteen999 2y ago
    It's a NUC with 00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics] Integrated graphics to be sure, but i'm usually only using it to stream games from PC or laptop and the performance is fine for that in game.
accrual 2y ago
This is really cool insight into both the Steam client and Linux programming. I understand why there may not be detailed release notes every release, but wow "Fixed some miscellaneous common crashes" is an understatement when you know about this work!
vlovich123 2y ago
Isn’t best practice to read all environment variables on boot and never use setenv? The only place where setenv would matter is for spawning new processes where you should probably be creating an new environ cloned from the current one and update the new values. Using getenv/setenv as an IPC messaging mechanism seems to be an opportunity for lots of issues aside from it historically not being multithreaded-safe on Linux and having all sorts of potential memory leaks hiding (which is what the post ignores when it says that it’s thread safe on MacOS).
- cryptonector 2y ago
  No. Because you might end up using a library that calls getenv(). Java does what you say, but it still presents problems for JNI code using libraries that want to getenv().
Aeolun 2y ago
My only issue with Steam at this point is that it’ll just randomly complain it has no connection, no matter which content server I set it to connect to. If I spam the ‘retry’ button it’ll eventually work, but it’s a massive PITA.
thrdbndndn 2y ago
> We removed the majority of setenv calls. It was mostly used when spawning processes Could someone elaborate this for a non-developer? Why would you use `setenv` (which I assume is functionally similar to `export key=value`, but correctly me if I'm wrong) (extensively) for spawning processes?
- fulafel 2y ago
  Environment variables are per-process, but inherited by child processes (not system wide). So when you start other programs you often set up env vars for them, or clear env vars that were used by your parent process to not inadvertantly perturb the child's behaviour.
  - bhawks 2y ago
    Yes the proper way to do that is execve, not by changing global variables in your parent process. https://man7.org/linux/man-pages/man2/execve.2.html https://man7.org/linux/man-pages/man2/execve.2.html Using setenv is mostly always a hack that relies on a bunch of assumptions that could easily change and be hard to debug.
    - ptsneves 2y ago
      Unless there is a fork exec family of syscalls are not child processes but the same process itself. Small detail but important because sometimes you really want a separate process altogether.
      - gpderetta 2y ago
        in that case you do not need setenv to pass values to the underlying process though. And if you are forking without immediately execv in a multithreaded program, setenv is the last of your problems. edit: what's probably happening is that execve is four or five abstraction layers deeper (possibly in a third party dependency) than where the env variable need to be set without a clean way to pass the values through.
    - 2y ago
bhawks 2y ago
FWIW the decision to leak memory on Mac actually goes back ~26 years to FreeBSD - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5604 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5604 which OSX inherited. I would not be surprised that Windows setenv has BSD roots due to licensing. 26 years ago people knew this API was broken but didn't fix it due to inertia of breaking buggy programs further. There really shouldn't be a need to change your own process's envvars. For subprocesses just use the proper exec function. For anything else there should be a clear API to call rather than changing a global variable and hoping some code far away from yours rereads it and handles things correctly.
- jandrese 2y ago
  I only partially disagree with the sentiment that it is "impossible to fix". For the current API that is true, but a fairly minor modification API would make it possible. All getenv() has to do is strdup() the return value before sending it back and leaving it on the programmer to free the memory when they are done with it. This does mean that the programmer will need to call getenv() again if they think the value might change, but I think that is a reasonable tradeoff. This change would make old programs leak memory every time they call getenv() without the subsequent free(), but since the current version also leaks memory that doesn't seem like a dealbreaker. As an added bonus the new version could be made thread safe by wrapping the strdup() in a mutex and doing similar work on the setenv() side.
  - bhawks 2y ago
    There exists a ton of working code that does not fiddle with setenv and now would have memory leaks if they don't change their code? Plus I would now need to test if my stdlib requires freeing memory or not because if I try to free on an older libc it is not going to work out well. I don't think the value works out. From a hygiene perspective - freeing the return of another API is an anti pattern. If you need the caller to release objects there should provide a FooLib_bar_destory(bar) or similar.
    - jandrese 2y ago
INTPenis 2y ago
The mere existence of Steam is astounding to someone who grew up playing nethack and chess on Linux. But the Steam client is really strange. Sometimes it works for months, and suddenly a game won't start, or something doesn't work, and I have to do weird stuff to get it working like purging all files or reinstalling. It doesn't make sense, it's like the Steam client rots.
- sunaookami 2y ago
  Steam is one of the worst software, it's frankly embarassing for such a huge company. It's a hotchpotch of different design principles and new features only get tacked on with a very different UI + UX. The client is mostly webviews but they are so slow and buggy and they use a very old insecure Chromium version under the hood. The launcher is still a 32 bit program. It's sad that other launchers are even worse though.
  - tapoxi 2y ago
    I checked and its using Chromium 126.0.6478.183 from July, which isn't that old.
    - sunaookami 2y ago
      It's four versions behind the current one with a lot of CVEs. There seems to be an LTS 126 but it's only for ChromeOS - it's at least behind this one which fixed one critical and two High CVEs: https://chromereleases.googleblog.com/2024/11/long-term-support-channel-update-for.html https://chromereleases.googleblog.com/2024/11/long-term-supp... Even being only one version behind is dangerous for such a huge attack vector.
  - 0x457 2y ago
    It seems like a lot of steam client issues come from weird desire to: 1) Keep it 32 bit 2) Have a single store-frontend At least, it came to a conclusion that it steam itself should be responsible for managing runtimes for games.
  - account42 2y ago
apatheticonion 2y ago
Would love an open source Steam client
DanielHB 2y ago
I can't wait to ditch windows for my tower PC.
WhyNotHugo 2y ago
I'm really curious why they're using setenv(3) so much. The main usages that I can think of is setting an environment variable before calling something like exec(3). That doesn't seem to be the case here. The article mentions that they use exevpe for spawning children processes. So what usages of setenv(3) would remain?
- Tobu 2y ago
  It is possible to check for setenv/unsetenv/putenv with nm -D, and a quick sample of my ~/.cargo/bin/* shows far too many programs using those. Yeah they could be single threaded, but who can guarantee they will remain so? Come to think of it listing symbols could detect pthread_create as well. I'd be interested in a way to do static binary analysis to get from those symbols to a call tree, as well. I don't see a way to check for **environ usage though, the compiler could turn this one into anything.
sonia598lewis 2y ago
[flagged]
russnes 2y ago
I love the steam client on linux these days, especially the compatibility for non-steam games is so great and Ive been using it to play WoW Classic while I have covid
deleted 2y ago
[deleted]
matheusmoreira 2y ago
I wish they'd make it more virtualization friendly. I don't want to run untrustworthy proprietary software on my main system. Common sandboxing mechanisms are insufficient since Steam and its games need access to the entire device tree anyway. Nothing short of a real virtual machine would do it for me. Will also make compatibility painless since I can just install the Linux distribution they support. I shopped around for computer parts with complete IOMMU support just so I could map the discrete GPU to the virtual machine and achieve near native performance... Only to discover they are exceendingly hostile to users who do this VFIO stuff. Just yet another reminder not to "buy" games on these platforms, I guess.
- lillecarl 2y ago
  > I shopped around for computer parts with complete IOMMU support just so I could map the discrete GPU to the virtual machine and achieve near native performance... So just any standard/decent motherboard bought within the last 3-4 years? > I wish they'd make it more virtualization friendly. Which games do you have issues with when virtualizing? I've only been locked out of Halo Infinite and I enable all hyperv flags in libvirt. > Just yet another reminder not to "buy" games on these platforms, I guess. If you buy a game on Steam and you're unable to play it they'll refund you straight away, which is why everyone loves Valve and Steam even when the client is crappy. I couldn't be happier with my setup, which I assume is similar to yours. (2 GPUs)
  - matheusmoreira 2y ago
    > So just any standard/decent motherboard bought within the last 3-4 years? It isn't at all obvious that IOMMU is supported by even current top of the line motherboards. I looked at a lot of products and have yet to see the VT-d and AMD-Vi keywords mentioned in technical specifications. To confirm support I had to read their UEFI firmware manuals and look for instructions on toggling the virtualization support. https://pcpartpicker.com/forums/topic/466120-ecc-ram-and-iommu-motherboard-support https://pcpartpicker.com/forums/topic/466120-ecc-ram-and-iom... At least ECC memory support has started to show up in technical specifications. Looks like IOMMU is not quite there yet. > Which games do you have issues with when virtualizing? From what I'm reading many games take virtualization as evidence of cheating and have no regard for false positives. I'm currently assuming any game with battleye or easy anti cheat will issue permanent bans on virtualization detection. > If you buy a game on Steam and you're unable to play it they'll refund you straight away Somehow that doesn't bring me much peace of mind.
Dwedit 2y ago
On Windows, "Environment" is stored in the Win32 Thread Information Block/Thread Environment Block (TIB/TEB), so it's thread-local rather than process-global.
AzzyHN 2y ago
All of this stuff goes way over my head. I'm on Pop!_OS and am happy to report "it just works" (tm), though it ignores scaling entirely.
arendtio 2y ago
Always cool to rediscover people via HN. This post reminded me of the work ttimo did for the Quake 3 engine more than two decades ago. I remember it because I read so many comments written by him (like 15 years ago): https://github.com/search?q=repo%3Aioquake%2Fioq3+ttimo&type=code https://github.com/search?q=repo%3Aioquake%2Fioq3+ttimo&type...

477 points by Venn1 2y ago

> If this can be addressed in glibc, it may involve a tradeoff on features, maybe an opt-in mechanism with a slight departure from the "impossible" POSIX spec. That's something we may pursue in the long term if we can propose something sensible. Yes please

jeffbee 2y ago

Maybe among the best decisions Java ever made was hiding setenv. You simply cannot set env vars in Java.

mschuster91 2y ago
Java in general tends to massively dislike environment variables. Which is a Good Thing, but unfortunately it makes dealing with proxy support so much harder - especially as there are just so damn many HTTP libraries that people use...
aidenn0 2y ago
So does that also mean you can't write e.g. a POSIX shell in Java?
- shakna 2y ago
  System.getenv exists, so getting is easy... But setting env vars generally involves a convoluted process. You can modify the JVM's buffer, or spawn a new ProcessBuilder, or a few other things. But it's a nasty place to be.
  - MBCook 2y ago
    Or make a little JNI class.
    - shakna 2y ago
      That doesn't work for preserving to child threads, unfortunately. Which is something people expect.
- bhawks 2y ago
  1) java shell starts 2) read getenv into a java hashmap 3) evaluate shell input - any sets update the hashmap, any gets read from the hashmap 4) when you get to a point where you need to exec a process you use the hashmap to pass envvars into process builder - which is going to call execvpe for you. So yeah you definitely could (there'd be other reasons why java wouldn't be a good choice). TBH reading that folks were calling setenv before an exec to propagate env vars to a child made me sad. I would guess that other use cases were leveraging environ as a poor man's global variable - which is also unfortunate.
- asveikau 2y ago
  execve(2) and friends allow you to specify a new processes's environment variables. How Java may choose to expose that I don't know. Assuming they do, that is all a shell needs. You do not strictly need to call setenv(3) to change a variable that gets evaluated with $, etc. You do need to pass the modified environment to a child process.
- lmm 2y ago
  You can pass a new environment to a process you spawn, which I think is adequate? You just can't modify the environment of your existing process.
- sunshowers 2y ago
  There are tons of reasons you can't write a POSIX shell in Java, starting from the lack of support for non-UTF-8 paths :)
NotACracker 2y ago
I admire your courage to say something positive about java. On another note, I believe Zig does the same thing.

o11c 2y ago

The real question is: is there any case where a program calls `setenv` in one thread and actually wants it to take effect in other already-existing threads? That said, GLIBC is pretty good at documenting all the dangerous functions, so it is possible to add locking/copying yourself.

fweimer 2y ago
Interesting idea. I strongly suspect that there are programs out there that expect that setenv changes the environ array (and they do not treat it as an opaque pointer passed to posix_spawn/execve). With a per-thread setenv, we would need a per-thread environ variable as well. Unfortunately, that's not really compatible with POSIX because environ is not declared in a header. Instead, programmers are expected to write a declaration extern char **environ; into their sources, and that declaration is incompatible with environ being a thread-local variable.
- o11c 2y ago
  Hm, in the end most of the problems do come down to stuff not coming from blessed headers. Regardless of anything else, how about: * deprecate direct access to `environ` and add functions to replace it. Have a macro that indicates this and provide a canonical compatibility shim for people to copy if they might use old libcs. * using linker magic, change the behavior of programs depending on whether they attempt to access `environ` or not, so old-API programs are still thread-unsafe but new ones are thread-safe. It's amazing how much you can do with the conditionally-linked object files from a static "library". Much of C's cross-TU "UB, no diagnostic required" is inexcusable since we can detect it quite easily with zero overhead (at least, for static linking) using today's linkers by deliberately causing multiple definition errors. Compatibility with old-ABI programs probably means fixing `environ` is not that simple, but you are libc and libc is in control of dynamic linking ...
  - ptsneves 2y ago
    All the inconsistencies you suggest sound like a trap, especially when you suggest ABI and API behaviour divergences. From what I understood of your macro idea this would lead to API changes that would lead to ifdefs for different (g)libcurl versions. Doesn’t feel good especially for software distributors. Also the changes you mention require changes in the linker scripts distributed by a variety of toolchains and would need to check if the libc target was a blessed one. In effect the deprecation would never move to obsolete and kept around forever. To clarify the above post libc is in control of dynamic linking through the dl*(dlopen) family of functions.

fweimer 2y ago

We've got patches under review: https://inbox.sourceware.org/libc-alpha/cover.1722193092.git.fweimer@redhat.com/ https://inbox.sourceware.org/libc-alpha/cover.1722193092.git... (triggered by https://issues.redhat.com/browse/RHEL-42410 https://issues.redhat.com/browse/RHEL-42410, a graphics stack stability issue that wasn't as visible in RHEL 9 for some reason) At least the first one (the getenv thread safety fix) will hopefully make it into glibc 2.41 and it should be quite safe to backport. It turns out that setenv is easier to handle because glibc already never frees environment strings. It's concurrent unsetenv that is rather tricky. Without some snapshot approach, getenv would return null pointers instead of environment variables values that are actually set. I don't want to introduce locking into getenv because getenv without setenv has been async-signal-safe for so long that it would likely break applications. The environ handling fixes are a bit more controversial because vfork+execve make it complicated to avoid memory leaks, but these further fixes are less important to the stability of the graphics stack.

Asooka 2y ago
What about using a linked list for variables added after the start of the process, which can be implemented atomically? Then once it gets "too long", a thread executing setenv could construct a new hashmap and replace the pointer to the old one with the pointer to the new one atomically, all without locking? To prevent two threads from rehashing at the same time, use an atomic flag for whether a thread is rehashing right now. That means if other threads call setenv in the meantime, the extra added variables would be appended to the list to be processed by a later setenv call (if one happens). That list could grow unbounded if enough threads call setenv quickly enough, but I think the simplicity of atomically swapping pointers might be worth it and setenv isn't called very frequently.
- jeffparsons 2y ago
  I like this approach, because it lets everything Just Work without anyone outside of the implementation having to think about it at all, and only incurs meaningful overhead if you're doing something really silly, but crucially still won't break — it'll just be slightly slow. That feels like the right trade-off.

Pannoniae 2y ago

To raise awareness: there's been a bug with the Linux Steam client which has been persistent for a long time. TL;DR: if you have Steam running for more than a ~day or so, you will run out of window handles so you won't be able to open any new graphical application/window until you restart Steam. Using Steam Chat appears to make the issue worse (it happens earlier). This has been documented under https://github.com/ValveSoftware/steam-for-linux/issues/9094 https://github.com/ValveSoftware/steam-for-linux/issues/9094 but for some reason that issue has been closed. I personally just restart Steam every day but if someone else encounters this issue and doesn't know why their windows are not opening, this is why :) I am using KDE/Wayland but I've observed this under X11 too.

LorenDB 2y ago
KDE Wayland user here. I have never observed this issue myself, and I leave Steam running all the time (although I never open Steam Chat).
- snvzz 2y ago
  Although, not albeit. You have a slave sentence there. These are incompatible with albeit.
  - LorenDB 2y ago
    Thanks, fixed! "Albeit" seemed like it would work, but "although" indeed fits much better.
    - soulofmischief 2y ago
      To put it another way, albeit shouldn't be used with independent clauses. 1. I leave Steam running all the time (although I never open Steam Chat). 2. I leave Steam running all the time (albeit never with Steam Chat). "I never open Steam Chat" can stand on its own as a sentence, but "never with Steam Chat" does not and thus can be appropriately modified with albeit.

snvzz 2y ago

Considering glibc's effort, I have to wonder what the other libc do, and whether they already implement something like this.

electromech 2y ago

Thank you! I deeply appreciate that Steam works so well on Linux these days. I don't take for granted the hard work happening behind the scenes to make that a reality for us.

Forgeties79 2y ago
I really hope they drop a full steam OS release again soon. I’d love to build a gaming tower around it. Might settle for bazzite but I’d like to do steam OS tbh
- lordofgibbons 2y ago
  What would that get you over any other distro with the steam software installed?
  - shortrounddev2 2y ago
    1st party support
  - zamadatix 2y ago
    There's a lot more to SteamOS than "it ran `apt install steam -y` for you". Even just having things like Gamescope running out of the box (which is more than just `apt install gamescope`) is a huge amount of headache set aside. Plus the guarantee when it doesn't work it's because of a bug rather than something you did or some incompatibility with your exact distro setup.
    - Forgeties79 2y ago
      Yeah you get it lol I just want to play my damn games and not tinker all the damn time!
    - akimbostrawman 2y ago
      >guarantee when it doesn't work it's because of a bug rather than something you did or some incompatibility with your exact distro setup. Flatpak fixes this problem. You don't need gamescope unless for stuff like scaling. The only few times you need to fiddle with anything on desktop is changing proton version or adding a launch variable from protondb.com (just like on steamdeck)

josephcsible 2y ago

> One of my colleagues rightly dubbed setenv "the worst Linux API". Is setenv really a Linux API, since it's neither defined by Linux (it's in POSIX) nor implemented by the Linux kernel (it's entirely in userspace)?

deleted 2y ago
[deleted]
amiga386 2y ago
Yes it is, if you program application software for Linux (and your other two supported platforms are Windows and macOS, as is the case for OP and their colleague who appear to work for Valve, supporting Steam) And the reason the colleague dubs it the worst has nothing to do with its specification in POSIX (which doesn't require the function to be thread-safe but also doesn't prevent it from being made thread-safe), but rather its specific implementation in glibc, which is the C library in use on all the Linux distributions that Valve support Steam on and all are equally dubbed as "Linux"
- deleted 2y ago
  [deleted]
WhyNotHugo 2y ago
Technically, this is a GNU libc limitation. A libc _could_ make this thread-safe. Languages like go just implement this feature internally and don't have this limitation. The limitation is contemplated in POSIX itself; from the posix documentation: > The setenv() function need not be thread-safe.
- deleted 2y ago
  [deleted]
- deathanatos 2y ago
  I think you get thread-safety and memory leaks (the article alludes to macOS taking this approach), or it's unsafe but doesn't leak. But that is inherent in the API, as specified by POSIX: setenv must necessarily either invalidate the pointer returned by getenv (which is cannot do safely) or leak it. Perhaps the leak is "better", in at least there won't be non-OOM crashes, but it still leaves a bad taste in one's mouth. For a long running program like Steam (that is for some odd reason calling setenv…?) … I'm not sure which is better. Better would be not calling setenv, which it sounds like they've worked on.

nineteen999 2y ago

The stability is fine for me, but the rendering performance in the steam client when the mouse is in the window is abysmal.

Vilian 2y ago
Nvidia?
- yxhuvud 2y ago
  Could just be really old gnome installation and higher poll rate than normal. There used to be issues there..
  - nineteen999 2y ago
    It's AlmaLinux 9.4. Gnome 40.10.
    - yxhuvud 2y ago
      Yeah, that is old enough that those issues may not have been fixed yet. If you have a high poll rate mouse I'd strongly advise upgrading.
- dpkirchner 2y ago
  Nvidia in my case, and I agree, the app performance is very poor. The easiest way to see the lag, IMO, is to open up the achievements list for a game and try to scroll around. It takes almost a second for mouse scroll wheel movements to have an effect. (Up to date mint, cinnamon)
- nineteen999 2y ago
  It's a NUC with 00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics] Integrated graphics to be sure, but i'm usually only using it to stream games from PC or laptop and the performance is fine for that in game.

accrual 2y ago

This is really cool insight into both the Steam client and Linux programming. I understand why there may not be detailed release notes every release, but wow "Fixed some miscellaneous common crashes" is an understatement when you know about this work!

vlovich123 2y ago

Isn’t best practice to read all environment variables on boot and never use setenv? The only place where setenv would matter is for spawning new processes where you should probably be creating an new environ cloned from the current one and update the new values. Using getenv/setenv as an IPC messaging mechanism seems to be an opportunity for lots of issues aside from it historically not being multithreaded-safe on Linux and having all sorts of potential memory leaks hiding (which is what the post ignores when it says that it’s thread safe on MacOS).

cryptonector 2y ago
No. Because you might end up using a library that calls getenv(). Java does what you say, but it still presents problems for JNI code using libraries that want to getenv().

Aeolun 2y ago

My only issue with Steam at this point is that it’ll just randomly complain it has no connection, no matter which content server I set it to connect to. If I spam the ‘retry’ button it’ll eventually work, but it’s a massive PITA.

thrdbndndn 2y ago

> We removed the majority of setenv calls. It was mostly used when spawning processes Could someone elaborate this for a non-developer? Why would you use `setenv` (which I assume is functionally similar to `export key=value`, but correctly me if I'm wrong) (extensively) for spawning processes?

fulafel 2y ago
Environment variables are per-process, but inherited by child processes (not system wide). So when you start other programs you often set up env vars for them, or clear env vars that were used by your parent process to not inadvertantly perturb the child's behaviour.
- bhawks 2y ago
  Yes the proper way to do that is execve, not by changing global variables in your parent process. https://man7.org/linux/man-pages/man2/execve.2.html https://man7.org/linux/man-pages/man2/execve.2.html Using setenv is mostly always a hack that relies on a bunch of assumptions that could easily change and be hard to debug.
  - ptsneves 2y ago
    Unless there is a fork exec family of syscalls are not child processes but the same process itself. Small detail but important because sometimes you really want a separate process altogether.
    - gpderetta 2y ago
      in that case you do not need setenv to pass values to the underlying process though. And if you are forking without immediately execv in a multithreaded program, setenv is the last of your problems. edit: what's probably happening is that execve is four or five abstraction layers deeper (possibly in a third party dependency) than where the env variable need to be set without a clean way to pass the values through.
  - 2y ago

bhawks 2y ago

FWIW the decision to leak memory on Mac actually goes back ~26 years to FreeBSD - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5604 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5604 which OSX inherited. I would not be surprised that Windows setenv has BSD roots due to licensing. 26 years ago people knew this API was broken but didn't fix it due to inertia of breaking buggy programs further. There really shouldn't be a need to change your own process's envvars. For subprocesses just use the proper exec function. For anything else there should be a clear API to call rather than changing a global variable and hoping some code far away from yours rereads it and handles things correctly.

jandrese 2y ago
I only partially disagree with the sentiment that it is "impossible to fix". For the current API that is true, but a fairly minor modification API would make it possible. All getenv() has to do is strdup() the return value before sending it back and leaving it on the programmer to free the memory when they are done with it. This does mean that the programmer will need to call getenv() again if they think the value might change, but I think that is a reasonable tradeoff. This change would make old programs leak memory every time they call getenv() without the subsequent free(), but since the current version also leaks memory that doesn't seem like a dealbreaker. As an added bonus the new version could be made thread safe by wrapping the strdup() in a mutex and doing similar work on the setenv() side.
- bhawks 2y ago
  There exists a ton of working code that does not fiddle with setenv and now would have memory leaks if they don't change their code? Plus I would now need to test if my stdlib requires freeing memory or not because if I try to free on an older libc it is not going to work out well. I don't think the value works out. From a hygiene perspective - freeing the return of another API is an anti pattern. If you need the caller to release objects there should provide a FooLib_bar_destory(bar) or similar.
  - jandrese 2y ago

INTPenis 2y ago

The mere existence of Steam is astounding to someone who grew up playing nethack and chess on Linux. But the Steam client is really strange. Sometimes it works for months, and suddenly a game won't start, or something doesn't work, and I have to do weird stuff to get it working like purging all files or reinstalling. It doesn't make sense, it's like the Steam client rots.

sunaookami 2y ago
Steam is one of the worst software, it's frankly embarassing for such a huge company. It's a hotchpotch of different design principles and new features only get tacked on with a very different UI + UX. The client is mostly webviews but they are so slow and buggy and they use a very old insecure Chromium version under the hood. The launcher is still a 32 bit program. It's sad that other launchers are even worse though.
- tapoxi 2y ago
  I checked and its using Chromium 126.0.6478.183 from July, which isn't that old.
  - sunaookami 2y ago
    It's four versions behind the current one with a lot of CVEs. There seems to be an LTS 126 but it's only for ChromeOS - it's at least behind this one which fixed one critical and two High CVEs: https://chromereleases.googleblog.com/2024/11/long-term-support-channel-update-for.html https://chromereleases.googleblog.com/2024/11/long-term-supp... Even being only one version behind is dangerous for such a huge attack vector.
- 0x457 2y ago
  It seems like a lot of steam client issues come from weird desire to: 1) Keep it 32 bit 2) Have a single store-frontend At least, it came to a conclusion that it steam itself should be responsible for managing runtimes for games.
- account42 2y ago

apatheticonion 2y ago

Would love an open source Steam client

DanielHB 2y ago

I can't wait to ditch windows for my tower PC.

WhyNotHugo 2y ago

I'm really curious why they're using setenv(3) so much. The main usages that I can think of is setting an environment variable before calling something like exec(3). That doesn't seem to be the case here. The article mentions that they use exevpe for spawning children processes. So what usages of setenv(3) would remain?

Tobu 2y ago
It is possible to check for setenv/unsetenv/putenv with nm -D, and a quick sample of my ~/.cargo/bin/* shows far too many programs using those. Yeah they could be single threaded, but who can guarantee they will remain so? Come to think of it listing symbols could detect pthread_create as well. I'd be interested in a way to do static binary analysis to get from those symbols to a call tree, as well. I don't see a way to check for **environ usage though, the compiler could turn this one into anything.

sonia598lewis 2y ago

[flagged]

russnes 2y ago

I love the steam client on linux these days, especially the compatibility for non-steam games is so great and Ive been using it to play WoW Classic while I have covid

deleted 2y ago

[deleted]

matheusmoreira 2y ago

I wish they'd make it more virtualization friendly. I don't want to run untrustworthy proprietary software on my main system. Common sandboxing mechanisms are insufficient since Steam and its games need access to the entire device tree anyway. Nothing short of a real virtual machine would do it for me. Will also make compatibility painless since I can just install the Linux distribution they support. I shopped around for computer parts with complete IOMMU support just so I could map the discrete GPU to the virtual machine and achieve near native performance... Only to discover they are exceendingly hostile to users who do this VFIO stuff. Just yet another reminder not to "buy" games on these platforms, I guess.

lillecarl 2y ago
> I shopped around for computer parts with complete IOMMU support just so I could map the discrete GPU to the virtual machine and achieve near native performance... So just any standard/decent motherboard bought within the last 3-4 years? > I wish they'd make it more virtualization friendly. Which games do you have issues with when virtualizing? I've only been locked out of Halo Infinite and I enable all hyperv flags in libvirt. > Just yet another reminder not to "buy" games on these platforms, I guess. If you buy a game on Steam and you're unable to play it they'll refund you straight away, which is why everyone loves Valve and Steam even when the client is crappy. I couldn't be happier with my setup, which I assume is similar to yours. (2 GPUs)
- matheusmoreira 2y ago
  > So just any standard/decent motherboard bought within the last 3-4 years? It isn't at all obvious that IOMMU is supported by even current top of the line motherboards. I looked at a lot of products and have yet to see the VT-d and AMD-Vi keywords mentioned in technical specifications. To confirm support I had to read their UEFI firmware manuals and look for instructions on toggling the virtualization support. https://pcpartpicker.com/forums/topic/466120-ecc-ram-and-iommu-motherboard-support https://pcpartpicker.com/forums/topic/466120-ecc-ram-and-iom... At least ECC memory support has started to show up in technical specifications. Looks like IOMMU is not quite there yet. > Which games do you have issues with when virtualizing? From what I'm reading many games take virtualization as evidence of cheating and have no regard for false positives. I'm currently assuming any game with battleye or easy anti cheat will issue permanent bans on virtualization detection. > If you buy a game on Steam and you're unable to play it they'll refund you straight away Somehow that doesn't bring me much peace of mind.

Dwedit 2y ago

On Windows, "Environment" is stored in the Win32 Thread Information Block/Thread Environment Block (TIB/TEB), so it's thread-local rather than process-global.

AzzyHN 2y ago

All of this stuff goes way over my head. I'm on Pop!_OS and am happy to report "it just works" (tm), though it ignores scaling entirely.

arendtio 2y ago

Always cool to rediscover people via HN. This post reminded me of the work ttimo did for the Quake 3 engine more than two decades ago. I remember it because I read so many comments written by him (like 15 years ago): https://github.com/search?q=repo%3Aioquake%2Fioq3+ttimo&type=code https://github.com/search?q=repo%3Aioquake%2Fioq3+ttimo&type...