11 ms·

Fedora 38 LLVM vs. Team Fortress 2

145 points by st_goliath 3y ago

amluto 3y ago
It should be straightforward to make a little LD_PRELOAD shim to implement the new operator new on top of old overloads and thus restore proper functioning. It would be a gross kludge, though.
- olliej 3y ago
  I'm not sure that's sound. You can't just redirect an aligned new to the unaligned operator new as you may get unaligned result. It _sounds_ like what is happening is a = ::operator new(some size, some alignment) ... ::operator delete(a); where delete is dropping the align_val_t parameter that would guarantee it hits the same allocator family. There are a variety of ways this can happen, and let's just take it as given that it is. The problem is that if operator new(size_t, align_val_t) is called then the struct has an alignment annotation. That can lead to codegen that reasonably assumes alignment, even without any source level decisions that depend on alignment. The result of having some equivalent of (either at runtime or link time) void * operator new(size_t sz, align_val_t a) { if (operator new(size_t) has been overridden) return ::operator new(sz); ... } could be an "aligned" allocation returning an unaligned value, causing crashes later on.
  - jenadine 3y ago
    That's not sound in general, but it is "probably" going to work for this specific case because the previous build was build with allocator that did not support this alignment, meaning that they did not need extra alignment. This is pretty rare actually. And you had anyway to use a custom allocator already with previous C++ versions to make it work.
    - olliej 3y ago
      While I do agree with you, and think it's probably worth seeing if detecting the override and falling back to unaligned allocation works, the problem is not that the code in TF is compiling assuming/requiring over aligned data. The problem is that there is system code that they are calling that is making using of over aligned allocation, so therefore could be generating code dependent on said alignment. The failure mode can very easily be someSystemLibrary.so`someFunction: alignedThing = ::operator new(size, alignment) ... i_dunno_dma_memcpy_or_something(a, somewhere else) ... ::operator delete(a) With no interaction with TF code at all. Except TF has replaced operator delete so that fails due to the allocator mismatch. If you make ::opeator new(size_t, align_val_t) redirect to ::operator new(size_t) if it detects an override then the aligned operation can fail. The above example is moderately difficult to induce so it's more likely that there's an explicit split with the system is doing one half of new/delete and TF is doing the other, but the important thing is that it implies the system code is built aware of alignment and it depends on the alignment even if TF does not.
  - viraptor 3y ago
    If you don't mind wasting a bit of time, you could forward size+alignment to the allocator, return the aligned version and keep a record of aligned-to-allocation mapping. (For freeing later) But as the other comment mentioned - it should be a problem for tf2 in the first place since that's not the behaviour they're after.
    - olliej 3y ago
      > If you don't mind wasting a bit of time, you could forward size+alignment to the allocator, return the aligned version and keep a record of aligned-to-allocation mapping. (For freeing later) I'm unsure what you're proposing here - the only methods you know in the replacement allocator are operator new(size_t) and operator delete(void). The two possible failure paths are: a = ::operator new(some size) ... ::operator delete(a, alignment) and a = ::operator new(some size, some alignment) ... ::operator delete(a) In the first case what you could do is say "if I did not allocator this pointer, optimistically forward it to operator delete(void)", in the latter case you can identify that a different operator new(size_t) exists but you have no idea how to make that allocator produce an aligned allocation. What I guess you could do is round the size up to a multiple of the specified alignment, and then just repeatedly allocate in the hope that you will eventually get a correctly aligned value out. But that would not be guaranteed.
      - viraptor 3y ago
        > and then just repeatedly allocate in the hope that you will eventually get a correctly aligned value out If you preload something that patches all the new/delete interfaces, you can do this without guesswork. new(size, alignment) -> res=alloc(size+alignment) res_aligned=res+... offsets[res_aligned] = res new(size) -> alloc(size) free(ptr) -> free(offsets[ptr] || ptr) offsets.del(ptr)
        olliej 3y ago
        Haha, you've missed the issue. The question is what does the system do when someone overrides the builtin allocator functions, but does not override all of them. You are absolutely correct that as a developer you can have your process override the allocator functions, and that is in fact what TF has done. The problem is that they have not overridden all of the allocation functions, and so they're crashing due to mismatching allocators being used. TF2 can "easily" fix this crash by implementing the aligned new, new[], delete, and delete[] operators in their custom allocator, or by simply removing their custom allocator's override of the global new & delete operators and using a common base class to get their faster allocator. The question we're talking about is "how does the standard library respond to this scenario in a way that maximizes correctness?".
  - Asooka 3y ago
    The C interface for aligned memory allocation is aligned_alloc(). The returned pointers are always freed with free(). So what is probably happening is that aligned new calls aligned_alloc(), and then aligned delete simply calls the regular delete, expecting to end up in free(), which by design should work with both kinds of pointers. I think the problem here is partly with the implementation of aligned new/delete. Since one is free to override only the old versions, the ones supplied by the standard library should make sure not to fall back to functions that may be partially overriden.
  - amluto 3y ago
    As pure speculation, one could forward to aligned_alloc and still free with ::delete. I haven’t tested this, nor have I looked at the code.
- zokier 3y ago
  LD_PRELOAD would probably run afoul with VAC though?
  - Polycryptus 3y ago
    Steam on Linux already uses LD_PRELOAD under-the-hood to load the overlay. Valve signs the overlay SO files, so they could be making an exception for Valve-signed-preloads in VAC, but it's also possible that VAC does something else to check for suspicious libraries loaded in.
stryan 3y ago
Valve does this for a couple of their games, see a similar issue with Dota 2[0]. [0] https://github.com/ValveSoftware/Dota-2/issues/2285 https://github.com/ValveSoftware/Dota-2/issues/2285
olliej 3y ago
This is a predictable outcome of overriding the global operator new. It remains annoying that this was ever allowed, and is a constant source of pain for c++ standard library implementations.
- phkahler 3y ago
  It seems more like the app and driver are mixing their new/delete pairs. That seems like a bug to me. Maybe even an API design issue if it's supposed to happen.
- DannyBee 3y ago
  It actually should still work, since fedora38 includes the llvm15 versioned libs. The only way to make this break is if something is loading random unversioned solibs or whatever the latest one it can find is, and expecting this to work forever. If it actually used a versioned solib, it would get llvm 15 just like it did before. This is the whole point of versioned solibs.
  - olliej 3y ago
    Versioning does not solve the problem. The aligned allocation operators have existed since llvm 8.x. The problem is not that the aligned allocation APIs are new. The problem is that TCMalloc is only partially replacing the global allocation APIs, it's just taken until this year for that bug to be exposed. What has happened is presumably some part of the OS has updated its target C++ version so is now using the aligned allocators, which exposes the gap in TCMalloc. I'm not sure if the spec explicitly allows an aligned allocation to be fed into an unaligned operator delete, but it seems like implementations do, so that's probably why adopting aligned operator new wouldn't be seen as an ABI break.
    - DannyBee 3y ago
      Thanks. This at least makes more sense than what I read, and explains what might actually have happened :)
IceWreck 3y ago
Isn't this the reason why people recommend using the flatpak version of Steam ?
- PlutoIsAPlanet 3y ago
  Yes, especially on Fedora. This isn't something Fedora is doing wrong, unfortunately some games build against older libraries or are built against Debian/Ubuntu and the Flatpak runtimes generally have better compatibility.
sosodev 3y ago
It's unfortunate but the Steam experience on Linux seems to be progressively getting worse (outside of Steam Deck ofc). The Steam client is often borderline unusable for Linux users. You can find many issue threads on GitHub reporting client freezes and crashes. It seems like a big part of the issues is a lack of maintenance. TF2 would actually run better on Linux via Proton but VAC isn't enabled so you can't join the vast majority of servers. Valve also has existing Source engine tooling that allows Linux ports to drop OpenGL entirely (dxvk-native as used by Portal 2 and L4D2) but they haven't added it to TF2... :(
- Entinel 3y ago
  Devil's advocate, I use Steam on Fedora and have had 0 issues. Very rarely freezes or crashes. It's probably the most stable application I use daily.
  - 2OEH8eoCRo0 3y ago
    I use Steam on Fedora as well and I notice a lot of jank with the Steam client (Nvidia 1080ti). Dropdown menus popping through windows, sound may or may not work for videos, freezing, etc. It's usable but it's not very pleasant.
    - Entinel 3y ago
      Just for GPU comparison I'm on an AMD RX card so it could be an Nvidia issue which is known to be jank on Linux.
  - sosodev 3y ago
    It seems to work perfectly for some people. I’ve regularly had issues with the client not rendering at all, freezing, and crashing on Pop_OS 22.04 LTS with an nvidia GTX 1660ti.
  - WaffleIronMaker 3y ago
    I also use Steam on Fedora, and I've not had any issues with Stardew Valley, Factorio, Celeste, N++, Undertale, and others. I remember having a brief issue with Portal, but I was able to resolve it. Overall, I've had a good experience.
mmh0000 3y ago
I loved the premise of the article, though I really wish the author had gone into detail about how he discovered the root cause.
admax88qqq 3y ago
Unfortunately this is exactly the type of stuff that makes supporting commercial apps on linux a nightmare. Weird crashes due to weird linking of system libraries. Common distros are very adamant about dynamic linking everything in order to support the use case of "core library has vulnerability, upgrade it in place without rebuilding consuming apps." Along with a desire to avoid "dll hell" and force a single canonical version of every library systemwide. This leads to these sorts of issues. Windows gets around it by letting applications put the DLLs they care about beside the executable, and having it check there first by default.
- gabcoh 3y ago
  Can Linux not trivially do the same thing as windows with LD_PRELOAD? If so why is this more of an issue on Linux than Windows? Is it really less a technical challenge and more just a matter of Linux getting less support from upstream developers?
  - bravetraveler 3y ago
    I was thinking/wondering this myself. Not to reinvent the wheel - more toss an idea around, but a 'venv for LD_PRELOAD' sounds like it'd deal with this pretty handily Not... in a way I'd use as a distribution/release maintainer. Probably as an administrator [of my LAN]
    - gabcoh 3y ago
      Such things already exist. Eg. Appimage or even docker.
      - bravetraveler 3y ago
        Right, but I don't really want to get into a distribution model - the hack suits me fine :) More an exercise in curiosity than anything Flatpak (or Snap, ew) probably deals with it fine today, Steam's there
      - lnxg33k1 3y ago
DannyBee 3y ago
Fedora 38 includes the LLVM15 libs to maintain backwards compatibility. Why is this automatically using a new, incompatible solib, instead of a versioned solib?
- AnssiH 3y ago
  The LLVM dependency is in the HW-specific driver solib which is loaded by the OpenGL library, which has the same soname as before.
  - DannyBee 3y ago
    Okay, then why does it fail when it should then still use llvm15? The author states this is an llvm16 issue, but unless the driver was built against llvm16, it should still be loading llvm15. If it was built against llvm16 (or loads llvm16), and doesn't work, that's not a failure of anything other than QA testing.
exabrial 3y ago
I thought TF2 was pretty much 100% hacked... like no legit non-hackers playing except at LAN parties.
- pnpnp 3y ago
  There are bots, but they’re easily avoidable in community servers.
- chrisdalke 3y ago
  It's still got a small but vibrant community in the community-run servers (not accessed through matchmaking). These are typically hand-moderated.
- themoonisachees 3y ago
  I played for a few hours just yesterday and a few bots joined but people are proactive at kicking them. Community servers are also thriving.
- exabrial 3y ago
  holy crap, I didn't expect downvotes for asking such a simple question. I actually used to enjoy tf2, but the hacking became unbearable.
Karliss 3y ago
Whole graphics drivers using LLVM in the backend has caused countless issues. The way I look at it one of the main problems is that graphic API libraries shouldn't leak symbols from implementation details like them using LLVM. They should expose only the graphics API and nothing more.
- vchuravy 3y ago
  Don't ask me about GNU_UNIQUE... Due to some wonderful C++ features the dynamic linker is forced to unify symbols across shared libraries, even if those symbols have different versions. This utterly breaks loading multiple libLLVM's except if you build the copy you care about with -no-gnu-unique (or whatever the flag was called) I have seen wonderful things like the initializers of an already loaded libLLVM being rerun when a new one is loaded.
  - planede 3y ago
    The presume wonderful C++ feature is spelled __attribute__((weak)) in GNU C.
AndyKelley 3y ago
related: this talk that I made 2 years ago about an experiment to ship static executables on linux that could do graphics: https://www.youtube.com/watch?v=pq1XqP4-qOo https://www.youtube.com/watch?v=pq1XqP4-qOo
boutique 3y ago
Funnily enough, on Half-Life 1 engine-based games (i.e. the engine that came before HL2 - on which Team Fortress 2 runs; such as Counter-Strike 1.6), a different allocator problem exists -- glibc's malloc() just decides to fail miserably[0] on some setups. [0] https://github.com/ValveSoftware/halflife/issues/3158 https://github.com/ValveSoftware/halflife/issues/3158
- kimixa 3y ago
  that's exactly the sort of error you get if something has written just out of bounds on a malloc'd chunk - it clobbers the allocator's internal state, which appears to be what that assert() is checking. It's probably an allocation before the failing one that is being misued - so the backtrace pointing to openal doesn't necessarily mean it's openal's fault. Running with valgrind or another heap memory checking tool will probably be helpful to track down that particular linked bug. EDIT: It looks like that there's at least one out-of-bounds write when starting up half life (On arch linux, so maybe slightly different library versions and not loaded the counterstrike mod). It looks like a valve bug - writing 2 bytes at index [30] of a malloc'd size of 31 goes one byte over, and it looks from the backtrace it's all valve's code and not deep in some library that might have been loaded in. Writing 2 bytes to a string is a bit odd, perhaps it's trying to null-terminate but somehow uses a wstring null? Or some attempt at SIMD that isn't correctly bound? It doesn't seem to crash for me though, it might just be luck that nothing important is put 1 byte over, and it feels a bit unlikely something would be due to allocation and type alignment requirements, but it's perfectly valid for the malloc implementation to keep something important in that byte. Or perhaps there's some other dynamics that change this - it looks like it's doing stuff with paths, so may change size (of the allocation or even the amount written) based on where the steam app is installed - stuff like your user name length changing that may be the difference between a crash. Or even another issue somewhere else I didn't see, or valgrind didn't catch. Just goes to show how many games ship for years with "big" bugs :P For reference: ==27467== Invalid write of size 2 ==27467== at 0x406526A: GetSteamContentPath() (pathmatch.cpp:523) ==27467== by 0x4065927: pathmatch(char const*, char\*, bool, char*, unsigned int) [clone .part.1] (pathmatch.cpp:594) ==27467== by 0x4066849: pathmatch (pathmatch.cpp:541) ==27467== by 0x4066849: CWrap (pathmatch.cpp:685) ==27467== by 0x4066849: __wrap___xstat (pathmatch.cpp:907) ==27467== by 0x406294A: stat (stat.h:455) ==27467== by 0x406294A: CFileSystem_Stdio::FS_stat(char const*, stat*) (FileSystem_Stdio.cpp:225) ==27467== by 0x4060819: CBaseFileSystem::AddPackFiles(char const*) (BaseFileSystem.cpp:1325) ==27467== by 0x4060AA4: CBaseFileSystem::AddSearchPathInternal(char const*, char const*, bool) (BaseFileSystem.cpp:254) ==27467== by 0x4060B37: CBaseFileSystem::AddSearchPath(char const*, char const*) (BaseFileSystem.cpp:186) ==27467== by 0x8049003: main (launcher.cpp:413) ==27467== Address 0x45e5f4e is 30 bytes inside a block of size 31 alloc'd ==27467== at 0x4041714: malloc (vg_replace_malloc.c:393) ==27467== by 0x4357C4A: strdup (strdup.c:42) ==27467== by 0x42F1A76: realpath_stk (canonicalize.c:410) ==27467== by 0x42F1A76: realpath@@GLIBC_2.3 (canonicalize.c:432) ==27467== by 0x406525B: GetSteamContentPath() (pathmatch.cpp:520) ==27467== by 0x4065927: pathmatch(char const*, char\*, bool, char*, unsigned int) [clone .part.1] (pathmatch.cpp:594) ==27467== by 0x4066849: pathmatch (pathmatch.cpp:541) ==27467== by 0x4066849: CWrap (pathmatch.cpp:685) ==27467== by 0x4066849: __wrap___xstat (pathmatch.cpp:907) ==27467== by 0x406294A: stat (stat.h:455) ==27467== by 0x406294A: CFileSystem_Stdio::FS_stat(char const*, stat*) (FileSystem_Stdio.cpp:225) ==27467== by 0x4060819: CBaseFileSystem::AddPackFiles(char const*) (BaseFileSystem.cpp:1325) ==27467== by 0x4060AA4: CBaseFileSystem::AddSearchPathInternal(char const*, char const*, bool) (BaseFileSystem.cpp:254) ==27467== by 0x4060B37: CBaseFileSystem::AddSearchPath(char const*, char const*) (BaseFileSystem.cpp:186) ==27467== by 0x8049003: main (launcher.cpp:413)

145 points by st_goliath 3y ago

It should be straightforward to make a little LD_PRELOAD shim to implement the new operator new on top of old overloads and thus restore proper functioning. It would be a gross kludge, though.

olliej 3y ago
I'm not sure that's sound. You can't just redirect an aligned new to the unaligned operator new as you may get unaligned result. It _sounds_ like what is happening is a = ::operator new(some size, some alignment) ... ::operator delete(a); where delete is dropping the align_val_t parameter that would guarantee it hits the same allocator family. There are a variety of ways this can happen, and let's just take it as given that it is. The problem is that if operator new(size_t, align_val_t) is called then the struct has an alignment annotation. That can lead to codegen that reasonably assumes alignment, even without any source level decisions that depend on alignment. The result of having some equivalent of (either at runtime or link time) void * operator new(size_t sz, align_val_t a) { if (operator new(size_t) has been overridden) return ::operator new(sz); ... } could be an "aligned" allocation returning an unaligned value, causing crashes later on.
- jenadine 3y ago
  That's not sound in general, but it is "probably" going to work for this specific case because the previous build was build with allocator that did not support this alignment, meaning that they did not need extra alignment. This is pretty rare actually. And you had anyway to use a custom allocator already with previous C++ versions to make it work.
  - olliej 3y ago
    While I do agree with you, and think it's probably worth seeing if detecting the override and falling back to unaligned allocation works, the problem is not that the code in TF is compiling assuming/requiring over aligned data. The problem is that there is system code that they are calling that is making using of over aligned allocation, so therefore could be generating code dependent on said alignment. The failure mode can very easily be someSystemLibrary.so`someFunction: alignedThing = ::operator new(size, alignment) ... i_dunno_dma_memcpy_or_something(a, somewhere else) ... ::operator delete(a) With no interaction with TF code at all. Except TF has replaced operator delete so that fails due to the allocator mismatch. If you make ::opeator new(size_t, align_val_t) redirect to ::operator new(size_t) if it detects an override then the aligned operation can fail. The above example is moderately difficult to induce so it's more likely that there's an explicit split with the system is doing one half of new/delete and TF is doing the other, but the important thing is that it implies the system code is built aware of alignment and it depends on the alignment even if TF does not.
- viraptor 3y ago
  If you don't mind wasting a bit of time, you could forward size+alignment to the allocator, return the aligned version and keep a record of aligned-to-allocation mapping. (For freeing later) But as the other comment mentioned - it should be a problem for tf2 in the first place since that's not the behaviour they're after.
  - olliej 3y ago
    > If you don't mind wasting a bit of time, you could forward size+alignment to the allocator, return the aligned version and keep a record of aligned-to-allocation mapping. (For freeing later) I'm unsure what you're proposing here - the only methods you know in the replacement allocator are operator new(size_t) and operator delete(void). The two possible failure paths are: a = ::operator new(some size) ... ::operator delete(a, alignment) and a = ::operator new(some size, some alignment) ... ::operator delete(a) In the first case what you could do is say "if I did not allocator this pointer, optimistically forward it to operator delete(void)", in the latter case you can identify that a different operator new(size_t) exists but you have no idea how to make that allocator produce an aligned allocation. What I guess you could do is round the size up to a multiple of the specified alignment, and then just repeatedly allocate in the hope that you will eventually get a correctly aligned value out. But that would not be guaranteed.
    - viraptor 3y ago
      > and then just repeatedly allocate in the hope that you will eventually get a correctly aligned value out If you preload something that patches all the new/delete interfaces, you can do this without guesswork. new(size, alignment) -> res=alloc(size+alignment) res_aligned=res+... offsets[res_aligned] = res new(size) -> alloc(size) free(ptr) -> free(offsets[ptr] || ptr) offsets.del(ptr)
      - olliej 3y ago
        Haha, you've missed the issue. The question is what does the system do when someone overrides the builtin allocator functions, but does not override all of them. You are absolutely correct that as a developer you can have your process override the allocator functions, and that is in fact what TF has done. The problem is that they have not overridden all of the allocation functions, and so they're crashing due to mismatching allocators being used. TF2 can "easily" fix this crash by implementing the aligned new, new[], delete, and delete[] operators in their custom allocator, or by simply removing their custom allocator's override of the global new & delete operators and using a common base class to get their faster allocator. The question we're talking about is "how does the standard library respond to this scenario in a way that maximizes correctness?".
- Asooka 3y ago
  The C interface for aligned memory allocation is aligned_alloc(). The returned pointers are always freed with free(). So what is probably happening is that aligned new calls aligned_alloc(), and then aligned delete simply calls the regular delete, expecting to end up in free(), which by design should work with both kinds of pointers. I think the problem here is partly with the implementation of aligned new/delete. Since one is free to override only the old versions, the ones supplied by the standard library should make sure not to fall back to functions that may be partially overriden.
- amluto 3y ago
  As pure speculation, one could forward to aligned_alloc and still free with ::delete. I haven’t tested this, nor have I looked at the code.
zokier 3y ago
LD_PRELOAD would probably run afoul with VAC though?
- Polycryptus 3y ago
  Steam on Linux already uses LD_PRELOAD under-the-hood to load the overlay. Valve signs the overlay SO files, so they could be making an exception for Valve-signed-preloads in VAC, but it's also possible that VAC does something else to check for suspicious libraries loaded in.

stryan 3y ago

Valve does this for a couple of their games, see a similar issue with Dota 2[0]. [0] https://github.com/ValveSoftware/Dota-2/issues/2285 https://github.com/ValveSoftware/Dota-2/issues/2285

olliej 3y ago

This is a predictable outcome of overriding the global operator new. It remains annoying that this was ever allowed, and is a constant source of pain for c++ standard library implementations.

phkahler 3y ago
It seems more like the app and driver are mixing their new/delete pairs. That seems like a bug to me. Maybe even an API design issue if it's supposed to happen.
DannyBee 3y ago
It actually should still work, since fedora38 includes the llvm15 versioned libs. The only way to make this break is if something is loading random unversioned solibs or whatever the latest one it can find is, and expecting this to work forever. If it actually used a versioned solib, it would get llvm 15 just like it did before. This is the whole point of versioned solibs.
- olliej 3y ago
  Versioning does not solve the problem. The aligned allocation operators have existed since llvm 8.x. The problem is not that the aligned allocation APIs are new. The problem is that TCMalloc is only partially replacing the global allocation APIs, it's just taken until this year for that bug to be exposed. What has happened is presumably some part of the OS has updated its target C++ version so is now using the aligned allocators, which exposes the gap in TCMalloc. I'm not sure if the spec explicitly allows an aligned allocation to be fed into an unaligned operator delete, but it seems like implementations do, so that's probably why adopting aligned operator new wouldn't be seen as an ABI break.
  - DannyBee 3y ago
    Thanks. This at least makes more sense than what I read, and explains what might actually have happened :)

IceWreck 3y ago

Isn't this the reason why people recommend using the flatpak version of Steam ?

PlutoIsAPlanet 3y ago
Yes, especially on Fedora. This isn't something Fedora is doing wrong, unfortunately some games build against older libraries or are built against Debian/Ubuntu and the Flatpak runtimes generally have better compatibility.

sosodev 3y ago

It's unfortunate but the Steam experience on Linux seems to be progressively getting worse (outside of Steam Deck ofc). The Steam client is often borderline unusable for Linux users. You can find many issue threads on GitHub reporting client freezes and crashes. It seems like a big part of the issues is a lack of maintenance. TF2 would actually run better on Linux via Proton but VAC isn't enabled so you can't join the vast majority of servers. Valve also has existing Source engine tooling that allows Linux ports to drop OpenGL entirely (dxvk-native as used by Portal 2 and L4D2) but they haven't added it to TF2... :(

Entinel 3y ago
Devil's advocate, I use Steam on Fedora and have had 0 issues. Very rarely freezes or crashes. It's probably the most stable application I use daily.
- 2OEH8eoCRo0 3y ago
  I use Steam on Fedora as well and I notice a lot of jank with the Steam client (Nvidia 1080ti). Dropdown menus popping through windows, sound may or may not work for videos, freezing, etc. It's usable but it's not very pleasant.
  - Entinel 3y ago
    Just for GPU comparison I'm on an AMD RX card so it could be an Nvidia issue which is known to be jank on Linux.
- sosodev 3y ago
  It seems to work perfectly for some people. I’ve regularly had issues with the client not rendering at all, freezing, and crashing on Pop_OS 22.04 LTS with an nvidia GTX 1660ti.
- WaffleIronMaker 3y ago
  I also use Steam on Fedora, and I've not had any issues with Stardew Valley, Factorio, Celeste, N++, Undertale, and others. I remember having a brief issue with Portal, but I was able to resolve it. Overall, I've had a good experience.

mmh0000 3y ago

I loved the premise of the article, though I really wish the author had gone into detail about how he discovered the root cause.

admax88qqq 3y ago

Unfortunately this is exactly the type of stuff that makes supporting commercial apps on linux a nightmare. Weird crashes due to weird linking of system libraries. Common distros are very adamant about dynamic linking everything in order to support the use case of "core library has vulnerability, upgrade it in place without rebuilding consuming apps." Along with a desire to avoid "dll hell" and force a single canonical version of every library systemwide. This leads to these sorts of issues. Windows gets around it by letting applications put the DLLs they care about beside the executable, and having it check there first by default.

gabcoh 3y ago
Can Linux not trivially do the same thing as windows with LD_PRELOAD? If so why is this more of an issue on Linux than Windows? Is it really less a technical challenge and more just a matter of Linux getting less support from upstream developers?
- bravetraveler 3y ago
  I was thinking/wondering this myself. Not to reinvent the wheel - more toss an idea around, but a 'venv for LD_PRELOAD' sounds like it'd deal with this pretty handily Not... in a way I'd use as a distribution/release maintainer. Probably as an administrator [of my LAN]
  - gabcoh 3y ago
    Such things already exist. Eg. Appimage or even docker.
    - bravetraveler 3y ago
      Right, but I don't really want to get into a distribution model - the hack suits me fine :) More an exercise in curiosity than anything Flatpak (or Snap, ew) probably deals with it fine today, Steam's there
    - lnxg33k1 3y ago

DannyBee 3y ago

Fedora 38 includes the LLVM15 libs to maintain backwards compatibility. Why is this automatically using a new, incompatible solib, instead of a versioned solib?

AnssiH 3y ago
The LLVM dependency is in the HW-specific driver solib which is loaded by the OpenGL library, which has the same soname as before.
- DannyBee 3y ago
  Okay, then why does it fail when it should then still use llvm15? The author states this is an llvm16 issue, but unless the driver was built against llvm16, it should still be loading llvm15. If it was built against llvm16 (or loads llvm16), and doesn't work, that's not a failure of anything other than QA testing.

exabrial 3y ago

I thought TF2 was pretty much 100% hacked... like no legit non-hackers playing except at LAN parties.

pnpnp 3y ago
There are bots, but they’re easily avoidable in community servers.
chrisdalke 3y ago
It's still got a small but vibrant community in the community-run servers (not accessed through matchmaking). These are typically hand-moderated.
themoonisachees 3y ago
I played for a few hours just yesterday and a few bots joined but people are proactive at kicking them. Community servers are also thriving.
exabrial 3y ago
holy crap, I didn't expect downvotes for asking such a simple question. I actually used to enjoy tf2, but the hacking became unbearable.

Karliss 3y ago

Whole graphics drivers using LLVM in the backend has caused countless issues. The way I look at it one of the main problems is that graphic API libraries shouldn't leak symbols from implementation details like them using LLVM. They should expose only the graphics API and nothing more.

vchuravy 3y ago
Don't ask me about GNU_UNIQUE... Due to some wonderful C++ features the dynamic linker is forced to unify symbols across shared libraries, even if those symbols have different versions. This utterly breaks loading multiple libLLVM's except if you build the copy you care about with -no-gnu-unique (or whatever the flag was called) I have seen wonderful things like the initializers of an already loaded libLLVM being rerun when a new one is loaded.
- planede 3y ago
  The presume wonderful C++ feature is spelled __attribute__((weak)) in GNU C.

AndyKelley 3y ago

related: this talk that I made 2 years ago about an experiment to ship static executables on linux that could do graphics: https://www.youtube.com/watch?v=pq1XqP4-qOo https://www.youtube.com/watch?v=pq1XqP4-qOo

boutique 3y ago

Funnily enough, on Half-Life 1 engine-based games (i.e. the engine that came before HL2 - on which Team Fortress 2 runs; such as Counter-Strike 1.6), a different allocator problem exists -- glibc's malloc() just decides to fail miserably[0] on some setups. [0] https://github.com/ValveSoftware/halflife/issues/3158 https://github.com/ValveSoftware/halflife/issues/3158

kimixa 3y ago
that's exactly the sort of error you get if something has written just out of bounds on a malloc'd chunk - it clobbers the allocator's internal state, which appears to be what that assert() is checking. It's probably an allocation before the failing one that is being misued - so the backtrace pointing to openal doesn't necessarily mean it's openal's fault. Running with valgrind or another heap memory checking tool will probably be helpful to track down that particular linked bug. EDIT: It looks like that there's at least one out-of-bounds write when starting up half life (On arch linux, so maybe slightly different library versions and not loaded the counterstrike mod). It looks like a valve bug - writing 2 bytes at index [30] of a malloc'd size of 31 goes one byte over, and it looks from the backtrace it's all valve's code and not deep in some library that might have been loaded in. Writing 2 bytes to a string is a bit odd, perhaps it's trying to null-terminate but somehow uses a wstring null? Or some attempt at SIMD that isn't correctly bound? It doesn't seem to crash for me though, it might just be luck that nothing important is put 1 byte over, and it feels a bit unlikely something would be due to allocation and type alignment requirements, but it's perfectly valid for the malloc implementation to keep something important in that byte. Or perhaps there's some other dynamics that change this - it looks like it's doing stuff with paths, so may change size (of the allocation or even the amount written) based on where the steam app is installed - stuff like your user name length changing that may be the difference between a crash. Or even another issue somewhere else I didn't see, or valgrind didn't catch. Just goes to show how many games ship for years with "big" bugs :P For reference: ==27467== Invalid write of size 2 ==27467== at 0x406526A: GetSteamContentPath() (pathmatch.cpp:523) ==27467== by 0x4065927: pathmatch(char const*, char\*, bool, char*, unsigned int) [clone .part.1] (pathmatch.cpp:594) ==27467== by 0x4066849: pathmatch (pathmatch.cpp:541) ==27467== by 0x4066849: CWrap (pathmatch.cpp:685) ==27467== by 0x4066849: __wrap___xstat (pathmatch.cpp:907) ==27467== by 0x406294A: stat (stat.h:455) ==27467== by 0x406294A: CFileSystem_Stdio::FS_stat(char const*, stat*) (FileSystem_Stdio.cpp:225) ==27467== by 0x4060819: CBaseFileSystem::AddPackFiles(char const*) (BaseFileSystem.cpp:1325) ==27467== by 0x4060AA4: CBaseFileSystem::AddSearchPathInternal(char const*, char const*, bool) (BaseFileSystem.cpp:254) ==27467== by 0x4060B37: CBaseFileSystem::AddSearchPath(char const*, char const*) (BaseFileSystem.cpp:186) ==27467== by 0x8049003: main (launcher.cpp:413) ==27467== Address 0x45e5f4e is 30 bytes inside a block of size 31 alloc'd ==27467== at 0x4041714: malloc (vg_replace_malloc.c:393) ==27467== by 0x4357C4A: strdup (strdup.c:42) ==27467== by 0x42F1A76: realpath_stk (canonicalize.c:410) ==27467== by 0x42F1A76: realpath@@GLIBC_2.3 (canonicalize.c:432) ==27467== by 0x406525B: GetSteamContentPath() (pathmatch.cpp:520) ==27467== by 0x4065927: pathmatch(char const*, char\*, bool, char*, unsigned int) [clone .part.1] (pathmatch.cpp:594) ==27467== by 0x4066849: pathmatch (pathmatch.cpp:541) ==27467== by 0x4066849: CWrap (pathmatch.cpp:685) ==27467== by 0x4066849: __wrap___xstat (pathmatch.cpp:907) ==27467== by 0x406294A: stat (stat.h:455) ==27467== by 0x406294A: CFileSystem_Stdio::FS_stat(char const*, stat*) (FileSystem_Stdio.cpp:225) ==27467== by 0x4060819: CBaseFileSystem::AddPackFiles(char const*) (BaseFileSystem.cpp:1325) ==27467== by 0x4060AA4: CBaseFileSystem::AddSearchPathInternal(char const*, char const*, bool) (BaseFileSystem.cpp:254) ==27467== by 0x4060B37: CBaseFileSystem::AddSearchPath(char const*, char const*) (BaseFileSystem.cpp:186) ==27467== by 0x8049003: main (launcher.cpp:413)