18 ms·

Fixing stutters in Papers Please on Linux

ximeng 5y ago
Fun write up. Here’s another example of a binary patch to fix a Linux game issue: https://steamcommunity.com/app/333300/discussions/0/2646360608736518971/?ctp=2#c3050608970471835150 https://steamcommunity.com/app/333300/discussions/0/26463606...
- sva_ 5y ago
  Gotta remember the byte patternsearch using grep (and replacement using printf/dd). Clever use of Unix tools.
- xtracto 5y ago
  There was also an Age of Empires 2 patch that fixed a bug in Linux that made the game pan indefinitely to some corner after running the game and alt-tabbing to another window momentarily.
salted-fry 5y ago
Hey, I know this issue! I ran into it in CK3 when it launched. You can also work around it by running chmod go-rx /dev/input/ while playing your game. Whether this is more or less invasive than binary-patching the game is up for debate.
charcircuit 5y ago
From the description of the problem (a freeze every 3 seconds) I knew exactly what it was. You can fix it by simply upgrading SDL as they fixed this bug 2 years ago. https://github.com/spurious/SDL-mirror/commit/59728f9802c7869a9b47d6176296086dc0d00d45 https://github.com/spurious/SDL-mirror/commit/59728f9802c786...
- iqanq 5y ago
  This is one of the reasons I use Gentoo in my desktops: you can run a "stable" (as in old) system, but pull a more recent version of a library or application if you need it. For example, I remember having problems accessing files that I had stored in my mobile phone. Solution: updating libmtp and libmtp only. I suppose you can't do this in a distro such as Debian without upgrading half the packages. Are there more distros that allow you to do this?
  - teddyh 5y ago
    If you are advanced enough to run Gentoo, you should be able to use Debian and force an install of (or re-compile yourself) a new package of the newer version, working around the fact that the official newer version would otherwise require other new packages.
    - vlovich123 5y ago
      You could. But the amount of time it takes is significantly more than yay -S <package> or emerge <package> vs the hell that this poses on Debian (not to mention the dependency hell you can run into)
      - yjftsjthsd-h 5y ago
        I don't know the situation on Gentoo, but partial updates are explicitly unsupported on Arch, not least because they don't do stable ABIs; Debian should have a much easier time upgrading just one package.
        vlovich123 5y ago
        Having used Arch and Debian, I‘ve definitely had an easier time installing the latest version of arbitrary packages on Arch. Something like SDL is part of base and thus is already running latest.
        elevader 5y ago
        It's not necessarily recommended to mix stable and testing but it mostly works fine in my experience. I'd guess Gentoo gets around quite a few problems as everything is compiled from source. So updating a single libary would cause a rebuild of everything that depends on it. Gentoo also has the concept of "Slots", so you could have multiple versions of the same libary installed and packages will choose their version to build against accordingly.
- Diggsey 5y ago
  Why is it even doing this on the main thread at all? The obvious thing would be to have a background thread polling for changes and then sending messages asynchronously to the main thread if a change actually occurred...
  - ninepoints 5y ago
    Because architectural simplicity is valuable and some operating systems deliver input events on a particular thread.
  - arghwhat 5y ago
    It is appropriate to use your main thread for your OS interaction - polling fds, talking to display server, whatever I/O you need, etc. An open/close call should never take this long, and you should never need to make a large amount of them in sequence after startup. What should not be on your main thread is any long blocking compute, which is why rendering/game logic often goes to another thread - although simple games could easily be single-threaded.
    - lawl 5y ago
      > An open/close call should never take this long > What should not be on your main thread is any long blocking compute Isn't that contradicting yourself? I'm pretty sure open() can block.
      - arghwhat 5y ago
        > Isn't that contradicting yourself? I'm pretty sure open() can block. No, blocking compute would be you doing something for a long period of time. "open/close can block" means very little. You only need a mitigation if it ends up blocking long enough to be a problem in reasonable setups. You do that have to care about what happens if someone runs the code on an intentionally terrible/horribly slow but technically in spec toy filesystem. No need to prematurely optimize for this scenario. And especially with devfs you cannot just listen to POSIX and must know what the kernel is providing you - knowing how long operations take on such fds is normal design input.
- salicideblock 5y ago
  From one of the printouts in the post, it seems that Papers Please is using a bundled and statically linked SDL. So it would be the game developer that would have to update the version of the SDL library. The binary patching done seems like a good-enough alternative in the meantime. I have the feeling such bundling of dependencies is fairly common when porting games for Linux.
  - charcircuit 5y ago
    At least for Steam you are recommended to link against Valve's Steam Linux Runtime which is a set of dynamic libraries including SDL.
  - viraptor 5y ago
    It's common in application of certain size and compatibility expectations. Windows and Mac games will bundle their dependencies as much as possible as well. Same for large apps. Nobody wants to end to in a situation where their relatively expensive purchase doesn't work because of the version of local libs.
    - ironmagma 5y ago
      Does bundling dependencies imply static linking, though? Why can’t they just dynamically link to the bundled dependency?
      - viraptor 5y ago
        Not really, it can be bundled either as static or dynamic.
  - edflsafoiewq 5y ago
    SDL has its own "dynapi" layer, where you can override it with your own copy of SDL even if it was statically linked: https://github.com/libsdl-org/SDL/blob/main/docs/README-dynapi.md https://github.com/libsdl-org/SDL/blob/main/docs/README-dyna...
- AshamedCaptain 5y ago
  Why is udev not used in this case?
  - charcircuit 5y ago
    Because this is the fallback when you compile without udev support
    - AshamedCaptain 5y ago
      I can guess that, but I was wondering why ... Is there a distro without udev? The steam/whatever sandbox does not support udev?
      - Sjonny 5y ago
        It's not the system that is not supporting udev, it's the choice of the game developers how they compiled SDL .. without dependencies, and so without udev.
        mschuster91 5y ago
        That's standard for game devs in the Linux world. The less dependencies you have to rely on the distribution for, the better - Windows stuff is either already present or shared OS-wide with binary backwards compatibility (=DirectX), so you can get away with shipping stuff that has a chance to run even 25 years in the future without major modification.
  - bdowling 5y ago
    Probably because a commercial developer is not going to want to link a LGPL library with their program. [0] https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h https://github.com/systemd/systemd/blob/main/src/libudev/lib...
arghwhat 5y ago
The issue has been identified before, but seems like it stalled: https://gitlab.freedesktop.org/libinput/libinput/-/issues/509 https://gitlab.freedesktop.org/libinput/libinput/-/issues/50..., https://patchwork.kernel.org/project/linux-input/patch/20201227144302.9419-1-kl@kl.wtf/ https://patchwork.kernel.org/project/linux-input/patch/20201....
- tankenmate 5y ago
  I just did a quick check the posted fix is not in the most recent -rc branch in the public git repo.
  - londons_explore 5y ago
    This is the issue with using mailing lists... Large numbers of perfectly good fixes, embodying many hours of effort, just get missed and forgotten about. At least with GitHub PR's, every request either needs to be merged or rejected.
    - loeg 5y ago
      I’m no fan of mailing lists, but GitHub PRs get ignored in much the same way.
      - misnome 5y ago
        There’s ignored, and there is “not aware that it’s unresolved”. How do mailing list flows handle the “give me a list of open patches”?
        colonwqbang 5y ago
        People treat emails like tickets, representing things that should be done. They put them in particular email directories depending on their personal workflow. When they are either done or rejected, they delete the email, archive it, or mark it as read. It's not unlike the github workflow except that it's up to each person to define the way they prefer to work. Not a single policy that's decided by the project owner. Planning/tracking also happens more in private instead of public. You may think that's worse, but perhaps you can also see why some might prefer it?
Narretz 5y ago
Why is the engine even checking input devices so often? Shouldn't the input device be registered via settings and then assumed to exist when the game runs? It seems wasteful to check all input devices every few seconds.
- pas 5y ago
  Exactly. It should enumerate them when the player opens settings. Or at startup. But even if it wants to do this, why is it doing it on the main thread!? :(
  - flohofwoe 5y ago
    If I start the game without a gamepad attached to the computer, and then attach the gamepad, I'd like to use the gamepad without restarting the game. And one would expect that polling the attached input devices should never take hundreds or thousands of milliseconds, there must be something seriously wrong in the Linux input device stack or maybe in one of the input device drivers.
    - Vvector 5y ago
      Or maybe just poll for new input devices when the game is paused, or before the game starts.
      - flohofwoe 5y ago
        Then you still have problems to handle like accidentally disconnecting/reconnecting the gamepad when somebody stumbles over the cable (for instance the game might want to automatically pause if the gamepad suddenly 'disappears' for any reason). Gamepads should be automatically detected at any time in the game as they are connected or disconnected. That's how it works on game consoles, and PC games shouldn't behave any different in that regard IMHO.
    - yjftsjthsd-h 5y ago
      The input stack is fine, this game disabled the things that would let it work nicely (see upthread discussion of SDL supporting udev and inotify)
the_af 5y ago
This is interesting. I never noticed these pauses using the native port from GOG on Ubuntu. I'm very sensitive to this kind of thing (low refresh rates on CRT monitors used to drive me crazy when nobody else noticed). Will have to fire up my copy again. A good excuse to play this marvelous game again.
pjmlp 5y ago
I guess that is the kind of challenge to have Windows games on GNU/Linux, fix them instead of playing.
- the_af 5y ago
  Isn't this a native port though? I remember playing Papers Please even before it had a native port, and enjoying it with zero problems.
  - pjmlp 5y ago
    Apparently I got that wrong, point still stands though.
    - xfer 5y ago
      It does if you sell on platforms that you don't intend to support.
      - pjmlp 5y ago
        The goal of stuff like WINE and Proton is exactly to enable using games on a platform whose creators never intended to support.
- shmerl 5y ago
  It's not like Windows doesn't have its own issues to fix. Except developers do it anyway, because of the market size. Windows isn't perfect or better for gaming.
  - qayxc 5y ago
    > Windows isn't perfect or better for gaming. This assessment depends entirely on the perspective. From a developer's POV, Windows definitively is the better platform, as it's very monolithic in that you can rely on the presence and longevity of APIs. Depending on the dev's influence on the market and the success of the game, you even get free optimisation, support, and bug fixes from h/w vendors in the form of game-specific driver patches. From a gamer's POV, Windows has advantages as well, since bugs are rarely OS-related and h/w vendors offer a lot more features OOTB. If you love tinkering with the OS and don't care if some titles or features just won't work, Linux is a valid option for gaming. Otherwise Windows is objectively the better option by default, since I can rely on the games working with all available features (e.g. multiplayer).
Traubenfuchs 5y ago
Why would you not let another thread do this nasty kind of polling and let the main loop check for a changed result, if at all?
- loeg 5y ago
  Close, on a synthetic fd with no I/O performed, should not take 100ms per call. This is a Linux performance bug.
  - mgaunard 5y ago
    close can take arbitrarily long, it's a blocking operation. Don't ever call close on the hot path.
reallifez4 5y ago
- reallifez4 5y ago
  [flagged]
kaszanka 5y ago
Wait, why is `close` in libpthread.so?
- jchw 5y ago
  I believe that a few libc functions are reimplemented in libpthread, the idea being that if you don’t link to pthreads, you don’t need the overhead (locking, etc.) that is needed in multithreaded situations. Feels a bit antiquated now… As for why close specifically though, that’s a good question. I wonder if it has something to do with special libc treatment of the standard fds or anything like that.
  - loeg 5y ago
    As you can see in the disassembly, it has to do with implementing async cancellation. I think they wrap many blocking syscalls in the same way. https://man7.org/linux/man-pages/man3/pthread_cancel.3.html https://man7.org/linux/man-pages/man3/pthread_cancel.3.html
- cesarb 5y ago
  > why is `close` in libpthread.so? That's because close() is a pthreads "cancellation point" (see https://man7.org/linux/man-pages/man7/pthreads.7.html https://man7.org/linux/man-pages/man7/pthreads.7.html for details), so it needs special handling when the process is using pthreads. If the process does not link to libpthread.so, the implementation in libc.so (which probably doesn't have cancellation point support) will be used.
reallifez4 5y ago
reallifez4 5y ago
papersplease 5y ago
papersplease 5y ago
- papersplease 5y ago
leonjza 5y ago
I really enjoyed the debugging process here, and am glad to have learnt about the -k flag which seems to only be available on systems with strace version 5.5, at least for me. As for the patch (and my love for all things Frida [1]), I think a call to Intercerptor.replace() after locating the symbol with Module.getExportByName() [2] would make for a simpler patch (at the cost of installing Frida). For example: const sym = Module.getExportByName("lime.ndll", "SDL_SemWait"); Interceptor.replace(sym, { onEnter: function() {}, onLeave: function() {} }); [1] https://frida.re/ https://frida.re/ [2] https://frida.re/docs/javascript-api/#module https://frida.re/docs/javascript-api/#module
shhhum 5y ago
Some time ago I've also stumbled upon this issue and found the workaround that I've posted here: https://www.gog.com/forum/papers_please/terrible_lags_on_linux_with_nvidia_gpu/post3 https://www.gog.com/forum/papers_please/terrible_lags_on_lin... Thanks for a more proper digging.
shmerl 5y ago
Interesting investigation. I don't remember having this problem when playing the GOG version, but may be I didn't have a lot of input devices?
facorreia 5y ago
This reminds me of the recent Linus Tech Tips series on gaming on Linux[1]. Their conclusion is that although many games work out of the box (although usually not at launch), Linux is not ready for mainstream gamers. Not many people would have the expertise or the interest to troubleshoot the problem as OP did. [1] https://www.youtube.com/watch?v=Rlg4K16ujFw https://www.youtube.com/watch?v=Rlg4K16ujFw
- spijdar 5y ago
  It definitely isn't. I've been a huge linux nerd since my preteens in the late 2000s, I jumped on to squeeze more performance out of the thoroughly mediocre hardware I had access to. I wanted to program, and I found Visual Studio to be incomprehensibly dense and confusing, while Linux tools were so much simpler, with GCC, GEdit, makefiles and the like being more to my liking. I fell deep into the rabbit hole, learned emacs, then vim (it was more responsive on my intel atom-powered netbook), became a "shell guru", eventually went to college at 16 and started doing cybersecurity work/pentesting professionally. I've even made a tiny contribution to the Linux kernel, which I'm pretty proud of. All this anecdata to say, I consider myself pretty okay at using Linux, I "prefer" Linux, but I don't use Linux for gaming. Not unless it makes sense. I play Minecraft on Linux, and FOSS games that were developed on Linux. There's a POWER9 desktop on my desk that runs Linux, and all my professional and hobby work goes there. I love it. But any commercial games? They go on my old college-days Intel desktop, running Win10. I can do the work to get games running on Linux, but why bother? Like Linus says in that video, when I have time to play video games, I really don't want to pull out a debugger and strace and crap to do more $DAYJOB work. Not to say I never do that for fun. I do. I've done some work with https://github.com/ptitSeb/box86 https://github.com/ptitSeb/box86, and that involves a similar process. But I just frankly don't find doing it to your average Steam game to be very fun. Sometimes the muse strikes, usually it doesn't. And for your average Linux user, much less your average computer user overall, you can forget about it. IMO, unless you have a strong ideological reason to only use FOSS OSes (and all the power to you!), the reason you use Linux is because it's a vastly superior tool for certain problems. Playing your average commercial game is not one of them.
  - 5y ago
CmdrKrool 5y ago
This looks very like a problem I encountered some time ago running the closed source 3DO emulator "Phoenix Project", and similarly the open source "FreeDO" project that it was forked from. I narrowed it down (also using strace, IIRC) to these programs repeatedly opening and closing the /dev/input/event* files, and that being weirdly slow. I made a seperate little test program just to open and close those files to confirm it. It was only slow on my main desktop machine; while on my lesser-powered laptop, running a practically identical Arch Linux setup, those file operations were quick and the programs ran fine. None of these programs use SDL. I couldn't/didn't progress any further then, but it's good to find some pointers here for further investigation (ie. the libinput issue). Funnily enough I'm pretty sure I played Papers Please on this machine at length without problems but I think that was probably the Windows version through Wine.
ux 5y ago
Great post, learned a bunch of things. Only one blog post, and no RSS (yet?) unfortunately.
Aissen 5y ago
strace tip of the day: you don't need lsof, strace can keep track of open fds quite well, just use the -y flag. -y --decode-fds --decode-fds=path Print paths associated with file descriptor arguments. -yy --decode-fds=all Print all available information associated with file descriptors: protocol-specific information associated with socket file descriptors, block/character device number associated with device file descriptors, and PIDs associated with pidfd file descriptors.
- loeg 5y ago
  That's a great tip, thanks.
nofinator 5y ago
Glory to Arstotzka
japhib 5y ago
Interesting tidbit about Papers, Please: it's written with Haxe (a programming language), on top of OpenFL (an open-source Flash alternative), which uses Lime as its foundational cross-platform library. The nice stack traces you see in the article, such as: openfl::display::Application_obj::__construct are because Haxe actually compiles down to C++! (That's also why it has nice interop with C/C++ libraries like SDL.) Haxe boasts an insane amount of compile targets -- just taking a glance at its Github page, it can compile to Javascript, C#, C++, Java, Lua, PHP, Python, and Flash, plus a couple different Haxe-specific interpreters. Not many game developers use Haxe, so it has a small, tight-knit community. Other well-known games written with Haxe include Dead Cells, Northgard, and Dicey Dungeons.
xzjis 5y ago
I wonder if using the Windows version with Wine/Proton does work better than using the Linux x64 version.

Fun write up. Here’s another example of a binary patch to fix a Linux game issue: https://steamcommunity.com/app/333300/discussions/0/2646360608736518971/?ctp=2#c3050608970471835150 https://steamcommunity.com/app/333300/discussions/0/26463606...

sva_ 5y ago
Gotta remember the byte patternsearch using grep (and replacement using printf/dd). Clever use of Unix tools.
xtracto 5y ago
There was also an Age of Empires 2 patch that fixed a bug in Linux that made the game pan indefinitely to some corner after running the game and alt-tabbing to another window momentarily.

salted-fry 5y ago

Hey, I know this issue! I ran into it in CK3 when it launched. You can also work around it by running chmod go-rx /dev/input/ while playing your game. Whether this is more or less invasive than binary-patching the game is up for debate.

charcircuit 5y ago

From the description of the problem (a freeze every 3 seconds) I knew exactly what it was. You can fix it by simply upgrading SDL as they fixed this bug 2 years ago. https://github.com/spurious/SDL-mirror/commit/59728f9802c7869a9b47d6176296086dc0d00d45 https://github.com/spurious/SDL-mirror/commit/59728f9802c786...

iqanq 5y ago
This is one of the reasons I use Gentoo in my desktops: you can run a "stable" (as in old) system, but pull a more recent version of a library or application if you need it. For example, I remember having problems accessing files that I had stored in my mobile phone. Solution: updating libmtp and libmtp only. I suppose you can't do this in a distro such as Debian without upgrading half the packages. Are there more distros that allow you to do this?
- teddyh 5y ago
  If you are advanced enough to run Gentoo, you should be able to use Debian and force an install of (or re-compile yourself) a new package of the newer version, working around the fact that the official newer version would otherwise require other new packages.
  - vlovich123 5y ago
    You could. But the amount of time it takes is significantly more than yay -S <package> or emerge <package> vs the hell that this poses on Debian (not to mention the dependency hell you can run into)
    - yjftsjthsd-h 5y ago
      I don't know the situation on Gentoo, but partial updates are explicitly unsupported on Arch, not least because they don't do stable ABIs; Debian should have a much easier time upgrading just one package.
      - vlovich123 5y ago
        Having used Arch and Debian, I‘ve definitely had an easier time installing the latest version of arbitrary packages on Arch. Something like SDL is part of base and thus is already running latest.
      - elevader 5y ago
        It's not necessarily recommended to mix stable and testing but it mostly works fine in my experience. I'd guess Gentoo gets around quite a few problems as everything is compiled from source. So updating a single libary would cause a rebuild of everything that depends on it. Gentoo also has the concept of "Slots", so you could have multiple versions of the same libary installed and packages will choose their version to build against accordingly.
Diggsey 5y ago
Why is it even doing this on the main thread at all? The obvious thing would be to have a background thread polling for changes and then sending messages asynchronously to the main thread if a change actually occurred...
- ninepoints 5y ago
  Because architectural simplicity is valuable and some operating systems deliver input events on a particular thread.
- arghwhat 5y ago
  It is appropriate to use your main thread for your OS interaction - polling fds, talking to display server, whatever I/O you need, etc. An open/close call should never take this long, and you should never need to make a large amount of them in sequence after startup. What should not be on your main thread is any long blocking compute, which is why rendering/game logic often goes to another thread - although simple games could easily be single-threaded.
  - lawl 5y ago
    > An open/close call should never take this long > What should not be on your main thread is any long blocking compute Isn't that contradicting yourself? I'm pretty sure open() can block.
    - arghwhat 5y ago
      > Isn't that contradicting yourself? I'm pretty sure open() can block. No, blocking compute would be you doing something for a long period of time. "open/close can block" means very little. You only need a mitigation if it ends up blocking long enough to be a problem in reasonable setups. You do that have to care about what happens if someone runs the code on an intentionally terrible/horribly slow but technically in spec toy filesystem. No need to prematurely optimize for this scenario. And especially with devfs you cannot just listen to POSIX and must know what the kernel is providing you - knowing how long operations take on such fds is normal design input.
salicideblock 5y ago
From one of the printouts in the post, it seems that Papers Please is using a bundled and statically linked SDL. So it would be the game developer that would have to update the version of the SDL library. The binary patching done seems like a good-enough alternative in the meantime. I have the feeling such bundling of dependencies is fairly common when porting games for Linux.
- charcircuit 5y ago
  At least for Steam you are recommended to link against Valve's Steam Linux Runtime which is a set of dynamic libraries including SDL.
- viraptor 5y ago
  It's common in application of certain size and compatibility expectations. Windows and Mac games will bundle their dependencies as much as possible as well. Same for large apps. Nobody wants to end to in a situation where their relatively expensive purchase doesn't work because of the version of local libs.
  - ironmagma 5y ago
    Does bundling dependencies imply static linking, though? Why can’t they just dynamically link to the bundled dependency?
    - viraptor 5y ago
      Not really, it can be bundled either as static or dynamic.
- edflsafoiewq 5y ago
  SDL has its own "dynapi" layer, where you can override it with your own copy of SDL even if it was statically linked: https://github.com/libsdl-org/SDL/blob/main/docs/README-dynapi.md https://github.com/libsdl-org/SDL/blob/main/docs/README-dyna...
AshamedCaptain 5y ago
Why is udev not used in this case?
- charcircuit 5y ago
  Because this is the fallback when you compile without udev support
  - AshamedCaptain 5y ago
    I can guess that, but I was wondering why ... Is there a distro without udev? The steam/whatever sandbox does not support udev?
    - Sjonny 5y ago
      It's not the system that is not supporting udev, it's the choice of the game developers how they compiled SDL .. without dependencies, and so without udev.
      - mschuster91 5y ago
        That's standard for game devs in the Linux world. The less dependencies you have to rely on the distribution for, the better - Windows stuff is either already present or shared OS-wide with binary backwards compatibility (=DirectX), so you can get away with shipping stuff that has a chance to run even 25 years in the future without major modification.
- bdowling 5y ago
  Probably because a commercial developer is not going to want to link a LGPL library with their program. [0] https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h https://github.com/systemd/systemd/blob/main/src/libudev/lib...

arghwhat 5y ago

The issue has been identified before, but seems like it stalled: https://gitlab.freedesktop.org/libinput/libinput/-/issues/509 https://gitlab.freedesktop.org/libinput/libinput/-/issues/50..., https://patchwork.kernel.org/project/linux-input/patch/20201227144302.9419-1-kl@kl.wtf/ https://patchwork.kernel.org/project/linux-input/patch/20201....

tankenmate 5y ago
I just did a quick check the posted fix is not in the most recent -rc branch in the public git repo.
- londons_explore 5y ago
  This is the issue with using mailing lists... Large numbers of perfectly good fixes, embodying many hours of effort, just get missed and forgotten about. At least with GitHub PR's, every request either needs to be merged or rejected.
  - loeg 5y ago
    I’m no fan of mailing lists, but GitHub PRs get ignored in much the same way.
    - misnome 5y ago
      There’s ignored, and there is “not aware that it’s unresolved”. How do mailing list flows handle the “give me a list of open patches”?
      - colonwqbang 5y ago
        People treat emails like tickets, representing things that should be done. They put them in particular email directories depending on their personal workflow. When they are either done or rejected, they delete the email, archive it, or mark it as read. It's not unlike the github workflow except that it's up to each person to define the way they prefer to work. Not a single policy that's decided by the project owner. Planning/tracking also happens more in private instead of public. You may think that's worse, but perhaps you can also see why some might prefer it?

Narretz 5y ago

Why is the engine even checking input devices so often? Shouldn't the input device be registered via settings and then assumed to exist when the game runs? It seems wasteful to check all input devices every few seconds.

pas 5y ago
Exactly. It should enumerate them when the player opens settings. Or at startup. But even if it wants to do this, why is it doing it on the main thread!? :(
- flohofwoe 5y ago
  If I start the game without a gamepad attached to the computer, and then attach the gamepad, I'd like to use the gamepad without restarting the game. And one would expect that polling the attached input devices should never take hundreds or thousands of milliseconds, there must be something seriously wrong in the Linux input device stack or maybe in one of the input device drivers.
  - Vvector 5y ago
    Or maybe just poll for new input devices when the game is paused, or before the game starts.
    - flohofwoe 5y ago
      Then you still have problems to handle like accidentally disconnecting/reconnecting the gamepad when somebody stumbles over the cable (for instance the game might want to automatically pause if the gamepad suddenly 'disappears' for any reason). Gamepads should be automatically detected at any time in the game as they are connected or disconnected. That's how it works on game consoles, and PC games shouldn't behave any different in that regard IMHO.
  - yjftsjthsd-h 5y ago
    The input stack is fine, this game disabled the things that would let it work nicely (see upthread discussion of SDL supporting udev and inotify)

the_af 5y ago

This is interesting. I never noticed these pauses using the native port from GOG on Ubuntu. I'm very sensitive to this kind of thing (low refresh rates on CRT monitors used to drive me crazy when nobody else noticed). Will have to fire up my copy again. A good excuse to play this marvelous game again.

pjmlp 5y ago

I guess that is the kind of challenge to have Windows games on GNU/Linux, fix them instead of playing.

the_af 5y ago
Isn't this a native port though? I remember playing Papers Please even before it had a native port, and enjoying it with zero problems.
- pjmlp 5y ago
  Apparently I got that wrong, point still stands though.
  - xfer 5y ago
    It does if you sell on platforms that you don't intend to support.
    - pjmlp 5y ago
      The goal of stuff like WINE and Proton is exactly to enable using games on a platform whose creators never intended to support.
shmerl 5y ago
It's not like Windows doesn't have its own issues to fix. Except developers do it anyway, because of the market size. Windows isn't perfect or better for gaming.
- qayxc 5y ago
  > Windows isn't perfect or better for gaming. This assessment depends entirely on the perspective. From a developer's POV, Windows definitively is the better platform, as it's very monolithic in that you can rely on the presence and longevity of APIs. Depending on the dev's influence on the market and the success of the game, you even get free optimisation, support, and bug fixes from h/w vendors in the form of game-specific driver patches. From a gamer's POV, Windows has advantages as well, since bugs are rarely OS-related and h/w vendors offer a lot more features OOTB. If you love tinkering with the OS and don't care if some titles or features just won't work, Linux is a valid option for gaming. Otherwise Windows is objectively the better option by default, since I can rely on the games working with all available features (e.g. multiplayer).

Traubenfuchs 5y ago

Why would you not let another thread do this nasty kind of polling and let the main loop check for a changed result, if at all?

loeg 5y ago
Close, on a synthetic fd with no I/O performed, should not take 100ms per call. This is a Linux performance bug.
- mgaunard 5y ago
  close can take arbitrarily long, it's a blocking operation. Don't ever call close on the hot path.

reallifez4 5y ago

reallifez4 5y ago
[flagged]

kaszanka 5y ago

Wait, why is `close` in libpthread.so?

jchw 5y ago
I believe that a few libc functions are reimplemented in libpthread, the idea being that if you don’t link to pthreads, you don’t need the overhead (locking, etc.) that is needed in multithreaded situations. Feels a bit antiquated now… As for why close specifically though, that’s a good question. I wonder if it has something to do with special libc treatment of the standard fds or anything like that.
- loeg 5y ago
  As you can see in the disassembly, it has to do with implementing async cancellation. I think they wrap many blocking syscalls in the same way. https://man7.org/linux/man-pages/man3/pthread_cancel.3.html https://man7.org/linux/man-pages/man3/pthread_cancel.3.html
cesarb 5y ago
> why is `close` in libpthread.so? That's because close() is a pthreads "cancellation point" (see https://man7.org/linux/man-pages/man7/pthreads.7.html https://man7.org/linux/man-pages/man7/pthreads.7.html for details), so it needs special handling when the process is using pthreads. If the process does not link to libpthread.so, the implementation in libc.so (which probably doesn't have cancellation point support) will be used.

reallifez4 5y ago

papersplease 5y ago

papersplease 5y ago

leonjza 5y ago

I really enjoyed the debugging process here, and am glad to have learnt about the -k flag which seems to only be available on systems with strace version 5.5, at least for me. As for the patch (and my love for all things Frida [1]), I think a call to Intercerptor.replace() after locating the symbol with Module.getExportByName() [2] would make for a simpler patch (at the cost of installing Frida). For example: const sym = Module.getExportByName("lime.ndll", "SDL_SemWait"); Interceptor.replace(sym, { onEnter: function() {}, onLeave: function() {} }); [1] https://frida.re/ https://frida.re/ [2] https://frida.re/docs/javascript-api/#module https://frida.re/docs/javascript-api/#module

shhhum 5y ago

Some time ago I've also stumbled upon this issue and found the workaround that I've posted here: https://www.gog.com/forum/papers_please/terrible_lags_on_linux_with_nvidia_gpu/post3 https://www.gog.com/forum/papers_please/terrible_lags_on_lin... Thanks for a more proper digging.

shmerl 5y ago

Interesting investigation. I don't remember having this problem when playing the GOG version, but may be I didn't have a lot of input devices?

facorreia 5y ago

This reminds me of the recent Linus Tech Tips series on gaming on Linux[1]. Their conclusion is that although many games work out of the box (although usually not at launch), Linux is not ready for mainstream gamers. Not many people would have the expertise or the interest to troubleshoot the problem as OP did. [1] https://www.youtube.com/watch?v=Rlg4K16ujFw https://www.youtube.com/watch?v=Rlg4K16ujFw

spijdar 5y ago
It definitely isn't. I've been a huge linux nerd since my preteens in the late 2000s, I jumped on to squeeze more performance out of the thoroughly mediocre hardware I had access to. I wanted to program, and I found Visual Studio to be incomprehensibly dense and confusing, while Linux tools were so much simpler, with GCC, GEdit, makefiles and the like being more to my liking. I fell deep into the rabbit hole, learned emacs, then vim (it was more responsive on my intel atom-powered netbook), became a "shell guru", eventually went to college at 16 and started doing cybersecurity work/pentesting professionally. I've even made a tiny contribution to the Linux kernel, which I'm pretty proud of. All this anecdata to say, I consider myself pretty okay at using Linux, I "prefer" Linux, but I don't use Linux for gaming. Not unless it makes sense. I play Minecraft on Linux, and FOSS games that were developed on Linux. There's a POWER9 desktop on my desk that runs Linux, and all my professional and hobby work goes there. I love it. But any commercial games? They go on my old college-days Intel desktop, running Win10. I can do the work to get games running on Linux, but why bother? Like Linus says in that video, when I have time to play video games, I really don't want to pull out a debugger and strace and crap to do more $DAYJOB work. Not to say I never do that for fun. I do. I've done some work with https://github.com/ptitSeb/box86 https://github.com/ptitSeb/box86, and that involves a similar process. But I just frankly don't find doing it to your average Steam game to be very fun. Sometimes the muse strikes, usually it doesn't. And for your average Linux user, much less your average computer user overall, you can forget about it. IMO, unless you have a strong ideological reason to only use FOSS OSes (and all the power to you!), the reason you use Linux is because it's a vastly superior tool for certain problems. Playing your average commercial game is not one of them.
- 5y ago

CmdrKrool 5y ago

This looks very like a problem I encountered some time ago running the closed source 3DO emulator "Phoenix Project", and similarly the open source "FreeDO" project that it was forked from. I narrowed it down (also using strace, IIRC) to these programs repeatedly opening and closing the /dev/input/event* files, and that being weirdly slow. I made a seperate little test program just to open and close those files to confirm it. It was only slow on my main desktop machine; while on my lesser-powered laptop, running a practically identical Arch Linux setup, those file operations were quick and the programs ran fine. None of these programs use SDL. I couldn't/didn't progress any further then, but it's good to find some pointers here for further investigation (ie. the libinput issue). Funnily enough I'm pretty sure I played Papers Please on this machine at length without problems but I think that was probably the Windows version through Wine.

ux 5y ago

Great post, learned a bunch of things. Only one blog post, and no RSS (yet?) unfortunately.

Aissen 5y ago

strace tip of the day: you don't need lsof, strace can keep track of open fds quite well, just use the -y flag. -y --decode-fds --decode-fds=path Print paths associated with file descriptor arguments. -yy --decode-fds=all Print all available information associated with file descriptors: protocol-specific information associated with socket file descriptors, block/character device number associated with device file descriptors, and PIDs associated with pidfd file descriptors.

loeg 5y ago
That's a great tip, thanks.

nofinator 5y ago

Glory to Arstotzka

japhib 5y ago

Interesting tidbit about Papers, Please: it's written with Haxe (a programming language), on top of OpenFL (an open-source Flash alternative), which uses Lime as its foundational cross-platform library. The nice stack traces you see in the article, such as: openfl::display::Application_obj::__construct are because Haxe actually compiles down to C++! (That's also why it has nice interop with C/C++ libraries like SDL.) Haxe boasts an insane amount of compile targets -- just taking a glance at its Github page, it can compile to Javascript, C#, C++, Java, Lua, PHP, Python, and Flash, plus a couple different Haxe-specific interpreters. Not many game developers use Haxe, so it has a small, tight-knit community. Other well-known games written with Haxe include Dead Cells, Northgard, and Dicey Dungeons.

xzjis 5y ago

I wonder if using the Windows version with Wine/Proton does work better than using the Linux x64 version.