8 ms·

Emulating Windows system calls in Linux

180 points by mfilion 6y ago

mrpippy 6y ago
There was a follow up a few weeks later: https://lwn.net/Articles/826313/ https://lwn.net/Articles/826313/
- Ericson2314 6y ago
  Let's make this the URL? I think the resolution makes for a more interesting discussion, and it does still talk about the provisional design in brief.
- vii 6y ago
  The plan described there is to use a signal (SIGSYS) to jump back into the process. This involves multiple kernel/userpace switches as most likely you need some Linux syscalls to emulate the MS Windows one. I like the idea of using eBPF to run an emulation layer. That has potential to be more efficient and doesn't require the running process to cooperate so closely with the syscall emulation. Then this would probably lead to extensions to eBPF and its tooling which should be useful for other things!
badrabbit 6y ago
Interesting, I would also look into if the usemode-helper function can maybe used to re-map syscalls and proxy them through a wrapper library just like LD_PRELOAD but done at the kernel level after this, every process will load the wrapper lib (not just every glibc process) much like windows programs load user32/kernel32.dll. Is this feasible or am I misunderstandig the feature?: https://kernelnewbies.org/KernelProjects/usermode-helper-enhancements https://kernelnewbies.org/KernelProjects/usermode-helper-enh... https://elixir.bootlin.com/linux/latest/source/security/Kconfig#L198 https://elixir.bootlin.com/linux/latest/source/security/Kcon...
- cryptonector 6y ago
  Oh, nice, so start an app that doesn't use libc (e.g., a golang app) using ptrace(2) to control it, inject code to intercept system calls, mark the app's text as requiring syscall dispatch, then stop ptracing. Presto: LD_PRELOAD for libc-non-using apps. Granted, to make this easy to use the interceptor should be able to use normal LD_PRELOAD objects, which means bootstrapping ld.so and libc, which is going to be very difficult to do.
  - monocasa 6y ago
    This is how User Mode Linux and gvisor work more or less (albeit with the 'injected' code running in another process entirely). It's also how a kernel I'm writing works in a port that works like user mode linux (in addition to the ports on native hardware). The issue is that ptrace is unreasonably slow (orders of magnitude slower than just regular syscalls), so they'd want the fast normal path to not go through ptrace. But they don't have a way to communicate that to the kernel at the moment.
    - cryptonector 6y ago
      And this scheme gets you a fast path.
Ericson2314 6y ago
If the firsts comment, https://lwn.net/Articles/824494/ https://lwn.net/Articles/824494/, is correct, and this can be made secure, I'm excited that this could mean a much more maintainable/easy-to-upstream version of Capsicum/CloudABI.
- trasz 6y ago
  Not really. Capsicum provides an actual security architecture (capability-based security), as opposed to a ad-hoc bandaid mechanism.
  - Ericson2314 6y ago
    Yeah to be clear I would rather see a proper implementation. From the LWN thread I thought Linux didn't have a personality system so this might be a stop-gap way to make a lighter weight implementation, but somebody commented that it in fact does (if barely), so that should be used instead.
- saagarjha 6y ago
  I don't see any reasonable protection you can have against TOCTOUs dealing with memory, since you could just always race for the small window between the check and the syscall being made.
kyberias 6y ago
> Windows applications are increasingly executing system calls directly rather than going through the API I didn't know this. What applications are doing this and why?
- wtallis 6y ago
  Anti-cheating libraries used in games seem to be the primary culprits, and they're doing it this way because their purpose is to break things while resisting the user's attempts to make things work nicely.
  - leeter 6y ago
    Unfortunately WINE can't just take the windows shim mechanism that largely just nukes these from orbit on binary load and removes the check all together. Maybe at some point MS will be more open about that.
  - mehrdadn 6y ago
    A bit off-topic, but do you happen to know if there is a way on Windows to intercept syscall instructions? (aside from dynamic analysis/recompilation/etc.?) If there isn't, I assume that's why cheat engines do this?
    - wtallis 6y ago
      I think it's inherent to the SYSCALL instruction that control gets transferred directly to the kernel and ring 0, with no possibility of intercepting in userspace. So on Windows or Linux, you need kernel help to redirect/intercept syscalls.
      - mehrdadn 6y ago
        I asked this as a Windows question rather than an x86 question. I was wondering if there's any Windows API that could help with this. (This includes kernel APIs.)
CodesInChaos 6y ago
Since calling syscalls directly instead of through the wrapper dll is unsupported, I wouldn't be surprised if MS prevented syscalls from outside the expected libraries at some point. OpenBSD already does this to improve security: https://marc.info/?l=openbsd-tech&m=157488907117170&w=2 https://marc.info/?l=openbsd-tech&m=157488907117170&w=2
- guenthert 6y ago
  I'm a bit at loss how this improves security or why it is necessary. If the kernel needs an external library as gatekeeper, something fundamentally went wrong. And why would, say a Lisp compiler use the C library (or to use a more popular example, why would a Java VM do the same)? It might be common practice (to ease FFI/JNI), but making this a requirement seems heading the wrong direction.
  - saagarjha 6y ago
    I believe libc is the only stable way to ask the kernel to do something for you.
    - kazinator 6y ago
      GNU Libc is only stable at the source code level, and tries fairly hard at the binary level with approaches like symbol versioning and whatnot. Linux syscalls are more stable at the actual binary level. Some "int $0x80" code you wrote in 1996 in x86 assembly language will work today, provided the binary format (like a.out) will load. The libc which was used then is not even around any more.
      - saagarjha 6y ago
        Ah, I was talking about OpenBSD's kernel interface. Linux is ABI stable at the syscall level, of course–I know it tries to be binary compatible, but I have never actually tested this much.
        6y ago
magicalhippo 6y ago
Despite having coded for Windows for quite some time, I realized I had never actually peeked that low before, so didn't have a good understanding of how system calls really worked. Found this article[1] which explained it nicely (and hopefully correctly). [1]: https://www.codeguru.com/cpp/w-p/system/devicedriverdevelopment/article.php/c8035/How-Do-Windows-NT-System-Calls-REALLY-Work.htm https://www.codeguru.com/cpp/w-p/system/devicedriverdevelopm...
nxc18 6y ago
This seems like it would be a lot of maintenance effort to get right, considering Windows shuffles their syscall table every release, including releases within the Windows 10 family. https://j00ru.vexillium.org/syscalls/nt/64/ https://j00ru.vexillium.org/syscalls/nt/64/ Has an excellent table of syscalls by version.
- gizmo686 6y ago
  If applications are calling these syscalls directly; then there needs to be some form of stability; otherwise, the applications would only work on the particular version of Windows they were compiled for.
  - frabert 6y ago
    The Win32 API (and nowadays the WinRT API as well) is that interface, system calls are explicitly unstable and internal-use-only I believe.
    - gizmo686 6y ago
      If all windows applications used the APIs like they were supposed to, then there would be no need to emulate the Windows syscalls. The point of this emulation is that some applications (games, apparently) were not using these APIs, and instead performing syscalls directly. In order for this to be feasable on Windows, there needs to be some stability in the syscalls; even if that stability involves angry MS engineers complaining that major applications are doing something that they were explicitly told not to do.
      - leeter 6y ago
        You'd be surprised, MS doesn't do this randomly. They'd rather break a game if the game is actively in production or the company is around. They only put in shims for applications if the vendor isn't still in business usually or can't get source code. But in quite a few cases they'll just put in a patch binary in the shims that just removes the anti-cheat all together from the binary when loaded. Game still runs... nobody cares.
jhallenworld 6y ago
Linux used to be able to run Xenix and SCO UNIX applications- I have not tried this in years, but I'm wondering if the mechanism can be shared such that any OS can be emulated.
- lizknope 6y ago
  https://en.wikipedia.org/wiki/Intel_Binary_Compatibility_Standard https://en.wikipedia.org/wiki/Intel_Binary_Compatibility_Sta... The Intel Binary Compatibility Standard (iBCS) is a standardized application binary interface (ABI) for Unix operating systems on Intel-386-compatible computers, published by AT&T, Intel and SCO in 1988, and updated in 1990.
  - jhallenworld 6y ago
    Yes, also I remember that SCO UNIX worked but SCO Xenix did not work. But really I wanted Xenix to work.. I seem to remember it worked once, but then stopped working when they implemented the standard. Or something- it was a long time ago.
    - lizknope 6y ago
      I had a friend get WordPerfect for SCO working under Linux. That was around 1996. I found these instructions. https://www.tldp.org/HOWTO/WordPerfect-5.html https://www.tldp.org/HOWTO/WordPerfect-5.html Same friend had a DEC Alpha and got Digital Unix Netscape to run under Linux on the Alpha. http://users.bart.nl/~geerten/FAQ-17.html http://users.bart.nl/~geerten/FAQ-17.html
- tzs 6y ago
  Were those 286 Xenix applications or 386 Xenix applications? Running 286 Xenix applications on a 386 or later Unix or Unix-like system is fairly straightforward as far as the 386 kernel is concerned. 286 Xenix used a different syscall mechanism than 386 Unixes did, so you didn't have the problem of the kernel needing to deal with 286 and 386 syscalls ending up in the same place. My recollection, from being half the two person team that implemented 286 Unix binary compatibility for System V Release 3.2 on 386 at ISC, and then being half the different two person team that did 286 Xenix binary compatibility, was that this is what was required in the kernel: 1. Adding a mechanism to allow a process to set up 16-bit segments as aliases to portions of its address space. 2. Modifying exec to recognize a 286 Unix or Xenix binary, and turn the exec into an exec of /bin/i286emul or /bin/x286emul, with the path to the 286 binary as an argument. 3. I don't remember if we had to do anything special for the 286 system call mechanism, because I don't actually remember what that mechanism was. I don't remember if what the binaries would try to do for a syscall would simply cause a trapable signal, and then the signal handler would recognize the system call and take care of it, of if the system call mechanism used something like a call gate that the 386 code had to set up, and so we had to add a mechanism to allowed the 386 code to do that. That was almost all the kernel work. Pretty much everything else could be handled in user mode code. When you would run a 286 Xenix binary, that turned into an exec of /bin/x286emul. x286emul would read the Xenix binary, allocate memory for it and load it, map segments to it, and jump to the code, switching the processor to 16-bit mode. When the Xenix code did a system call and it ended up in the handler in x286emul, for most system calls it was simply a matter of copying arguments from where the 286 call put them to where the corresponding 386 call expected them, making the 386 system call, then putting results in the right place, and returning to the Xenix code. For some system calls, more work was needed. Signal handling, for instance. If the 286 code wanted to trap a signal, the 386 code would have to set a trap of its own, and its handler would then have to deal with the stack fiddling and such to deliver it to the 286 code. Speaking of signal handling, that led to the stupidest, most annoying meeting I've ever had to attend. ISC had done the official 386 port of System V Release 3 under contract from AT&T, and we did the 286 Unix binary compatibility as part of that. Later, there was a deal between AT&T and Sun and Microsoft to make Unixes more compatible, which included Xenix compatibilty, and we were doing the Xenix compatibility as part of that, under contract from Microsoft. For the 286 binary compatibility, ISC did the whole thing--all the user mode code in i286emul and all the supporting kernel changes. But for Xenix compatibilty, ISC was just doing x286emul. Microsoft was doing any kernel modifications needed, which wasn't very much because the mods already there for i286emul mostly worked find for x286emul. There was one difference between Xenix and Unix signal handling that we could not fully address in x286emul. We needed kernel support--some sort of per-process flag that would tell it the processes wanted signals to behave like Xenix signals. We asked Microsoft to add such a flag. A bit later they got back to us and said there was issues with such a flag that could not be settled by email or phone. We had to have an in-person meeting. So me and the other guy working no x286emul had to fly early one morning from Los Angeles to Seattle, take a rental car to Redmond, and attend a meeting with Microsoft. At the meeting, we introduced ourselves and the half dozen or so Microsoft people introduced themselves, and then they presented this issue that could not be handled by email or phone: should this flag be controlled by an ioctl, or should it be an optional new parameter for the signal() call [1]. We gave our opinion, that was the end of the meeting, and we headed for the airport. [1] or something like that. I don't remember the exact second option.

180 points by mfilion 6y ago

mrpippy 6y ago

There was a follow up a few weeks later: https://lwn.net/Articles/826313/ https://lwn.net/Articles/826313/

Ericson2314 6y ago
Let's make this the URL? I think the resolution makes for a more interesting discussion, and it does still talk about the provisional design in brief.
vii 6y ago
The plan described there is to use a signal (SIGSYS) to jump back into the process. This involves multiple kernel/userpace switches as most likely you need some Linux syscalls to emulate the MS Windows one. I like the idea of using eBPF to run an emulation layer. That has potential to be more efficient and doesn't require the running process to cooperate so closely with the syscall emulation. Then this would probably lead to extensions to eBPF and its tooling which should be useful for other things!

badrabbit 6y ago

Interesting, I would also look into if the usemode-helper function can maybe used to re-map syscalls and proxy them through a wrapper library just like LD_PRELOAD but done at the kernel level after this, every process will load the wrapper lib (not just every glibc process) much like windows programs load user32/kernel32.dll. Is this feasible or am I misunderstandig the feature?: https://kernelnewbies.org/KernelProjects/usermode-helper-enhancements https://kernelnewbies.org/KernelProjects/usermode-helper-enh... https://elixir.bootlin.com/linux/latest/source/security/Kconfig#L198 https://elixir.bootlin.com/linux/latest/source/security/Kcon...

cryptonector 6y ago
Oh, nice, so start an app that doesn't use libc (e.g., a golang app) using ptrace(2) to control it, inject code to intercept system calls, mark the app's text as requiring syscall dispatch, then stop ptracing. Presto: LD_PRELOAD for libc-non-using apps. Granted, to make this easy to use the interceptor should be able to use normal LD_PRELOAD objects, which means bootstrapping ld.so and libc, which is going to be very difficult to do.
- monocasa 6y ago
  This is how User Mode Linux and gvisor work more or less (albeit with the 'injected' code running in another process entirely). It's also how a kernel I'm writing works in a port that works like user mode linux (in addition to the ports on native hardware). The issue is that ptrace is unreasonably slow (orders of magnitude slower than just regular syscalls), so they'd want the fast normal path to not go through ptrace. But they don't have a way to communicate that to the kernel at the moment.
  - cryptonector 6y ago
    And this scheme gets you a fast path.

Ericson2314 6y ago

If the firsts comment, https://lwn.net/Articles/824494/ https://lwn.net/Articles/824494/, is correct, and this can be made secure, I'm excited that this could mean a much more maintainable/easy-to-upstream version of Capsicum/CloudABI.

trasz 6y ago
Not really. Capsicum provides an actual security architecture (capability-based security), as opposed to a ad-hoc bandaid mechanism.
- Ericson2314 6y ago
  Yeah to be clear I would rather see a proper implementation. From the LWN thread I thought Linux didn't have a personality system so this might be a stop-gap way to make a lighter weight implementation, but somebody commented that it in fact does (if barely), so that should be used instead.
saagarjha 6y ago
I don't see any reasonable protection you can have against TOCTOUs dealing with memory, since you could just always race for the small window between the check and the syscall being made.

kyberias 6y ago

> Windows applications are increasingly executing system calls directly rather than going through the API I didn't know this. What applications are doing this and why?

wtallis 6y ago
Anti-cheating libraries used in games seem to be the primary culprits, and they're doing it this way because their purpose is to break things while resisting the user's attempts to make things work nicely.
- leeter 6y ago
  Unfortunately WINE can't just take the windows shim mechanism that largely just nukes these from orbit on binary load and removes the check all together. Maybe at some point MS will be more open about that.
- mehrdadn 6y ago
  A bit off-topic, but do you happen to know if there is a way on Windows to intercept syscall instructions? (aside from dynamic analysis/recompilation/etc.?) If there isn't, I assume that's why cheat engines do this?
  - wtallis 6y ago
    I think it's inherent to the SYSCALL instruction that control gets transferred directly to the kernel and ring 0, with no possibility of intercepting in userspace. So on Windows or Linux, you need kernel help to redirect/intercept syscalls.
    - mehrdadn 6y ago
      I asked this as a Windows question rather than an x86 question. I was wondering if there's any Windows API that could help with this. (This includes kernel APIs.)

CodesInChaos 6y ago

Since calling syscalls directly instead of through the wrapper dll is unsupported, I wouldn't be surprised if MS prevented syscalls from outside the expected libraries at some point. OpenBSD already does this to improve security: https://marc.info/?l=openbsd-tech&m=157488907117170&w=2 https://marc.info/?l=openbsd-tech&m=157488907117170&w=2

guenthert 6y ago
I'm a bit at loss how this improves security or why it is necessary. If the kernel needs an external library as gatekeeper, something fundamentally went wrong. And why would, say a Lisp compiler use the C library (or to use a more popular example, why would a Java VM do the same)? It might be common practice (to ease FFI/JNI), but making this a requirement seems heading the wrong direction.
- saagarjha 6y ago
  I believe libc is the only stable way to ask the kernel to do something for you.
  - kazinator 6y ago
    GNU Libc is only stable at the source code level, and tries fairly hard at the binary level with approaches like symbol versioning and whatnot. Linux syscalls are more stable at the actual binary level. Some "int $0x80" code you wrote in 1996 in x86 assembly language will work today, provided the binary format (like a.out) will load. The libc which was used then is not even around any more.
    - saagarjha 6y ago
      Ah, I was talking about OpenBSD's kernel interface. Linux is ABI stable at the syscall level, of course–I know it tries to be binary compatible, but I have never actually tested this much.
      - 6y ago

magicalhippo 6y ago

Despite having coded for Windows for quite some time, I realized I had never actually peeked that low before, so didn't have a good understanding of how system calls really worked. Found this article[1] which explained it nicely (and hopefully correctly). [1]: https://www.codeguru.com/cpp/w-p/system/devicedriverdevelopment/article.php/c8035/How-Do-Windows-NT-System-Calls-REALLY-Work.htm https://www.codeguru.com/cpp/w-p/system/devicedriverdevelopm...

nxc18 6y ago

This seems like it would be a lot of maintenance effort to get right, considering Windows shuffles their syscall table every release, including releases within the Windows 10 family. https://j00ru.vexillium.org/syscalls/nt/64/ https://j00ru.vexillium.org/syscalls/nt/64/ Has an excellent table of syscalls by version.

gizmo686 6y ago
If applications are calling these syscalls directly; then there needs to be some form of stability; otherwise, the applications would only work on the particular version of Windows they were compiled for.
- frabert 6y ago
  The Win32 API (and nowadays the WinRT API as well) is that interface, system calls are explicitly unstable and internal-use-only I believe.
  - gizmo686 6y ago
    If all windows applications used the APIs like they were supposed to, then there would be no need to emulate the Windows syscalls. The point of this emulation is that some applications (games, apparently) were not using these APIs, and instead performing syscalls directly. In order for this to be feasable on Windows, there needs to be some stability in the syscalls; even if that stability involves angry MS engineers complaining that major applications are doing something that they were explicitly told not to do.
    - leeter 6y ago
      You'd be surprised, MS doesn't do this randomly. They'd rather break a game if the game is actively in production or the company is around. They only put in shims for applications if the vendor isn't still in business usually or can't get source code. But in quite a few cases they'll just put in a patch binary in the shims that just removes the anti-cheat all together from the binary when loaded. Game still runs... nobody cares.

jhallenworld 6y ago

Linux used to be able to run Xenix and SCO UNIX applications- I have not tried this in years, but I'm wondering if the mechanism can be shared such that any OS can be emulated.

lizknope 6y ago
https://en.wikipedia.org/wiki/Intel_Binary_Compatibility_Standard https://en.wikipedia.org/wiki/Intel_Binary_Compatibility_Sta... The Intel Binary Compatibility Standard (iBCS) is a standardized application binary interface (ABI) for Unix operating systems on Intel-386-compatible computers, published by AT&T, Intel and SCO in 1988, and updated in 1990.
- jhallenworld 6y ago
  Yes, also I remember that SCO UNIX worked but SCO Xenix did not work. But really I wanted Xenix to work.. I seem to remember it worked once, but then stopped working when they implemented the standard. Or something- it was a long time ago.
  - lizknope 6y ago
    I had a friend get WordPerfect for SCO working under Linux. That was around 1996. I found these instructions. https://www.tldp.org/HOWTO/WordPerfect-5.html https://www.tldp.org/HOWTO/WordPerfect-5.html Same friend had a DEC Alpha and got Digital Unix Netscape to run under Linux on the Alpha. http://users.bart.nl/~geerten/FAQ-17.html http://users.bart.nl/~geerten/FAQ-17.html
tzs 6y ago
Were those 286 Xenix applications or 386 Xenix applications? Running 286 Xenix applications on a 386 or later Unix or Unix-like system is fairly straightforward as far as the 386 kernel is concerned. 286 Xenix used a different syscall mechanism than 386 Unixes did, so you didn't have the problem of the kernel needing to deal with 286 and 386 syscalls ending up in the same place. My recollection, from being half the two person team that implemented 286 Unix binary compatibility for System V Release 3.2 on 386 at ISC, and then being half the different two person team that did 286 Xenix binary compatibility, was that this is what was required in the kernel: 1. Adding a mechanism to allow a process to set up 16-bit segments as aliases to portions of its address space. 2. Modifying exec to recognize a 286 Unix or Xenix binary, and turn the exec into an exec of /bin/i286emul or /bin/x286emul, with the path to the 286 binary as an argument. 3. I don't remember if we had to do anything special for the 286 system call mechanism, because I don't actually remember what that mechanism was. I don't remember if what the binaries would try to do for a syscall would simply cause a trapable signal, and then the signal handler would recognize the system call and take care of it, of if the system call mechanism used something like a call gate that the 386 code had to set up, and so we had to add a mechanism to allowed the 386 code to do that. That was almost all the kernel work. Pretty much everything else could be handled in user mode code. When you would run a 286 Xenix binary, that turned into an exec of /bin/x286emul. x286emul would read the Xenix binary, allocate memory for it and load it, map segments to it, and jump to the code, switching the processor to 16-bit mode. When the Xenix code did a system call and it ended up in the handler in x286emul, for most system calls it was simply a matter of copying arguments from where the 286 call put them to where the corresponding 386 call expected them, making the 386 system call, then putting results in the right place, and returning to the Xenix code. For some system calls, more work was needed. Signal handling, for instance. If the 286 code wanted to trap a signal, the 386 code would have to set a trap of its own, and its handler would then have to deal with the stack fiddling and such to deliver it to the 286 code. Speaking of signal handling, that led to the stupidest, most annoying meeting I've ever had to attend. ISC had done the official 386 port of System V Release 3 under contract from AT&T, and we did the 286 Unix binary compatibility as part of that. Later, there was a deal between AT&T and Sun and Microsoft to make Unixes more compatible, which included Xenix compatibilty, and we were doing the Xenix compatibility as part of that, under contract from Microsoft. For the 286 binary compatibility, ISC did the whole thing--all the user mode code in i286emul and all the supporting kernel changes. But for Xenix compatibilty, ISC was just doing x286emul. Microsoft was doing any kernel modifications needed, which wasn't very much because the mods already there for i286emul mostly worked find for x286emul. There was one difference between Xenix and Unix signal handling that we could not fully address in x286emul. We needed kernel support--some sort of per-process flag that would tell it the processes wanted signals to behave like Xenix signals. We asked Microsoft to add such a flag. A bit later they got back to us and said there was issues with such a flag that could not be settled by email or phone. We had to have an in-person meeting. So me and the other guy working no x286emul had to fly early one morning from Los Angeles to Seattle, take a rental car to Redmond, and attend a meeting with Microsoft. At the meeting, we introduced ourselves and the half dozen or so Microsoft people introduced themselves, and then they presented this issue that could not be handled by email or phone: should this flag be controlled by an ioctl, or should it be an optional new parameter for the signal() call [1]. We gave our opinion, that was the end of the meeting, and we headed for the airport. [1] or something like that. I don't remember the exact second option.