8 ms·

Emulating win32

145 points by zdw 4y ago

kreelman 4y ago
Good luck with this. Looks interesting.
jamesfinlayson 4y ago
> In the specific case of Windows the kernel interface is pretty fiddly (the syscall ids vary between Windows releases) Interesting - I wouldn't have expected that. Anyone know why that's the case?
- p_l 4y ago
  Windows never used syscalls officially - the Windows API evolved from essentially having the operating system be inside few shared libraries that your program linked with. Even once the idea of syscall started showing up with protected mode and NT, the applications wouldn't directly invoke them, but through interfaces in the libraries. Thus there was never really a proper need to have stable syscall interface, because at least officially no application was supposed to use them - they were private implementation details of various DLLs.
  - ryao 4y ago
    These days, you have DRM solutions doing syscalls directly. I think Jurassic World evolution is an example of software using DRM that does that. The wine developers had to implement the ability to emulate windows syscalls because of it. I wonder how Microsoft will react to third parties beginning to do syscalls.
    - userbinator 4y ago
      "beginning to"? This has been around since the early NT days, basically as long as DRM (and thus anti-DRM) has existed. Microsoft's reaction? Nothing but occasionally changing the syscall numbers.
    - yjftsjthsd-h 4y ago
      I'm shocked that that works - I was given to understand that MS outright shuffles syscalls per build to make them completely unreliable from userspace.
      - p_l 4y ago
        A lot of DRM (especially game DRM) solutions don't care about reliability or stability. If they figure a way that makes it harder to reverse or run or whatever, they will do it. Even if they are essentially rootkits bad enough someone really should try to see if CFAA would stick.
  - userbinator 4y ago
    In particular, Windows 9x has a completely different syscall interface (if you could even call it that) than the NT line; the majority of kernel32 functions seem to go through a far call to a fixed address: https://news.ycombinator.com/item?id=13264007 https://news.ycombinator.com/item?id=13264007 The above-linked handful of pages may be the only detailed documentation of the 9x syscall interface that is still reachable by search engines today.
- TeMPOraL 4y ago
  IIRC it's the other way around - it's Linux exposing syscalls for direct access that's unusual.
  - yjftsjthsd-h 4y ago
    Yep; Darwin doesn't do raw syscalls, the BSDs don't (OpenBSD going so far as to block anything but libc from making syscalls, as a security feature), illumos doesn't... actually the only OS I can think of off-hand where applications regularly use syscalls is DOS, and even then it looks more like a side-effect of x86 architecture and the constraints that it was developed under. Go hit this particularly hard, having mostly grown up on Linux and then finding out the hard way that using libc isn't really optional on other systems.
    - kevin_thibedeau 4y ago
      DOS syscalls are an outgrowth of the CP/M hardware abstraction. It isn't tied to x86.
    - wongarsu 4y ago
      On well designed systems (and I will count Windows in that group), libc is optional and syscalls are decoupled into another library. After all (unlike what some BSDs think) not everything is C, and forcing everyone to use the libc abstractions isn't very nice.
- detaro 4y ago
  Applications are not supposed to ever use them, but rather make use of the system libraries to access them. So having stable IDs is somewhere between not a priority and something to be avoided to stop developers ignoring that rule.
- monocasa 4y ago
  Interestingly they started in more or less in alphabetical order. It certainly looks for the outside like there's a big file that gets updated wherever the dev adding the syscall thinks looks clean, and gets reorganized every now and then. Combined with the obvious 'it's not a stable interface so we like changing it up regularly so people don't feel like they can depend on it'. Here's the table itself for 32 bit x86 versions if anyone's curious: https://j00ru.vexillium.org/syscalls/nt/32/ https://j00ru.vexillium.org/syscalls/nt/32/
- wongarsu 4y ago
  I'm pretty sure Windows syscall ids are automatically generated in the build process, with no special effort to keep them consistent between releases. Since the code that receives syscalls (kernel) and the code that calls them (shared libraries like user32.dll or ntdll.dll) are shipped together that doesn't cause any problems, and is probably more convinient than manually mapping numbers to function calls.
MuffinFlavored 4y ago
> To execute a Windows .exe then you just load it into memory (which requires unpacking the .exe file format) and tell the processor to jump to the first instruction. Obviously a simplification, worth mentioning maybe at a slightly deeper level https://learn.microsoft.com/en-us/cpp/c-language/thread-local-storage?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-language/thread-loca... I think a PE32 (.exe) can have multiple sections like this that are implicitly started before main()/entrypoint basically
- userbinator 4y ago
  TLS callbacks are a relatively common trick used by packers/obfuscators. AFAIK, the only other code that can get executed before the main entrypoint of the PE would be the entrypoints ("DllMain") of its imported DLLs.
  - andrewf 4y ago
    Lots of 32-bit Windows apps shipped with 16-bit installers (old versions of InstallShield and the like). When MS dropped Win16 support from x64 Windows, they built native implementations which "ran" the 16-bit installer based on the resources packed into it. In console emulation terms, they supported 16-bit InstallShield and friends with a HLE (High Level Emulation) approach rather than continuing to execute the original binaries on top of a low-level emulation. For a hobby/minimalist Win32 implementation I wonder if HLE support for popular packers/obfuscators would be easier than getting the low level details right?
chungy 4y ago
> The second approach is to instead emulate just the x86 instruction set, and use Wine as described above as an implementation of the massive Windows API backing onto something more tractable to then emulate, such as the Linux kernel API. This is also the approach of qemu user-space, which went unmentioned in the article. It can emulate another ISA and translate the syscalls to your host version of Linux. This way you can run x86 Wine on, say, ARM or RISC-V systems. From a user perspective, if everything is set up so, Windows applications for x86 will run just as well as ARM Linux applications.
- zozbot234 4y ago
  Ah, but can you emulate calls to a shared library, translating ABI's and binary structure definitions seamlessly across architectures? I suppose that this may well be impossible in the general case.
  - saagarjha 4y ago
    If shared libraries are the level at which your OS exposes its interfaces, then yes.
tombert 4y ago
I have been saying that we need a “dosbox for the next generation”. Wine is great but due to Apple dropping 64 bit support it’s been decidedly less useful for me. DosBox works well enough for most pre-Windows stuff, DosBox-X can handle the Windows 95 era well enough, but Windows 2000-Windows 7 stuff is actually kind of hard to get fully working on modern computers, and I feel emulation has not been able to keep up. I wish a lot of success on this project. It would be great to get Jazz Jackrabbit 2 working on my Mac without mucking with VMWare.
- my123 4y ago
  Apple does support Wine on modern macOS, with specific segmentation support to run 32-bit apps in that environment.
  - tombert 4y ago
    How do I go about doing that? Whenever I try and run Wine it gives me weird errors.
    - kitsunesoba 4y ago
      I have no idea what's happening under the hood, but PlayOnMac[0] seems to poke WINE the right way to make it work as expected on newer macOS. [0]: https://www.playonmac.com/en/ https://www.playonmac.com/en/
- zamadatix 4y ago
  I use CrossOver to run a 32 bit Windows app designed in the early 2000s on my M1 Mac, works great. Used to use Parallels but found it to just not be as good, plus they got really spammy with the marketing emails for a while so I just cut it out.
- TheDesolate0 4y ago
  There's a pretty easy, cheap, and straight forward fix for this. Stop using apple products. They are luke-warm in design and terrible in execution. Get a PC: Supports Linux, windows, all your games (ALL) and has a mouse with more than one button.
andrewf 4y ago
winevdm came to mind as a rather different Windows API + CPU emulator project - it glues WinAPI thunking to MAME's x86 emulator to run Win16 apps against the Win32 API. On 32-bit x86 this is redundant, because the built-in WoW32 will do the same job. winevdm enables running Win16 apps on 64-bit Windows on x64 and ARM. The closed Github issues are a trip. https://github.com/otya128/winevdm/issues?q=is%3Aissue+is%3Aclosed https://github.com/otya128/winevdm/issues?q=is%3Aissue+is%3A... In the words of its own README: CPU Emulator wine based Win16->Win32 conversion codes: Relay routines from 16-bit to 32-bit are autogenerated by convspec DOS emulation for Win16 16-bit <=> native HANDLE conversion Fix compatibility problems, fix compatibility problems
- qwezxcrty 4y ago
  It's a wonderful project. Allowing me to run a 30 years old Win16 companion program (for a HP 53310A frequency domain analyzer) on modern 64bits Windows.
omgmajk 4y ago
This article is a rabbit hole of links that I happily wasted two hours of work reading. Well done.
blueflow 4y ago
Nitpick: Wine is as much a windows emulator as xterm is a terminal emulator.
- phoe-krk 4y ago
  From `man xterm`: xterm - terminal emulator for X From Wikipedia entry for Wine: Wine (formerly a recursive backronym for Wine Is Not an Emulator (...))
  - deleted 4y ago
    [deleted]
  - blueflow 4y ago
    Neither wine nor xterm have virtualized cpu or memory, they both imitate (= emulate) some system via their interfaces, yet one is primarily an emulator and the other is not one. That does not make sense. The wine project changed name to avoid the bad reputation emulators had back then due to their poor performance. "Wine is not an emulator" is not factually accurate, its a social workaround.
    - actionfromafar 4y ago
      Words can have different meanings in different contexts. "Wine is not an emulator" was a very useful tagline to indicate that it was fast, as opposed to virtual machines like Virtual PC, PCTask etc.
      - blueflow 4y ago
        "My bicycle is not a vehicle because it doesn't have a engine"
  - vasvir 4y ago
    Actually "Wine is not an emulator" is factually correct. Wine is a re-implementation of win32 on top of (near) POSIX OSes like Linux.
tambourine_man 4y ago
Imagine if WINE had become the defacto cross platform standard, instead of Electron. Or some modernizing shim on top of it. The amount of memory and CPU we could have saved.
- flatiron 4y ago
  Wine doesn’t help you on iOS or the web. It could never have been as cross platform as a browser
  - eurasiantiger 4y ago
    BoxedWine would like to disagree.
    - ape4 4y ago
      Maybe its not too late
  - tambourine_man 4y ago
    Yeah, I though about mobile as soon as I hit “add comment”. Still, there must be a better way. I'm rooting for all the lighter weight alternatives.
  - saagarjha 4y ago
    You're not trying hard enough.
- augusto-moura 4y ago
  I truly believe that the success of Electron is not limited to only being cross-platform. It is a very compelling motivation, really. But I think one of the key aspects is that the HTML/CSS/JS ecosystem is so huge for building UIs that is a lot simpler than to rely on the classic UI desktop frameworks
  - tambourine_man 4y ago
    You’re correct. We need to solve the resource intensive issue while still allowing people to write apps with a web stack
zerr 4y ago
> if you have the source code of a Windows program you can compile it against Wine's implementation of the Windows API and get a native executable out the other side Interesting if anyone tried this on Notepad++.

145 points by zdw 4y ago

Good luck with this. Looks interesting.

> In the specific case of Windows the kernel interface is pretty fiddly (the syscall ids vary between Windows releases) Interesting - I wouldn't have expected that. Anyone know why that's the case?

p_l 4y ago
Windows never used syscalls officially - the Windows API evolved from essentially having the operating system be inside few shared libraries that your program linked with. Even once the idea of syscall started showing up with protected mode and NT, the applications wouldn't directly invoke them, but through interfaces in the libraries. Thus there was never really a proper need to have stable syscall interface, because at least officially no application was supposed to use them - they were private implementation details of various DLLs.
- ryao 4y ago
  These days, you have DRM solutions doing syscalls directly. I think Jurassic World evolution is an example of software using DRM that does that. The wine developers had to implement the ability to emulate windows syscalls because of it. I wonder how Microsoft will react to third parties beginning to do syscalls.
  - userbinator 4y ago
    "beginning to"? This has been around since the early NT days, basically as long as DRM (and thus anti-DRM) has existed. Microsoft's reaction? Nothing but occasionally changing the syscall numbers.
  - yjftsjthsd-h 4y ago
    I'm shocked that that works - I was given to understand that MS outright shuffles syscalls per build to make them completely unreliable from userspace.
    - p_l 4y ago
      A lot of DRM (especially game DRM) solutions don't care about reliability or stability. If they figure a way that makes it harder to reverse or run or whatever, they will do it. Even if they are essentially rootkits bad enough someone really should try to see if CFAA would stick.
- userbinator 4y ago
  In particular, Windows 9x has a completely different syscall interface (if you could even call it that) than the NT line; the majority of kernel32 functions seem to go through a far call to a fixed address: https://news.ycombinator.com/item?id=13264007 https://news.ycombinator.com/item?id=13264007 The above-linked handful of pages may be the only detailed documentation of the 9x syscall interface that is still reachable by search engines today.
TeMPOraL 4y ago
IIRC it's the other way around - it's Linux exposing syscalls for direct access that's unusual.
- yjftsjthsd-h 4y ago
  Yep; Darwin doesn't do raw syscalls, the BSDs don't (OpenBSD going so far as to block anything but libc from making syscalls, as a security feature), illumos doesn't... actually the only OS I can think of off-hand where applications regularly use syscalls is DOS, and even then it looks more like a side-effect of x86 architecture and the constraints that it was developed under. Go hit this particularly hard, having mostly grown up on Linux and then finding out the hard way that using libc isn't really optional on other systems.
  - kevin_thibedeau 4y ago
    DOS syscalls are an outgrowth of the CP/M hardware abstraction. It isn't tied to x86.
  - wongarsu 4y ago
    On well designed systems (and I will count Windows in that group), libc is optional and syscalls are decoupled into another library. After all (unlike what some BSDs think) not everything is C, and forcing everyone to use the libc abstractions isn't very nice.
detaro 4y ago
Applications are not supposed to ever use them, but rather make use of the system libraries to access them. So having stable IDs is somewhere between not a priority and something to be avoided to stop developers ignoring that rule.
monocasa 4y ago
Interestingly they started in more or less in alphabetical order. It certainly looks for the outside like there's a big file that gets updated wherever the dev adding the syscall thinks looks clean, and gets reorganized every now and then. Combined with the obvious 'it's not a stable interface so we like changing it up regularly so people don't feel like they can depend on it'. Here's the table itself for 32 bit x86 versions if anyone's curious: https://j00ru.vexillium.org/syscalls/nt/32/ https://j00ru.vexillium.org/syscalls/nt/32/
wongarsu 4y ago
I'm pretty sure Windows syscall ids are automatically generated in the build process, with no special effort to keep them consistent between releases. Since the code that receives syscalls (kernel) and the code that calls them (shared libraries like user32.dll or ntdll.dll) are shipped together that doesn't cause any problems, and is probably more convinient than manually mapping numbers to function calls.

MuffinFlavored 4y ago

> To execute a Windows .exe then you just load it into memory (which requires unpacking the .exe file format) and tell the processor to jump to the first instruction. Obviously a simplification, worth mentioning maybe at a slightly deeper level https://learn.microsoft.com/en-us/cpp/c-language/thread-local-storage?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-language/thread-loca... I think a PE32 (.exe) can have multiple sections like this that are implicitly started before main()/entrypoint basically

userbinator 4y ago
TLS callbacks are a relatively common trick used by packers/obfuscators. AFAIK, the only other code that can get executed before the main entrypoint of the PE would be the entrypoints ("DllMain") of its imported DLLs.
- andrewf 4y ago
  Lots of 32-bit Windows apps shipped with 16-bit installers (old versions of InstallShield and the like). When MS dropped Win16 support from x64 Windows, they built native implementations which "ran" the 16-bit installer based on the resources packed into it. In console emulation terms, they supported 16-bit InstallShield and friends with a HLE (High Level Emulation) approach rather than continuing to execute the original binaries on top of a low-level emulation. For a hobby/minimalist Win32 implementation I wonder if HLE support for popular packers/obfuscators would be easier than getting the low level details right?

chungy 4y ago

> The second approach is to instead emulate just the x86 instruction set, and use Wine as described above as an implementation of the massive Windows API backing onto something more tractable to then emulate, such as the Linux kernel API. This is also the approach of qemu user-space, which went unmentioned in the article. It can emulate another ISA and translate the syscalls to your host version of Linux. This way you can run x86 Wine on, say, ARM or RISC-V systems. From a user perspective, if everything is set up so, Windows applications for x86 will run just as well as ARM Linux applications.

zozbot234 4y ago
Ah, but can you emulate calls to a shared library, translating ABI's and binary structure definitions seamlessly across architectures? I suppose that this may well be impossible in the general case.
- saagarjha 4y ago
  If shared libraries are the level at which your OS exposes its interfaces, then yes.

tombert 4y ago

I have been saying that we need a “dosbox for the next generation”. Wine is great but due to Apple dropping 64 bit support it’s been decidedly less useful for me. DosBox works well enough for most pre-Windows stuff, DosBox-X can handle the Windows 95 era well enough, but Windows 2000-Windows 7 stuff is actually kind of hard to get fully working on modern computers, and I feel emulation has not been able to keep up. I wish a lot of success on this project. It would be great to get Jazz Jackrabbit 2 working on my Mac without mucking with VMWare.

my123 4y ago
Apple does support Wine on modern macOS, with specific segmentation support to run 32-bit apps in that environment.
- tombert 4y ago
  How do I go about doing that? Whenever I try and run Wine it gives me weird errors.
  - kitsunesoba 4y ago
    I have no idea what's happening under the hood, but PlayOnMac[0] seems to poke WINE the right way to make it work as expected on newer macOS. [0]: https://www.playonmac.com/en/ https://www.playonmac.com/en/
zamadatix 4y ago
I use CrossOver to run a 32 bit Windows app designed in the early 2000s on my M1 Mac, works great. Used to use Parallels but found it to just not be as good, plus they got really spammy with the marketing emails for a while so I just cut it out.
TheDesolate0 4y ago
There's a pretty easy, cheap, and straight forward fix for this. Stop using apple products. They are luke-warm in design and terrible in execution. Get a PC: Supports Linux, windows, all your games (ALL) and has a mouse with more than one button.

andrewf 4y ago

winevdm came to mind as a rather different Windows API + CPU emulator project - it glues WinAPI thunking to MAME's x86 emulator to run Win16 apps against the Win32 API. On 32-bit x86 this is redundant, because the built-in WoW32 will do the same job. winevdm enables running Win16 apps on 64-bit Windows on x64 and ARM. The closed Github issues are a trip. https://github.com/otya128/winevdm/issues?q=is%3Aissue+is%3Aclosed https://github.com/otya128/winevdm/issues?q=is%3Aissue+is%3A... In the words of its own README: CPU Emulator wine based Win16->Win32 conversion codes: Relay routines from 16-bit to 32-bit are autogenerated by convspec DOS emulation for Win16 16-bit <=> native HANDLE conversion Fix compatibility problems, fix compatibility problems

qwezxcrty 4y ago
It's a wonderful project. Allowing me to run a 30 years old Win16 companion program (for a HP 53310A frequency domain analyzer) on modern 64bits Windows.

omgmajk 4y ago

This article is a rabbit hole of links that I happily wasted two hours of work reading. Well done.

blueflow 4y ago

Nitpick: Wine is as much a windows emulator as xterm is a terminal emulator.

phoe-krk 4y ago
From `man xterm`: xterm - terminal emulator for X From Wikipedia entry for Wine: Wine (formerly a recursive backronym for Wine Is Not an Emulator (...))
- deleted 4y ago
  [deleted]
- blueflow 4y ago
  Neither wine nor xterm have virtualized cpu or memory, they both imitate (= emulate) some system via their interfaces, yet one is primarily an emulator and the other is not one. That does not make sense. The wine project changed name to avoid the bad reputation emulators had back then due to their poor performance. "Wine is not an emulator" is not factually accurate, its a social workaround.
  - actionfromafar 4y ago
    Words can have different meanings in different contexts. "Wine is not an emulator" was a very useful tagline to indicate that it was fast, as opposed to virtual machines like Virtual PC, PCTask etc.
    - blueflow 4y ago
      "My bicycle is not a vehicle because it doesn't have a engine"
- vasvir 4y ago
  Actually "Wine is not an emulator" is factually correct. Wine is a re-implementation of win32 on top of (near) POSIX OSes like Linux.

tambourine_man 4y ago

Imagine if WINE had become the defacto cross platform standard, instead of Electron. Or some modernizing shim on top of it. The amount of memory and CPU we could have saved.

flatiron 4y ago
Wine doesn’t help you on iOS or the web. It could never have been as cross platform as a browser
- eurasiantiger 4y ago
  BoxedWine would like to disagree.
  - ape4 4y ago
    Maybe its not too late
- tambourine_man 4y ago
  Yeah, I though about mobile as soon as I hit “add comment”. Still, there must be a better way. I'm rooting for all the lighter weight alternatives.
- saagarjha 4y ago
  You're not trying hard enough.
augusto-moura 4y ago
I truly believe that the success of Electron is not limited to only being cross-platform. It is a very compelling motivation, really. But I think one of the key aspects is that the HTML/CSS/JS ecosystem is so huge for building UIs that is a lot simpler than to rely on the classic UI desktop frameworks
- tambourine_man 4y ago
  You’re correct. We need to solve the resource intensive issue while still allowing people to write apps with a web stack

zerr 4y ago

> if you have the source code of a Windows program you can compile it against Wine's implementation of the Windows API and get a native executable out the other side Interesting if anyone tried this on Notepad++.