7 ms·

Windows: Prefer the Native API over Win32

71 points by nikbackm 8mo ago

pseudohadamard 8mo ago
For people not familiar with Windows development, another name for the NT native API is "the API that pretty much every document on Windows programming tells you not to use". It's like coding to the Linux syscall interface instead of libc.
- nullpoint420 8mo ago
  Yeah, I know go has had issues because they subvert libc themselves in similar fashion. I wonder how this will turn out.
  - LoganDark 8mo ago
    I think they had to revert back to libc on macOS/iOS because those have syscall interfaces that truly are not stable (and golang found that out the hard way). I wonder if they had to do the same on BSDs because of syscall filtering.
    - samus 8mo ago
      Indeed, OpenBSD recently added hardening measures and started restricting the generic syscall interface to libc.
  - wrs 8mo ago
    Go backed out of their strategy on MacOS and started using libc (libsystem?), because when Apple says something is internal and may change without notice, they really mean it. It may be a better risk with Microsoft, but it’s still a risk.
  - lelanthran 8mo ago
    Same way it turned out for Go: they had to walk it back.
- HexDecOctBin 8mo ago
  Linux syscall interface is actually stable and can easily be targeted. It’s BSDs (and Mac OS) that force everyone to link to only libc.
  - pjmlp 8mo ago
    More like everyone else, Linux kernel is the exception here.
- kvemkon 8mo ago
  > It's like ... Considering the level of the API. But it is total opposite comparing a bit deeper. Linux has a famous rule "WE DO NOT BREAK USERSPACE!" e.g. [1]. [1] https://news.ycombinator.com/item?id=44611692 https://news.ycombinator.com/item?id=44611692
  - dunder_cat 8mo ago
    One thing that is amusing about the prevalence of advanced anti-cheat in Windows gaming is it's actually causing said API/ABIs to undergo ossification. A good data point is the invention of Syscall User Dispatch^1 on Linux which would allow a program to basically install a syscall handler when they originate from various regions of memory. I do not know how usable this is in practice, admittedly -- but I think the fact it was contributed at all speaks to the growing need. ^1 https://docs.kernel.org/admin-guide/syscall-user-dispatch.html https://docs.kernel.org/admin-guide/syscall-user-dispatch.ht...
- koakuma-chan 8mo ago
  > It's like coding to the Linux syscall interface instead of libc. The right thing to do? I don't see why I would want to use libc.
  - delta_p_delta_x 8mo ago
    On Windows, the stability guarantees are opposite to that of Linux. The kernel ABI is not guaranteed to be stable, whereas the Win32 ABI is. And frankly, the Windows way is better. On Linux, the 'ABI' for nearly all user-mode programs is not the kernel's ABI but rather glibc's (plus the variety of third-party libraries, because Win32 has a massive surface area and is an all-in-one API). Now, glibc's ABI constantly changes, so linking against a newer glibc (almost certainly the 'host' glibc, because it is almost impossible to supply a different 'target' glibc without Docker) will result in a program that doesn't run on older glibc. So much for Torvalds' 'don't break userspace'. Not so for a program compiled for 'newer' Win32; all that matters are API compatibilities. If one only uses old-hat interfaces that are documented to be present on Windows 2000, one can write and compile one's code on Windows 11, and the executable will run on the former with no issues. And vice versa, actually.
    - monocasa 8mo ago
      A lot of the native API is considered stable these days. The actual signals aren't, but the wrappers in ntdll are.
    - samus 8mo ago
      The Win32 ABI is also just a wrapper on the native API, which is only stable in practice, but not officially according to any Microsoft documentation. Glibc is userspace seen from the perspective of the Linux kernel.
      - delta_p_delta_x 8mo ago
        It doesn't really matter if it's 'just a wrapper', because said wrapper provides an ABI. Even if the underlying Native API changes, the interface the wrapper presents to other compiled binaries won't. The latter will contain caller/callee register setup, type layouts, function arguments and more for that wrapper. Cygwin is also 'just a wrapper' for the Native API and Win32, and look how drastically it changes the ABI of applications.
- pjmlp 8mo ago
  Except unlike Linux syscall interface and like almost every other OS out there, ABI compatibility is an accident, not a guarantee.
- eps 8mo ago
  "Every document" notwithstanding, Native API is very widely used in practice and generally considered stable. If in doubt, try and find examples of its breakage, semantic changes, etc.
- samus 8mo ago
  With the crucial difference that Linux places high value on syscall interface binary compatibility, while the NT native API is not guaranteed to be stable in any way. A bit more comparable is OpenBSD where applications are very much expected to only use libc wrappers, which threw a wrench into the works for the Go runtime.
- Narishma 8mo ago
  > It's like coding to the Linux syscall interface instead of libc. It's the opposite of that. The Linux syscall is more stable than the (gnu)libc.
- josephcsible 8mo ago
  > It's like coding to the Linux syscall interface instead of libc. Which is perfectly fine to do and guaranteed to work forever because of Linus's policy that kernel updates aren't allowed to break userspace programs.
deleted 8mo ago
[deleted]
lostmsu 8mo ago
Fool's errand. Apps built with this will have to be maintained forever (vs the apps from Win 9x which still work in Windows 11).
- audunw 8mo ago
  The reason apps from Win 9x runs on Windows 11 is that MS puts a ton of effort into explicitly supporting old apps. For popular apps that includes supporting undocumented APIs and even app-specific bug compatibility. Putting it on app developers to account for infinite forward compatibility is not at all reasonable. The best outcome would be if many Zig apps become popular enough that Windows is forced to maintain backward compatibility for ntdll. The API is clearly superior to win32 as many other developers have discovered and discussed before. It’d be nice to force MS to take low lever programming seriously instead of chasing AI slop.
  - lostmsu 8mo ago
    > The reason apps from Win 9x runs on Windows 11 is that MS puts a ton of effort into explicitly supporting old apps. No, that's one of the reasons. The other one is that public kernel32 -> private ntdll design works.
roelschroeven 8mo ago
> > Won't this get flagged by anti-virus scanners as suspicious? > Unfortunately, yes. We consider this a problem for the anti-virus scanners to solve. I don't think the anti-virus scanners consider Zig important enough, or even know about. They will not be the ones experiencing problems. Having executables quarantined and similar problems will fall on Zig developers and users of their software. That seems like a major drawback for using Zig.
- monocasa 8mo ago
  Yeah, I had this problem when shipping go binaries on Windows. Antivirus vendors really do not care that your program regularly shows up as a false positive due to their crappy heuristics, even if you have millions of users.
  - anfragment 8mo ago
    Have you tried code-signing with an EV certificate? If so, did it help? Asking for a friend.
    - monocasa 8mo ago
      Statistically notable improvement, but it didn't help a whole lot.
      - yndoendo 8mo ago
        I always upload a copy to https://www.virustotal.com https://www.virustotal.com to help combat the false positives. It was really bad a couple years ago because anything wrapped in Inno Setup kept being flagged. Now maybe one or two flag vendors do; Bkav Pro and CrowdStrike Falcon are the dominate culprits always.
        monocasa 8mo ago
        Uploading to virustotal doesn't really do anything to combat false positives AFAICT. It only lets you test against many AV vendors at once.
cmovq 8mo ago
Is there an official stance on whether ntdll is stable? Obviously they're not going to change things arbitrarily since applications depend on it, but I'm wondering if there is a guarantee like the linux syscall interface or how you can run a win32 application compiled in 2004 on Win11.
- monocasa 8mo ago
  It's partially stable. Basically any thing documented on msdn in the API docs is considered stable. Such as: https://learn.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile https://learn.microsoft.com/en-us/windows/win32/api/winternl...
  - delta_p_delta_x 8mo ago
    Indeed. Anything documented has a function wrapper. `NtCreateFile` is a function wrapper for the syscall number, so any user-mode code that has `NtCreateFile` instead of directly loading the syscall number 0x55 will be stable. The latter might not. In fact, it is not; the number has increased by 3 since Windows XP[1]. One could probably produce some sort of function pointer loader library with these tables, but at that point... Why not just use the documented APIs? [1]: https://github.com/j00ru/windows-syscalls/blob/8a6806ac9148633e8afbbdb42e6002930b002741/x64/csv/nt.csv#L100 https://github.com/j00ru/windows-syscalls/blob/8a6806ac91486...
    - Dwedit 8mo ago
      Only Malware uses the system call numbers directly. Using the system call numbers directly is foolish if they're going to change and break your app. Just import and call a function that will perform the actual SYSENTER (or WOW64 context change).
      - monocasa 8mo ago
        Unfortunately, that's not the case. Wine for instance has to keep up to date to maintain compatibility with some applications. https://gitlab.winehq.org/wine/wine/-/releases/wine-11.0 https://gitlab.winehq.org/wine/wine/-/releases/wine-11.0 > NT system calls use the same syscall numbering as recent Windows, to support applications that hardcode syscall numbers.
bob1029 8mo ago
Why not use both DLLs? Prefer win32 wherever possible and use the lower level APIs only if absolutely necessary. Benchmark after you have figured this out. Performance is probably not a thing at this level of abstraction.
- nvme0n1p1 8mo ago
  What makes you think they haven't benchmarked? Here's one fun example from following development on Zulip: advapi.dll loads bcrypt.dll, which loads bcryptprimitives.dll. bcryptprimitives.dll runs an internal test suite every time it's loaded into any process. So if you can avoid loading advapi.dll, your process will start faster.
  - delta_p_delta_x 8mo ago
    Is there a source for this? My Google- and GitHub-fu turns up nothing.
    - lelanthran 8mo ago
      He might be talking about cipher test that respected cryptography libs do on initialisation to verify integrity. Skipping those seem like a really bad idea.
      - josephcsible 8mo ago
        > Skipping those seem like a really bad idea. Why? Is there any realistic scenario where your cryptography libs worked correctly yesterday but the exact same ones will be buggy today? What would be wrong with them just running once per build instead?
    - nvme0n1p1 8mo ago
      Join their Zulip and search for bcryptprimitives. That's where I got my info.
  - lelanthran 8mo ago
dblohm7 8mo ago
This is a terrible idea! _Maybe_, _maybe_ using only the documented APIs with only the documented parameters. Unfortunately it makes too many false assumptions about interoperability between Win32 and the underlying native API that aren't true. For example (and the Go runtime does this, much to my chagrin), querying the OS version via the native API always gives you "accurate" version information without needing to link a manifest into your application. Unfortunately that lack of manifest will still cause many Win32 APIs above the native layer to drop into a compatibility mode, creating a fundamental inconsistency between what the application thinks the OS capabilities are versus which Win32 subsystem behaviours the OS thinks it should be offering.
boje 8mo ago
Honestly, this sounds like a future headache that would otherwise go unnoticed unless the programmer is dealing with porting or binding over source code meant for older Windows systems to Zig (or supporting older systems in general). Eventually it might result in a bunch of people typing out blogposts venting their frustrations, and the creation of tutorials and shims for hooking to Win32 instead of the Zig standard library with varying results. Which is fine, I suppose. Legacy compiler targets are a thing. This is already a problem with Linux binaries for systems that don't have a recent enough Glibc (unless the binaries themselves don't link to it and do syscalls directly).
cmovq 8mo ago
>> Microsoft are free to change the Native API at will, and you will be left holding both pieces when things break. > [...] the worst case scenario is really mild: A new version of windows comes out, breaking ntdll compatibility. Zig project adds a fix to the std lib. Application developer recompiles their zig project from source, and ships an update to their users. That assumes the application developer will continue to maintain it until the end of time. Also "the fix" would mean developers wanting to support earlier Windows versions would need to use an older std library? Or is the library going to have runtime checks to see what Windows build its running on?
- eps 8mo ago
  > Microsoft are free to change the Native API at will,... But they won't, because if there is one thing that Microsoft has always been extremely good at and cared for is backward compatibility. And changing Native API will break a ton of existing software, because even though undocumented it is very widely used.
  - dbdoskey 8mo ago
    Actually they do change the native API quite a bit. Not in minor releases so much but in major releases
    - eps 8mo ago
      They depricate some methods (very rarely and reasonably) and add new enums or struct versions to existing ones, but never change existing semantics, leave alone method signatures. As I said elsewhere, I invite you to find examples of actually destructive Native API changes.
  - blibble 8mo ago
    you are confusing the ntdll interface (which is undocumented and subject to change), and win32 (which is stable, mostly) they tell you not to use ntdll, and say they will change it whenever they want and they have in the past (they have had to moderate this policy with "containers", but it's still what they say)
qq66 8mo ago
> Comparing the comprehensive Win32 API reference against the incidentally documented Native APIs, its clear which one Microsoft would prefer you use. The native API is treated as an implementation detail, whilst core parts of Windows' backwards compatibility strategy are implemented in Windows subsystem. > A general-purpose programming language and toolchain for maintaining robust, optimal, and reusable software. Zig clearly doesn't actually care that much about building robust and reusable software if they're going to forgo Microsoft's decades-long backwards compatibility functionality for the dubious gains of using bare-metal APIs.
- slopinthebag 8mo ago
  lately it feels like zig is attempting to speed run irrelevance. which is a shame.
  - gjsman-1000 8mo ago
    I don't know, everyone here seems plenty okay to tolerate worse levels of instability from Linux binaries. :)
    - slopinthebag 8mo ago
      im too much of a mac user to understand what this references :p
      - gjsman-1000 8mo ago
        Take a random Linux binary which does anything non-trivial (has a GUI, does system monitoring, etc.), try running it on a different distribution from 3 years earlier without a packaging system, and tell me how it goes.
        lelanthran 8mo ago
        Zig is proposing the opposite problem: future versions of windows wont run even trivial zig programs from today. I can tell you that old Linux binaries run just fine on current distros. Looking at how many times you repeated your misunderstanding in this thread it's clear that, not only do you not understand the solution, you don't understand the problem either.
self_awareness 8mo ago
For me this is too much. I wish Zig all the best, but decisions like this make me want to jump off this sinking ship.
advisedwang 8mo ago
I'm really struggling to see the pros of this: > Performance - using the native API bypasses the standard Windows API, thus removing a software layer, speeding things up. But the article cites no bemchmarks > Power - some capabilities are not provided by the standard Windows API, but are available with the native API. Makes sense when you are doing something that needs that power, but that makes more sense as an exception to prefering win32 than a general reason to prefer native. > Dependencies - using the native API removes dependencies on subsystem DLLs, creating potentially smaller, leaner executables. Linking win32 is a miniscule cost. (unless you have a benchmark to show me...) > Flexibility - in the early stages of Windows boot, native applications (those dependent on NtDll.dll only) can execute, while others cannot. Is Zig being used for such applications? If so, why are the calls that the document says will be kept on win32 not an issue?
- sebazzz 8mo ago
  If I am not incorrect, for early boot applications, the application must be set to use the native subsystem.
moogly 8mo ago
Yikes. Are they going to rename the language to "cyg" also? Does not inspire confidence.
born-jre 8mo ago
Don’t know about windows programming to give opinion but by the sentiment here maybe they should give dev to choose bashed on some comptime flag or sth and maintain two versions
bmacho 8mo ago
> the worst case scenario is really mild: A new version of windows comes out, breaking ntdll compatibility. Zig project adds a fix to the std lib. Application developer recompiles their zig project from source, and ships an update to their users. The ~only good thing that programmers have achieved in the past ~60 years has been Windows stability. Create a popular programming language, and then make programs written in it not run on newer Windowses is just something else. I so hate this. Was "robust, optimal and reusable" always "run an older Windows on your newer Windows to run Zig software"?
- gjsman-1000 8mo ago
  ... this is just Linux binaries. It's humorous to me that we literally do exactly this, for Linux, with even less stability, but heaven forbid we do something approaching that on Windows despite the snobbery against Windows.
TazeTSchnitzel 8mo ago
Go famously tried to bypass macOS's libc and directly use the underlying syscall ABI, which is unstable, and then a macOS update came out and broke everything, which taught them the error of their ways (https://github.com/golang/go/issues/17490 https://github.com/golang/go/issues/17490). I wonder if this will happen to Zig too.
- CorrectHorseBat 8mo ago
  The same on OpenBSD
chad1n 8mo ago
Anyone who has some experience with native apis knows that a standard library should never rely on unstable apis. Ntdll is not "stable" as in Microsoft can change it at any time since they expect anyone to use kernel32. It's questionable that they referenced a random book on this top claiming that ntdll is more performant than kernel32 which is doubtful. There are some specific cases where this is true (the ntfs stuff), but, in general, it's not, at least not in a significant matter. A standard library should never do this, it might break binaries for no reason, other than making a cool blog post. I, as a developer, can choose to use ntfs, but a standard library should never. https://news.ycombinator.com/item?id=25997506 https://news.ycombinator.com/item?id=25997506 https://github.com/golang/go/issues/68678 https://github.com/golang/go/issues/68678
lelanthran 8mo ago
> While this can happen, we have not (yet) been affected by any changes in the Win32 -> Native layers. Frankly this is dumb. Zig hasn't been around long enough to have even seen any changes, so using this as a reason is just plain dumb. The view that, if windows ever changes, the code must be recompiled is a naive view one would expect from a child, not from a group of experienced devs.
Dwedit 8mo ago
The one thing that really benefits from using NT Native API over Win32 is listing files in a directory. You get to use a 64KB buffer to get directory listing results, while Win32 just does the one at a time. 64KB buffer size means fewer system calls. (Reading the MFT is still faster. Yes, you need admin for that)
- xfeeefeee 8mo ago
  Oh hey this is exactly why I made node-windows-readdir-fast - especially with the way node works, this makes reading filenames and length and times around 50x faster https://github.com/xfeeefeee/node-windows-readdir-fast https://github.com/xfeeefeee/node-windows-readdir-fast https://www.npmjs.com/package/node-windows-readdir-fast https://www.npmjs.com/package/node-windows-readdir-fast Windows only of course, but the concept is sound. Was also fun benchmarking to find out that parsing a binary stream was faster than creating a ton of objects through the node api (or json deserialization)
jojomodding 7mo ago
For comparison, in Rust they track down the differences in which flags are ignored in a certain kind of `fcntl` syscall across all the architectures that have this C function, which includes Solaris, Mac OS, the BSDs, Linux, ... This is so that this is correctly handled in Miri, which can then be used to run the test-suite of the OS-specific parts of the standard library, and observe if this uses unsupported features in some way. This ensures that the standard library relies on documented features and not on whatever happens to work right now. See for example this PR comment: https://github.com/rust-lang/miri/pull/4840#discussion_r2836627676 https://github.com/rust-lang/miri/pull/4840#discussion_r2836...

71 points by nikbackm 8mo ago

For people not familiar with Windows development, another name for the NT native API is "the API that pretty much every document on Windows programming tells you not to use". It's like coding to the Linux syscall interface instead of libc.

nullpoint420 8mo ago
Yeah, I know go has had issues because they subvert libc themselves in similar fashion. I wonder how this will turn out.
- LoganDark 8mo ago
  I think they had to revert back to libc on macOS/iOS because those have syscall interfaces that truly are not stable (and golang found that out the hard way). I wonder if they had to do the same on BSDs because of syscall filtering.
  - samus 8mo ago
    Indeed, OpenBSD recently added hardening measures and started restricting the generic syscall interface to libc.
- wrs 8mo ago
  Go backed out of their strategy on MacOS and started using libc (libsystem?), because when Apple says something is internal and may change without notice, they really mean it. It may be a better risk with Microsoft, but it’s still a risk.
- lelanthran 8mo ago
  Same way it turned out for Go: they had to walk it back.
HexDecOctBin 8mo ago
Linux syscall interface is actually stable and can easily be targeted. It’s BSDs (and Mac OS) that force everyone to link to only libc.
- pjmlp 8mo ago
  More like everyone else, Linux kernel is the exception here.
kvemkon 8mo ago
> It's like ... Considering the level of the API. But it is total opposite comparing a bit deeper. Linux has a famous rule "WE DO NOT BREAK USERSPACE!" e.g. [1]. [1] https://news.ycombinator.com/item?id=44611692 https://news.ycombinator.com/item?id=44611692
- dunder_cat 8mo ago
  One thing that is amusing about the prevalence of advanced anti-cheat in Windows gaming is it's actually causing said API/ABIs to undergo ossification. A good data point is the invention of Syscall User Dispatch^1 on Linux which would allow a program to basically install a syscall handler when they originate from various regions of memory. I do not know how usable this is in practice, admittedly -- but I think the fact it was contributed at all speaks to the growing need. ^1 https://docs.kernel.org/admin-guide/syscall-user-dispatch.html https://docs.kernel.org/admin-guide/syscall-user-dispatch.ht...
koakuma-chan 8mo ago
> It's like coding to the Linux syscall interface instead of libc. The right thing to do? I don't see why I would want to use libc.
- delta_p_delta_x 8mo ago
  On Windows, the stability guarantees are opposite to that of Linux. The kernel ABI is not guaranteed to be stable, whereas the Win32 ABI is. And frankly, the Windows way is better. On Linux, the 'ABI' for nearly all user-mode programs is not the kernel's ABI but rather glibc's (plus the variety of third-party libraries, because Win32 has a massive surface area and is an all-in-one API). Now, glibc's ABI constantly changes, so linking against a newer glibc (almost certainly the 'host' glibc, because it is almost impossible to supply a different 'target' glibc without Docker) will result in a program that doesn't run on older glibc. So much for Torvalds' 'don't break userspace'. Not so for a program compiled for 'newer' Win32; all that matters are API compatibilities. If one only uses old-hat interfaces that are documented to be present on Windows 2000, one can write and compile one's code on Windows 11, and the executable will run on the former with no issues. And vice versa, actually.
  - monocasa 8mo ago
    A lot of the native API is considered stable these days. The actual signals aren't, but the wrappers in ntdll are.
  - samus 8mo ago
    The Win32 ABI is also just a wrapper on the native API, which is only stable in practice, but not officially according to any Microsoft documentation. Glibc is userspace seen from the perspective of the Linux kernel.
    - delta_p_delta_x 8mo ago
      It doesn't really matter if it's 'just a wrapper', because said wrapper provides an ABI. Even if the underlying Native API changes, the interface the wrapper presents to other compiled binaries won't. The latter will contain caller/callee register setup, type layouts, function arguments and more for that wrapper. Cygwin is also 'just a wrapper' for the Native API and Win32, and look how drastically it changes the ABI of applications.
pjmlp 8mo ago
Except unlike Linux syscall interface and like almost every other OS out there, ABI compatibility is an accident, not a guarantee.
eps 8mo ago
"Every document" notwithstanding, Native API is very widely used in practice and generally considered stable. If in doubt, try and find examples of its breakage, semantic changes, etc.
samus 8mo ago
With the crucial difference that Linux places high value on syscall interface binary compatibility, while the NT native API is not guaranteed to be stable in any way. A bit more comparable is OpenBSD where applications are very much expected to only use libc wrappers, which threw a wrench into the works for the Go runtime.
Narishma 8mo ago
> It's like coding to the Linux syscall interface instead of libc. It's the opposite of that. The Linux syscall is more stable than the (gnu)libc.
josephcsible 8mo ago
> It's like coding to the Linux syscall interface instead of libc. Which is perfectly fine to do and guaranteed to work forever because of Linus's policy that kernel updates aren't allowed to break userspace programs.

deleted 8mo ago

[deleted]

lostmsu 8mo ago

Fool's errand. Apps built with this will have to be maintained forever (vs the apps from Win 9x which still work in Windows 11).

audunw 8mo ago
The reason apps from Win 9x runs on Windows 11 is that MS puts a ton of effort into explicitly supporting old apps. For popular apps that includes supporting undocumented APIs and even app-specific bug compatibility. Putting it on app developers to account for infinite forward compatibility is not at all reasonable. The best outcome would be if many Zig apps become popular enough that Windows is forced to maintain backward compatibility for ntdll. The API is clearly superior to win32 as many other developers have discovered and discussed before. It’d be nice to force MS to take low lever programming seriously instead of chasing AI slop.
- lostmsu 8mo ago
  > The reason apps from Win 9x runs on Windows 11 is that MS puts a ton of effort into explicitly supporting old apps. No, that's one of the reasons. The other one is that public kernel32 -> private ntdll design works.

roelschroeven 8mo ago

> > Won't this get flagged by anti-virus scanners as suspicious? > Unfortunately, yes. We consider this a problem for the anti-virus scanners to solve. I don't think the anti-virus scanners consider Zig important enough, or even know about. They will not be the ones experiencing problems. Having executables quarantined and similar problems will fall on Zig developers and users of their software. That seems like a major drawback for using Zig.

monocasa 8mo ago
Yeah, I had this problem when shipping go binaries on Windows. Antivirus vendors really do not care that your program regularly shows up as a false positive due to their crappy heuristics, even if you have millions of users.
- anfragment 8mo ago
  Have you tried code-signing with an EV certificate? If so, did it help? Asking for a friend.
  - monocasa 8mo ago
    Statistically notable improvement, but it didn't help a whole lot.
    - yndoendo 8mo ago
      I always upload a copy to https://www.virustotal.com https://www.virustotal.com to help combat the false positives. It was really bad a couple years ago because anything wrapped in Inno Setup kept being flagged. Now maybe one or two flag vendors do; Bkav Pro and CrowdStrike Falcon are the dominate culprits always.
      - monocasa 8mo ago
        Uploading to virustotal doesn't really do anything to combat false positives AFAICT. It only lets you test against many AV vendors at once.

cmovq 8mo ago

Is there an official stance on whether ntdll is stable? Obviously they're not going to change things arbitrarily since applications depend on it, but I'm wondering if there is a guarantee like the linux syscall interface or how you can run a win32 application compiled in 2004 on Win11.

monocasa 8mo ago
It's partially stable. Basically any thing documented on msdn in the API docs is considered stable. Such as: https://learn.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile https://learn.microsoft.com/en-us/windows/win32/api/winternl...
- delta_p_delta_x 8mo ago
  Indeed. Anything documented has a function wrapper. `NtCreateFile` is a function wrapper for the syscall number, so any user-mode code that has `NtCreateFile` instead of directly loading the syscall number 0x55 will be stable. The latter might not. In fact, it is not; the number has increased by 3 since Windows XP[1]. One could probably produce some sort of function pointer loader library with these tables, but at that point... Why not just use the documented APIs? [1]: https://github.com/j00ru/windows-syscalls/blob/8a6806ac9148633e8afbbdb42e6002930b002741/x64/csv/nt.csv#L100 https://github.com/j00ru/windows-syscalls/blob/8a6806ac91486...
  - Dwedit 8mo ago
    Only Malware uses the system call numbers directly. Using the system call numbers directly is foolish if they're going to change and break your app. Just import and call a function that will perform the actual SYSENTER (or WOW64 context change).
    - monocasa 8mo ago
      Unfortunately, that's not the case. Wine for instance has to keep up to date to maintain compatibility with some applications. https://gitlab.winehq.org/wine/wine/-/releases/wine-11.0 https://gitlab.winehq.org/wine/wine/-/releases/wine-11.0 > NT system calls use the same syscall numbering as recent Windows, to support applications that hardcode syscall numbers.

bob1029 8mo ago

Why not use both DLLs? Prefer win32 wherever possible and use the lower level APIs only if absolutely necessary. Benchmark after you have figured this out. Performance is probably not a thing at this level of abstraction.

nvme0n1p1 8mo ago
What makes you think they haven't benchmarked? Here's one fun example from following development on Zulip: advapi.dll loads bcrypt.dll, which loads bcryptprimitives.dll. bcryptprimitives.dll runs an internal test suite every time it's loaded into any process. So if you can avoid loading advapi.dll, your process will start faster.
- delta_p_delta_x 8mo ago
  Is there a source for this? My Google- and GitHub-fu turns up nothing.
  - lelanthran 8mo ago
    He might be talking about cipher test that respected cryptography libs do on initialisation to verify integrity. Skipping those seem like a really bad idea.
    - josephcsible 8mo ago
      > Skipping those seem like a really bad idea. Why? Is there any realistic scenario where your cryptography libs worked correctly yesterday but the exact same ones will be buggy today? What would be wrong with them just running once per build instead?
  - nvme0n1p1 8mo ago
    Join their Zulip and search for bcryptprimitives. That's where I got my info.
- lelanthran 8mo ago

dblohm7 8mo ago

This is a terrible idea! _Maybe_, _maybe_ using only the documented APIs with only the documented parameters. Unfortunately it makes too many false assumptions about interoperability between Win32 and the underlying native API that aren't true. For example (and the Go runtime does this, much to my chagrin), querying the OS version via the native API always gives you "accurate" version information without needing to link a manifest into your application. Unfortunately that lack of manifest will still cause many Win32 APIs above the native layer to drop into a compatibility mode, creating a fundamental inconsistency between what the application thinks the OS capabilities are versus which Win32 subsystem behaviours the OS thinks it should be offering.

boje 8mo ago

Honestly, this sounds like a future headache that would otherwise go unnoticed unless the programmer is dealing with porting or binding over source code meant for older Windows systems to Zig (or supporting older systems in general). Eventually it might result in a bunch of people typing out blogposts venting their frustrations, and the creation of tutorials and shims for hooking to Win32 instead of the Zig standard library with varying results. Which is fine, I suppose. Legacy compiler targets are a thing. This is already a problem with Linux binaries for systems that don't have a recent enough Glibc (unless the binaries themselves don't link to it and do syscalls directly).

cmovq 8mo ago

>> Microsoft are free to change the Native API at will, and you will be left holding both pieces when things break. > [...] the worst case scenario is really mild: A new version of windows comes out, breaking ntdll compatibility. Zig project adds a fix to the std lib. Application developer recompiles their zig project from source, and ships an update to their users. That assumes the application developer will continue to maintain it until the end of time. Also "the fix" would mean developers wanting to support earlier Windows versions would need to use an older std library? Or is the library going to have runtime checks to see what Windows build its running on?

eps 8mo ago
> Microsoft are free to change the Native API at will,... But they won't, because if there is one thing that Microsoft has always been extremely good at and cared for is backward compatibility. And changing Native API will break a ton of existing software, because even though undocumented it is very widely used.
- dbdoskey 8mo ago
  Actually they do change the native API quite a bit. Not in minor releases so much but in major releases
  - eps 8mo ago
    They depricate some methods (very rarely and reasonably) and add new enums or struct versions to existing ones, but never change existing semantics, leave alone method signatures. As I said elsewhere, I invite you to find examples of actually destructive Native API changes.
- blibble 8mo ago
  you are confusing the ntdll interface (which is undocumented and subject to change), and win32 (which is stable, mostly) they tell you not to use ntdll, and say they will change it whenever they want and they have in the past (they have had to moderate this policy with "containers", but it's still what they say)

qq66 8mo ago

> Comparing the comprehensive Win32 API reference against the incidentally documented Native APIs, its clear which one Microsoft would prefer you use. The native API is treated as an implementation detail, whilst core parts of Windows' backwards compatibility strategy are implemented in Windows subsystem. > A general-purpose programming language and toolchain for maintaining robust, optimal, and reusable software. Zig clearly doesn't actually care that much about building robust and reusable software if they're going to forgo Microsoft's decades-long backwards compatibility functionality for the dubious gains of using bare-metal APIs.

slopinthebag 8mo ago
lately it feels like zig is attempting to speed run irrelevance. which is a shame.
- gjsman-1000 8mo ago
  I don't know, everyone here seems plenty okay to tolerate worse levels of instability from Linux binaries. :)
  - slopinthebag 8mo ago
    im too much of a mac user to understand what this references :p
    - gjsman-1000 8mo ago
      Take a random Linux binary which does anything non-trivial (has a GUI, does system monitoring, etc.), try running it on a different distribution from 3 years earlier without a packaging system, and tell me how it goes.
      - lelanthran 8mo ago
        Zig is proposing the opposite problem: future versions of windows wont run even trivial zig programs from today. I can tell you that old Linux binaries run just fine on current distros. Looking at how many times you repeated your misunderstanding in this thread it's clear that, not only do you not understand the solution, you don't understand the problem either.

self_awareness 8mo ago

For me this is too much. I wish Zig all the best, but decisions like this make me want to jump off this sinking ship.

advisedwang 8mo ago

I'm really struggling to see the pros of this: > Performance - using the native API bypasses the standard Windows API, thus removing a software layer, speeding things up. But the article cites no bemchmarks > Power - some capabilities are not provided by the standard Windows API, but are available with the native API. Makes sense when you are doing something that needs that power, but that makes more sense as an exception to prefering win32 than a general reason to prefer native. > Dependencies - using the native API removes dependencies on subsystem DLLs, creating potentially smaller, leaner executables. Linking win32 is a miniscule cost. (unless you have a benchmark to show me...) > Flexibility - in the early stages of Windows boot, native applications (those dependent on NtDll.dll only) can execute, while others cannot. Is Zig being used for such applications? If so, why are the calls that the document says will be kept on win32 not an issue?

sebazzz 8mo ago
If I am not incorrect, for early boot applications, the application must be set to use the native subsystem.

moogly 8mo ago

Yikes. Are they going to rename the language to "cyg" also? Does not inspire confidence.

born-jre 8mo ago

Don’t know about windows programming to give opinion but by the sentiment here maybe they should give dev to choose bashed on some comptime flag or sth and maintain two versions

bmacho 8mo ago

> the worst case scenario is really mild: A new version of windows comes out, breaking ntdll compatibility. Zig project adds a fix to the std lib. Application developer recompiles their zig project from source, and ships an update to their users. The ~only good thing that programmers have achieved in the past ~60 years has been Windows stability. Create a popular programming language, and then make programs written in it not run on newer Windowses is just something else. I so hate this. Was "robust, optimal and reusable" always "run an older Windows on your newer Windows to run Zig software"?

gjsman-1000 8mo ago
... this is just Linux binaries. It's humorous to me that we literally do exactly this, for Linux, with even less stability, but heaven forbid we do something approaching that on Windows despite the snobbery against Windows.

TazeTSchnitzel 8mo ago

Go famously tried to bypass macOS's libc and directly use the underlying syscall ABI, which is unstable, and then a macOS update came out and broke everything, which taught them the error of their ways (https://github.com/golang/go/issues/17490 https://github.com/golang/go/issues/17490). I wonder if this will happen to Zig too.

CorrectHorseBat 8mo ago
The same on OpenBSD

chad1n 8mo ago

Anyone who has some experience with native apis knows that a standard library should never rely on unstable apis. Ntdll is not "stable" as in Microsoft can change it at any time since they expect anyone to use kernel32. It's questionable that they referenced a random book on this top claiming that ntdll is more performant than kernel32 which is doubtful. There are some specific cases where this is true (the ntfs stuff), but, in general, it's not, at least not in a significant matter. A standard library should never do this, it might break binaries for no reason, other than making a cool blog post. I, as a developer, can choose to use ntfs, but a standard library should never. https://news.ycombinator.com/item?id=25997506 https://news.ycombinator.com/item?id=25997506 https://github.com/golang/go/issues/68678 https://github.com/golang/go/issues/68678

lelanthran 8mo ago

> While this can happen, we have not (yet) been affected by any changes in the Win32 -> Native layers. Frankly this is dumb. Zig hasn't been around long enough to have even seen any changes, so using this as a reason is just plain dumb. The view that, if windows ever changes, the code must be recompiled is a naive view one would expect from a child, not from a group of experienced devs.

Dwedit 8mo ago

The one thing that really benefits from using NT Native API over Win32 is listing files in a directory. You get to use a 64KB buffer to get directory listing results, while Win32 just does the one at a time. 64KB buffer size means fewer system calls. (Reading the MFT is still faster. Yes, you need admin for that)

xfeeefeee 8mo ago
Oh hey this is exactly why I made node-windows-readdir-fast - especially with the way node works, this makes reading filenames and length and times around 50x faster https://github.com/xfeeefeee/node-windows-readdir-fast https://github.com/xfeeefeee/node-windows-readdir-fast https://www.npmjs.com/package/node-windows-readdir-fast https://www.npmjs.com/package/node-windows-readdir-fast Windows only of course, but the concept is sound. Was also fun benchmarking to find out that parsing a binary stream was faster than creating a ton of objects through the node api (or json deserialization)

jojomodding 7mo ago

For comparison, in Rust they track down the differences in which flags are ignored in a certain kind of `fcntl` syscall across all the architectures that have this C function, which includes Solaris, Mac OS, the BSDs, Linux, ... This is so that this is correctly handled in Miri, which can then be used to run the test-suite of the OS-specific parts of the standard library, and observe if this uses unsupported features in some way. This ensures that the standard library relies on documented features and not on whatever happens to work right now. See for example this PR comment: https://github.com/rust-lang/miri/pull/4840#discussion_r2836627676 https://github.com/rust-lang/miri/pull/4840#discussion_r2836...