6 ms·
Making Win32 APIs More Accessible to More Languages
- mappu 6y agoGolang is also considering consuming this new metadata: https://github.com/golang/go/issues/43838 https://github.com/golang/go/issues/43838
- wruza 6y agoA good thing that is late for 25 years. Still better than never, WinAPIs never were an easiest thing to wrap.
- emddudley 6y agoRust for Windows uses this. https://news.ycombinator.com/item?id=25862291 https://news.ycombinator.com/item?id=25862291
- userbinator 6y agoThere was some discussion on this earlier today: https://news.ycombinator.com/item?id=25885196 https://news.ycombinator.com/item?id=25885196
- jiggawatts 6y agoOne issue I've had with any kind of auto-generated bindings based on metadata is that you get low-level types such as "Win32 BOOL" and "Win32 GUID" polluting the output. Mainstream languages already either have native bool and guid types, or have defacto standard libraries. E.g.: Rust has the UUID crate: https://crates.io/crates/uuid https://crates.io/crates/uuid Similarly, Java has had a UUID type since v1.5: https://docs.oracle.com/javase/8/docs/api/java/util/UUID.html https://docs.oracle.com/javase/8/docs/api/java/util/UUID.htm... It's not just concrete types, but core language concepts often "don't translate". E.g.: Win32 is essentially a C-like set of bindings. In many languages such as C++, D, Zig, etc... this is not an issue. However the type system of C is much less strict that Rust, so the metadata has no concept of ownership and lifetimes. This means that if you want to consume Win32 APIs from Rust, you're still forced to wrap everything manually at least once to escape the "unsafe" sandpit. A thought I had is that C is used for API definitions because "it is lowest common denominator". But that's the wrong thing to do, the metadata doesn't have to "sink to the lowest level"! An ideal metadata definition would be something akin to Rust's type system, or even stronger than that! "Degrading" a binding from a strong type system to a weak one is trivial and can be performed completely automatically. The converse is difficult and error-prone even for humans, and impossible for code generators. You can't synthesise this missing data! Even the read-only guarantee is too weak. E.g.: // Rust could treat 'buf' as read-only borrow. void SendData( [readonly] void* buf ); // But here the borrow outlives the function call lifetime! void SendDataAsync( [readonly]void* buf ); The two methods have identical metadata signatures, but the information required by a Rust programmer or a Rust code generator is missing... There are many such abstract API concepts missing from C, such as thread safety, process integrity safety, non-returning calls, blocking vs async, etc...
- brundolf 6y ago> One issue I've had with any kind of auto-generated bindings based on metadata is that you get low-level types such as "Win32 BOOL" and "Win32 GUID" polluting the output. Mainstream languages already either have native bool and guid types, or have defacto standard libraries. This shouldn't actually be a huge problem in Rust, because of the traits system. It should be possible to implement all the expected behaviors (Into/From, Add/Sub/Mul/Div, Ord/Eq, etc) for these special types and if library code is properly generic, it shouldn't care whether or not you're using the native type. If it does require a specific concrete type, From/Into should still make things reasonably ergonomic. Re: safety, the typical approach has just been to keep one library that's a direct mapping to the C API, which is publicly unsafe, and then create a safe library abstracted on top of that which codifies the documented behaviors. That layer relies on documentation, and on humans to follow it, but then so does all C code ever written :)
- jiggawatts 6y agoIn my experience with both classic Win32 programming in the 1990s in C++ and then later in C# and now in Rust is that it is obscenely verbose. It takes a solid page of code to call a single method, and most of that is just noise. A typical method call is: Allocate some struct. Zero it out. Set a "length" field to the length of the struct. Populate the "parameter" fields. Call the API with the struct and 0 for the return buffer size. You get back an expected error code for "buffer too small". You allocate the required space in your buffer. Call the method again. The amount of data changed! Loop and try again.[1] You get your data! Now to check the error codes. Look up error code using a Win32 method. Free the buffer. If on top of that I'd have to call "into" and "from" for every field and every parameter at every step... I think I might just snap and go become a bricklayer or something. PS: Win32 strings and Rust strings aren't compatible. I've already noticed that the current version of the Microsoft code-generator seems to use an awful lot of [u16] arrays, which are not the system OsString struct. For comparison, the C# "String" type isn't directly compatible with Win32 strings either, but the FFI engine handles this conversion back-and-forth for you. Similarly, few (any?) languages have good built-in support for the various kinds of "struct-followed-by-dynamic-data" typically returned by system APIs. Notably, C does not have any way to encode this in the type system, so API metadata tends to just leave this out or describe it in the documentation. If the metadata was extented to cover this kind of thing, then the generators could do all sorts of clever things with this scenario... [1] A lot of people don't do this, they assume that the second call will always succeed. Programming in this style leads to rare random errors throughout the entire application. A loop is required for correctness for most (but not all) Win32 calls that return dynamic data...
- Kaknut 6y agoOh so it arrived late, but better come now than never. Also it's super nice that I can use RUST for it.
- smasher164 6y agoNow if MacOS followed suit, all would be right in the universe.
- benibela 6y agoThe best way to access Win32 APIs has always been Delphi's VCL
- ed_blackburn 6y agoReminds me of Dan Appleman’s VB Win32 API book from ~20 years ago.
- dblock 6y agoI was the de facto maintainer of JNA (https://github.com/java-native-access/jna https://github.com/java-native-access/jna) win32 bindings for years. The API metadata is welcome - we have always based all work on header files in the visual studio / windows SDK. Now there’s a change to generate that code - someone might want to try to replace hand crafted mappings in JNA with generated ones - there’s a huge test suite to work with.
- pjmlp 6y agoFirst of all, thanks for the hard work. I guess there is also the possibility to use the JNI replacement when it comes out of preview, although that would be constrained to Java 17 or later.
- BelenusMordred 6y agoIt's a great move but nearly every lang on Earth has a solid 3rd party Win32 library built over years and battle hardened. Waiting patiently for Microsoft to actually put some effort into the obscure TPM documentation along with making it easier for developers to implement.
- tinus_hn 6y agoThe TPM infrastructure is full of leaks that can’t be fixed, they’re not going to expose that with documentation. It’s like asking them to document a vendor to open-source their DRM.
- BelenusMordred 6y ago> full of leaks that can’t be fixed Not claiming it's perfect, saying it exists on nearly every cpu running Windows today. There's serious value for plenty of people out there to have access to these interfaces that are only internally documented and available to a small group of people from one company alone. Only have to look at the history of adding randomness to CryptographyAPI-NextGen to understand where I'm coming from. No one has been able to add an additional RNG source to the Windows entropy pool for ~15 years now, highly venerable Microsoft developers say this is a good thing in public for all their enterprise clients when basic maths proves it wrong in about 2 minutes. They do not have your best interests at heart, no matter how big you are, in fact it's less likely the bigger you are.
- GordonS 6y agoI mainly use C#, and this is late, but really welcome! I usually use Vanara for Win32 PInvoke wrappera, or hand-code them if they are undocumented or I only need a couple. Rarely I've come across memory issues with the wrappers provided by Vanara and PInvoke.net, where they've forgotten to return a buffer - these kind of problems are absolute bastards to track down, and the root cause is of course manually building wrapper code. Having an automated means to do this will be a huge boon. I wonder though - quite often if I'm digging into Win32 stuff, it means I'm working on performance sensitive code, where I want to keep allocations low - I haven't looked I to it yet, but I hope C#/Win32 has an option to generate wrappers that take pointers instead of only using managed types!
- ziml77 6y agoI gave their Rust bindings a try, but they still have work to do to even match the manual work that has been put into the winapi crate. I can say from experience is long, tedious, and error prone so it will definitely be better to be auto-generating the bindings.
- jedisct1 6y agoInteresting. The real thing here is the Win32 Metadata project: https://github.com/microsoft/win32metadata https://github.com/microsoft/win32metadata It contains definitions that can be used to automatically generate bindings to any language. So, we have C# and Rust as examples, but the very same definitions could be used with languages like Zig. Very cool initiative.