10 ms·

Designing a COM Library for Rust

184 points by arunc 7y ago

Crinus 7y ago
Weirdly, the github page says: > COM has been superseded by WinRT which builds on COM to provide even more guarantees about the binary interface. Yet the WinRT page says: > The Windows Runtime is based on Component Object Model (COM) APIs, and it's designed to be accessed through language projections. A projection hides the COM details, and provides a more natural programming experience for a given language. Which implies that WinRT doesn't supersed (ie. replace) COM but it uses COM itself (though the "language projection" part makes me think that the way it is used is so convoluted that you need language extensions to make it tolerable :-P).
- pjmlp 7y ago
  WinRT is an evolution of COM. Basically there is a new base interface, IInspectable instead of IUnknown, and many more datatypes are allowed to cross the wire between processes, making it easier to also use generics, lists and arrays across COM libraries. Additonally, .NET metadata libraries are using instead of the old COM type libraries. If you prefer, it is the original design of .NET before the CLR came into play, and related to the Longhorn architecture, just using COM instead of .NET. A path started in Vista. Now for this Rust approach to win the hearts of Windows devs, it needs to be as productive as C++/CX or C++/WinRT VS tooling, in all COM/UWP usage scenarios.
  - Crinus 7y ago
    One thing i'm not sure about WinRT and cannot figure out from the docs, AFAIK it is possible to "bit bang" (or "byte bang" if you will) a COM object by arranging everything in memory using plain old C structs (or equivalent for other languages) from compilers that weren't COM aware. Is it possible to do the same with WinRT or it requires the language to be WinRT aware? F.e. could i get something like TCC and write some code that uses a WinRT object even though TCC has no idea about WinRT?
    - pjmlp 7y ago
      Not really, as it is more elaborated. https://docs.microsoft.com/en-us/uwp/winrt-cref/winrt-type-system https://docs.microsoft.com/en-us/uwp/winrt-cref/winrt-type-s... The documentation is a bit lacking, as it appeared with WinRT, then we had UAP, and finally UWP. While it all boils down to the same, those marketing changes, did cause the documentation to keep changing. Alongside C++/CX, there was the low level "Windows Runtime C++ Template Library (WRL)", which was basically an evolution of ATL, with MDIL 2.0. https://docs.microsoft.com/en-us/cpp/cppcx/wrl/windows-runtime-cpp-template-library-wrl?view=vs-2015 https://docs.microsoft.com/en-us/cpp/cppcx/wrl/windows-runti... Eventually MIDL improved to 3.0, as the work on C++/WinRT progressed. https://docs.microsoft.com/en-us/uwp/midl-3/ https://docs.microsoft.com/en-us/uwp/midl-3/ https://channel9.msdn.com/Shows/ifdefWINDOWS/MIDL3-with-Larry-Osterman https://channel9.msdn.com/Shows/ifdefWINDOWS/MIDL3-with-Larr... Here are some technical documents and sessions related to WinRT. "The Windows Runtime" https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/The-Windows-Runtime https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/Th... "Windows Runtime internals: understanding "Hello World"" https://channel9.msdn.com/events/Build/BUILD2011/PLAT-875T https://channel9.msdn.com/events/Build/BUILD2011/PLAT-875T The C++/WinRT talks from Kenny Kerr, related how they migrated from C++/CX language extensions to the C++17 compliant WinRT in chronological order. "CppCon 2017: Scott Jones & Kenny Kerr “C++/WinRT and the Future of C++ on Windows”" https://www.youtube.com/watch?v=7TdpWB_vRZM https://www.youtube.com/watch?v=7TdpWB_vRZM "Effective C++/WinRT for UWP and Win32" https://www.youtube.com/watch?v=nOFNc2uTmGs https://www.youtube.com/watch?v=nOFNc2uTmGs "Meet C++/WinRT 2.0: Faster and smarter in the open" https://www.youtube.com/watch?v=X41j_gzSwOY https://www.youtube.com/watch?v=X41j_gzSwOY Uff, hope this sheds some light on your questions and anyone else that wants to get a clearer picture of COM and its evolution into UWP.
      - ak39 7y ago
        This reply should go into HN hall of fame. Thank you very much!
    - asveikau 7y ago
      > it is possible to "bit bang"... a COM object by arranging everything in memory using plain old C structs ... from compilers that weren't COM aware. Is it possible to do the same with WinRT or it requires the language to be WinRT aware? Yes, actually, you can do that. I had such code working. Basically declared the IInspectable VTable as a C struct and filled it with my own implementation. Then wrote an "activator" for the relevant classes the same way. [What COM used to call factories.] Did the same thing with a few VTables for MS-authored WinRT classes calling into them in the other direction. All this was tedious, I had complicated reasons for wanting to do this and I eventually switched to WRL when those reasons disappeared. Writing COM/WinRT from C++ with something like WRL is much nicer.
  - mwcampbell 7y ago
    > Now for this Rust approach to win the hearts of Windows devs, it needs to be as productive as C++/CX or C++/WinRT VS tooling, in all COM/UWP usage scenarios. Possibly related, I just spotted this [1] on the personal blog of Kenny Kerr, creator of C++/WinRT: > On a personal note, I’m spending a lot of my time working with the Rust language and can’t wait to share more about that eventually. Disclosure: I work on Windows at Microsoft, but not in this area. Posting on my own behalf though. [1]: https://kennykerr.ca/2019/10/09/xlang-cppwinrt-repos/ https://kennykerr.ca/2019/10/09/xlang-cppwinrt-repos/
    - pjmlp 7y ago
      Very interesting, given how he made C++/WinRT possible. Thanks for bringing it up.
thrower123 7y ago
COM interop used to be a big buzzword say, a decade ago, but I haven't really heard or seen much usage of it since. Most of my experience in using it has been horribly abusing it to drive Office applications and Notes to do some awful things because at the time there weren't any libraries to manipulate certain file types. I have done some truly heinous things with the Lync/Skype COM-based client SDK, back in the times when the Office 365 version of that product was viable, and there was no other API options available.
- pjmlp 7y ago
  Since Windows Vista the large majority of new native APIs on Windows are based on COM. You probably haven't heard much about it, because now it has the fancy name of UWP, or it comes with .NET wrappers on top.
  - withinrafael 7y ago
    UWP is ambiguously defined and the acronym's usage is on the way out. An evolution of UAP and Windows Store apps, it boiled down to a whitelist of APIs allowed on multiple devices (the U in Universal) and encompassed Win32, WinRT and COM APIs. UWP itself isn't an entity with APIs, is not a technology, and the original concept is dead thanks to new platform leadership at Microsoft. (Still in play on Hololens, Surface Hub, and other sandboxed devices for now...) That said, "UWP" is confusingly also used to refer to Windows apps that tap into WinRT APIs, participate in the modern app lifecycle, are modeled and have identity, or generally tap into newer technologies such as AppContainer. You'll see this, for example, perpetuated in Visual Studio in its project templating. It's a BIG OL' MESS right now; I'm working with/pushing Microsoft to get this communicated clearly and mopped up but old habits die hard.
    - pjmlp 7y ago
      Well, it is quite clear actually. https://docs.microsoft.com/en-us/windows/apps/desktop/choose-your-platform https://docs.microsoft.com/en-us/windows/apps/desktop/choose...
holy_city 7y ago
This is similar to the work done in winapi [1], com-imp[2] and my own tangential work on porting VST3 [3] to Rust [4] I'm really glad MS is doing this. What needs to be a bit clearer to me is how they maintain ABI compatibility under the hood of MSVC for COM interfaces (which uses the vtable layout of an inherited class) and how that's compatible with MinGW/GCC stacks on Windows, mostly what can break it. I got stuck porting VST3 with multiple inheritance, and it was a headache trying to reverse engineer the appropriate struct layouts for COM implementations. [1] https://github.com/retep998/winapi-rs/blob/0.3/src/macros.rs https://github.com/retep998/winapi-rs/blob/0.3/src/macros.rs [2] https://github.com/Connicpu/com-impl https://github.com/Connicpu/com-impl [3] https://github.com/steinbergmedia/vst3sdk https://github.com/steinbergmedia/vst3sdk [4] https://github.com/m-hilgendorf/cli-host https://github.com/m-hilgendorf/cli-host (sorry for the messy code, it was a weekend of hacking away trying to host a VST3 in pure Rust)
- pjmlp 7y ago
  COM layout is followed upon most mainstream Windows compiled languages, namely major C++ compilers, .NET, Delphi, Eiffel, Ada, so it is not MSVC++ keeping ABI compatibility under the hood on their own.
  - holy_city 7y ago
    My issue isn't ABI stability but the ABI itself w.r.t vtable layout. Best I can tell it should be similar to Itanium's spec? [1]. It's been months since I did this, but iirc my problems stemmed from having multiple interfaces on top of the same implementation and the ordering/layout of those interfaces, though the IUnknown interface which is supposed to handle that. [1] https://itanium-cxx-abi.github.io/cxx-abi/abi.html#vtable https://itanium-cxx-abi.github.io/cxx-abi/abi.html#vtable
    - barrkel 7y ago
      COM is agnostic as to how you do multiple inheritance - it doesn't have the concept. It specifies the QueryInterface protocol, but you don't need to return the same instance for the result of the QI call, just one that uses the same lifetime refcount. Tear-off interfaces and delegated implementations are things in this world.
jchw 7y ago
Very cool. I have played with COM in Go to a lesser extent. It’s not too bad actually. My intent was to write Go wrappers around all of the TSF COM interfaces, but I burnt out after getting basic IME working - TSF is enormous!
- MehdiHK 7y ago
  Hi, I believe you are talking about this repo https://github.com/jchv/go-tsf https://github.com/jchv/go-tsf Do you know any other project that's working on creating basic IME on top of TSF in Go?
  - jchw 7y ago
    You are correct, and no I am not. Is your goal to build an application with IME support, or an entire IME?
    - MehdiHK 7y ago
      An entire IME.
      - jchw 7y ago
        Ah I see.. sadly, I never really figured out a great way to support full COM interop in Go. It would be ideal if you could implement an interface and register it as a COM object, but I hadn’t hashed out exactly how.
ajxs 7y ago
I'm interested in more information regarding the Safe Systems Programming Languages team's work. Did they assess other languages other than Rust? I'm scanning through their blog entries, without having read every single one in depth so please pardon me if I miss something, and I'm not seeing anything mentioning formal methods or other systems programming languages. It's a very interesting development for sure!

184 points by arunc 7y ago

Weirdly, the github page says: > COM has been superseded by WinRT which builds on COM to provide even more guarantees about the binary interface. Yet the WinRT page says: > The Windows Runtime is based on Component Object Model (COM) APIs, and it's designed to be accessed through language projections. A projection hides the COM details, and provides a more natural programming experience for a given language. Which implies that WinRT doesn't supersed (ie. replace) COM but it uses COM itself (though the "language projection" part makes me think that the way it is used is so convoluted that you need language extensions to make it tolerable :-P).

pjmlp 7y ago
WinRT is an evolution of COM. Basically there is a new base interface, IInspectable instead of IUnknown, and many more datatypes are allowed to cross the wire between processes, making it easier to also use generics, lists and arrays across COM libraries. Additonally, .NET metadata libraries are using instead of the old COM type libraries. If you prefer, it is the original design of .NET before the CLR came into play, and related to the Longhorn architecture, just using COM instead of .NET. A path started in Vista. Now for this Rust approach to win the hearts of Windows devs, it needs to be as productive as C++/CX or C++/WinRT VS tooling, in all COM/UWP usage scenarios.
- Crinus 7y ago
  One thing i'm not sure about WinRT and cannot figure out from the docs, AFAIK it is possible to "bit bang" (or "byte bang" if you will) a COM object by arranging everything in memory using plain old C structs (or equivalent for other languages) from compilers that weren't COM aware. Is it possible to do the same with WinRT or it requires the language to be WinRT aware? F.e. could i get something like TCC and write some code that uses a WinRT object even though TCC has no idea about WinRT?
  - pjmlp 7y ago
    Not really, as it is more elaborated. https://docs.microsoft.com/en-us/uwp/winrt-cref/winrt-type-system https://docs.microsoft.com/en-us/uwp/winrt-cref/winrt-type-s... The documentation is a bit lacking, as it appeared with WinRT, then we had UAP, and finally UWP. While it all boils down to the same, those marketing changes, did cause the documentation to keep changing. Alongside C++/CX, there was the low level "Windows Runtime C++ Template Library (WRL)", which was basically an evolution of ATL, with MDIL 2.0. https://docs.microsoft.com/en-us/cpp/cppcx/wrl/windows-runtime-cpp-template-library-wrl?view=vs-2015 https://docs.microsoft.com/en-us/cpp/cppcx/wrl/windows-runti... Eventually MIDL improved to 3.0, as the work on C++/WinRT progressed. https://docs.microsoft.com/en-us/uwp/midl-3/ https://docs.microsoft.com/en-us/uwp/midl-3/ https://channel9.msdn.com/Shows/ifdefWINDOWS/MIDL3-with-Larry-Osterman https://channel9.msdn.com/Shows/ifdefWINDOWS/MIDL3-with-Larr... Here are some technical documents and sessions related to WinRT. "The Windows Runtime" https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/The-Windows-Runtime https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/Th... "Windows Runtime internals: understanding "Hello World"" https://channel9.msdn.com/events/Build/BUILD2011/PLAT-875T https://channel9.msdn.com/events/Build/BUILD2011/PLAT-875T The C++/WinRT talks from Kenny Kerr, related how they migrated from C++/CX language extensions to the C++17 compliant WinRT in chronological order. "CppCon 2017: Scott Jones & Kenny Kerr “C++/WinRT and the Future of C++ on Windows”" https://www.youtube.com/watch?v=7TdpWB_vRZM https://www.youtube.com/watch?v=7TdpWB_vRZM "Effective C++/WinRT for UWP and Win32" https://www.youtube.com/watch?v=nOFNc2uTmGs https://www.youtube.com/watch?v=nOFNc2uTmGs "Meet C++/WinRT 2.0: Faster and smarter in the open" https://www.youtube.com/watch?v=X41j_gzSwOY https://www.youtube.com/watch?v=X41j_gzSwOY Uff, hope this sheds some light on your questions and anyone else that wants to get a clearer picture of COM and its evolution into UWP.
    - ak39 7y ago
      This reply should go into HN hall of fame. Thank you very much!
  - asveikau 7y ago
    > it is possible to "bit bang"... a COM object by arranging everything in memory using plain old C structs ... from compilers that weren't COM aware. Is it possible to do the same with WinRT or it requires the language to be WinRT aware? Yes, actually, you can do that. I had such code working. Basically declared the IInspectable VTable as a C struct and filled it with my own implementation. Then wrote an "activator" for the relevant classes the same way. [What COM used to call factories.] Did the same thing with a few VTables for MS-authored WinRT classes calling into them in the other direction. All this was tedious, I had complicated reasons for wanting to do this and I eventually switched to WRL when those reasons disappeared. Writing COM/WinRT from C++ with something like WRL is much nicer.
- mwcampbell 7y ago
  > Now for this Rust approach to win the hearts of Windows devs, it needs to be as productive as C++/CX or C++/WinRT VS tooling, in all COM/UWP usage scenarios. Possibly related, I just spotted this [1] on the personal blog of Kenny Kerr, creator of C++/WinRT: > On a personal note, I’m spending a lot of my time working with the Rust language and can’t wait to share more about that eventually. Disclosure: I work on Windows at Microsoft, but not in this area. Posting on my own behalf though. [1]: https://kennykerr.ca/2019/10/09/xlang-cppwinrt-repos/ https://kennykerr.ca/2019/10/09/xlang-cppwinrt-repos/
  - pjmlp 7y ago
    Very interesting, given how he made C++/WinRT possible. Thanks for bringing it up.

thrower123 7y ago

COM interop used to be a big buzzword say, a decade ago, but I haven't really heard or seen much usage of it since. Most of my experience in using it has been horribly abusing it to drive Office applications and Notes to do some awful things because at the time there weren't any libraries to manipulate certain file types. I have done some truly heinous things with the Lync/Skype COM-based client SDK, back in the times when the Office 365 version of that product was viable, and there was no other API options available.

pjmlp 7y ago
Since Windows Vista the large majority of new native APIs on Windows are based on COM. You probably haven't heard much about it, because now it has the fancy name of UWP, or it comes with .NET wrappers on top.
- withinrafael 7y ago
  UWP is ambiguously defined and the acronym's usage is on the way out. An evolution of UAP and Windows Store apps, it boiled down to a whitelist of APIs allowed on multiple devices (the U in Universal) and encompassed Win32, WinRT and COM APIs. UWP itself isn't an entity with APIs, is not a technology, and the original concept is dead thanks to new platform leadership at Microsoft. (Still in play on Hololens, Surface Hub, and other sandboxed devices for now...) That said, "UWP" is confusingly also used to refer to Windows apps that tap into WinRT APIs, participate in the modern app lifecycle, are modeled and have identity, or generally tap into newer technologies such as AppContainer. You'll see this, for example, perpetuated in Visual Studio in its project templating. It's a BIG OL' MESS right now; I'm working with/pushing Microsoft to get this communicated clearly and mopped up but old habits die hard.
  - pjmlp 7y ago
    Well, it is quite clear actually. https://docs.microsoft.com/en-us/windows/apps/desktop/choose-your-platform https://docs.microsoft.com/en-us/windows/apps/desktop/choose...

holy_city 7y ago

This is similar to the work done in winapi [1], com-imp[2] and my own tangential work on porting VST3 [3] to Rust [4] I'm really glad MS is doing this. What needs to be a bit clearer to me is how they maintain ABI compatibility under the hood of MSVC for COM interfaces (which uses the vtable layout of an inherited class) and how that's compatible with MinGW/GCC stacks on Windows, mostly what can break it. I got stuck porting VST3 with multiple inheritance, and it was a headache trying to reverse engineer the appropriate struct layouts for COM implementations. [1] https://github.com/retep998/winapi-rs/blob/0.3/src/macros.rs https://github.com/retep998/winapi-rs/blob/0.3/src/macros.rs [2] https://github.com/Connicpu/com-impl https://github.com/Connicpu/com-impl [3] https://github.com/steinbergmedia/vst3sdk https://github.com/steinbergmedia/vst3sdk [4] https://github.com/m-hilgendorf/cli-host https://github.com/m-hilgendorf/cli-host (sorry for the messy code, it was a weekend of hacking away trying to host a VST3 in pure Rust)

pjmlp 7y ago
COM layout is followed upon most mainstream Windows compiled languages, namely major C++ compilers, .NET, Delphi, Eiffel, Ada, so it is not MSVC++ keeping ABI compatibility under the hood on their own.
- holy_city 7y ago
  My issue isn't ABI stability but the ABI itself w.r.t vtable layout. Best I can tell it should be similar to Itanium's spec? [1]. It's been months since I did this, but iirc my problems stemmed from having multiple interfaces on top of the same implementation and the ordering/layout of those interfaces, though the IUnknown interface which is supposed to handle that. [1] https://itanium-cxx-abi.github.io/cxx-abi/abi.html#vtable https://itanium-cxx-abi.github.io/cxx-abi/abi.html#vtable
  - barrkel 7y ago
    COM is agnostic as to how you do multiple inheritance - it doesn't have the concept. It specifies the QueryInterface protocol, but you don't need to return the same instance for the result of the QI call, just one that uses the same lifetime refcount. Tear-off interfaces and delegated implementations are things in this world.

jchw 7y ago

Very cool. I have played with COM in Go to a lesser extent. It’s not too bad actually. My intent was to write Go wrappers around all of the TSF COM interfaces, but I burnt out after getting basic IME working - TSF is enormous!

MehdiHK 7y ago
Hi, I believe you are talking about this repo https://github.com/jchv/go-tsf https://github.com/jchv/go-tsf Do you know any other project that's working on creating basic IME on top of TSF in Go?
- jchw 7y ago
  You are correct, and no I am not. Is your goal to build an application with IME support, or an entire IME?
  - MehdiHK 7y ago
    An entire IME.
    - jchw 7y ago
      Ah I see.. sadly, I never really figured out a great way to support full COM interop in Go. It would be ideal if you could implement an interface and register it as a COM object, but I hadn’t hashed out exactly how.

ajxs 7y ago

I'm interested in more information regarding the Safe Systems Programming Languages team's work. Did they assess other languages other than Rust? I'm scanning through their blog entries, without having read every single one in depth so please pardon me if I miss something, and I'm not seeing anything mentioning formal methods or other systems programming languages. It's a very interesting development for sure!