8 ms·

Not that it helps here, but Microsoft never considered the MSVCRT that ships with Windows to be public API. This is not the "Windows allocator", this is the (ve

Not that it helps here, but Microsoft never considered the MSVCRT that ships with Windows to be public API. This is not the "Windows allocator", this is the (very) old MSVC runtime library's allocator. Of course that doesn't keep anyone from using this library because it's present on any Windows system, unlike the newer MSVC versions' runtime library. Using the allocator from a later MSVC's runtime library would provide much better results, as would writing a custom allocator on top of Windows' heap implementation. MSVCRT basically just exists for backwards compatibility. It's impossible to improve this library at this point.

jart 4y ago
It's effectively mandatory. Microsoft provides about twelve different C Runtimes. But if you're building something like an open source library, you can't link two different C runtimes where you might accidentally malloc() memory with one and then free() with the other. If you want to be able to pass pointers around your dynamic link libraries, you have to link the one C runtime everyone else uses, which is MSVCRT. Also worth mentioning that on Windows 10 last time I checked ADVAPI32 links MSVCRT. So it's pretty much impossible to not link.
- TonyTrapp 4y ago
  It isn't mandatory. I have never actively linked against MSVCRT on Windows. From my experience it's mostly software that isn't built with Visual Studio that uses MSVCRT, or software that that takes extreme care of its binary size (e.g. 64k intros). MSVCRT is not even an up-to-date C runtime library. You wouldn't be able to use it for writing software requiring C11 library features without implementing them somewhere on top of it. It's true that you cannot just happily pass pointers around and expect someone else to be able to safely delete your pointer - but that is why any serious library with a C interface provides its own function to free objects you obtained from the library. Saying that this is impossible without MSVCRT implies that every software needs to be built with it, which is not even remotely the case. If I wanted, I could build all the C libraries I use with LLVM and still link against them in my application compiled with the latest MSVC runtime or UCRT. The much bigger problem is mixing C++ runtimes in the same piece of software, there you effectively must guarantee that each library uses the same runtime, or chaos ensues.
  - kazinator 4y ago
    If you're writing in C++ on Windows, expose only COM (or at least COM-style) interface called through virtual functions on an object pointer. Then you can use whatever C++ run-time you want, internally. What you don't want is the other library calling C++ functions by name. Like you pass it some ostream object and it calls ostream::put or whatever, where that symbolically resolves to the wrong one.
    - p_l 4y ago
      It's also why all, or nearly all, new API on windows are done through COM. It also means that C++ or other runtimes don't pollute your ABI and make it annoyingly hard to access features from any random code.
      - kazinator 4y ago
        Another reasonable choice for some components would be to have a purely C api; everything extern "C", and only PODs (plain old datastructures) in the arguments.
        olvy0 4y ago
        Exactly. Using a pure C interface escapes COM's requirement to register every component (regsvr32) and its overblown "GUID for everything" model, and its baroque way of creating components (somewhat alleviated by various macros and templates, but still). Making an object oriented interface is slightly cumbersome, but you can do it by sending back and forth a cookie (void* or int) of your object as the first parameter. This will also make your interface accessible to other languages, if the need arises, since the C ABI is a de-facto stable ABI on all platforms (disregarding library objects and issues). Another alternative if you want to stick with C++: make up your own lightweight COM! I've done this successfully in my own code at work, it works great, and has been working for 8 years now. This method allows people to write C++ components against my app without using COM and without having the exact same compiler versions and having symbol problems. It may seem like a lot of work but it really isn't. 1) Expose a pure virtual interface IFoo, that doesn't include any implementation. Once it is released, never touch it, only create new versions which inherit from it, IFoo2, IFoo3 etc. 2) Expose an extern "C" CreateFoo() that will return FooImpl : IFoo. Also a matching DeleteFoo(IFoo). 3) For all structs / classes you need to receive/send from IFoo methods, create another interface. This includes all C++ classes, including std::string, hide it behind IMyString or something. This is a minor inconvenience but it sidesteps all ABI incompatibilities. 4) Have all interfaces inherit from some IBaseXX which has a GetVersion() method A component which uses this component could call this, and if it returns e.g. 2, then it can safely cast IFoo to IFoo2* and use IFoo2 methods. Else it can return an error message or use something from IFoo* This relies on the fact that C++ vtable layout is essentially an ABI that will never change, at least under Windows, since the whole of COM relies on this and MS will never change it. Anything other than vtable layout in an object is subject to change, so the trick is to only have pure virtual interfaces. I have no idea if this trick will also work on Linux, I don't know how stable GCC / Clang's vtable layout is from version to version, but I suspect it will. This was taken from a CodeProject article I read a few years back, but I can't find it anymore... the closest I can find is [0] (DynObj - C++ Cross Platform Plugin Objects), but it is more complicated than what I read back then. I didn't use it, I wrote my own as I outlined above. Like I said, it isn't really that complicated. [0] https://www.codeproject.com/Articles/20648/DynObj-C-Cross-Platform-Plugin-Objects https://www.codeproject.com/Articles/20648/DynObj-C-Cross-Pl...
- plonk 4y ago
  Our programs ship their DLL dependencies in their own installer anyway, like most others on Windows. Just ship your FOSS library with a CMake configuration and let the users build it with whatever runtime they want.
- kazinator 4y ago
  C applications targeting Windows must provide their own C library with malloc and free (if they are using the "hosted implementation" features of C). MSVCRT.DLL isn't the library "everyone" uses; just Microsoft programs, and some misguided freeware built with MinGW. Even if ADVAPI32.DLL uses MSVCRT.DLL, it's not going to mistakenly call the malloc that you provide in your application; Windows DLL's don't even have that sort of global symbol resolution power. I would be very surprised if any public API in ADVAPI32 returns a pointer that the application is required to directly free, or accept a pointer that the application must malloc. If that were the case, you'd have to attach to MSVCRT.DLL with LoadLibrary, look up those functions with GetProcAddress and call them that way. Windows has non-malloc allocators for sharing memory that way among DLL's: the "Heap API" in KERNEL32. One component can HeapAlloc something which another can HeapFree: they have to agree on the same heap handle, though. You can use GetProcessHeap to get the default heap for the process. It may be that the MSVCRT.DLL malloc uses this; or else it's based on VirtualAlloc directly.
  - cesarb 4y ago
    > MSVCRT.DLL isn't the library "everyone" uses; just Microsoft programs, and some misguided freeware built with MinGW. There's a third set of users: programs built with old enough versions of the Microsoft compiler. Before Microsoft decided that every single version of its C compiler should use a different C runtime (and much later they changed their mind again), all Microsoft C and C++ compilers linked their output with MSVCRT.DLL. In fact, that's probably the reason MinGW chose to use MSVCRT.DLL: to increase compatibility with the Microsoft compilers, by using the same C runtime.
    - mananaysiempre 4y ago
      > Before Microsoft decided that every single version of its C compiler should use a different C runtime (and much later they changed their mind again). It is... not as simple as that[1]. MSVCRT itself is the fifth version of a Win32 C runtime, after CRTDLL (used to ship with the OS but no longer), MSVCRT10, MSVCRT20, and MSVCRT40. It’s just that both the toolchains linking with MSVCRT and the OSes shipping it (and nothing newer) endured for a very long time while Microsoft went on a collective .NET piligrimage. Of course, “NT OS/2” started the same year ANSI C was ratified (and Win32 only a couple of years later), so some degree of flailing around was inevitable. But the MSVCRT Garden of Eden story is not true. [1] http://www.malsmith.net/blog/visual-c-visual-history/ http://www.malsmith.net/blog/visual-c-visual-history/
FreakLegion 4y ago
There's also UCRT, which ships with the OS since Windows 10. The logic of this rant was a real head-scratcher. If you must blame one side, it's LLVM. Fragmentation of C runtimes is annoying but inescapable. Glibc for example isn't any better.
- ChrisSD 4y ago
  The UCRT has even been present since Windows 7, if users keep up with updates. Or if applications bundle the UCRT installer with their own.
  - josephcsible 4y ago
    > Or if applications bundle the UCRT installer with their own. But the UCRT is proprietary, so there are often legal issues with doing so.
    - ChrisSD 4y ago
      There is an official Universal CRT Redistributable.
      - rurban 4y ago
        And java ships it, and installs it into its own java bindir, not the global system dir. So you have two competing copies, which leads to race conditions (different "global" locks!) and nice cashes, until you can analyze it with DrMemory or the kernel debugger. Nobody does that.
- leajkinUnk 4y ago
  Could you elaborate why Glibc isn't any better? I remember some funny problems with Glibc, like, 20 years ago, but it's been invisible to me (as a user) since then. You get a new Glibc, old binaries still work, it's fine.
  - usrn 4y ago
garaetjjte 4y ago
>but Microsoft never considered the MSVCRT that ships with Windows to be public API It was in the past. At first msvcrt.dll was the runtime library used up to Visual C++ 6. Later, VC++ moved to their own separate dlls, but you could still link with system msvcrt.dll using corresponding DDK/WDK up to Windows 7. I'm also not sure that this is just ancient library left for compatibility, some system components still link to it, and msvcrt.dll itself seems to link with UCRT libraries.
- TonyTrapp 4y ago
  > It was in the past. At first msvcrt.dll was the runtime library used up to Visual C++ 6. At that time it was already a big mess, because at first it was the runtime library of Visual C++ 4 in fact! The gory details are here: https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=1273 https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12... > some system components still link to it Some system components themselves are very much ancient and unmaintained and only exist for backwards compatibility as well.
  - garaetjjte 4y ago
    Ancient or not, I don't think it really matters for allocation performance: malloc in both msvcrt.dll and ucrtbase.dll after some indirection ends up calling RtlAllocateHeap in ntdll.dll
    - TonyTrapp 4y ago
      It does, because modern allocators do a lot of house keeping on top of the system heaps, e.g. to quickly reuse identically-sized memory chunks that were just freed.
  - jasomill 4y ago
    For posterity, here's a link to a version of that page with the original comments section intact: https://web.archive.org/web/20160410051513/https://blogs.msdn.microsoft.com/oldnewthing/20140411-00/?p=1273/ https://web.archive.org/web/20160410051513/https://blogs.msd...
Sesse__ 4y ago
Win32 has an allocator (HeapAlloc), and it is similarly slow and low-concurrent. Even if you enable the newer stuff like LFH.

jart 4y ago

It's effectively mandatory. Microsoft provides about twelve different C Runtimes. But if you're building something like an open source library, you can't link two different C runtimes where you might accidentally malloc() memory with one and then free() with the other. If you want to be able to pass pointers around your dynamic link libraries, you have to link the one C runtime everyone else uses, which is MSVCRT. Also worth mentioning that on Windows 10 last time I checked ADVAPI32 links MSVCRT. So it's pretty much impossible to not link.

TonyTrapp 4y ago
It isn't mandatory. I have never actively linked against MSVCRT on Windows. From my experience it's mostly software that isn't built with Visual Studio that uses MSVCRT, or software that that takes extreme care of its binary size (e.g. 64k intros). MSVCRT is not even an up-to-date C runtime library. You wouldn't be able to use it for writing software requiring C11 library features without implementing them somewhere on top of it. It's true that you cannot just happily pass pointers around and expect someone else to be able to safely delete your pointer - but that is why any serious library with a C interface provides its own function to free objects you obtained from the library. Saying that this is impossible without MSVCRT implies that every software needs to be built with it, which is not even remotely the case. If I wanted, I could build all the C libraries I use with LLVM and still link against them in my application compiled with the latest MSVC runtime or UCRT. The much bigger problem is mixing C++ runtimes in the same piece of software, there you effectively must guarantee that each library uses the same runtime, or chaos ensues.
- kazinator 4y ago
  If you're writing in C++ on Windows, expose only COM (or at least COM-style) interface called through virtual functions on an object pointer. Then you can use whatever C++ run-time you want, internally. What you don't want is the other library calling C++ functions by name. Like you pass it some ostream object and it calls ostream::put or whatever, where that symbolically resolves to the wrong one.
  - p_l 4y ago
    It's also why all, or nearly all, new API on windows are done through COM. It also means that C++ or other runtimes don't pollute your ABI and make it annoyingly hard to access features from any random code.
    - kazinator 4y ago
      Another reasonable choice for some components would be to have a purely C api; everything extern "C", and only PODs (plain old datastructures) in the arguments.
      - olvy0 4y ago
        Exactly. Using a pure C interface escapes COM's requirement to register every component (regsvr32) and its overblown "GUID for everything" model, and its baroque way of creating components (somewhat alleviated by various macros and templates, but still). Making an object oriented interface is slightly cumbersome, but you can do it by sending back and forth a cookie (void* or int) of your object as the first parameter. This will also make your interface accessible to other languages, if the need arises, since the C ABI is a de-facto stable ABI on all platforms (disregarding library objects and issues). Another alternative if you want to stick with C++: make up your own lightweight COM! I've done this successfully in my own code at work, it works great, and has been working for 8 years now. This method allows people to write C++ components against my app without using COM and without having the exact same compiler versions and having symbol problems. It may seem like a lot of work but it really isn't. 1) Expose a pure virtual interface IFoo, that doesn't include any implementation. Once it is released, never touch it, only create new versions which inherit from it, IFoo2, IFoo3 etc. 2) Expose an extern "C" CreateFoo() that will return FooImpl : IFoo. Also a matching DeleteFoo(IFoo). 3) For all structs / classes you need to receive/send from IFoo methods, create another interface. This includes all C++ classes, including std::string, hide it behind IMyString or something. This is a minor inconvenience but it sidesteps all ABI incompatibilities. 4) Have all interfaces inherit from some IBaseXX which has a GetVersion() method A component which uses this component could call this, and if it returns e.g. 2, then it can safely cast IFoo to IFoo2* and use IFoo2 methods. Else it can return an error message or use something from IFoo* This relies on the fact that C++ vtable layout is essentially an ABI that will never change, at least under Windows, since the whole of COM relies on this and MS will never change it. Anything other than vtable layout in an object is subject to change, so the trick is to only have pure virtual interfaces. I have no idea if this trick will also work on Linux, I don't know how stable GCC / Clang's vtable layout is from version to version, but I suspect it will. This was taken from a CodeProject article I read a few years back, but I can't find it anymore... the closest I can find is [0] (DynObj - C++ Cross Platform Plugin Objects), but it is more complicated than what I read back then. I didn't use it, I wrote my own as I outlined above. Like I said, it isn't really that complicated. [0] https://www.codeproject.com/Articles/20648/DynObj-C-Cross-Platform-Plugin-Objects https://www.codeproject.com/Articles/20648/DynObj-C-Cross-Pl...
plonk 4y ago
Our programs ship their DLL dependencies in their own installer anyway, like most others on Windows. Just ship your FOSS library with a CMake configuration and let the users build it with whatever runtime they want.
kazinator 4y ago
C applications targeting Windows must provide their own C library with malloc and free (if they are using the "hosted implementation" features of C). MSVCRT.DLL isn't the library "everyone" uses; just Microsoft programs, and some misguided freeware built with MinGW. Even if ADVAPI32.DLL uses MSVCRT.DLL, it's not going to mistakenly call the malloc that you provide in your application; Windows DLL's don't even have that sort of global symbol resolution power. I would be very surprised if any public API in ADVAPI32 returns a pointer that the application is required to directly free, or accept a pointer that the application must malloc. If that were the case, you'd have to attach to MSVCRT.DLL with LoadLibrary, look up those functions with GetProcAddress and call them that way. Windows has non-malloc allocators for sharing memory that way among DLL's: the "Heap API" in KERNEL32. One component can HeapAlloc something which another can HeapFree: they have to agree on the same heap handle, though. You can use GetProcessHeap to get the default heap for the process. It may be that the MSVCRT.DLL malloc uses this; or else it's based on VirtualAlloc directly.
- cesarb 4y ago
  > MSVCRT.DLL isn't the library "everyone" uses; just Microsoft programs, and some misguided freeware built with MinGW. There's a third set of users: programs built with old enough versions of the Microsoft compiler. Before Microsoft decided that every single version of its C compiler should use a different C runtime (and much later they changed their mind again), all Microsoft C and C++ compilers linked their output with MSVCRT.DLL. In fact, that's probably the reason MinGW chose to use MSVCRT.DLL: to increase compatibility with the Microsoft compilers, by using the same C runtime.
  - mananaysiempre 4y ago
    > Before Microsoft decided that every single version of its C compiler should use a different C runtime (and much later they changed their mind again). It is... not as simple as that[1]. MSVCRT itself is the fifth version of a Win32 C runtime, after CRTDLL (used to ship with the OS but no longer), MSVCRT10, MSVCRT20, and MSVCRT40. It’s just that both the toolchains linking with MSVCRT and the OSes shipping it (and nothing newer) endured for a very long time while Microsoft went on a collective .NET piligrimage. Of course, “NT OS/2” started the same year ANSI C was ratified (and Win32 only a couple of years later), so some degree of flailing around was inevitable. But the MSVCRT Garden of Eden story is not true. [1] http://www.malsmith.net/blog/visual-c-visual-history/ http://www.malsmith.net/blog/visual-c-visual-history/

FreakLegion 4y ago

There's also UCRT, which ships with the OS since Windows 10. The logic of this rant was a real head-scratcher. If you must blame one side, it's LLVM. Fragmentation of C runtimes is annoying but inescapable. Glibc for example isn't any better.

ChrisSD 4y ago
The UCRT has even been present since Windows 7, if users keep up with updates. Or if applications bundle the UCRT installer with their own.
- josephcsible 4y ago
  > Or if applications bundle the UCRT installer with their own. But the UCRT is proprietary, so there are often legal issues with doing so.
  - ChrisSD 4y ago
    There is an official Universal CRT Redistributable.
    - rurban 4y ago
      And java ships it, and installs it into its own java bindir, not the global system dir. So you have two competing copies, which leads to race conditions (different "global" locks!) and nice cashes, until you can analyze it with DrMemory or the kernel debugger. Nobody does that.
leajkinUnk 4y ago
Could you elaborate why Glibc isn't any better? I remember some funny problems with Glibc, like, 20 years ago, but it's been invisible to me (as a user) since then. You get a new Glibc, old binaries still work, it's fine.
- usrn 4y ago

garaetjjte 4y ago

>but Microsoft never considered the MSVCRT that ships with Windows to be public API It was in the past. At first msvcrt.dll was the runtime library used up to Visual C++ 6. Later, VC++ moved to their own separate dlls, but you could still link with system msvcrt.dll using corresponding DDK/WDK up to Windows 7. I'm also not sure that this is just ancient library left for compatibility, some system components still link to it, and msvcrt.dll itself seems to link with UCRT libraries.

TonyTrapp 4y ago
> It was in the past. At first msvcrt.dll was the runtime library used up to Visual C++ 6. At that time it was already a big mess, because at first it was the runtime library of Visual C++ 4 in fact! The gory details are here: https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=1273 https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12... > some system components still link to it Some system components themselves are very much ancient and unmaintained and only exist for backwards compatibility as well.
- garaetjjte 4y ago
  Ancient or not, I don't think it really matters for allocation performance: malloc in both msvcrt.dll and ucrtbase.dll after some indirection ends up calling RtlAllocateHeap in ntdll.dll
  - TonyTrapp 4y ago
    It does, because modern allocators do a lot of house keeping on top of the system heaps, e.g. to quickly reuse identically-sized memory chunks that were just freed.
- jasomill 4y ago
  For posterity, here's a link to a version of that page with the original comments section intact: https://web.archive.org/web/20160410051513/https://blogs.msdn.microsoft.com/oldnewthing/20140411-00/?p=1273/ https://web.archive.org/web/20160410051513/https://blogs.msd...

Sesse__ 4y ago

Win32 has an allocator (HeapAlloc), and it is similarly slow and low-concurrent. Even if you enable the newer stuff like LFH.