5 ms·

> When we free memory, we should make sure that if the block we return to the free list is next to any other free blocks, we combine them together. This is call

> When we free memory, we should make sure that if the block we return to the free list is next to any other free blocks, we combine them together. This is called "coalescing." A little offtopic but the default Delphi 7 memory allocator did this, except that it also merged blocks that it obtained from different OS allocation calls. This worked fine for regular usage, but if that memory was ever used for Bitmaps for UI stuff, it wouldn't work: Since Windows does some of its UI stuff in kernel mode, before doing anything Windows would attempt to lock the entire allocation's VAD entry to prevent you from messing with it in another thread while it was using it. If the Bitmap you were working on happened to belong to two different OS-level allocations, this lock function would fail since it wasn't meant to handle that case. This would lead to random, extremely cryptic errors such as ERROR_NO_SCROLLBARS "The window does not have scroll bars." since the lock call was deep in the stack and the callers replaced the error with another one as it bubbled up. In the end we had to replace the allocator to avoid that issue. That was a fun day I spent debugging that.

meekaaku 3y ago
You mean a bug this deep took one day to debug?
- alex7734 3y ago
  Yes and no, it took me from 8am to 3am once we decided it needed to get fixed but really it sat on the app for years, it only happened on a background process that sent print jobs on a timer, since it used Windows GDI to compose the image we sent to the printer it was affected (our "frontend" should've been affected too but never was, I guess because it had a different memory usage pattern). We just had it restart itself and try again whenever it got one of those errors when printing but eventually we wanted to add a feature that required the process to not die, and by that time I was already 99% sure that it wasn't something in our code and I had already ruled out threading issues. I ended up putting it in a VM with a kernel debugger attached and having a script make a snapshot and make it print over and over until it errored, then following along in IDA until I saw what was going on. Having a way to trigger it (by restoring the snapshot) on demand helped a lot, otherwise it would have taken forever to make sense of it as it could sit without crashing for nearly an hour.
  - zeusk 3y ago
    How do you attach kd to VM? While I was at MS, it was such a big PITA - we just had a bunch of IT managed machines with KVM console access and KDNET for debugging.
    - EvanAnderson 3y ago
      In Hyper-V it's fairly easy. You make a virtual serial port ("COMPort"), set the bootloader to enable kernel debugging over serial, then connect to the virtual serial port from the host via a named pipe. https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/attaching-to-a-virtual-machine--kernel-mode- https://learn.microsoft.com/en-us/windows-hardware/drivers/d... I haven't tried it with vSphere but I suspect it'd be similar.
      - jasomill 3y ago
        Kernel debugging over serial should be possible in vSphere, but Ethernet is easier to set up: 1. Make sure at least one virtual NIC on the target VM (with IP connectivity to the debug host machine/VM) is on Microsoft's NIC whitelist[1]. I use e1000e; note that vmxnet3 is not on the list. 2. Follow Microsoft's directions[2] to connect. I can confirm this works on vSphere[3] and there's no reason it shouldn't also work on VMware Workstation, Player, and Fusion. [1] https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/supported-ethernet-nics-for-network-kernel-debugging-in-windows-10 https://learn.microsoft.com/en-us/windows-hardware/drivers/d... [2] https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-up-a-network-debugging-connection-automatically https://learn.microsoft.com/en-us/windows-hardware/drivers/d... [3] Tested last week with a Windows Server 2022 target (e1000e virtual NIC) and Windows 10 debug host (vmxnet3 virtual NIC), both running on ESXi 8.0 Update 1 VM hosts.
      - vlovich123 3y ago
        Delphi 7 was 2002. Was hyper-v around back then? Or is this just a legacy app still running?
        3y ago
ilyt 3y ago
I think the far weirder part of this was the kernel-side handling of scrollbars
- alex7734 3y ago
  If I recall correctly the kernel part of things would return an out of memory error which the user mode DLL translated to that weird error (sometimes, other times it would just say "out of system resources", it depended on what widget the bitmap that overlapped the two memory regions belonged to). Here's a 2003 forum post from someone else having the same problem: http://www.delphigroups.info/2/1/749064.html http://www.delphigroups.info/2/1/749064.html
- derefr 3y ago
  Until Windows 95, Windows was essentially just a DOS application that grabbed the framebuffer and ran an event loop where it drew "controls" (which includes windows, buttons, text views, and yes, scrollbars.) That was the whole point of it. It wasn't an "OS" per se; DOS was the OS. Windows was what a Linux-head would think of as a combination of an X server and window manager. And Windows loaded your "application" as essentially a DLL, with the Windows global event loop calling into your application's event-loop delegate handler (WndProc) whenever it has an interesting event that your application might like to react to. (Your application wasn't even a "process" per se. Until Windows 95, everything was just happening in one shared address space, in real mode. In fact, it was only in Windows 3.1 where user applications stopped running in ring 0!) If you think about it, this "the kernel is a game engine and your application is the game" approach isn't necessarily a bad design... for a single-tasking OS's library exokernel, like the Wii's https://wiibrew.org/wiki/IOS https://wiibrew.org/wiki/IOS. But, of course, Windows claimed to be a multitasking OS. But it actually wasn't! And I don't mean the obvious thing about it not having pre-emption. Lots of multitasking OSes didn't have pre-emption. No, what I mean is that the concurrency primitive for the cooperative scheduling wasn't the "task" (i.e. the process or thread. Which, again, there weren't any of.) Instead, the concurrency primitive was the window. Until Windows 95, Windows was a multi-windowing OS. Each control was owned by a window. Each window had a WndProc. If your Windows executable (i.e. application delegate module) set up two windows, then each window participated separately in the global Windows event loop, up-to-and-including things like having its own set of loaded fonts, its own clipboard state, and its own interned strings table. In OOP terms†, your application was just a dead "class object", running no logic of its own save for one-time load and unload hooks. It was the windows themselves that were the "instances" of your class. This might make you realize why MDI (or Multiple Document Interface, where there are multiple small per-document "windows" inside one big window) was so popular back then. The MDI "windows" weren't actually windows — they didn't have their own WndProc. They were just controls, like a tab view is a control. Only the big container window was a real window, and so all the resources within that big window were shared between all the virtual windows. MDI was a memory-saving trick! --- † The actual more interesting analogy is that Windows was essentially a (single-threaded, cooperatively-scheduled) actor system, where windows were the actors. There is a very close parallel between (Windows 3.1 executables, Windows 3.1 windows) and (Erlang modules, Erlang processes).

meekaaku 3y ago

You mean a bug this deep took one day to debug?

alex7734 3y ago
Yes and no, it took me from 8am to 3am once we decided it needed to get fixed but really it sat on the app for years, it only happened on a background process that sent print jobs on a timer, since it used Windows GDI to compose the image we sent to the printer it was affected (our "frontend" should've been affected too but never was, I guess because it had a different memory usage pattern). We just had it restart itself and try again whenever it got one of those errors when printing but eventually we wanted to add a feature that required the process to not die, and by that time I was already 99% sure that it wasn't something in our code and I had already ruled out threading issues. I ended up putting it in a VM with a kernel debugger attached and having a script make a snapshot and make it print over and over until it errored, then following along in IDA until I saw what was going on. Having a way to trigger it (by restoring the snapshot) on demand helped a lot, otherwise it would have taken forever to make sense of it as it could sit without crashing for nearly an hour.
- zeusk 3y ago
  How do you attach kd to VM? While I was at MS, it was such a big PITA - we just had a bunch of IT managed machines with KVM console access and KDNET for debugging.
  - EvanAnderson 3y ago
    In Hyper-V it's fairly easy. You make a virtual serial port ("COMPort"), set the bootloader to enable kernel debugging over serial, then connect to the virtual serial port from the host via a named pipe. https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/attaching-to-a-virtual-machine--kernel-mode- https://learn.microsoft.com/en-us/windows-hardware/drivers/d... I haven't tried it with vSphere but I suspect it'd be similar.
    - jasomill 3y ago
      Kernel debugging over serial should be possible in vSphere, but Ethernet is easier to set up: 1. Make sure at least one virtual NIC on the target VM (with IP connectivity to the debug host machine/VM) is on Microsoft's NIC whitelist[1]. I use e1000e; note that vmxnet3 is not on the list. 2. Follow Microsoft's directions[2] to connect. I can confirm this works on vSphere[3] and there's no reason it shouldn't also work on VMware Workstation, Player, and Fusion. [1] https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/supported-ethernet-nics-for-network-kernel-debugging-in-windows-10 https://learn.microsoft.com/en-us/windows-hardware/drivers/d... [2] https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-up-a-network-debugging-connection-automatically https://learn.microsoft.com/en-us/windows-hardware/drivers/d... [3] Tested last week with a Windows Server 2022 target (e1000e virtual NIC) and Windows 10 debug host (vmxnet3 virtual NIC), both running on ESXi 8.0 Update 1 VM hosts.
    - vlovich123 3y ago
      Delphi 7 was 2002. Was hyper-v around back then? Or is this just a legacy app still running?
      - 3y ago

ilyt 3y ago

I think the far weirder part of this was the kernel-side handling of scrollbars

alex7734 3y ago
If I recall correctly the kernel part of things would return an out of memory error which the user mode DLL translated to that weird error (sometimes, other times it would just say "out of system resources", it depended on what widget the bitmap that overlapped the two memory regions belonged to). Here's a 2003 forum post from someone else having the same problem: http://www.delphigroups.info/2/1/749064.html http://www.delphigroups.info/2/1/749064.html
derefr 3y ago
Until Windows 95, Windows was essentially just a DOS application that grabbed the framebuffer and ran an event loop where it drew "controls" (which includes windows, buttons, text views, and yes, scrollbars.) That was the whole point of it. It wasn't an "OS" per se; DOS was the OS. Windows was what a Linux-head would think of as a combination of an X server and window manager. And Windows loaded your "application" as essentially a DLL, with the Windows global event loop calling into your application's event-loop delegate handler (WndProc) whenever it has an interesting event that your application might like to react to. (Your application wasn't even a "process" per se. Until Windows 95, everything was just happening in one shared address space, in real mode. In fact, it was only in Windows 3.1 where user applications stopped running in ring 0!) If you think about it, this "the kernel is a game engine and your application is the game" approach isn't necessarily a bad design... for a single-tasking OS's library exokernel, like the Wii's https://wiibrew.org/wiki/IOS https://wiibrew.org/wiki/IOS. But, of course, Windows claimed to be a multitasking OS. But it actually wasn't! And I don't mean the obvious thing about it not having pre-emption. Lots of multitasking OSes didn't have pre-emption. No, what I mean is that the concurrency primitive for the cooperative scheduling wasn't the "task" (i.e. the process or thread. Which, again, there weren't any of.) Instead, the concurrency primitive was the window. Until Windows 95, Windows was a multi-windowing OS. Each control was owned by a window. Each window had a WndProc. If your Windows executable (i.e. application delegate module) set up two windows, then each window participated separately in the global Windows event loop, up-to-and-including things like having its own set of loaded fonts, its own clipboard state, and its own interned strings table. In OOP terms†, your application was just a dead "class object", running no logic of its own save for one-time load and unload hooks. It was the windows themselves that were the "instances" of your class. This might make you realize why MDI (or Multiple Document Interface, where there are multiple small per-document "windows" inside one big window) was so popular back then. The MDI "windows" weren't actually windows — they didn't have their own WndProc. They were just controls, like a tab view is a control. Only the big container window was a real window, and so all the resources within that big window were shared between all the virtual windows. MDI was a memory-saving trick! --- † The actual more interesting analogy is that Windows was essentially a (single-threaded, cooperatively-scheduled) actor system, where windows were the actors. There is a very close parallel between (Windows 3.1 executables, Windows 3.1 windows) and (Erlang modules, Erlang processes).