6 ms·

DOS/4GW and Protected Mode (2021)

126 points by atan2 4y ago

dusted 4y ago
So cool to finally read about this.. I remember wondering about this when I was a kid, starting games..
- hericium 4y ago
  And nowhere to search for info on it!
  - adzm 4y ago
    This was the most frustrating part of learning this stuff while growing up; I quickly got into more esoteric issues and complicated questions than I could find answers to from the limited circle a ten year old has. The big breakthrough was a teacher who let me use his Internet connection to post on newsgroups. I was always amazed at how incredibly helpful people were on there, even at big name companies at the time. This was in the early-to-mid 90s. I even had someone from Microsoft explain and correct code I had been sharing to make a silly TSR that would bounce a smiley face around the prompt (which was an existing .com executable I was trying to reproduce)
  - pjmlp 4y ago
    Depends, I was an havid magazine collector of PC Techniques, The C Users Journal (latter The C/C++ Users Journal), DrDobbs, Crash, Amiga Format, Computer Shopper UK, Portuguese and Spanisch versions of them like Spooler, Microhobby, Micromania, Programacion and the local library computing section.
forgotmypw17 4y ago
https://archive.ph/SpUm4 https://archive.ph/SpUm4
lizknope 4y ago
I first learned C in 1991 on a VAX running VMS and also on an IBM AIX machine. If I went outside the array bounds I might get a crash or segfault and could use a debugger to load the crash dump and see where it died. When I tried using my roommate's DOS PC and C compiler I crashed the entire machine so many times. There was no memory protection of any kind. Write something outside the array bounds and you could overwrite critical DOS data structures and lock up the whole machine. Hard reboot so many times. Then my roommate tried to explain near and far pointers. I never understood it until I looked it up a few years ago. It was all related to the 16-bit vs 32-bit segmented vs flat memory models. Everything just seemed so much easier and faster on the VMS and Unix systems. But the also cost 10 to 100 times as much. I also thought it was really pathetic that I could only run one program at a time. On the VAX and Unix systems 10 to 200 people could be logged on at the same time all doing there own thing and it was very difficult to accidentally bring down the whole machine. It all made me NOT want my own PC because DOS/Win 3.1 was so limited. It wasn't until Linux in 1993 that I wanted my own PC.
- karmakaze 4y ago
  I agree DOS/Windows wasn't anything to look forward to running. It was only good for FreeCell or selling software for. I did have a Windows laptop that dial-up connected to the Internet via Winsock.dll though, so that was cool. I was running OS/2 1.2 and 1.3 in 1988-89 and beta versions before that. It only let you run a single DOS box but you could run any number of protected mode OS/2 text and Presentation Manager (PM) GUI programs. These were of the 16-bit (near/far) segmented memory model. In 1992, OS/2 2.0 ran the 32-bit flat mode and multiple DOS boxes in windowed areas of the desktop, as well as Win16 programs. The company also had developed their software for IBM mainframes, Wang minis, VAX, AIX, HP-UX, and DEC Alpha. Of all of these OS/2 and Windows NT were the most interesting to me as they seemed close to consumer platforms. The experimental NeXT target we dabbled with but never ported to. That was the future that we had to wait many years to be popularized by Apple. Interface Builder seemed so much better than XCode though. DOS4GW was the flavour that came free with the Watcom compilers. It was very popular with DOS games that let you use the available hardware to the maximum capabilities.
- indymike 4y ago
JayGuerette 4y ago
"... a terminate-and-stay-resident program (or TSR) was a computer program running under DOS that uses a system call to return control to DOS as though it has finished, but remains in computer memory so it can be reactivated later. Needless to say, this was extremely unreliable." There were very likely some hacky TSRs that caused problems, but in my experience most were extremely reliable. We used an off-the-shelf TSR to enhance a motion control system that laser scribed ceramic vacuum checks for silicon wafer fabrication. Those things cost $5k in 1990, and took ~20h of processing, increasing their value to $15k; we wouldn't screw around with something that was inherently "extremely unreliable".
- anyfoo 4y ago
  Yes. DOS itself came with several TSRs, for example keyboard drivers. They were absolutelt commonplace and, if written properly, nowhere as unreliable as claimed.
- pinewurst 4y ago
  I used a TSR to run protected mode software under early Microsoft Windows back in the day. Effectively it was letting me have the big allocations I needed while using Windows as a portable display driver.
- Falkon1313 4y ago
  I had one TSR that let you allocate RAM and switch between up to 3 programs. Of course, they had to be small programs due to the 640KB limit. But it was still quite useful in the days before hard drives, to be able to have a couple of utilities (like a text editor) loaded without having to keep swapping floppy disks. Also wrote a couple of TSRs of my own as a kid learning to program. Sure you could crash the system, as you could with any program, but they were as reliable as anything else. The only thing special was that you generally didn't want to run two TSRs that naively hooked the same interrupt without chaining properly. But back then we didn't have thousands of mysterious background processes always running in the background. You knew what few programs you had run, so it wasn't really a problem.
- deleted 4y ago
  [deleted]
jmclnx 4y ago
Very interesting article. My objects were always written for real mode on DOS, by the time I would have wanted to use these extenders, Coherent and then Linux came out.
mrlonglong 4y ago
I loved Watcom C/C++ for what I could do with it. Unreal mode became easy with the appropriate magic.
nazgulsenpai 4y ago
Anyone too young to remember or non-DOS gamers, you can see it in action here on Duke Nukem 3D running in browser on the Internet Archive: https://archive.org/details/DUKE3D_DOS https://archive.org/details/DUKE3D_DOS :)
- atan2 4y ago
  Thank you for this! It's also super helpful that it runs directly in the web browser. :)
sbf501 4y ago
EDIT: Why do I post sometimes...
- nazgulsenpai 4y ago
  Changed because parent changed. Don't sweat it, fren. We all have done it!
  - deleted 4y ago
    [deleted]
  - sbf501 4y ago
    EDIT... :)
starmole 4y ago
Note that extenders did a lot of emulation stuff for you, which was not really needed if all one wanted was flat 32bit mode. We did it for 4k intros back in the day: https://www.pouet.net/prod.php?which=289 https://www.pouet.net/prod.php?which=289. One fun optimization trick was the 0x66 prefix: Switching from 16bit to 32bit mode also switched the trade-off in opcode sizes. So in that intro most of the audio code runs in 16bit mode, while the graphics (which is actually not palette but full 32bit color) runs in 32bit mode.
- andrewf 4y ago
  I'd be interested in reading the source, but links to the source code seem to be dead (domains bought up by dodgy parking page types)
  - starmole 4y ago
    Yeah, I noticed. Tried to look at it myself - I don't have it archived anywhere myself. But last time I looked at it I cringed pretty hard. It's not great code :) But here you go: https://web.archive.org/web/20120710162700/http://www.active-web.cc/html/research/bin/v3src.zip https://web.archive.org/web/20120710162700/http://www.active...
    - andrewf 4y ago
      Neat, thanks!
bayesian_horse 4y ago
I actually remember this shit. Man I'm old.
captainmuon 4y ago
I always wondered why DOS extenders were separate .exes. It looks like you were supposed to be able to replace them, or start them in advance and share one with multiple programs. But that wasn't the case, when you launch it you just got a message to launch the main executable. Why did they bother to make it a proper executable? Wouldn't it make more sense to give it a .sys or .dat extension and to just load it yourself (in DOS a matter of reading it into memory and jumping to that address). Or, since it is proprietary and often delivered with the compiler, just link it in statically? Random anecdote: as a kid, DOS/4GW stood for "DOS for great win(dows)" (which I know makes absolutely no sense).
- TonyTrapp 4y ago
  You can replace DOS extenders to a certain extent. DPMI is standardized, so if an application is directly developed against the DPMI interface, it doesn't matter which DPMI server is present. I remember that some programs would run without any additional executable on Windows 98 (which provides its own DPMI server) but would require CWSDPMI.EXE when run from DOS. I guess that other programs would directly communicate with DOS/4GW through a different interface and that's when you can't just replace it with a different extender, but there are other extenders that can fully replace DOS/4GW such as DOS/32.
- nu11ptr 4y ago
  You can replace them. I forget the details on how I did it, but I recall doing this for a popular first person star wars game at the time. I replaced the DPMI runtime with a different one and the level load times skyrocketed in speed. My suspicion at the time was that the runtime included switched to real mode to do disk transfers but the other perhaps talked directly to the IDE hard disk staying in protected mode? Not sure, but it was very, very fast with I/O. I was pretty happy with my hack at the time.
  - leeter 4y ago
    My suspicion is you're right. Mostly based on Raymond's blog on the role of DOS in Windows 95[1]. Where people using weird drivers and other things that got in the way of 32bit disk access could massively slow down the system. It's plausible that one extender was being "nice" and using your native drivers or at least trying to be compatible whereas another just went "Nah that crap is slow" and just went for it with a native direct driver and was thus massively faster. [1] https://devblogs.microsoft.com/oldnewthing/20071224-00/?p=24063 https://devblogs.microsoft.com/oldnewthing/20071224-00/?p=24...
rob74 4y ago
So the "4G" stands for 4 gigabytes... that's funny, as at that time (early nineties) I bet no one could even imagine a humble PC having 4 GB of memory. That was the domain of supercomputers. And nowadays, even smartphones have more RAM than that... Also, "Fun fact: The original Wolfenstein 3D engine, created by id Software, was developed using pure real mode". Well, that figures... if you still wanted your code to run on 16 bit CPUs (286 and below), you had to use real mode. That's also why, for several years, only the then-"AAA" games like Doom, Duke Nukem 3D or Tomb Raider used DOS extenders. Titles that were less demanding on the hardware (platformers and other 2D games) kept using real mode for quite some time longer. Actually, I can add a fun fact of my own: when I started in software development in 2000, it was at a company developing Windows applications using Delphi - and at that time they were still keeping up compatibility with Windows 3.1, i.e. compiling in 16 bit mode. They finally switched to 32 bit soon after I joined, and I can't tell you what a relief that was. Although getting rid of the weird hacks in the source code that were made necessary by the constrained memory space of 16 bit applications took some more time...
abcd_f 4y ago
Watcom compiler shipped with DOS/4GW extender and the audacious simplicity of being able to malloc one whole megabyte in one go felt like a real magic. Good times. * Not 100% if it was 1 meg, but 4gw ultimately removed the run-time allocation cap. If you had X megs, you could allocate X megs with one call. That was truly revolutionary.
hyperman1 4y ago
There was always a delay of a few seconds before the main program started. I found out why when I tried to write protected mode code myself: You could only ask 4kb of memory at a time from emm386 (or himem, I forgot). There was no bulk access. So you had to loop trough the complete allocation in steps of 4k at a time. I think emm386 switched to protected mode, changed a page table, then went back to real/v86 mode for each block. When my code was written, it had exactly the same delay as other dos extenders.
simne 4y ago
Actually, Gates really said "640k will be enough..", but as I remember, context was, "to run (current for that year) msoffice comfortable", with typical for that year business tasks :) So I think, it will be better for him to admit this, and give exact source, so will not Streisand effect issues.
- willywanker 4y ago
  He said this in 1981, there was no Windows let alone MS Office at the time. The average simple text mode DOS application was barely 50-100kb in total, so 640k was quite a comfortable limit for several years more.
tssva 4y ago
> The 286 was a 16-bit processor, which means that the 286 could now address up to 16 MB of RAM! The 80286 was a 16-bit processor but had a 24-bit address bus which is what allowed it to access 16MB of RAM. The 8088/8086 were also 16-bit processors but had a 20-bit address bus which limited them to 1MB. The Z80 and 6502 were 8-bit CPUs with 16-bit address buses which limited them to 64KB.
nicky-g 4y ago
Love pikuma! Great learning resource
int_19h 4y ago
Some late DOS extenders provided Win32 APIs for stuff like the console, threads, memory etc. HX-DOS (https://www.japheth.de/HX.html https://www.japheth.de/HX.html) was famously capable of running Quake 2 on its own.

126 points by atan2 4y ago

dusted 4y ago

So cool to finally read about this.. I remember wondering about this when I was a kid, starting games..

hericium 4y ago
And nowhere to search for info on it!
- adzm 4y ago
  This was the most frustrating part of learning this stuff while growing up; I quickly got into more esoteric issues and complicated questions than I could find answers to from the limited circle a ten year old has. The big breakthrough was a teacher who let me use his Internet connection to post on newsgroups. I was always amazed at how incredibly helpful people were on there, even at big name companies at the time. This was in the early-to-mid 90s. I even had someone from Microsoft explain and correct code I had been sharing to make a silly TSR that would bounce a smiley face around the prompt (which was an existing .com executable I was trying to reproduce)
- pjmlp 4y ago
  Depends, I was an havid magazine collector of PC Techniques, The C Users Journal (latter The C/C++ Users Journal), DrDobbs, Crash, Amiga Format, Computer Shopper UK, Portuguese and Spanisch versions of them like Spooler, Microhobby, Micromania, Programacion and the local library computing section.

forgotmypw17 4y ago

https://archive.ph/SpUm4 https://archive.ph/SpUm4

lizknope 4y ago

I first learned C in 1991 on a VAX running VMS and also on an IBM AIX machine. If I went outside the array bounds I might get a crash or segfault and could use a debugger to load the crash dump and see where it died. When I tried using my roommate's DOS PC and C compiler I crashed the entire machine so many times. There was no memory protection of any kind. Write something outside the array bounds and you could overwrite critical DOS data structures and lock up the whole machine. Hard reboot so many times. Then my roommate tried to explain near and far pointers. I never understood it until I looked it up a few years ago. It was all related to the 16-bit vs 32-bit segmented vs flat memory models. Everything just seemed so much easier and faster on the VMS and Unix systems. But the also cost 10 to 100 times as much. I also thought it was really pathetic that I could only run one program at a time. On the VAX and Unix systems 10 to 200 people could be logged on at the same time all doing there own thing and it was very difficult to accidentally bring down the whole machine. It all made me NOT want my own PC because DOS/Win 3.1 was so limited. It wasn't until Linux in 1993 that I wanted my own PC.

karmakaze 4y ago
I agree DOS/Windows wasn't anything to look forward to running. It was only good for FreeCell or selling software for. I did have a Windows laptop that dial-up connected to the Internet via Winsock.dll though, so that was cool. I was running OS/2 1.2 and 1.3 in 1988-89 and beta versions before that. It only let you run a single DOS box but you could run any number of protected mode OS/2 text and Presentation Manager (PM) GUI programs. These were of the 16-bit (near/far) segmented memory model. In 1992, OS/2 2.0 ran the 32-bit flat mode and multiple DOS boxes in windowed areas of the desktop, as well as Win16 programs. The company also had developed their software for IBM mainframes, Wang minis, VAX, AIX, HP-UX, and DEC Alpha. Of all of these OS/2 and Windows NT were the most interesting to me as they seemed close to consumer platforms. The experimental NeXT target we dabbled with but never ported to. That was the future that we had to wait many years to be popularized by Apple. Interface Builder seemed so much better than XCode though. DOS4GW was the flavour that came free with the Watcom compilers. It was very popular with DOS games that let you use the available hardware to the maximum capabilities.
indymike 4y ago

JayGuerette 4y ago

"... a terminate-and-stay-resident program (or TSR) was a computer program running under DOS that uses a system call to return control to DOS as though it has finished, but remains in computer memory so it can be reactivated later. Needless to say, this was extremely unreliable." There were very likely some hacky TSRs that caused problems, but in my experience most were extremely reliable. We used an off-the-shelf TSR to enhance a motion control system that laser scribed ceramic vacuum checks for silicon wafer fabrication. Those things cost $5k in 1990, and took ~20h of processing, increasing their value to $15k; we wouldn't screw around with something that was inherently "extremely unreliable".

anyfoo 4y ago
Yes. DOS itself came with several TSRs, for example keyboard drivers. They were absolutelt commonplace and, if written properly, nowhere as unreliable as claimed.
pinewurst 4y ago
I used a TSR to run protected mode software under early Microsoft Windows back in the day. Effectively it was letting me have the big allocations I needed while using Windows as a portable display driver.
Falkon1313 4y ago
I had one TSR that let you allocate RAM and switch between up to 3 programs. Of course, they had to be small programs due to the 640KB limit. But it was still quite useful in the days before hard drives, to be able to have a couple of utilities (like a text editor) loaded without having to keep swapping floppy disks. Also wrote a couple of TSRs of my own as a kid learning to program. Sure you could crash the system, as you could with any program, but they were as reliable as anything else. The only thing special was that you generally didn't want to run two TSRs that naively hooked the same interrupt without chaining properly. But back then we didn't have thousands of mysterious background processes always running in the background. You knew what few programs you had run, so it wasn't really a problem.
deleted 4y ago
[deleted]

jmclnx 4y ago

Very interesting article. My objects were always written for real mode on DOS, by the time I would have wanted to use these extenders, Coherent and then Linux came out.

mrlonglong 4y ago

I loved Watcom C/C++ for what I could do with it. Unreal mode became easy with the appropriate magic.

nazgulsenpai 4y ago

Anyone too young to remember or non-DOS gamers, you can see it in action here on Duke Nukem 3D running in browser on the Internet Archive: https://archive.org/details/DUKE3D_DOS https://archive.org/details/DUKE3D_DOS :)

atan2 4y ago
Thank you for this! It's also super helpful that it runs directly in the web browser. :)

sbf501 4y ago

EDIT: Why do I post sometimes...

nazgulsenpai 4y ago
Changed because parent changed. Don't sweat it, fren. We all have done it!
- deleted 4y ago
  [deleted]
- sbf501 4y ago
  EDIT... :)

starmole 4y ago

Note that extenders did a lot of emulation stuff for you, which was not really needed if all one wanted was flat 32bit mode. We did it for 4k intros back in the day: https://www.pouet.net/prod.php?which=289 https://www.pouet.net/prod.php?which=289. One fun optimization trick was the 0x66 prefix: Switching from 16bit to 32bit mode also switched the trade-off in opcode sizes. So in that intro most of the audio code runs in 16bit mode, while the graphics (which is actually not palette but full 32bit color) runs in 32bit mode.

andrewf 4y ago
I'd be interested in reading the source, but links to the source code seem to be dead (domains bought up by dodgy parking page types)
- starmole 4y ago
  Yeah, I noticed. Tried to look at it myself - I don't have it archived anywhere myself. But last time I looked at it I cringed pretty hard. It's not great code :) But here you go: https://web.archive.org/web/20120710162700/http://www.active-web.cc/html/research/bin/v3src.zip https://web.archive.org/web/20120710162700/http://www.active...
  - andrewf 4y ago
    Neat, thanks!

bayesian_horse 4y ago

I actually remember this shit. Man I'm old.

captainmuon 4y ago

I always wondered why DOS extenders were separate .exes. It looks like you were supposed to be able to replace them, or start them in advance and share one with multiple programs. But that wasn't the case, when you launch it you just got a message to launch the main executable. Why did they bother to make it a proper executable? Wouldn't it make more sense to give it a .sys or .dat extension and to just load it yourself (in DOS a matter of reading it into memory and jumping to that address). Or, since it is proprietary and often delivered with the compiler, just link it in statically? Random anecdote: as a kid, DOS/4GW stood for "DOS for great win(dows)" (which I know makes absolutely no sense).

TonyTrapp 4y ago
You can replace DOS extenders to a certain extent. DPMI is standardized, so if an application is directly developed against the DPMI interface, it doesn't matter which DPMI server is present. I remember that some programs would run without any additional executable on Windows 98 (which provides its own DPMI server) but would require CWSDPMI.EXE when run from DOS. I guess that other programs would directly communicate with DOS/4GW through a different interface and that's when you can't just replace it with a different extender, but there are other extenders that can fully replace DOS/4GW such as DOS/32.
nu11ptr 4y ago
You can replace them. I forget the details on how I did it, but I recall doing this for a popular first person star wars game at the time. I replaced the DPMI runtime with a different one and the level load times skyrocketed in speed. My suspicion at the time was that the runtime included switched to real mode to do disk transfers but the other perhaps talked directly to the IDE hard disk staying in protected mode? Not sure, but it was very, very fast with I/O. I was pretty happy with my hack at the time.
- leeter 4y ago
  My suspicion is you're right. Mostly based on Raymond's blog on the role of DOS in Windows 95[1]. Where people using weird drivers and other things that got in the way of 32bit disk access could massively slow down the system. It's plausible that one extender was being "nice" and using your native drivers or at least trying to be compatible whereas another just went "Nah that crap is slow" and just went for it with a native direct driver and was thus massively faster. [1] https://devblogs.microsoft.com/oldnewthing/20071224-00/?p=24063 https://devblogs.microsoft.com/oldnewthing/20071224-00/?p=24...

rob74 4y ago

So the "4G" stands for 4 gigabytes... that's funny, as at that time (early nineties) I bet no one could even imagine a humble PC having 4 GB of memory. That was the domain of supercomputers. And nowadays, even smartphones have more RAM than that... Also, "Fun fact: The original Wolfenstein 3D engine, created by id Software, was developed using pure real mode". Well, that figures... if you still wanted your code to run on 16 bit CPUs (286 and below), you had to use real mode. That's also why, for several years, only the then-"AAA" games like Doom, Duke Nukem 3D or Tomb Raider used DOS extenders. Titles that were less demanding on the hardware (platformers and other 2D games) kept using real mode for quite some time longer. Actually, I can add a fun fact of my own: when I started in software development in 2000, it was at a company developing Windows applications using Delphi - and at that time they were still keeping up compatibility with Windows 3.1, i.e. compiling in 16 bit mode. They finally switched to 32 bit soon after I joined, and I can't tell you what a relief that was. Although getting rid of the weird hacks in the source code that were made necessary by the constrained memory space of 16 bit applications took some more time...

abcd_f 4y ago

Watcom compiler shipped with DOS/4GW extender and the audacious simplicity of being able to malloc one whole megabyte in one go felt like a real magic. Good times. * Not 100% if it was 1 meg, but 4gw ultimately removed the run-time allocation cap. If you had X megs, you could allocate X megs with one call. That was truly revolutionary.

hyperman1 4y ago

There was always a delay of a few seconds before the main program started. I found out why when I tried to write protected mode code myself: You could only ask 4kb of memory at a time from emm386 (or himem, I forgot). There was no bulk access. So you had to loop trough the complete allocation in steps of 4k at a time. I think emm386 switched to protected mode, changed a page table, then went back to real/v86 mode for each block. When my code was written, it had exactly the same delay as other dos extenders.

simne 4y ago

Actually, Gates really said "640k will be enough..", but as I remember, context was, "to run (current for that year) msoffice comfortable", with typical for that year business tasks :) So I think, it will be better for him to admit this, and give exact source, so will not Streisand effect issues.

willywanker 4y ago
He said this in 1981, there was no Windows let alone MS Office at the time. The average simple text mode DOS application was barely 50-100kb in total, so 640k was quite a comfortable limit for several years more.

tssva 4y ago

> The 286 was a 16-bit processor, which means that the 286 could now address up to 16 MB of RAM! The 80286 was a 16-bit processor but had a 24-bit address bus which is what allowed it to access 16MB of RAM. The 8088/8086 were also 16-bit processors but had a 20-bit address bus which limited them to 1MB. The Z80 and 6502 were 8-bit CPUs with 16-bit address buses which limited them to 64KB.

nicky-g 4y ago

Love pikuma! Great learning resource

int_19h 4y ago

Some late DOS extenders provided Win32 APIs for stuff like the console, threads, memory etc. HX-DOS (https://www.japheth.de/HX.html https://www.japheth.de/HX.html) was famously capable of running Quake 2 on its own.