I recall many of my compsci teachers explaining how it's just not possible to implement "legacy" support, hence why Win9x is so unstable. I was recently running some 16 bit apps (Wine can do it, ROS can't) and thinking on what would allow stable 16-bit and 8-bit support. Here's my thoughts; use them, abuse them, ignore them, comment on them, I don't much care really, this is one hell of a fun mental exercise for me anyway. Comments are appreciated though, since it'd probably help my understanding of the problem, which is nice...Full native API implementation
First off, this whole discussion assumes all APIs get implemented as full native code. This means 32-bit or 64-bit, even for 16-bit DOS and Windows 3.1 calls. All 16-bit calls should get thunked (I seriously hope that's the right term) into 32-bit libraries.Emulator backend
The backend for native Win16 and DOS should be a virtualizer or emulator. For 64-bit hosts you will need to use a real emulator like Qemu or a custom implementation; in 32-bit protected mode you can probably use VM86 mode for real mode applications. Some magic can happen with the GDT and LDT to do 286 protected mode, but again only in 32-bit protected mode; the LDT is no longer valid in 64-bit long mode, and you must fully emulate 16-bit code.
For 286 protected mode, I believe you must have a GDT; you can probably change the GDT when changing to the emulator task. Since the OS has to manage the LDT, the program loader should set that up upon loading the program. Any 16-bit code goes into a segment with the D-bit cleared of course.
In cases where virtualization fails-- for example, a loader needing a memory segment taken by the OS--dirtier tricks should be used. For example, clear the interrupt table and let interrupts cause a fault, such that the emulator itself can pick up and handle the interrupt. This will require tight cooperation with the OS. In the most extreme case, a process may need to fall back completely to emulation.
Any and all activity that can be done outside Ring-0 should occur in Ring-3. This means if you go into full emulator mode, you better be running that outside the kernel; that stuff can crash so easy it's not funny.DOS emulation
DOS emulation should fall to VM86 mode or a fully emulated 8086 mode. DOS emulation is, from my point of view, needlessly complex; however I would like to be able to run Doom (not Doom95) and Duke Nukem; Duke Nukem and especially Shadow Warrior (using the same engine) work on Windows 9x but seriously dislike Windows NT.
When running DOS emulation in VM86 mode, you can simply catch VM86-mode interrupts and pass them to the application itself. When a processor in VM86 mode calls an interrupt, EFLAGS will show 1 for the VM86 bit (, pg. 242), and you can thus pass control down to the application (emulator) to handle this. More recent processors uses a more complex method if VM86 mode extensions are enabled, whereby a 32 byte map tells which interrupts to handle in the old way, and which to handle by using a real-mode IDT to call real-mode interrupt handlers; this is great for running DOS, but not for getting an interrupt to pass control into a protected mode application, and thus isn't useful here (it's useful for DOSEMU).
If emulating rather than using VM86 mode, you can simply handle calling an interrupt by calling the appropriate code directly.
In either case, a full implementation of the DOS ABI needs to exist. This means something needs to handle the interrupts used to make OS calls. Something should optionally emulate EMM386, HIMEM.SYS, SVGA, etc. Something will have to emulate a SB16. It should be enough that a graphical game like Duke Nukem can run inside a DOS box; even more interesting would be running Windows 3.1 or 3.11 inside a DOS box.
What you don't need to emulate is a hard drive. Anything attempting to defrag or partition or whatever should fail; sorry, but there's a sanity limit, and you just hit it. OS calls to the file system will specify concrete goals like "Change to D:" or "cd \games\quake" and should not need to care that this is a compressed folder on NTFS. In fact, you could theoretically allow a "sandbox" mode to move a DOS application's C:\ root into any folder; deny access to other drives; and in general just muck around with how you want the app to interact with your system.Win16
Win16 is not as touchy a discussion as DOS. Win16 wants to run in 386ENH mode and use protected mode to run Win16 applications; seeing as Win16 isn't running, things are easier than that.
Again, this code runs in 16-bit mode, and you need to use either VM86 mode, segmentation with 16-bit segments, or emulation to pull it off. Beyond that, Wine implements the Win16 API, so actually running stuff shouldn't be too difficult. The API calls can directly thunk, rather than play with interrupts; the Win16 API exposes sound, CD-ROM, video, etc.
There's not much I really want to say on Win16 that I didn't say on DOS. You don't need to worry about real mode artifacts like HIMEM.SYS or EMM386; you can probably avoid running into interrupt handlers (Wine somehow runs as a user mode program under Linux and does just fine with Win16); in general it's just a simpler beast. Make of it what you will.References
 Pentium Processor System Architecture By Don Anderson, Tom Shanley, MindShare, Inc. Read on books.google.com
 Wikipedia on Windows 3.0
 Wikipedia on Windows 3.1
 Wikipedia on Virtual 8086 Mode
 Brian's Kernel Development Tutorial: Global Descriptor Table
 How to use the vm86 system call
 Microsoft Help and Support on Real Mode Not Supported By Windows 3.1
 Giese, Chris. Protected Mode
. Bona Fide OS Development.