[ros-dev] FreeWin95

[Alex Ionescu]

Emanuele Aliberti wrote:

> Alex Ionescu wrote:
>
>> I view this as a "misunderstanding with some design choices", if not 
>> even a "misunderstanding of NT Subsystem design".
>
>
> Subsystem design is subtle. Cutler is not an ordinary person. Also the 
> subsystem design was conceived in another era, in CS terms. Cutler and 
> his group designed it when the OS (beginning-to-ruin) myth was CM Mach 
> (at the same time, RMS invented Hurd; recently Apple could sell it 
> rebranded as Mac OS X...).

Mark Lucovsky wrote the subsystem design (smss, csrss), not Cutler, just 
a by the way.

>
>>>
>>> The design choice, in short, is the rôle of the "Client/Server 
>>> Runtime SubSystem" (CSR=CSRSS.EXE+CSRSRV.DLL) in the design of the 
>>> "O.S. Personality" code, that is the emulation code that manages 
>>> processes of the same operating system in user mode, making them 
>>> belive to run on the emulated O.S.
>>
>>
>>
>> The Client/Server Runtime SubSystem, which is unfortunately 
>> incorrectly named by Microsoft, 
>
>
> It is funny to note now that what you say here is a thought that 
> spinned around in my mind for years in the past. My personal answer 
> was exactly yours.
>
>> is actually the Win32 Personality. 
>
>
> Later I discovered a theologian and philosopher - W. Ockam 
> (1295/1300-1349) - that today most of us (don't) know is the father of 
> the KIS(S) principle. Do not invent an hypothesis if it *not needed*. 
> I used to say "Those developers at MS are really eccentric! Why did 
> they call the Win32 subsystem server CSRSS.EXE? They are really dumb!".
>
> The unnecessary assumption is, of course, that they *incorrectly 
> named* it.

(Note for others: The KISS principle is "Keep it simple, stupid" or 
sometimes "Keep it simple and stupid")

If you really want to argue using Ockam's Razor (note for others: 
Ockam's razor basically says that the simplest assumption is usually the 
correct one), then I'm afraid I'm the one winning this argument. My 
assumption is that CSRSS has always been built to service Win32's needs. 
This has much evidence, as you've added and agreed with me, mostly 
importantly:
- The fact that other subsystems have their own *ss.exe
- The fact that CSRSS is specially "polluted" for Win32

Your assumption, as far as I could understand, is that CSR was 
"originally supposed to be somethign else" and was later "polluted". 
Now, what makes more sense and seems the easiest assumption, by using 
Ockam's Razor:
- CSR is what all evidence to it points, a Win32 subsystem runtime 
manager. (Alex)
- CSR is everything but what it currently seems to be. CSR was actually 
designed to be something totally different. It was later polluted and 
became what it is now.(Emanuele)

Ockam would agree with theory #1, and your assumption would become the 
unnecessary one.

>
>> It manages Win32 processes in user-mode, and allows them to access 
>> kernel features which would pollute the native-mode ntdll.dll, would 
>> they have been included there. Only Win32 processes use csrss and 
>> registration is done in kernel32. Console windows, another Win32-ism 
>> also use csrss. 
>
>
> This is what we observe today in a live system of the NT family. But, 
> if you see a man that wears a black suit, are you sure he's a priest? 
> It could be a MiB, or a punk, or simply an old fashioned gentleman.

If he also wears a cross, he could be a punk or priest. Ockam's Razor 
would dictate the simplest assumption to be the true one. Therefore, he 
is a priest.

>
> For instance, everybody here knows that the dynamic library NTDLL.DLL 
> exposes the so called "native API", the NT real API. But even you are 
> diverted by that sweet assonance and commit the mistake of saying it 
> has the native-mode mark. Indeed its subsystem ID is 3.
>
> When I discovered it, I had a shock, a few years ago.
>
> Does it fit in you picture?

(Note to others: 3 is IMAGE_SUBSYSTEM_WINDOWS_CUI)

I've always found it strange that ntdll.dll has the CUI subsystem, and 
not IMAGE_SUBSYSTEM_NATIVE.

Of course, the most plausible reasons for this are that due to some 
kernel/loader semantics, it's simply easier to use this flag for X, Y 
reasons. Even simpler, perhaps it's just the default used for building 
other DLLs. You can note that basesrv.dll and winsrv.dll also have the 
same flag set, even though they are not CUI DLLs. I'm not going to go in 
some theory that "Oh God, oh God, ntdll has the Win32 Console flag set, 
it must be a hidden console DLL!".

>
>> Because NT was designed with a set of universal native APIs, this was 
>> necessary so that routines useful to only one personality wouldn't be 
>> exported as native APIs. For example, fork() is a POSIX-only API, not 
>> Windows 32. Therefore, it was not included in ntdll.dll, even though 
>> the kernel supports it. Instead, it's used by the POSIX subsystem. 
>> The same goes for Console windows and their APIs, which are usable 
>> only in Win32. Access to the video hardware is done in kernel-mode, 
>> but the functions are called through CSRSS.
>>
> A very good example, but a less precise description. NtCreateProcess 
> is an API that is more abstract than Win32 CreateProcess and than 
> POSIX fork/exec. As it is more abstract, the respective less general 
> semantics can be implemented using it. As Casper pointed out, more 
> general, more abstract, means more complex.

Which is why the Native API isn't the default API, because programming 
with it would be much more complicated. Win32 is a good thing in that 
regard.

>
>> Now, CSRSS is in a way a Client/Server Subsystem, because it's merely 
>> a 4KB host to CSR Servers, which implement parts of the Win32 
>> behaviour. The 3 servers currently used are: basesrv.dll, which 
>> handles SxS (Side by Side assemblies, a Windows XP feature), VDM 
>> (16-bit emulation) and stuff like Process/Thread creation, 
>> winsrv.dll, which handles some graphical things and 
>> windowmessage-related functionality (such as hardware event 
>> dispatching) plus all the Console code (winsrv is internally split 
>> into 2 CSR Servers). I think a big problem here is that ea believes 
>> that CSRSS is the place where you would add a POSIX subsystem. This 
>> isn't the case. CSRSS is reserved for Win32 CSR Servers.
>
>
> Of course, not in the instance where basesrv.dll and winsrv.dll are 
> loaded. 

Not in ANY instance!

> I think the csrss.exe + csrsrv.dll (the core) were the original CSR. 

Assumption assumption! csrsrv.dll is merely the real implementation of 
CSRSS, and provides a common API for all the CSR servers (base and 
winsrv). You are *convinced* that CSR+csrsrv.dll were designed to allow 
other subsystems to be implemented as DLLs. No, no, no! It was *never* 
that way. You mention the proof yourself: csrsrv.dll does things like 
setting up the Win32 Base Name Objects directories and other 
win32-specific things!

> Of course the os2ss and psxss MS sold us don't use it.

I don't think they "sell" them; psxss is free, and os2ss was part of 
Windows. And the reason they don't use it is because they're -not 
supposed to-!

> They couldn't, because the current CSR core was intentionally 
> "polluted" to support only basesrv.dll and winsrv.dll. But it is not 
> our fault.

Another assumption. How about assuming that CSR was intentionally 
CREATED for Win32.

>
> Also I dont think the Win32 server should run in more than one instance.

All subsystems run in more then one instance if multiple Session IDs are 
used. This is by the very own design of "Sessions" in NT. Sessions are 
completely seperate entities. They run their own set of programs, have 
their own desktops, windowstations, windows, etc. Even their -memory- is 
seperated. Memory in one session is usually seperated from memory in 
another session. Even the Paged and Nonpaged Pools are seperated! It 
makes perfect sense that even subsystems are seperated. Plus, this is a 
feature that I personally LOVE. It allows me to debug csrss.exe on a 
-live- system, simply by connecting to another session. Without this 
wonderful design, such abilities would be lost. Also, assume csrss.exe 
crashes in one session. The others sessions are still ok. There are a 
myriad of reasons why having an isolated environment for each session is 
a good thing. It's why windows terminal services are probably one of the 
best features of NT.

> When you look at the process list, in a TS machine, those many 
> csrss.exe you see are (but one) μSessions of the Win32 server (the 
> mother is the one with lowest PID). Imagine Linus' face if you propose 
> to load a new instance of the Linux kernel each time you login...

I really wouldn't associate csrss.exe with the Linux kernel. If this was 
a high-school debate I could easily attack you personally because you 
just made that comparison ;)

> well it is exactly what happens when you connect from a TS client 
> (win32k.sys is loaded again). 

Win32k is made to load into every session by DESIGN, and it comes with a 
share of benefits. Did you also know that other drivers can be made 
session-specific as well?

> I don't know how you define that way of programming: perhaps, in a 
> sentence, "Does it compile? Let's sell it!". Well, two.

I think you define it as "secure, isolated, scalable design". Which is 
quite different from Linux's monolithic, unscalable design.

>
>>
>> Now, let's move on...what actually loads CSRSS? This is called SMSS, 
>> the Session Manager SubSystem. The Session Manager handles all the 
>> subsystems that are installed on an NT Machine. By default, this 
>> means CSRSS, the Win32 subsystem. If you install SFU 3.5, or are 
>> using Windows Server 2003 R2, you will also get psxss.exe.
>
>
> The original PSXSS.EXE was, as you know, a nice trick. MS had to show 
> that NT was POSIX 1003.1 compliant, to get DOD contracts.

Possibly true, although it sounds like a conspiracy theory. I'm sure a 
big part of POSIX compliance was also compatibility, and also to prove 
that NT is a solid design. Remember that NT was originally built for 
non-x86 CPUs only because Cutler thought it was the best way to catch 
bugs and make a solid OS. Subsystems were a design choice in NT for a 
long time.

> But, of course, it should do it fast and could not show "too much". 
> This decency shows how they got two prizes with one shot. PSX is not a 
> vertical subsystem, but a side subsystem, or a second-level 
> personality. In fact it requires the Win32 subsystem running to work. 

Ah, finally we get to a point where I'll agree you with you on 
pollution! Yes, POSIX only works with Win32 loaded. But can we easily 
work around this later on, when we have a POSIX subystem? Sure! Does it 
have anything to do with CSR? NO!

> The first prize is that you show NT is a POSIX system. The second 
> prize is that you can not drop Win32 to run your POSIX system. Market 
> and market, not computer science.

Yes, Microsoft was probably afraid of allowing NT to run in POSIX-only 
mode, but we can pull it off much later once that's needed.

>
> This recalls me that recently MS was sentenced, because it is 
> impossible to remove IE from Windows. A recurring trick.

Well, when NT was first publically shipped its job was to run Win32 
applications. Allowing Win32 to be removed might've caused many 
technical/support headaches apart from the marketing stuff. I wouldn't 
jump to conspiracy theories right away. The design of NT is much 
different from the shell/ie.

>
>> On older Windows, you can also have os2ss.exe. These are the POSIX 
>> and OS/2 subsystems.
>
>
> Once again, let's look back. OS/2 was the original user mode API 
> Cutler was sked by MS to make available on top of the NT core. As 
> Windows 3.0 was a worldwide success and the alliance with IBM broke, 
> they had to switch to Win32. As there was a common ancestor, do you 
> think they rewrote the CSR from ground up? It is more likely that they 
> freezed the CSR used by OS/2 and, after a lazy copy & paste, extended 
> it ("polluted it") to build what we today say it should be named 
> w32ss.exe.

I think I see what you mean, yes, CSR back in those days was probably 
the OS/2 subsystem. How is that better though? It still wasn't a GENERIC 
subsystem loader (SMSS is that). They merely took out the OS/2 
"pollution" and added win32 "pollution". Then they added the OS/2 
"pollution" back into what became os2ss.exe. In that sense, CSR was 
always "polluted" by whatever the subsystem it was holding was supposed 
to be. Now it's a win32 subsystem, and has been that way for 15 years.

>
>> They have nothing to do with CSR at all, and they are actually, in 
>> themselves, CSRSSes for their own subsystems.
>
>
> This is by design, an evil design... ;) Would have MS shown how easy 
> is writing your own subsystem in a moment when they had to face IBM 
> OS/2 rival API? My friends, close every gate, every window, every hole 
> and harmour the Win32 server!

Slippery slope fallacy/conspiracy theory. It was simply the design that 
every subsystem woudl have its own "CSR EXE" followed by server DLLs.

>
>> This is the naming mistake I was talking about; Microsoft should have 
>> named csrss as w32ss.exe.
>
>
> Ockam! ;)

Not really, if viewed from the historical perspective, it makes sense 
why it was never renamed as such. Assuming that CSR was something else 
then the original OS/2 subsystem for a very short while is much more flawed.

>
>>
>> So, SMSS is the first process that will load, and it will then load 
>> the subsystems, say csrss psxss and os2ss. These subsystems will then 
>> load server dlls which expose functionality that can be called from 
>> user-mode processes.
>>
> SM is a super server. But then, if it is so super, why does it humble 
> itself in defining DOS devices

This was moved to CSRSS.

> , in loading a process of one of its controlled children (I refer to 
> WINLOGON.EXE)... If that is not "pollution", maybe a nuclear bomb 
> exploded somewhere in the vicinity. :(

And this is the second point I agree with you, the second design mistake 
of NT. Smss's "initial command" is to run winlogon, and that really 
creates a problem where win32 becomes -required-. However, instead of 
thinking this was some marketing ploy, I think that NT was getting close 
to shipping, and doing it this way was a much easier solution for the 
prime goal: Win32. So yes, it's a bad design decision:

BTW: It has a "bug", in that if you define an "inital debugger", the 
initial command will be replaced by "initial debugger.exe winlogon", so 
that winlogon can be launched with a debugger. Stick your 
native/posix/whatever program as the initial debugger, and ignore the 
winlogon command line, and you've beaten the flaw ;) The string is 
configurable in the registry, IIRC. Once again, this is something we can 
do better in ReactOS after the windows stuff works normally.

>
>> As I'm writing this, my Win2K3 machine is running posix grep and 
>> working perfectly. It's not touchign CSRSS at all, nor should it have 
>> to. CSRSS is a win32 subsystem.
>>
> CSRSS.EXE+CSRSRV.DLL, as they load BASESRV.DLL and WINSRV.DLL, *are* 
> the Win32 personality subsystem server.

That's what I meant.

>
>>>
>>> Think about it like the program run by the Architect in Matrix: when 
>>> you double click on Microsoft Word's icon and the process is 
>>> created, it happily belives to run under Windows, it can see the 
>>> familiar libraries, it can call CreateFile and get back a reasonable 
>>> file. In fact, CreateFile calls NtCreateFile and so on. Microsoft 
>>> Word lives in a fake World. ;)
>>>
>>> The two opposite directions the CSR can turn for are:
>>> - be a process totally bound to the Windows personality
>>
>>
>>
>> Which it's -supposed- to be. CSR is W32SS.
>
>
> If we clone Windows NT, this is correct. 

ReactOS is a Windows NT clone.

> But dumping is illegal.
>
>>
>>> - be a general purpose process, a facility for any kind of personality
>>
>>
>>
>> Other personalities don't need CSR! They have PSX and OS2 and VMS and 
>> etc...
>
>
> The pieces of code MS sold us match this definition. I ask myself: is 
> this a technical limitation, or imposed by market staff?

How about a design choice? One which makes a lot of sense to me.

>
> As Gunnar saw, probably there is a common set of facilities you need 
> every time you write a personality server. I would like to write it 
> once for them all (If I could, but you showed you are much more 
> skilled than me!). 

The common set of facilities are exposed by SMSS callbacks and a smlib! 
I've written about that in a previous email.

> Casper, as already noted, pointed out it is a complex piece of code to 
> write. You are required to write the equivalent of the abstract 
> NtCreateProcess, to be called by the Windows personality server and by 
> the POSIX personality server and by the OS/2 server and by the...

One of these abstract callbacks is SbCreateProcess. I plan to implement 
the Sb callbacks and SMSS later this year. (Nice bit of trivia: 
SbApiPort and SmApiPort are quite different! One relates to SMSS (Sm) 
itself, the other handles the common subsystem (Sb) functionality which 
you seem to want).

>
> My question is: is it worth writing such a live run-time code? (DLLs 
> are "dead code", because they live un the context of the process that 
> loads them; vice versa, a subsystem is a "live DLL", because it lives 
> before the process that will use it; if you like biology, there is the 
> same difference between a virus and a bacterium).
>
>>
>>>
>>> More on demand.
>>>
>> Another issue which ea has brought up is that NT is somehow to 
>> "connected" to Win32.
>
>
> Well, Alex, I repeatedly used the adjective "polluted", which does not 
> mean "connected". I meant they put helper pieces of code, here and 
> there, to make the Win32 server's life easier. If you want a more 
> technical equivalent of "polluted", read it "less general, less 
> abstract". Casper is definitely right!

Polluted, connected, I still disagree. There is almost nothing in NT 
which makes Win32's life easier, apart from some win32k.sys stuff. Yes, 
there are certaintly some things which help out win32, but that's 
because NT is PRIMARLY DESIGNED to be a Win32 OS. By definition, so must 
ReactOS, at least until 1.0 when our goal, to clone Windows NT, is 
reached. After that, we are free to explore other areas, if 
compatibility can remain intact.

>
>> I beg to differ. I've spent a week working on a sample native 
>> application. It currently can display running processes, drivers, 
>> create a new process and create/delete/list files. Here are some 
>> screenshots:
>>
>> http://members.tripod.com/WinAlOS/rtlcli.png
>> http://members.tripod.com/WinAlOS/rtlcli2.png
>> http://members.tripod.com/WinAlOS/rtlcli3.png
>>
>> Note that there is no CSR, no win32k.sys loaded. NT is running in its 
>> rawest form. You need less then 16MB of memory (notice that the 
>> processes are using <1MB, the rest is up to the drivers, which you 
>> can disable) and about the same amount of diskspace. This works 
>> perfectly in Windows 2003 without any need to "separate win32".
>
>
> I actually didn't understand if Russinovich was paid by MS to show that, 

I feel a bit insulted. What does Russniovich have to do with anything? 
The screenshots above show a program which I alone have worked on, and 
that no other "NT hacker" has attempted in doing before.

> or if he actually could get there alone, the real NT hacker! Because, 
> if you look at what that really means, you get a strange feeling.

I have no idea what you're talking about. There are many more 
knowledgable people in NT then Mark. They just don't get the media/MS 
attention he does. If he's the "real" NT hacker then what are Skwing, 
Filip, KJK::Hyperion, yourself and I?

>
> First, the rumors that pollution (Win32/Win64 infiltration) is gone to 
> a point where the old NT-designed core is less stable, are immediatley 
> silenced. Russinovich, indipendently (and with the naive "Oooh" by 
> some MS engineers...), showed it!

Actually, I think I just did by writing that program. I once again don't 
see what the -hell- Russinovich has to do with me. And no, MS didn't pay 
me to write that program.

>
> Second, in a hidden and really smart way, you promote the equivalence 
> CSR=Windows: "Note that there is no CSR, no win32k.sys loaded.". 

Yeah, CSR+Win32k.sys == Windows.

> Bill is a genius! I love him.

Bill is an idiot, but a very good businessman. Cutler and Mark are 
geniuses, and if it wasn't for, especially, Cutler's angry attitude and 
specific design goals for NT, win32 would probably be in ntoskrnl.exe

>
> With best regards,
>
Best regards,
Alex Ionescu