ReactOS Community > ReactOS Blogs

My new blog and website are available here:
Alex Ionescu

Comments (0) | Trackbacks (0)

UPDATE4: Got bored of the MSVC fixes. Only lib and ntoskrnl are left. Going back to kernel work now!

UPDATE3: Due to the heavy kernel patches I've made lately I am taking a month off from future commits and will be working to find and fix possible regressions, as well as to get ReactOS building with MSVC.

UPDATE2: Added some more work, changed some dates around.

UPDATE: Been going through a real life crisis. Dates modified accordingly.

Here's an updated TODO list in the near-term for me (yeah, I know, I change stuff around really often):

• Finish User-Mode Networking for ReactOS. Deadline: Dec 4th.
  
  
  
  • Complete!
  
  
• Make new Process Manager patch work. Deadline: Dec 6th. 
  
  
  
  
  • Complete!
  
  
• Make new ntgdi.h patch work. Deadline: Dec 8th.
  
  
  
  • Complete!
  
  
• Implement new Freeldr GUI changes. Deadline: Dec 21st.
  
  
  
  • Complete!
  
  
• Finish and test Pushlock Implementation. Deadline: Dec 25th. 
  
  
  
  • Complete!
  
  
  
• Commit Push Lock Implementation. Deadline: Dec 27th.
  
  
  
  • Complete!
  
  
• Cleanup Rundown Implementation. Deadline: Dec 27th.
  
  
  
  • Complete!
  
  
• Cleanup atom.c, event.c, error.c, evtpair.c, lookas.c, mutant.c, profile.c, sem.c, time.c, zone.c, win32k.c Deadline: Dec 28th.
  
  
  
  • Complete!
  
  
  
• Optimize regular Spinlocks. Deadline: Dec 29th.
  
  
  
  • Complete!
  
  
• Cleanup interlck.c, list.c. Deadline: Dec 29th.
  
  
  
  • Complete!
  
  
• Implement advanced features for worker threads (work.c). Deadline: Dec 30th.
  
  
  
  • Complete!
  
  
• Rewrite most of resource.c. Deadline: Dec 30th.
  
  
  
  
  • Complete!
  
  
• Fix some semantic bugs in the wait code, and heavily optimize it: Deadline: Jan 4th.
  
  
  
  • Complete!
  
  
• Commit and test resource.c. Deadline: Jan 5th
  
  
  
  • Complete!
  
  
• Implement Queued Spinlocks. Deadline: Jan 15th.
  
  
  
  
  • Complete!
  
  
• Fix resource and wait regressions that the previous patches caused. Deadline: ASAP.
  
  
  
  • Complete!

• Re-implement most of the timer code (in ke) to take advantage of timer tables and per-timer table locks. Deadline: Jan 17th.
  
• Implement MmGrowKernelStack and PsConvertGuiToThread (Converstion of CLI to GUI threads and stack creation (60KB) and expansion support). Deadline: Jan 16th.
  
  
  
  
  • Complete!
  
  
• Finally re-implement my User-Mode callback code. Deadline: Jan 18th.
  
  
  
  • Complete!
  
  
• Fixup the TrapFrame<->Context Functions to support external ContextFlags, and add stubs for FPU register conversion. Deadline: Jan 20th.
  
  
  
  • Complete!
  
  
• Detect and disable cmpxchg8 on 486. Deadline: Jan 15th





• Half-way there!


• Fix syscall.S. Deadline: Jan 19th.




• Complete!


• Finish, commit and test the new Loader. Deadline: Jan 20th.
  
• Commit and test the User-Mode Networking branch, with appropriate afd.sys changes. Deadline: Jan 23th.
  
• Rewrite FPU support. Deadline: Jan 24th.





• Half-way there!


• Refactor low-level I/O code and finish old changes. Deadline: Jan 25th. 
  
• Refactor exceptions/traps. Deadline: Jan 28th. 




• Complete!


• Use macros for syscall.S. Deadline: Jan 29th. 




• Complete!


• Refactor bootup sequence. Deadline: Feb 11th.
  
  
• Refactor afd.sys. Deadline: Feb 28th. 
  
• Refactor Ob Parsing (again). Deadline: March 10th. 
  
• Implement new SMSS and CSR DLLs. Deadline: May 1st.

This is my -entire- work on ReactOS counting in ALL issues which currently nag me and I've wanted to fix forever. Beyond this list there is nothing that I can currently think of, except some private projects that I will think about later.

Comments (2) | Trackbacks (0)

Apologies for not having continued the article below; while installing my new box, I accidentally formatted, deleted, re-paritionned and re-imaged into a new RAID array all 8000 lines of code I had written, then I over-wrote the clusters on the disk when I reinstalled Windows. So it took me another week to re-write everything and begin testing. Until two days, ago, almost every application except Firefox, Thunderbird and IE was working fine. Today, thanks to a tool which I SERIOUSLY recommend, called ApiMonitor, I was able to see the differences between what my DLL was doing, and what the official one was. I saw that I was succesfully completing connect(), while the windows dll was saying "WSAEWOULDBLOCK". Additonnally, the handles with my winsock were a bit lower, as if two handles were missing. ProcessXP showed these to bee the AFD AsyncConnectHelper handles, which meant that the real DLL was doing asyncronous connects, while mine wasn't. After some digging, I found out that:

1) I wasn't properly setting the overlapped flag (which determines if operations will be async or not)

2) Even after setting the flag correctly, mswsock itself has to be notified that you'll be doing async stuff (by using WSAIoCtl/ioctlsocket). Turns out that I hadn't modified this function during my rewrite, and whoever wrote it originally was overwriting the value.

3) Even after fixing those two things, turns out that the socket() implementation was also somethign I had missed. Before calling WSASocket, it's supposed to set WSA_FLAG_OVERLAPPED by default, which it currently didn't.

After fixing those three things, Firefox was behaving exactly the same, API-wise, when starting up (I had set it to start to a blank page). I tried yahoo.com, and everything loaded perfectly. Downloading works, everything. And this of course fixed Thunderbird too.

IE is using the new "getnameinfo" and "getaddrinfo" APIs which were added in XP. They took almost a whole day to implement, but thankfully the source-code is already available in a header (although I had to heavily modify it, which is why it took so long). I'm now going to try IE, but I feel it should work too.

Gotta love it when hard work pays off!

Oh yeah, here's what works:

- Firefox

- Thunderbird

- Filezilla, ftp

- MIRC

- SVN, CVS

Those apps use almost 85% of ws2_32's apis, so I don't know what else to throw at it. Any suggestions? (Except for IE, which I'm going to test now)

Comments (0) | Trackbacks (0)

Ever wondered what makes WS2_32 tick? This technical spec will detail the inner workings of the ReactOS Winsock DLL (ws2_32.dll), which is based on the joint Intel-Microsoft implementation internals, as well as the official samples for Layered Providers (which include the DPROCESS, DTHREAD, etc C++ classes which I've duplicated in C with different names). So without further ado, here is a brief introduction.

1. The ws2_32.dll “kernel”

Before even getting into the details of Winsock API calls, we'll familiarize ourselves with the basic “objects” (implemented as types) and internal APIs which manage them. These objects (such as the Process, Thread and Socket objects) allow us to manipulate some OS internals with relative portability, provide an internal reference counting mechanism, as well as automatic allocation/deallocation of objects. In short, it works much like the NT Kernel, and wraps common operations under certain objects. The basic ws2_32.dll (henceforth “Winsock”) objects are:

• WSPROCESS (The process object)


• WSTHREAD (The thread object)


• WSSOCKET (The socket object)

Apart from the core objects above, which will go in detail later, there are also several objects for managing the Layered Providers (which have entries in the registry). The registry data for layered providers (LSPs) is called a “catalog”, of which two are present: the namespace catalog (for NSPs) and the protocol (or transport) catalog (for TSPs). Internally, these are exposed under the following objects:

• NSCATALOG (The namespace catalog object)


• NSCATALOG_ENTRY (The namespace catalog entry object)


• TPCATALOG (The transport protocol catalog object)


• TPCATALOG_ENTRY (The transport protocol catalog entry object)

Each catalog entry defines a provider, both of which are exposed as:

• NSPROVIDER (The namespace provider object)


• TPPROVIDER (The transport protocol provider object)

Due to the way that namespace Winsock APIs are handled however, it becomes advantageous to wrap the namespace providers into two more objects:

• NSQUERY (The namespace query object)


• NSQUERYPROVIDER (The namespace query provider object).

Because most namespace APIs are done as “queries” mapped by a handle and being served by a provider, the NSQUERY object allows us to contain that information into a “query” object and perform various accounting on it. Additionally, as each query call can have per-context data connected to a certain namespace provider, as well as to simplify the internal API, the NSQUERYPROVIDER contains a shrinkwrap between NSQUERYies and NSPROVIDERs, containing data such as the current query handle.

Finally, a more of a pseudo-object, WSASYNCBLOCK contains the data for the internal APIs which managae WSAAsyncGetXbyY APIs.

To handle these objects, the following internal API prefixes exist:

• WsProc*


• WsThread*


• WsSock*


• WsNc*


• WsTc*


• WsNp*



• WsTp*


• WsNcEntry*


• WsTcEntry*


• WsNq*


• WsNqProv*


• WsAsync*

Note that the names for the objects are not yet final, and might receive some underscores here and there. In the next post, I'll describe the structure of the core 3 objects as well as document their internal APIs.

Comments (0) | Trackbacks (0)

I apologize for the recent lack of updates... life has been quite a b*tch.

I suppose I should start by saying where I've been in my schedule. Well, over a month late! But not because I've slacked off, but because I've worked on unplanned things.

First of all, it turns out that the NDK stuff had a lot of small issues which still required my attention, including DDK and PSDK compatibility. That took me about a week, but I got most of the ReactOS stuff to build with MSVC + NDK + DDK/PSDK/IFS/WDK. It's a long term goal of mine to setup a super-duper build beast on msvc + incredibuild, and this was a first step that a lot of people needed to get the MSVC backend for our build system running.

Then of course, as I browsed through the source, I found more stuff that was annoying me. It turns out we were still duplicating almost 30 files in ntdll/rtl and ntoskrnl/rtl. I spent a great deal of time sharing everying in lib\rtl, and ntdll became almost only 10 files. But then, while looking at ntdll, I noticed two more things, which became three:

1) Our NTDLL callback from kernel-mode (KiUserApcDispatcher, LdrInitializeThunk, KiUserExceptionDispatcher) were written in C and buggy (they even had the wrong prototype and were using the wrong number of parameters!)
2) Our CSR APIs were completely incompatible and/or unexistent.
3) Turns out our whole exception handling code was buggy and wrong, with ugly hacks in many places (one example is that RtlUnwind was basically “optimized” not to actually care for the 3 optional parameters which some apps might send).

I started with #2, because it wasn't extremly hard. I implemented the entire range of CsrCapture* APIs and made CsrClientCallServer and CsrConnectToServer binary compatible with Windows XP, then I modified kernel32 to properly use the APIs when initializing and also for some console applications. I added a regression but fixed it a week later.

I then attacked issue #1 and #3 and rewrote in almost entirety the exposed exception handling and dispatching interface, fixing bugs, as well as critical APIs such as NtContinue. I also optimized the system call handler again (for sysenter/int2e), only to discover that we were not doing V86 Trap Frame Bias. I spend almost a week working out all the issues, and we -still- can't use 100% of my optimize code because we seem to have a bug regarding segment registers (which I think NT fixes by using “lazy loading“). As you can gather, this was a pretty large patch and touched a great deal of kernel internals. I wanted to go further and start touching the IRQ/TRAP handlers and low-level CPU exception code (which I'm also unhappy with), but I decided to keep that for later. Oh, and finally, I rewrote all the PCONTEXT<->PKTRAP_FRAME APIs to add sanization, proper conversion in some special cases, some assumptions and support for edited frames. I also added support for PKEXCEPTION_FRAMEs which will be needed on PPC and other RISC (and even IA-64) CPUs. There's probably a couple other things I fixed too, but there are so many that I've forgotten. Oh yes, I added SEH to all the ntdll callback handlers, so that if they crash in user-mode, they'll cleanly return to kernel-mode instead of just dying.

Emanuele Aliberti had comitted a new skelon for CSR, which was finally going to work right and use csrss/csrsrv.dll + basesrv.dll and winsrv.dll. I was really afraid for the design to become as messed up as the current CSR is, so I took the selfish (although also quite beneficary) step of writing the whole thing by myself, documenting all its interfaces, and verifying for 100% binary compatibility on NT. The end result (around 6000 lines) is something I'm really proud of. A completely open-sourced and heavily documented csrsrv.dll clone. That took me around 2 weeks, and I've added smss to my futture TODO list to match this new design.

If writing csrsrv from scratch and doing all those kernel fixes and implementing CSR apis properly in ntdll wasn't enough, I also became unhappy with another large part of duplicated code between ntdll/ntoskrnl: the Dbg* APIs like DbgPrint(Ex), vDbgPrintWithPrefix, DbgQueryFilterState, DbgPrompt, etc. Our KiDebugRoutine was also a big hack, so I decided to share the code between the two like it should be, and add it to the RTL Library (theoretically, Dbg is part of RTL). I also informed myself on the real way that Debug Output is done (basically piggybacking on an INT3 exception), but I decided not to implement that yet, since this patch was large enough as it was and required another batch of kernel changes. It still had one bug however, so for now the kernel doesn't use KiDebugRoutine, since it seems to make it freeze for some reason.

You think that's all? No way... on my way to ASM-izing the LdrInitializeThunk, I started feeling hate for the Ldr API. So I rewrote all of it using the hash tables that are used on NT, as well as implementing all the new Win2K3 functionality, such as SxS, Activation Contexts, Fiber Local Storage, Thread Local Storage, Redirection, .LOCAL support, etc. I haven't committed it yet though because it still needs some testing and 3 functions are missing, but it was good to do some PE programming again.

After that, I took a look at the Kernel's Process Management functions again (Ps), which I wrote most of, and noticed how broken they were and how many things I had missed, mostly dereferencing things, or checking for flags. I was also not using the new Win2003 Internal APIs which help a lot with enumeration, so I added all that, as well as fixed at least 10 issues with locks and races, stuff that wasn't getting cleaned, and more. Then I did a round of optimization and added some nice tweaks (especially for the process startup and termination APCs.). I also haven't committed that yet, because it needs testing and is missing a couple of new functions as well.

That was now 2 95% complete patches, so I decided to work on something that would be 100%. I had implemented Guarded Mutex and Kernel Gates some time ago, but the code wasn't working well. One of the reasons was that there were two subtle bugs, which Magey helped me find. Once that was done however, I realized that the entire Kernel APC code was messed up and not respecting the new Win2003 semantics related to Special Apc Disable and guarded regions, so I improved and rewrote much of my old code, also making it much much cleaner. I committed the patch, and also fixed a regression, and now APCs work -perfectly- and as up-to-date as possible (except for Vista's new APC Rate Limiting stuff, I'll get to that in another time).

So finally, after all this was done, I decided to dump my Ob/Cm patch, since too many things had changed, and it was really eating away at my soul. I put winsock back on my priority, and I've been rewriting some parts of my 6 month old code for the last 2 days. I'll go in detail of the WS 2.2 architecture tomorrow and how it all works internally inside ws2_32.dll.

Thanks for your time!

Comments (0) | Trackbacks (0)

I'm proud to announce that RC2 of 0.2.7 will be released imminently, with fixes for all known regessions and blocker bugs that affected RC1 and previous. If no new issues will be found by testers with RC2, then it will probably become the final release build. It is also possible that we merge some additional bug fixes and create an RC3, but that is up to the release coordinator and the testing coordinator to determine.

As for myself, I've been very busy on the Object Manager patch. Unhappy with the current state it was in, I scratched most of it in the trash can and restarted it. The end result was that all the hacks from the new code were removed, and the old Registry/File code which is not yet adapted was added to a deprecated file. Then, a second patch on top of this completely refactors the registry object code to make it compatible with the new patch. Finally, a third patch on top of this optimizes and cleans up the said registry object code to yield a much smaller, cleaner Cm->Ob interface. The reason this is released as three patches is becaused all three are highly critical and could cause regressions. Also, all three together would easily cover almost 10 000 lines of modified source code. By themselves, they should be much more manageable and easy to test. I have just completed the second patch today, and it is due to be tested tomorrow, while I finish the third patch (of which I have started half). This change won't be anywhere near 0.2.7, but it will be a great push towards higher Object Manager NT compatibility. It has been on my mind for over 3 months now. Since the code is complete and working, I can now easily estimate completion for Monday.

As per my schedule, the next step will be to commit the Winsock Rewrite branch, which I will be working on for the whole week. With any luck, it should be regression-less and provide its new features. At this stage I am currently two weeks behind schedule, but I was smart enough to give myself a buffer in August in case this happened. As soon as the long-awaited branch is committed and proven to work, I'll jump back in the Object Manager to improve the directory object implementation, make it compatible, and finally fix the object header structure. This however, will most probably require the I/O File creation code to be modified to use the new parse routine, which will be an extremly ardous task. I do not expect completion of this work until September.

Then, of course, full into I/O mode, I'll do those I/O fixes and improvements that I talked about. Pushlocks and the new scheduler will be delayed beyond that, since they are not so critical yet. I am also working on implementing a compatible NT Heap Manager (ours is based on a slow, uncompatible and feature-less win95 reverse engineered implementation) with Thomas, but only in my spare time when other work bores me.

With any luck however, the Winsock stuff might only take two or three days, if I work really hard on it, which would ultimately shave a whole week off. I really want the I/O fixes done by september as well.

See you soon at 0.2.7!

Comments (0) | Trackbacks (0)

As I write this, ROSRTL has finally been removed, and the Headers are done... there's probably one or two more files to work, and some fixmes, but nothign too drastic... the design is pretty much set in stone right now, except for the win32k headers which Filip has to look at. So that's two things down the list. Next up are my Object Manager patches... It'll probably take me 3-5 days to commit them sucessfuly, since I've been away from kernel code for so long that I need a little refresher on what I was trying to do and all. Yes, you've probably noticed the delay...this is because of the 0.2.7 release which contained major blocker bugs that Waxdragon, BrandonTurner and I have been hunting down (with some help from Filip and Hartmut, and Herve). 50% of them are fixed, except some input bugs that we really need our input developer's “input” on...but he hasnt' been alive in ages, so we'll see how that goes.

Wish me luck for that Object Manager patch

Comments (0) | Trackbacks (0)

I just finished comitting a 2000 line patch which merely properly renamed the fields in the RTL_USER_PROCESS_INFORMATION structure... everything seemed to work like before, except any application AFTER explorer loaded would crash on startup with a user-mode exception. No stack trace, no explenation, nothing. So how do you find the typo you made in 2000 renamings?

First, I had someone set up KDBG to break on user-mode exceptions. This gave me the piece of the code that was crashing. It was the startup thunk inside NTDLL which was reading Peb->ProcessParameters->ImageFileName. Turns out the pointer of the unicode string buffer was “0x409” instead of “0x40409”. Process parameters have this great thing where they can be “normalized” or “Denormalized”, which means if the pointers are actually offests or not. It so happened that for any process launched by the compand prompt or explorer (but not during system startup), NTDLL was not able to Normalize the process parameters, because the flag was already set! This made no sense to me at all...how could renaming structures cause the flag to be set? I started looking at every call to normalize-denormalize, but nothing out of the ordinary was there.

So I reverted before the patch, and added a Debug Print in the kernel-mode APC which executes the user-mode APC thunk, printing out the current flag, and the actual pointer. For the first thread in every process, the flag was off, and the value was an offset. Then every other thread in the same process had the flag on, and a valid pointer. I then put back my patch. Oops, when launching our programs, the flag was different from the what it should be. So somewhere, the flag was set incorrectly...but how? We only renamed stuff! So I went inside the Rtl function which sets the flag, and I made it add an additional meaningless flag, as a sort of “magic value” that we would read during the APC. Sure enough, the system startup services had the magic value, but everything after that didn't. So this wasn't the place which was setting the flag...

I then ran a “grep” on the kernel32 library for “->Flags”. And then I saw it. One clear line. Params->Flags = lpStartupInfo->dwFlags. Wait a second...those win32 flags are for windows, not for the parameters! That line should say ->WindowFlags!!! And then it struck me... that was the needle in the haystack. The one place where I had incorrectly renamed a field, and was now overwritng it with a value from win32 kernel32, which happened to be 0 instead of 1 like it should be. And there you have it folks..perserverence and original ideas will get you to your goal. And asking other people too, because after 6 hours trying to find a bug, your judgement might get clouded.

Comments (0) | Trackbacks (0)

I've seen that a lot of users read my blog and I've heard... disturbing... complaints that it's too technical.

I can't think of anything non-technical right now (I've been hacking away at kernel32 for the last 5 days), but I guess I could update my summer timeline in a way that is meaningful to non-techies. 'Lo and behold, Alex's 2005 Summer Timeline.

Alex's Work To Be Completed

Target: September 2005

• 
  
  Convert ReactOS to NDK:




• 
  
  Remove old ReactOS Headers and use NDK and PSDK/DDK/IFS.


• 
  
  Clean the /include directory, delete duplicates, and merge with PSDK.



• 
  
  Clean the ReactOS DDK directory so that no files that don't need to be there remain.

What this means for you:

1. If you compile ReactOS from source, compilation will be performed slightly faster and depedency checking will be better handled.


2. Also for people which compile from source, you will eventually be able to compile ReactOS with the Microsoft PSDK, DDK and IFS Kits.


3. If you are a user, you can expect enhanced compatibility, as many of our incorrect structures have been fixed with the correct, updated ones.

• 
  
  Remove ROSRTL (The ReactOS RTL Library):





• 
  
  Re-implement the Native Loader not to use ROSRTL and to take advantage of Kernel Stack Freeing. Implement new flags and fix context.


• 
  
  Re-implement the Kernel32 code for Creating Processes/Threads not to use ROSRTL and greatly cleanup and modularize the code, as well as fix and support more command-line parsing issues. Implement new XP features in Process and Thread Creation.


• 
  
  Re-implement the Kernel32 code for Creating/Switching to Fibers not to use ROSRTL, and to support new XP and 2003 features, such as FPU State Saving, Guaranteed Stack Bytes and SxS Assemblies/Side-By-Side executables.


• 
  
  Fix all code using the other ROSRTL functions/headers to use the appropriate APIs or macros instead.

What this means for you:

1. Enhanced compatibility with weird command-line settings and exotic folder/filename combinations (ie: c:\haha.exe\ha ha ha.bat\me.cmd /foo:“hi“)



2. Notification of applications using non-implemented CreateProcess flags, leading to better compatibility and bug reports.


3. Increased compatibility with newer applications making user of Fibers, such as the MS HTML Engine, and Database Applications.


4. An existing platform to build upon a compatible SxS implementation for ReactOS, which will lead to the possibiliy of using the new Common Control 6.0 classes for XP as well as Themes.


5. Enhanced Security for Thread/Process creation, and a stubbed interface for new security functionaty for processes added in XP.


6. A stubbed interface which will allow Application Compatibility support to be implemented in ReactOS.

• 
  
  Fix some outstanding issues in the Object Manager




• 
  
  Fix Object Callbacks



• 
  
  Fix the Parse Routine and code using it incorrectly


• 
  
  Remove usage of the Create Routine and code using it instead of the proper Parse Routine


• 
  
  Make Object Directory structures and implementation compatible with Windows 2003.


• 
  
  Fix the Object Header structure.

What this means for you:

1. Noticeable speed increase in bootup, and heavy registry applications.



2. Increased compatibility with NT Internals that might be exploited by drivers.


3. A more robust platform for implementing more advanced features into ReactOS, like Sessions.

• 
  
  User-Mode Networking Rewrite




• 
  
  Re-implement large parts of ws2_32.dll to properly support Layered Service Providers installed through the registry.


• 
  
  Implement NSP (Namespace Provider) support in mswsock.dll to handle DNS lookups.


• 
  
  Fix bugs in mswsock.dll and take advantage of more features (ws2help and wshtcpip)



• 
  
  Implement wshtcpip's functions completely.


• 
  
  Implement ws2help's functions completely.

What this means for you:

1. More compatibility with networking apps that use LSPs, such as Firewalls, Proxies, Anti-Spam Software, Filtering Software (and sadly, Spyware).


2. Richer and stabler networking code, capable of supporting more platforms once they become available.


3. Richer and more flexible Namespace lookups, capable of supporting 3rd-party NSP such as Netware Directory Lookups, LDAP, etc.

• TENTATIVE: New Scheduler





• Fix system initiazation code in regards to the Process Manager Startup, and the Idle Thread/Process and First Thread/Process


• Use a new scheduler based on the Windows XP one, supporting pre-emption, quantum, starvation, priority boosting and all its Thread States.


• Fix all low-level kernel functions dealing with priorities and other scheduling/dispatching semantics.

What this means for you:

1. Faster and more responsive system.


2. Richer and more mature kernel platform, more scalable and performing.


3. More compatibility with Drivers and applications heavily timing-based that depend on correct implementation of low-level kernel APIs.

This looks like a long list, but most of the work has already been done. The NDK is complete and only 3 more headers remain to be deprecated. ROSRTL has entirely been removed except for the the parts touching kernel32/rtl, but the code described above has been written and is currently undergoing testing, so the NDK/ROSRTL goal should be done in less then a week from today. The Object Manager changes have already been done and I'm sitting on a patch to implement them. I know the patch breaks installation, but apart from that everything works, so it should be a simple fix. The User-Mode Networking rewrite is complete and was undergoing testing until the NDK stuff took priority. Most of the APIs have been tested and work, and some Namespace bugs remain will be fixed. The thread scheduler code has also already been written, but 6 months ago and on branch that is no longer valid because a lot of the changes have been manually merged by me to decrease the number of core changes. The code simply needs to be split up and heavily tested, which is why it's possible that it might not make it before the end of summer. The rest will though, I promise. If all goes well, ROSRTL/NDK will be done by next Monday, Object Manager the Monday after, one more Monday until winsock, and that leaves August for any unforseen work, and for attempting to finish the scheduler work.

However, before attempting the Scheduler, I will probably complete the implementation of Pushlocks ( I shoud've made an entry for this above. What it means for you: Much faster system and responsiveness in some cases, up to 30% speed improvement in stuff like handle lookups) as well as fix Guarded Mutex and make the kernel use it instead of fast mutex (What this means for you: Same as pushlocks, using these objects will increase system performance and responsiveness). There are also some fixes in the I/O Manager that I'll need to complete once the object manager fixes are in (What this means for you: better file I/O as well as enhanced driver support).

What's beyond that? Well, my queue of work plans a total refactoring of our LPC code with Emanuele Aliberti, since our code is full of race conditions and is totally incompatible with NT. I will also take visit to the Object Manager again to fix some nagging issues. Also, there are some ideas about FreeLoader's transofrmation into a different beast, but those block on the Xen port of ReactOS. Beyond that, there are few things in ReactOS which will need fixing, from my point of view and knowledge with the subsystems I work on. My focus will be on new implementations.

I hope that wasn't too technical and that it means something to you users! Thanks for being so supportive!

Comments (0) | Trackbacks (0)

INITIAL_TEB seems to be one of the most secretive, weird, subtle and least-understood native structure of NT, by the looks of it.

It first started its life under the pseudonym USER_STACK, and was defined as follows:

typedef struct _USER_STACK
{

	PVOID  FixedStackBase;

	PVOID  FixedStackLimit;

	PVOID  ExpandableStackBase;

	PVOID  ExpandableStackLimit;

	PVOID  ExpandableStackBottom;

} USER_STACK, *PUSER_STACK;

So far so good, until some new undocumented header came along and renamed it to INITIAL_TEB, where it now looked like this:

typedef struct _INITIAL_TEB
{

    ULONG Unknown_1;

    ULONG Unknown_2;

    PVOID StackTop;

    PVOID StackBase;

    PVOID Unknown_3;

} INITIAL_TEB, *PINITIAL_TEB;

Whoops, there goes some data! And notice how the ExpandableStackLimit now became the base, while the Base became the Top. And how did they know/guess the first two fields were ULONG? Finally, the structure was updated (by none other then ReactOS, apparently) to:

typedef struct _INITIAL_TEB
{

    PVOID StackBase;

    PVOID StackLimit;

    PVOID StackCommit;

    PVOID StackCommitMax;

    PVOID StackReserved;



} INITIAL_TEB, *PINITIAL_TEB;

And this is where it gets really sad. It almost looks as if all the fields were entirely reshuffled and lost their meaning. Simple terms ike “StackBase” became “StackCommitMax”, but at least this is more related to the original “ExpandableStackLimit”.

All in all, the first structure was the closest, but had the wrong name, the second was a total mess, but had a right name, while the third tried to have the best of both worlds. Ultimately however, it's a failure too, mostly because it decided to name the fields in some kind of cubist terminology. Then there's the issue of the first two fields. I've decided to break this status quo and do some research on my own. It took less then 10 minutes. So hopefully, to end all further discussion, I present a fourth, and final structure:

typedef struct _INITIAL_TEB
{
    PVOID PreviousStackBase;
    PVOID PreviousStackLimit;
    PVOID StackBase;
    PVOID StackLimit;
    PVOID AllocatedStackBase;
} INITIAL_TEB, *PINITIAL_TEB;

And I'll actually take the time to explain it! As you know, the TEB's first data structure is called the NT_TIB, and it contains valuable stack information such as the Stack Base and the Stack Limit. There is also another pointer in the TEB towards the “DeallocationStack“, which is used by RtlFreeUserStack. Quite simply, StackBase is your Stack Base, or in other words, where your stack starts. The AllocatedStackBase, on the other hand, is where NT started the allocation of your stack with the NtAllocateVirtualMemory. Why the difference? Well, it's simple; stacks are top-down, which means that your Stack Base is going to be the highest address in the memory block. However, the “Allocation Base“ simply refers to the Base Address where the memory allocation was down, so it's the lowest number.

To finish, we also have “Stack Limit“, which, being the limit of the stack, means it's the lowest address in the memory block -- in theory. It so happens that it's usually not. Why? Once again, the reason is not so hard to understand. NT supports expandable stacks, which means that you have two arguments when creating your thread's stack: the maximum stack space to reserve, and the actual stack space to commit. This implies that your memory allocation might've been 20KB, but your actual commited stack memory is only 10KB. Hence, Stack Limit will be 10KB below Stack Base, and 10KB above AllocatedStackBase. If this is still confusing, here is some pretty ASCII Art:

R /+----------------+--> StackBase = 0x55000             |  D   
E  +                +                                    |  I     S
S  + COMMITTED STACK+                                    |  R     T
E  +                +                                    |  E  O  A
R  +----------------+--> StackLimit = 0x45000            |  C  F  C
V  +                +                                    |  T     K
E  +  FREE  STACK   +                                    |  I
D  +                +                                    |  O
  \+----------------+--> StackAllocationBase = 0x35000   \/ N

The stack in this picture was requested to have a total maximum size of 0x20000 bytes, but only a current size of 0x10000 bytes. The system reserved 0x20000 bytes of memory starting at 0x35000, up to 0x55000. It commited 0x10000 bytes of memory, starting at 0x45000. The current Stack Base is the beginning of the stack, which on Intel CPUs is the highest valid address, in our case 0x55000. The limit of our stack, or how low it can go, is 0x45000. When the OS will free the stack, it will free it starting at the StackAllocationBase, to free the entire memory.

There is, however, one more issue we haven't looked at: the Guard Page. This is a special protected area of memory between your Committed Stack and the Reserved Stack. It exists to easily catch stack corruption, where you've went beyond your Stack Limit. However, it's only created if there's an actual free stack (meaning you reserved more then you committed). So keep in mind that NT will not create a guard page for fixed stacks.

Finally, there is the issue of the two first fields in INITIAL_TEB. These are usually initialized to zero, and usually stay that way. However, if they do contain a value, then NT will use this value as the actual numbers which will go in the NT_TIB and TEB. It seems that it allows for multiple stacks to be created, and I've aptly named it “PreviousStackBase” and “PreviousStackLimit”. I could not find any cases where these are set to anything, only places where they are read, and used instead of the current values. They are probably used during Stack Expansion.

Well, there you have it. I hope this closes the chapter on INITIAL_TEB.

Comments (0) | Trackbacks (0)