ReactOS Community > ReactOS Blogs

I has been working on the Compatibility Database (part of the Support Database) since November 2005. Now, the Compatibility Database has reached the beta status and everyone can use it, simply visit www.reactos.org/support.

Comments (3) | Trackbacks (0)

ThePhysicist wrote
"And it would be nice to have more dev blogs. Just a few sentences from
time to time apart from the svn commits to see what's going on."

Comments (4) | Trackbacks (0)

You're all probably wondering what happened to this .....

*updated*

Comments (2) | Trackbacks (0)

Ok, well I’m gonna start blogging, considering that’s what we set this thing up for. This, my first ReactOS blog, will attempt to shed some light on the audit process answering the many questions which commonly appear on IRC

Comment (1) | Trackbacks (0)

I am currently working on the ReactOS Support Database. That is a sub-section of the ReactOS website which main goal is to help ReactOS user.

Comments (2) | Trackbacks (0)

Currently working on

Trying to work out something with the $100 laptop project.
- 20051223
ReactOS First Quarter Report
- 20060105
ReactOS Foundation First Quarter Report
- 20060105

Comment (1) | Trackbacks (0)

UPDATE4: Got bored of the MSVC fixes. Only lib and ntoskrnl are left. Going back to kernel work now!

UPDATE3: Due to the heavy kernel patches I've made lately I am taking a month off from future commits and will be working to find and fix possible regressions, as well as to get ReactOS building with MSVC.

UPDATE2: Added some more work, changed some dates around.

UPDATE: Been going through a real life crisis. Dates modified accordingly.

Here's an updated TODO list in the near-term for me (yeah, I know, I change stuff around really often):

• Finish User-Mode Networking for ReactOS. Deadline: Dec 4th.
  
  
  
  • Complete!
  
  
• Make new Process Manager patch work. Deadline: Dec 6th. 
  
  
  
  
  • Complete!
  
  
• Make new ntgdi.h patch work. Deadline: Dec 8th.
  
  
  
  • Complete!
  
  
• Implement new Freeldr GUI changes. Deadline: Dec 21st.
  
  
  
  • Complete!
  
  
• Finish and test Pushlock Implementation. Deadline: Dec 25th. 
  
  
  
  • Complete!
  
  
  
• Commit Push Lock Implementation. Deadline: Dec 27th.
  
  
  
  • Complete!
  
  
• Cleanup Rundown Implementation. Deadline: Dec 27th.
  
  
  
  • Complete!
  
  
• Cleanup atom.c, event.c, error.c, evtpair.c, lookas.c, mutant.c, profile.c, sem.c, time.c, zone.c, win32k.c Deadline: Dec 28th.
  
  
  
  • Complete!
  
  
  
• Optimize regular Spinlocks. Deadline: Dec 29th.
  
  
  
  • Complete!
  
  
• Cleanup interlck.c, list.c. Deadline: Dec 29th.
  
  
  
  • Complete!
  
  
• Implement advanced features for worker threads (work.c). Deadline: Dec 30th.
  
  
  
  • Complete!
  
  
• Rewrite most of resource.c. Deadline: Dec 30th.
  
  
  
  
  • Complete!
  
  
• Fix some semantic bugs in the wait code, and heavily optimize it: Deadline: Jan 4th.
  
  
  
  • Complete!
  
  
• Commit and test resource.c. Deadline: Jan 5th
  
  
  
  • Complete!
  
  
• Implement Queued Spinlocks. Deadline: Jan 15th.
  
  
  
  
  • Complete!
  
  
• Fix resource and wait regressions that the previous patches caused. Deadline: ASAP.
  
  
  
  • Complete!

• Re-implement most of the timer code (in ke) to take advantage of timer tables and per-timer table locks. Deadline: Jan 17th.
  
• Implement MmGrowKernelStack and PsConvertGuiToThread (Converstion of CLI to GUI threads and stack creation (60KB) and expansion support). Deadline: Jan 16th.
  
  
  
  
  • Complete!
  
  
• Finally re-implement my User-Mode callback code. Deadline: Jan 18th.
  
  
  
  • Complete!
  
  
• Fixup the TrapFrame<->Context Functions to support external ContextFlags, and add stubs for FPU register conversion. Deadline: Jan 20th.
  
  
  
  • Complete!
  
  
• Detect and disable cmpxchg8 on 486. Deadline: Jan 15th





• Half-way there!


• Fix syscall.S. Deadline: Jan 19th.




• Complete!


• Finish, commit and test the new Loader. Deadline: Jan 20th.
  
• Commit and test the User-Mode Networking branch, with appropriate afd.sys changes. Deadline: Jan 23th.
  
• Rewrite FPU support. Deadline: Jan 24th.





• Half-way there!


• Refactor low-level I/O code and finish old changes. Deadline: Jan 25th. 
  
• Refactor exceptions/traps. Deadline: Jan 28th. 




• Complete!


• Use macros for syscall.S. Deadline: Jan 29th. 




• Complete!


• Refactor bootup sequence. Deadline: Feb 11th.
  
  
• Refactor afd.sys. Deadline: Feb 28th. 
  
• Refactor Ob Parsing (again). Deadline: March 10th. 
  
• Implement new SMSS and CSR DLLs. Deadline: May 1st.

This is my -entire- work on ReactOS counting in ALL issues which currently nag me and I've wanted to fix forever. Beyond this list there is nothing that I can currently think of, except some private projects that I will think about later.

Comments (2) | Trackbacks (0)

Apologies for not having continued the article below; while installing my new box, I accidentally formatted, deleted, re-paritionned and re-imaged into a new RAID array all 8000 lines of code I had written, then I over-wrote the clusters on the disk when I reinstalled Windows. So it took me another week to re-write everything and begin testing. Until two days, ago, almost every application except Firefox, Thunderbird and IE was working fine. Today, thanks to a tool which I SERIOUSLY recommend, called ApiMonitor, I was able to see the differences between what my DLL was doing, and what the official one was. I saw that I was succesfully completing connect(), while the windows dll was saying "WSAEWOULDBLOCK". Additonnally, the handles with my winsock were a bit lower, as if two handles were missing. ProcessXP showed these to bee the AFD AsyncConnectHelper handles, which meant that the real DLL was doing asyncronous connects, while mine wasn't. After some digging, I found out that:

1) I wasn't properly setting the overlapped flag (which determines if operations will be async or not)

2) Even after setting the flag correctly, mswsock itself has to be notified that you'll be doing async stuff (by using WSAIoCtl/ioctlsocket). Turns out that I hadn't modified this function during my rewrite, and whoever wrote it originally was overwriting the value.

3) Even after fixing those two things, turns out that the socket() implementation was also somethign I had missed. Before calling WSASocket, it's supposed to set WSA_FLAG_OVERLAPPED by default, which it currently didn't.

After fixing those three things, Firefox was behaving exactly the same, API-wise, when starting up (I had set it to start to a blank page). I tried yahoo.com, and everything loaded perfectly. Downloading works, everything. And this of course fixed Thunderbird too.

IE is using the new "getnameinfo" and "getaddrinfo" APIs which were added in XP. They took almost a whole day to implement, but thankfully the source-code is already available in a header (although I had to heavily modify it, which is why it took so long). I'm now going to try IE, but I feel it should work too.

Gotta love it when hard work pays off!

Oh yeah, here's what works:

- Firefox

- Thunderbird

- Filezilla, ftp

- MIRC

- SVN, CVS

Those apps use almost 85% of ws2_32's apis, so I don't know what else to throw at it. Any suggestions? (Except for IE, which I'm going to test now)

Comments (0) | Trackbacks (0)

Ever wondered what makes WS2_32 tick? This technical spec will detail the inner workings of the ReactOS Winsock DLL (ws2_32.dll), which is based on the joint Intel-Microsoft implementation internals, as well as the official samples for Layered Providers (which include the DPROCESS, DTHREAD, etc C++ classes which I've duplicated in C with different names). So without further ado, here is a brief introduction.

1. The ws2_32.dll “kernel”

Before even getting into the details of Winsock API calls, we'll familiarize ourselves with the basic “objects” (implemented as types) and internal APIs which manage them. These objects (such as the Process, Thread and Socket objects) allow us to manipulate some OS internals with relative portability, provide an internal reference counting mechanism, as well as automatic allocation/deallocation of objects. In short, it works much like the NT Kernel, and wraps common operations under certain objects. The basic ws2_32.dll (henceforth “Winsock”) objects are:

• WSPROCESS (The process object)


• WSTHREAD (The thread object)


• WSSOCKET (The socket object)

Apart from the core objects above, which will go in detail later, there are also several objects for managing the Layered Providers (which have entries in the registry). The registry data for layered providers (LSPs) is called a “catalog”, of which two are present: the namespace catalog (for NSPs) and the protocol (or transport) catalog (for TSPs). Internally, these are exposed under the following objects:

• NSCATALOG (The namespace catalog object)


• NSCATALOG_ENTRY (The namespace catalog entry object)


• TPCATALOG (The transport protocol catalog object)


• TPCATALOG_ENTRY (The transport protocol catalog entry object)

Each catalog entry defines a provider, both of which are exposed as:

• NSPROVIDER (The namespace provider object)


• TPPROVIDER (The transport protocol provider object)

Due to the way that namespace Winsock APIs are handled however, it becomes advantageous to wrap the namespace providers into two more objects:

• NSQUERY (The namespace query object)


• NSQUERYPROVIDER (The namespace query provider object).

Because most namespace APIs are done as “queries” mapped by a handle and being served by a provider, the NSQUERY object allows us to contain that information into a “query” object and perform various accounting on it. Additionally, as each query call can have per-context data connected to a certain namespace provider, as well as to simplify the internal API, the NSQUERYPROVIDER contains a shrinkwrap between NSQUERYies and NSPROVIDERs, containing data such as the current query handle.

Finally, a more of a pseudo-object, WSASYNCBLOCK contains the data for the internal APIs which managae WSAAsyncGetXbyY APIs.

To handle these objects, the following internal API prefixes exist:

• WsProc*


• WsThread*


• WsSock*


• WsNc*


• WsTc*


• WsNp*



• WsTp*


• WsNcEntry*


• WsTcEntry*


• WsNq*


• WsNqProv*


• WsAsync*

Note that the names for the objects are not yet final, and might receive some underscores here and there. In the next post, I'll describe the structure of the core 3 objects as well as document their internal APIs.

Comments (0) | Trackbacks (0)

I apologize for the recent lack of updates... life has been quite a b*tch.

I suppose I should start by saying where I've been in my schedule. Well, over a month late! But not because I've slacked off, but because I've worked on unplanned things.

First of all, it turns out that the NDK stuff had a lot of small issues which still required my attention, including DDK and PSDK compatibility. That took me about a week, but I got most of the ReactOS stuff to build with MSVC + NDK + DDK/PSDK/IFS/WDK. It's a long term goal of mine to setup a super-duper build beast on msvc + incredibuild, and this was a first step that a lot of people needed to get the MSVC backend for our build system running.

Then of course, as I browsed through the source, I found more stuff that was annoying me. It turns out we were still duplicating almost 30 files in ntdll/rtl and ntoskrnl/rtl. I spent a great deal of time sharing everying in lib\rtl, and ntdll became almost only 10 files. But then, while looking at ntdll, I noticed two more things, which became three:

1) Our NTDLL callback from kernel-mode (KiUserApcDispatcher, LdrInitializeThunk, KiUserExceptionDispatcher) were written in C and buggy (they even had the wrong prototype and were using the wrong number of parameters!)
2) Our CSR APIs were completely incompatible and/or unexistent.
3) Turns out our whole exception handling code was buggy and wrong, with ugly hacks in many places (one example is that RtlUnwind was basically “optimized” not to actually care for the 3 optional parameters which some apps might send).

I started with #2, because it wasn't extremly hard. I implemented the entire range of CsrCapture* APIs and made CsrClientCallServer and CsrConnectToServer binary compatible with Windows XP, then I modified kernel32 to properly use the APIs when initializing and also for some console applications. I added a regression but fixed it a week later.

I then attacked issue #1 and #3 and rewrote in almost entirety the exposed exception handling and dispatching interface, fixing bugs, as well as critical APIs such as NtContinue. I also optimized the system call handler again (for sysenter/int2e), only to discover that we were not doing V86 Trap Frame Bias. I spend almost a week working out all the issues, and we -still- can't use 100% of my optimize code because we seem to have a bug regarding segment registers (which I think NT fixes by using “lazy loading“). As you can gather, this was a pretty large patch and touched a great deal of kernel internals. I wanted to go further and start touching the IRQ/TRAP handlers and low-level CPU exception code (which I'm also unhappy with), but I decided to keep that for later. Oh, and finally, I rewrote all the PCONTEXT<->PKTRAP_FRAME APIs to add sanization, proper conversion in some special cases, some assumptions and support for edited frames. I also added support for PKEXCEPTION_FRAMEs which will be needed on PPC and other RISC (and even IA-64) CPUs. There's probably a couple other things I fixed too, but there are so many that I've forgotten. Oh yes, I added SEH to all the ntdll callback handlers, so that if they crash in user-mode, they'll cleanly return to kernel-mode instead of just dying.

Emanuele Aliberti had comitted a new skelon for CSR, which was finally going to work right and use csrss/csrsrv.dll + basesrv.dll and winsrv.dll. I was really afraid for the design to become as messed up as the current CSR is, so I took the selfish (although also quite beneficary) step of writing the whole thing by myself, documenting all its interfaces, and verifying for 100% binary compatibility on NT. The end result (around 6000 lines) is something I'm really proud of. A completely open-sourced and heavily documented csrsrv.dll clone. That took me around 2 weeks, and I've added smss to my futture TODO list to match this new design.

If writing csrsrv from scratch and doing all those kernel fixes and implementing CSR apis properly in ntdll wasn't enough, I also became unhappy with another large part of duplicated code between ntdll/ntoskrnl: the Dbg* APIs like DbgPrint(Ex), vDbgPrintWithPrefix, DbgQueryFilterState, DbgPrompt, etc. Our KiDebugRoutine was also a big hack, so I decided to share the code between the two like it should be, and add it to the RTL Library (theoretically, Dbg is part of RTL). I also informed myself on the real way that Debug Output is done (basically piggybacking on an INT3 exception), but I decided not to implement that yet, since this patch was large enough as it was and required another batch of kernel changes. It still had one bug however, so for now the kernel doesn't use KiDebugRoutine, since it seems to make it freeze for some reason.

You think that's all? No way... on my way to ASM-izing the LdrInitializeThunk, I started feeling hate for the Ldr API. So I rewrote all of it using the hash tables that are used on NT, as well as implementing all the new Win2K3 functionality, such as SxS, Activation Contexts, Fiber Local Storage, Thread Local Storage, Redirection, .LOCAL support, etc. I haven't committed it yet though because it still needs some testing and 3 functions are missing, but it was good to do some PE programming again.

After that, I took a look at the Kernel's Process Management functions again (Ps), which I wrote most of, and noticed how broken they were and how many things I had missed, mostly dereferencing things, or checking for flags. I was also not using the new Win2003 Internal APIs which help a lot with enumeration, so I added all that, as well as fixed at least 10 issues with locks and races, stuff that wasn't getting cleaned, and more. Then I did a round of optimization and added some nice tweaks (especially for the process startup and termination APCs.). I also haven't committed that yet, because it needs testing and is missing a couple of new functions as well.

That was now 2 95% complete patches, so I decided to work on something that would be 100%. I had implemented Guarded Mutex and Kernel Gates some time ago, but the code wasn't working well. One of the reasons was that there were two subtle bugs, which Magey helped me find. Once that was done however, I realized that the entire Kernel APC code was messed up and not respecting the new Win2003 semantics related to Special Apc Disable and guarded regions, so I improved and rewrote much of my old code, also making it much much cleaner. I committed the patch, and also fixed a regression, and now APCs work -perfectly- and as up-to-date as possible (except for Vista's new APC Rate Limiting stuff, I'll get to that in another time).

So finally, after all this was done, I decided to dump my Ob/Cm patch, since too many things had changed, and it was really eating away at my soul. I put winsock back on my priority, and I've been rewriting some parts of my 6 month old code for the last 2 days. I'll go in detail of the WS 2.2 architecture tomorrow and how it all works internally inside ws2_32.dll.

Thanks for your time!

Comments (0) | Trackbacks (0)