Page 1 of 6

Reactos Update (Was: ReactOS applications needed?)

Posted: Mon Jan 14, 2013 8:16 am
by gonzoMD
Vic: u talked 'bout ReactOS Update, naums is working on an update script for new trunk builds.
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279

Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point

Re: ReactOS applications needed?

Posted: Mon Jan 14, 2013 10:16 am
by vicmarcal
gonzoMD wrote:Vic: u talked 'bout ReactOS Update, naums is working on an update script for new trunk builds.
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279

Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point
Yup.I didnt know he opened a thread in the German Forum :)
I'd been talking with him to find a way to do it.How to replace critical parts,etc.How to use alresdy compiled binaries to avoid depending on buildbots.How to request the files on demand.How to serve them..etc.
I can work in the Frontend area.Basically a Window with a button.(The GUI is not difficult to do really).
Most of the work is done in the Server,which is the area I dont know at all. :)

Re: ReactOS applications needed?

Posted: Mon Jan 14, 2013 10:21 am
by gonzoMD
vicmarcal wrote:
gonzoMD wrote:Vic: u talked 'bout ReactOS Update, naums is working on an update script for new trunk builds.
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279

Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point
Yup.I didnt know he opened a thread in the German Forum :)
I'd been talking with him to find a way to do it.How to replace critical parts,etc.How to use alresdy compiled binaries to avoid depending on buildbots.How to request the files on demand.How to serve them..etc.
I can work in the Frontend area.Basically a Window with a button.(The GUI is not difficult to do really).
Most of the work is done in the Server,which is the area I dont know at all. :)
I don't know much about GUI apps, but what I know is that they are not complicated but they contain maaaaaaany lines of code.
I read something about services in msdn atm because we need an update sheduler/notifier which informs the updater about new revisions.

Re: ReactOS applications needed?

Posted: Mon Jan 14, 2013 11:28 am
by vicmarcal
GUI app are really complex.But its complexity are related to the GUI complexity. Creating a small GUI is not really difficult. Charles Petzold has a nice book to learn about GUI basics.
@ the service: I wouldnt go so far yet. I would begin with an On Demand app: The user launches the Updater manually.And the launcher checks for updates. In version 2.0 you can add the checking as a Service. And in version 3.0 we can implement a BITS technology so the downloads are done on idle moments not stealing bandwith when the user is "using"it.
Improving step by step :)

Re: ReactOS applications needed?

Posted: Mon Jan 14, 2013 12:07 pm
by hto
gonzoMD wrote: […] now he just analyse the changes and download the changed files from a server. Forum Topic here: […]
I don't speak Deutsch so will answer here. His idea was to use hashes to compare files from different builds. Although there are some files (such as TrueType fonts) which rarely change from revision to revision, most of .exe, .dll, etc. files are different in every build, even when the source code was not modified in any way. In particular, the PE header has a time/date stamp field which is not the same. Also, some files (ntoskrnl.exe, cmd.exe, shell32.dll, etc.) has a revision number compiled into them. Of course, you can invent some sofisticated hashing algorithm which will checksum only certain relevant parts of files, but there is a simpler and better solution.

Even though files from the two consecutive builds are diferent, the difference is small, so you can store just compressed binary diffs between revisions, rather then whole files. Version control systems, such as SVN, are already able to do that. Simply unpack the .7z archive, extract all files from the .iso, then unpack (and then remove) reactos.cab archive (better to make a new subdirectory and unpack into it). And commit all that stuff into a repository. I experimented with it a long time ago, there was a thread on this forum with results. It can be useful for regression hunting purposes.

The update procedure will work most of time, but in some rare cases an incompatibility is introduced, so you must reinstall the system.

About MoveFileEx and ntoskrnl.exe / hal.dll problem — the boot loader has Kernel= and Hal= or SystemPath= options, try to use them.

Re: ReactOS applications needed?

Posted: Mon Jan 14, 2013 5:21 pm
by PascalDragon
hto wrote:About MoveFileEx and ntoskrnl.exe / hal.dll problem — the boot loader has Kernel= and Hal= or SystemPath= options, try to use them.
Ah! Nice :)
That could solve that problem as well. It's still not trivial, but it seems that the necessary tools are there :)

Regards,
Sven

Re: ReactOS applications needed?

Posted: Mon Jan 14, 2013 7:39 pm
by vicmarcal
Well, the idea shared with Naums was pretty close:

1)The user launchs Updater.exe
2) Updater.exe checks if there is a list of MD5s of current system files:
2A)If so: We send that list to the Server.
2B)If not: We create the list and send it to the Server.
3)The Server recives this list.
4)The Server checks if it has the latest iso from iso.reactos.org.
4B)If not: The Server downloads the latest ISO from iso.reactos.org,creates a folder for the ISO, unzips the ISO and its .cab files there, and creates a MD5 list of those files.
4B)If so: Checks the integrity of the folder: All the files and MD5 list exist.
5)The Server compares MD5 list coming from the user, with the one in the Server folder.
6)Are there any files with different MD5?or any files in the Server list that doesnt exist in the User list(a new dll,app,module added)?
6A)If so: The Server creates a .cab with the modified-new files and the reactos.inf file.
6B)If not: The Server sends a "Your version is totally updated".The connection is closed.END
7)Updater.exe recives the .cab and stores somewhere in C:
8)The connection to the Server is closed.
9)The Updater calls MoveFileEx with the flag MOVEFILE_DELAY_UNTIL_REBOOT that will perform the copy of the files when rebooting for those apps or dlls which are in use.

This way the Server will store just the ISOs requested by the users, and not all the isos from iso.reactos.org, preserving HDD space. These ISOs are stored for concurrent request, so a second request will be served much faster.
Statistiscally, with zillions of users, all the ISOs from Iso.reactos.org will be requested and stored in The Server.But we're not there yet ;)
For that time, the Server will have plenty space to store all the ISOs or the updater will work linked to the Builders detecting the modified files on compiling time and creating diffs with those modified modules instead.

Improvements:
-Instead sending the MD5 of all the ReactOS files, Updater.exe can retrieve the revision from a registry key. The problem would be that if there is any error during the MoveFileEx, some files will be updated, other wont, but the revision in the registry is updated(or not). Managing ReactOS as a whole is more prone to errors that managing by file.

Problems:
-If Hto is right, and he is because he knows better, two dlls with the same code will have different MD5 because the "PE Header". The Hash algorithm(that is included not just by Security reasons but also to discover differences between the Dll installed in the User PC and the latest version of the Dll stored in the Server) must be, then, a different one.
Is there a Hash algorithm that could be used for this purpose?
Maybe we can create a really easy Hash,one that takes the "Name of the file","Extension of the file" and "Size in bytes". It won't be supersecure but at least will work to check for differences.

- Do you see any other issue?

Re: ReactOS applications needed?

Posted: Mon Jan 14, 2013 9:21 pm
by milon
vicmarcal wrote:Problems:
-If Hto is right, and he is because he knows better, two dlls with the same code will have different MD5 because the "PE Header". The Hash algorithm(that is included not just by Security reasons but also to discover differences between the Dll installed in the User PC and the latest version of the Dll stored in the Server) must be, then, a different one.
Is there a Hash algorithm that could be used for this purpose?
Maybe we can create a really easy Hash,one that takes the "Name of the file","Extension of the file" and "Size in bytes". It won't be supersecure but at least will work to check for differences.

- Do you see any other issue?
If I were creating a hash like that for my own uses, I'd be tempted to use the date & time the file was created. Still not supersecure, but a little better.

Posted: Mon Jan 14, 2013 10:17 pm
by hto
Hash sums are not needed.

I think the update should work on the level of the whole ReactOS directory tree, it should not play around with every single file. It will require more disk space, but it is more safe.

Imagine that you want to update the system from r58173 to r58174. The Updater.exe program creates a full copy of ReactOS directory tree nameed ReactOS-r58174. Then it downloads a binary patch 58173-58174 from the server; even if most of files are different, the patch should be quite small. Then applies it to the ReactOS-r58174 and makes last preparations before the 1st reboot. The machine will then boot some minimal system similar to the 1st stage text-mode installer. It will rename the ReactOS directory into ReactOS-r58173, and ReactOS-r58174 into ReactOS; it takes just a small number of disk operations, which is important for such a critical procedure. Then goes the 2nd reboot into the new system. The old system is still there for backup purposes.

Re: ReactOS applications needed?

Posted: Tue Jan 15, 2013 12:27 am
by vicmarcal
Sounds nice.
And how will the Server calculate the binary patch UserRevision-ActualRevision(e.g 58173-58174)?

Re: ReactOS applications needed?

Posted: Tue Jan 15, 2013 10:39 am
by Black_Fox
@vicmarcal: E.g. http://www.daemonology.net/bsdiff/ - from the top of my head.

Re: ReactOS applications needed?

Posted: Tue Jan 15, 2013 1:43 pm
by fred02
vicmarcal wrote: - Do you see any other issue?
Will do at this stage of the project, when it is not so widespread and used by people that know what they do (mostly :D ). When ROS gets more widespread and hit general public (beta), some more security/trustworthiness checks will be necessary to ensure that no system file is changed by a malicious version.

Re: ReactOS applications needed?

Posted: Tue Jan 15, 2013 4:14 pm
by vicmarcal
About the .diff thing:
"JDIFF tries to find a minimal set of differences between two files using a heuristic algorithm with constant space and linear time complexity. This means that accuracy is traded over speed. JDIFF will therefore, in general, not always find the smallest set of differences, but will try to be fast and will use a fixed amount of memory. "
Hmmm..
This raises the "Who does ensure me that the Dlls after patched via .diff will be exactly the same as the original?".

Re: ReactOS applications needed?

Posted: Tue Jan 15, 2013 4:16 pm
by vicmarcal
http://i-tools.org/bsdiff
And a new tool via Google:
http://dev.chromium.org/developers/desi ... -courgette

Interesting link about how it works,reducing the size of the diff about 90%.

Posted: Tue Jan 15, 2013 8:00 pm
by hto
vicmarcal wrote: This raises the "Who does ensure me that the Dlls after patched via .diff will be exactly the same as the original?".
The word "accuracy" above refers to the fact that the tool does not generate the smallest possible patch, just a small enough one. It does not mean that the result is incorrect.
And how will the Server calculate the binary patch UserRevision-ActualRevision(e.g 58173-58174)?
I already mentioned version control systems, which internally keep only some revisions, while other, intermediate revisions are stored as diffs.