Reactos Update (Was: ReactOS applications needed?)

Here you can discuss ReactOS related topics.

Moderator: Moderator Team

gonzoMD
Posts: 1018
Joined: Fri Oct 20, 2006 7:49 am
Location: Germany
Contact:

Reactos Update (Was: ReactOS applications needed?)

Post by gonzoMD » Mon Jan 14, 2013 8:16 am

Vic: u talked 'bout ReactOS Update, naums is working on an update script for new trunk builds.
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279

Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: ReactOS applications needed?

Post by vicmarcal » Mon Jan 14, 2013 10:16 am

gonzoMD wrote:Vic: u talked 'bout ReactOS Update, naums is working on an update script for new trunk builds.
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279

Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point
Yup.I didnt know he opened a thread in the German Forum :)
I'd been talking with him to find a way to do it.How to replace critical parts,etc.How to use alresdy compiled binaries to avoid depending on buildbots.How to request the files on demand.How to serve them..etc.
I can work in the Frontend area.Basically a Window with a button.(The GUI is not difficult to do really).
Most of the work is done in the Server,which is the area I dont know at all. :)
Image

gonzoMD
Posts: 1018
Joined: Fri Oct 20, 2006 7:49 am
Location: Germany
Contact:

Re: ReactOS applications needed?

Post by gonzoMD » Mon Jan 14, 2013 10:21 am

vicmarcal wrote:
gonzoMD wrote:Vic: u talked 'bout ReactOS Update, naums is working on an update script for new trunk builds.
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279

Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point
Yup.I didnt know he opened a thread in the German Forum :)
I'd been talking with him to find a way to do it.How to replace critical parts,etc.How to use alresdy compiled binaries to avoid depending on buildbots.How to request the files on demand.How to serve them..etc.
I can work in the Frontend area.Basically a Window with a button.(The GUI is not difficult to do really).
Most of the work is done in the Server,which is the area I dont know at all. :)
I don't know much about GUI apps, but what I know is that they are not complicated but they contain maaaaaaany lines of code.
I read something about services in msdn atm because we need an update sheduler/notifier which informs the updater about new revisions.

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: ReactOS applications needed?

Post by vicmarcal » Mon Jan 14, 2013 11:28 am

GUI app are really complex.But its complexity are related to the GUI complexity. Creating a small GUI is not really difficult. Charles Petzold has a nice book to learn about GUI basics.
@ the service: I wouldnt go so far yet. I would begin with an On Demand app: The user launches the Updater manually.And the launcher checks for updates. In version 2.0 you can add the checking as a Service. And in version 3.0 we can implement a BITS technology so the downloads are done on idle moments not stealing bandwith when the user is "using"it.
Improving step by step :)
Image

hto
Developer
Posts: 2193
Joined: Sun Oct 01, 2006 3:43 pm

Re: ReactOS applications needed?

Post by hto » Mon Jan 14, 2013 12:07 pm

gonzoMD wrote: […] now he just analyse the changes and download the changed files from a server. Forum Topic here: […]
I don't speak Deutsch so will answer here. His idea was to use hashes to compare files from different builds. Although there are some files (such as TrueType fonts) which rarely change from revision to revision, most of .exe, .dll, etc. files are different in every build, even when the source code was not modified in any way. In particular, the PE header has a time/date stamp field which is not the same. Also, some files (ntoskrnl.exe, cmd.exe, shell32.dll, etc.) has a revision number compiled into them. Of course, you can invent some sofisticated hashing algorithm which will checksum only certain relevant parts of files, but there is a simpler and better solution.

Even though files from the two consecutive builds are diferent, the difference is small, so you can store just compressed binary diffs between revisions, rather then whole files. Version control systems, such as SVN, are already able to do that. Simply unpack the .7z archive, extract all files from the .iso, then unpack (and then remove) reactos.cab archive (better to make a new subdirectory and unpack into it). And commit all that stuff into a repository. I experimented with it a long time ago, there was a thread on this forum with results. It can be useful for regression hunting purposes.

The update procedure will work most of time, but in some rare cases an incompatibility is introduced, so you must reinstall the system.

About MoveFileEx and ntoskrnl.exe / hal.dll problem — the boot loader has Kernel= and Hal= or SystemPath= options, try to use them.

PascalDragon
Posts: 123
Joined: Wed Aug 04, 2010 7:34 pm

Re: ReactOS applications needed?

Post by PascalDragon » Mon Jan 14, 2013 5:21 pm

hto wrote:About MoveFileEx and ntoskrnl.exe / hal.dll problem — the boot loader has Kernel= and Hal= or SystemPath= options, try to use them.
Ah! Nice :)
That could solve that problem as well. It's still not trivial, but it seems that the necessary tools are there :)

Regards,
Sven
Free Pascal compiler developer

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: ReactOS applications needed?

Post by vicmarcal » Mon Jan 14, 2013 7:39 pm

Well, the idea shared with Naums was pretty close:

1)The user launchs Updater.exe
2) Updater.exe checks if there is a list of MD5s of current system files:
2A)If so: We send that list to the Server.
2B)If not: We create the list and send it to the Server.
3)The Server recives this list.
4)The Server checks if it has the latest iso from iso.reactos.org.
4B)If not: The Server downloads the latest ISO from iso.reactos.org,creates a folder for the ISO, unzips the ISO and its .cab files there, and creates a MD5 list of those files.
4B)If so: Checks the integrity of the folder: All the files and MD5 list exist.
5)The Server compares MD5 list coming from the user, with the one in the Server folder.
6)Are there any files with different MD5?or any files in the Server list that doesnt exist in the User list(a new dll,app,module added)?
6A)If so: The Server creates a .cab with the modified-new files and the reactos.inf file.
6B)If not: The Server sends a "Your version is totally updated".The connection is closed.END
7)Updater.exe recives the .cab and stores somewhere in C:
8)The connection to the Server is closed.
9)The Updater calls MoveFileEx with the flag MOVEFILE_DELAY_UNTIL_REBOOT that will perform the copy of the files when rebooting for those apps or dlls which are in use.

This way the Server will store just the ISOs requested by the users, and not all the isos from iso.reactos.org, preserving HDD space. These ISOs are stored for concurrent request, so a second request will be served much faster.
Statistiscally, with zillions of users, all the ISOs from Iso.reactos.org will be requested and stored in The Server.But we're not there yet ;)
For that time, the Server will have plenty space to store all the ISOs or the updater will work linked to the Builders detecting the modified files on compiling time and creating diffs with those modified modules instead.

Improvements:
-Instead sending the MD5 of all the ReactOS files, Updater.exe can retrieve the revision from a registry key. The problem would be that if there is any error during the MoveFileEx, some files will be updated, other wont, but the revision in the registry is updated(or not). Managing ReactOS as a whole is more prone to errors that managing by file.

Problems:
-If Hto is right, and he is because he knows better, two dlls with the same code will have different MD5 because the "PE Header". The Hash algorithm(that is included not just by Security reasons but also to discover differences between the Dll installed in the User PC and the latest version of the Dll stored in the Server) must be, then, a different one.
Is there a Hash algorithm that could be used for this purpose?
Maybe we can create a really easy Hash,one that takes the "Name of the file","Extension of the file" and "Size in bytes". It won't be supersecure but at least will work to check for differences.

- Do you see any other issue?
Image

milon
Posts: 969
Joined: Sat Sep 05, 2009 9:26 pm

Re: ReactOS applications needed?

Post by milon » Mon Jan 14, 2013 9:21 pm

vicmarcal wrote:Problems:
-If Hto is right, and he is because he knows better, two dlls with the same code will have different MD5 because the "PE Header". The Hash algorithm(that is included not just by Security reasons but also to discover differences between the Dll installed in the User PC and the latest version of the Dll stored in the Server) must be, then, a different one.
Is there a Hash algorithm that could be used for this purpose?
Maybe we can create a really easy Hash,one that takes the "Name of the file","Extension of the file" and "Size in bytes". It won't be supersecure but at least will work to check for differences.

- Do you see any other issue?
If I were creating a hash like that for my own uses, I'd be tempted to use the date & time the file was created. Still not supersecure, but a little better.

hto
Developer
Posts: 2193
Joined: Sun Oct 01, 2006 3:43 pm

Post by hto » Mon Jan 14, 2013 10:17 pm

Hash sums are not needed.

I think the update should work on the level of the whole ReactOS directory tree, it should not play around with every single file. It will require more disk space, but it is more safe.

Imagine that you want to update the system from r58173 to r58174. The Updater.exe program creates a full copy of ReactOS directory tree nameed ReactOS-r58174. Then it downloads a binary patch 58173-58174 from the server; even if most of files are different, the patch should be quite small. Then applies it to the ReactOS-r58174 and makes last preparations before the 1st reboot. The machine will then boot some minimal system similar to the 1st stage text-mode installer. It will rename the ReactOS directory into ReactOS-r58173, and ReactOS-r58174 into ReactOS; it takes just a small number of disk operations, which is important for such a critical procedure. Then goes the 2nd reboot into the new system. The old system is still there for backup purposes.

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: ReactOS applications needed?

Post by vicmarcal » Tue Jan 15, 2013 12:27 am

Sounds nice.
And how will the Server calculate the binary patch UserRevision-ActualRevision(e.g 58173-58174)?
Image

Black_Fox
Posts: 1584
Joined: Fri Feb 15, 2008 9:44 pm
Location: Czechia

Re: ReactOS applications needed?

Post by Black_Fox » Tue Jan 15, 2013 10:39 am

@vicmarcal: E.g. http://www.daemonology.net/bsdiff/ - from the top of my head.

fred02
Posts: 551
Joined: Thu Nov 22, 2007 5:54 pm

Re: ReactOS applications needed?

Post by fred02 » Tue Jan 15, 2013 1:43 pm

vicmarcal wrote: - Do you see any other issue?
Will do at this stage of the project, when it is not so widespread and used by people that know what they do (mostly :D ). When ROS gets more widespread and hit general public (beta), some more security/trustworthiness checks will be necessary to ensure that no system file is changed by a malicious version.

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: ReactOS applications needed?

Post by vicmarcal » Tue Jan 15, 2013 4:14 pm

About the .diff thing:
"JDIFF tries to find a minimal set of differences between two files using a heuristic algorithm with constant space and linear time complexity. This means that accuracy is traded over speed. JDIFF will therefore, in general, not always find the smallest set of differences, but will try to be fast and will use a fixed amount of memory. "
Hmmm..
This raises the "Who does ensure me that the Dlls after patched via .diff will be exactly the same as the original?".
Image

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: ReactOS applications needed?

Post by vicmarcal » Tue Jan 15, 2013 4:16 pm

http://i-tools.org/bsdiff
And a new tool via Google:
http://dev.chromium.org/developers/desi ... -courgette

Interesting link about how it works,reducing the size of the diff about 90%.
Image

hto
Developer
Posts: 2193
Joined: Sun Oct 01, 2006 3:43 pm

Post by hto » Tue Jan 15, 2013 8:00 pm

vicmarcal wrote: This raises the "Who does ensure me that the Dlls after patched via .diff will be exactly the same as the original?".
The word "accuracy" above refers to the fact that the tool does not generate the smallest possible patch, just a small enough one. It does not mean that the result is incorrect.
And how will the Server calculate the binary patch UserRevision-ActualRevision(e.g 58173-58174)?
I already mentioned version control systems, which internally keep only some revisions, while other, intermediate revisions are stored as diffs.

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot], DotBot [Crawler], Google [Bot], r3mendozal and 3 guests