Reactos Update (Was: ReactOS applications needed?)
Moderator: Moderator Team
Reactos Update (Was: ReactOS applications needed?)
Vic: u talked 'bout ReactOS Update, naums is working on an update script for new trunk builds.
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279
Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279
Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point
Re: ReactOS applications needed?
Yup.I didnt know he opened a thread in the German ForumgonzoMD wrote:Vic: u talked 'bout ReactOS Update, naums is working on an update script for new trunk builds.
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279
Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point
I'd been talking with him to find a way to do it.How to replace critical parts,etc.How to use alresdy compiled binaries to avoid depending on buildbots.How to request the files on demand.How to serve them..etc.
I can work in the Frontend area.Basically a Window with a button.(The GUI is not difficult to do really).
Most of the work is done in the Server,which is the area I dont know at all.
Re: ReactOS applications needed?
I don't know much about GUI apps, but what I know is that they are not complicated but they contain maaaaaaany lines of code.vicmarcal wrote:Yup.I didnt know he opened a thread in the German ForumgonzoMD wrote:Vic: u talked 'bout ReactOS Update, naums is working on an update script for new trunk builds.
at first he wanted to compile the new code byself, but now he just analyse the changes and download the changed files from a server. Forum Topic here: http://www.reactos.org/forum/viewtopic.php?f=6&t=12279
Milon: He is looking for someone who implements the Frontend in plain WinAPI. so this could be a good starting point
I'd been talking with him to find a way to do it.How to replace critical parts,etc.How to use alresdy compiled binaries to avoid depending on buildbots.How to request the files on demand.How to serve them..etc.
I can work in the Frontend area.Basically a Window with a button.(The GUI is not difficult to do really).
Most of the work is done in the Server,which is the area I dont know at all.
I read something about services in msdn atm because we need an update sheduler/notifier which informs the updater about new revisions.
Re: ReactOS applications needed?
GUI app are really complex.But its complexity are related to the GUI complexity. Creating a small GUI is not really difficult. Charles Petzold has a nice book to learn about GUI basics.
@ the service: I wouldnt go so far yet. I would begin with an On Demand app: The user launches the Updater manually.And the launcher checks for updates. In version 2.0 you can add the checking as a Service. And in version 3.0 we can implement a BITS technology so the downloads are done on idle moments not stealing bandwith when the user is "using"it.
Improving step by step
@ the service: I wouldnt go so far yet. I would begin with an On Demand app: The user launches the Updater manually.And the launcher checks for updates. In version 2.0 you can add the checking as a Service. And in version 3.0 we can implement a BITS technology so the downloads are done on idle moments not stealing bandwith when the user is "using"it.
Improving step by step
Re: ReactOS applications needed?
I don't speak Deutsch so will answer here. His idea was to use hashes to compare files from different builds. Although there are some files (such as TrueType fonts) which rarely change from revision to revision, most of .exe, .dll, etc. files are different in every build, even when the source code was not modified in any way. In particular, the PE header has a time/date stamp field which is not the same. Also, some files (ntoskrnl.exe, cmd.exe, shell32.dll, etc.) has a revision number compiled into them. Of course, you can invent some sofisticated hashing algorithm which will checksum only certain relevant parts of files, but there is a simpler and better solution.gonzoMD wrote: […] now he just analyse the changes and download the changed files from a server. Forum Topic here: […]
Even though files from the two consecutive builds are diferent, the difference is small, so you can store just compressed binary diffs between revisions, rather then whole files. Version control systems, such as SVN, are already able to do that. Simply unpack the .7z archive, extract all files from the .iso, then unpack (and then remove) reactos.cab archive (better to make a new subdirectory and unpack into it). And commit all that stuff into a repository. I experimented with it a long time ago, there was a thread on this forum with results. It can be useful for regression hunting purposes.
The update procedure will work most of time, but in some rare cases an incompatibility is introduced, so you must reinstall the system.
About MoveFileEx and ntoskrnl.exe / hal.dll problem — the boot loader has Kernel= and Hal= or SystemPath= options, try to use them.
-
- Posts: 123
- Joined: Wed Aug 04, 2010 7:34 pm
Re: ReactOS applications needed?
Ah! Nicehto wrote:About MoveFileEx and ntoskrnl.exe / hal.dll problem — the boot loader has Kernel= and Hal= or SystemPath= options, try to use them.
That could solve that problem as well. It's still not trivial, but it seems that the necessary tools are there
Regards,
Sven
Free Pascal compiler developer
Re: ReactOS applications needed?
Well, the idea shared with Naums was pretty close:
1)The user launchs Updater.exe
2) Updater.exe checks if there is a list of MD5s of current system files:
2A)If so: We send that list to the Server.
2B)If not: We create the list and send it to the Server.
3)The Server recives this list.
4)The Server checks if it has the latest iso from iso.reactos.org.
4B)If not: The Server downloads the latest ISO from iso.reactos.org,creates a folder for the ISO, unzips the ISO and its .cab files there, and creates a MD5 list of those files.
4B)If so: Checks the integrity of the folder: All the files and MD5 list exist.
5)The Server compares MD5 list coming from the user, with the one in the Server folder.
6)Are there any files with different MD5?or any files in the Server list that doesnt exist in the User list(a new dll,app,module added)?
6A)If so: The Server creates a .cab with the modified-new files and the reactos.inf file.
6B)If not: The Server sends a "Your version is totally updated".The connection is closed.END
7)Updater.exe recives the .cab and stores somewhere in C:
8)The connection to the Server is closed.
9)The Updater calls MoveFileEx with the flag MOVEFILE_DELAY_UNTIL_REBOOT that will perform the copy of the files when rebooting for those apps or dlls which are in use.
This way the Server will store just the ISOs requested by the users, and not all the isos from iso.reactos.org, preserving HDD space. These ISOs are stored for concurrent request, so a second request will be served much faster.
Statistiscally, with zillions of users, all the ISOs from Iso.reactos.org will be requested and stored in The Server.But we're not there yet
For that time, the Server will have plenty space to store all the ISOs or the updater will work linked to the Builders detecting the modified files on compiling time and creating diffs with those modified modules instead.
Improvements:
-Instead sending the MD5 of all the ReactOS files, Updater.exe can retrieve the revision from a registry key. The problem would be that if there is any error during the MoveFileEx, some files will be updated, other wont, but the revision in the registry is updated(or not). Managing ReactOS as a whole is more prone to errors that managing by file.
Problems:
-If Hto is right, and he is because he knows better, two dlls with the same code will have different MD5 because the "PE Header". The Hash algorithm(that is included not just by Security reasons but also to discover differences between the Dll installed in the User PC and the latest version of the Dll stored in the Server) must be, then, a different one.
Is there a Hash algorithm that could be used for this purpose?
Maybe we can create a really easy Hash,one that takes the "Name of the file","Extension of the file" and "Size in bytes". It won't be supersecure but at least will work to check for differences.
- Do you see any other issue?
1)The user launchs Updater.exe
2) Updater.exe checks if there is a list of MD5s of current system files:
2A)If so: We send that list to the Server.
2B)If not: We create the list and send it to the Server.
3)The Server recives this list.
4)The Server checks if it has the latest iso from iso.reactos.org.
4B)If not: The Server downloads the latest ISO from iso.reactos.org,creates a folder for the ISO, unzips the ISO and its .cab files there, and creates a MD5 list of those files.
4B)If so: Checks the integrity of the folder: All the files and MD5 list exist.
5)The Server compares MD5 list coming from the user, with the one in the Server folder.
6)Are there any files with different MD5?or any files in the Server list that doesnt exist in the User list(a new dll,app,module added)?
6A)If so: The Server creates a .cab with the modified-new files and the reactos.inf file.
6B)If not: The Server sends a "Your version is totally updated".The connection is closed.END
7)Updater.exe recives the .cab and stores somewhere in C:
8)The connection to the Server is closed.
9)The Updater calls MoveFileEx with the flag MOVEFILE_DELAY_UNTIL_REBOOT that will perform the copy of the files when rebooting for those apps or dlls which are in use.
This way the Server will store just the ISOs requested by the users, and not all the isos from iso.reactos.org, preserving HDD space. These ISOs are stored for concurrent request, so a second request will be served much faster.
Statistiscally, with zillions of users, all the ISOs from Iso.reactos.org will be requested and stored in The Server.But we're not there yet
For that time, the Server will have plenty space to store all the ISOs or the updater will work linked to the Builders detecting the modified files on compiling time and creating diffs with those modified modules instead.
Improvements:
-Instead sending the MD5 of all the ReactOS files, Updater.exe can retrieve the revision from a registry key. The problem would be that if there is any error during the MoveFileEx, some files will be updated, other wont, but the revision in the registry is updated(or not). Managing ReactOS as a whole is more prone to errors that managing by file.
Problems:
-If Hto is right, and he is because he knows better, two dlls with the same code will have different MD5 because the "PE Header". The Hash algorithm(that is included not just by Security reasons but also to discover differences between the Dll installed in the User PC and the latest version of the Dll stored in the Server) must be, then, a different one.
Is there a Hash algorithm that could be used for this purpose?
Maybe we can create a really easy Hash,one that takes the "Name of the file","Extension of the file" and "Size in bytes". It won't be supersecure but at least will work to check for differences.
- Do you see any other issue?
Re: ReactOS applications needed?
If I were creating a hash like that for my own uses, I'd be tempted to use the date & time the file was created. Still not supersecure, but a little better.vicmarcal wrote:Problems:
-If Hto is right, and he is because he knows better, two dlls with the same code will have different MD5 because the "PE Header". The Hash algorithm(that is included not just by Security reasons but also to discover differences between the Dll installed in the User PC and the latest version of the Dll stored in the Server) must be, then, a different one.
Is there a Hash algorithm that could be used for this purpose?
Maybe we can create a really easy Hash,one that takes the "Name of the file","Extension of the file" and "Size in bytes". It won't be supersecure but at least will work to check for differences.
- Do you see any other issue?
Hash sums are not needed.
I think the update should work on the level of the whole ReactOS directory tree, it should not play around with every single file. It will require more disk space, but it is more safe.
Imagine that you want to update the system from r58173 to r58174. The Updater.exe program creates a full copy of ReactOS directory tree nameed ReactOS-r58174. Then it downloads a binary patch 58173-58174 from the server; even if most of files are different, the patch should be quite small. Then applies it to the ReactOS-r58174 and makes last preparations before the 1st reboot. The machine will then boot some minimal system similar to the 1st stage text-mode installer. It will rename the ReactOS directory into ReactOS-r58173, and ReactOS-r58174 into ReactOS; it takes just a small number of disk operations, which is important for such a critical procedure. Then goes the 2nd reboot into the new system. The old system is still there for backup purposes.
I think the update should work on the level of the whole ReactOS directory tree, it should not play around with every single file. It will require more disk space, but it is more safe.
Imagine that you want to update the system from r58173 to r58174. The Updater.exe program creates a full copy of ReactOS directory tree nameed ReactOS-r58174. Then it downloads a binary patch 58173-58174 from the server; even if most of files are different, the patch should be quite small. Then applies it to the ReactOS-r58174 and makes last preparations before the 1st reboot. The machine will then boot some minimal system similar to the 1st stage text-mode installer. It will rename the ReactOS directory into ReactOS-r58173, and ReactOS-r58174 into ReactOS; it takes just a small number of disk operations, which is important for such a critical procedure. Then goes the 2nd reboot into the new system. The old system is still there for backup purposes.
Re: ReactOS applications needed?
Sounds nice.
And how will the Server calculate the binary patch UserRevision-ActualRevision(e.g 58173-58174)?
And how will the Server calculate the binary patch UserRevision-ActualRevision(e.g 58173-58174)?
Re: ReactOS applications needed?
@vicmarcal: E.g. http://www.daemonology.net/bsdiff/ - from the top of my head.
Re: ReactOS applications needed?
Will do at this stage of the project, when it is not so widespread and used by people that know what they do (mostly ). When ROS gets more widespread and hit general public (beta), some more security/trustworthiness checks will be necessary to ensure that no system file is changed by a malicious version.vicmarcal wrote: - Do you see any other issue?
Re: ReactOS applications needed?
About the .diff thing:
This raises the "Who does ensure me that the Dlls after patched via .diff will be exactly the same as the original?".
Hmmm.."JDIFF tries to find a minimal set of differences between two files using a heuristic algorithm with constant space and linear time complexity. This means that accuracy is traded over speed. JDIFF will therefore, in general, not always find the smallest set of differences, but will try to be fast and will use a fixed amount of memory. "
This raises the "Who does ensure me that the Dlls after patched via .diff will be exactly the same as the original?".
Re: ReactOS applications needed?
http://i-tools.org/bsdiff
And a new tool via Google:
http://dev.chromium.org/developers/desi ... -courgette
Interesting link about how it works,reducing the size of the diff about 90%.
And a new tool via Google:
http://dev.chromium.org/developers/desi ... -courgette
Interesting link about how it works,reducing the size of the diff about 90%.
The word "accuracy" above refers to the fact that the tool does not generate the smallest possible patch, just a small enough one. It does not mean that the result is incorrect.vicmarcal wrote: This raises the "Who does ensure me that the Dlls after patched via .diff will be exactly the same as the original?".
I already mentioned version control systems, which internally keep only some revisions, while other, intermediate revisions are stored as diffs.And how will the Server calculate the binary patch UserRevision-ActualRevision(e.g 58173-58174)?
Who is online
Users browsing this forum: Trendiction [Bot] and 40 guests