Blog: Google SoC lwIP Report Week 10

Here you can discuss ReactOS related topics.

Moderator: Moderator Team

zhu48
Posts: 18
Joined: Fri Jun 03, 2016 5:28 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by zhu48 »

Regarding mTCP being a user library...
But how much is the first one relevant to ReactOS?
The problem with a user-mode library is that there's no quick and easy way to turn it into a good driver. You can re-wrap it into a standard API like WinSock, or you can port the algorithms into kernel code.

Wrapping the whole thing into WinSock would be complicated as all hell, as there are probably all sorts of things that don't play nice with each other on a design level. For example, mTCP probably does direct system calls, while WinSock actually delegates a lot of functionality to the AFD DLL. Just with that difference, we're talking about trying to split mTCP into two different parts. Imagine the myriad of smaller things that could go wrong when trying to wrap a library that wasn't designed to be wrapped.

The second solution, porting the algorithms, is extremely labor-intensive. First, one must read through and thoroughly understand the code one is trying to port. Second, one must turn user code into kernel code. User code can take all sorts of liberties because they enjoy access to standard libraries and memory protection. Since all kernel code runs in the same address space, kernel code must also obey some very strict rules to not crash the rest of the system or cause huge performance problems. For one, kernel code needs to understand exactly when it is allowed to deallocate a chunk of memory, something user code can have some liberty with. Locking also works very differently in kernel mode, since you can directly control processor state, thereby affecting interrupts and scheduling. Porting any algorithm from user to kernel mode requires a lot of thought and a lot of testing.
I tried to accent namely this -- maybe you or someone else could get some nice ideas for handling concurency. Eg. the use of Shared Memory seemed to me handy in this aspect. I am sorry if I failed.
Isn't that what this forum is for, discussion? I don't think you failed anything, I think it was good of you to bring up mTCP and start a productive discussion like this. I certainly learned a few important things while reading that paper, and I'm sure you've learned a few things from all the other posts people have made here.
Wish you best luck in bringing clarity on the subject! And have a great weekend:)
Thank you!
CircularTriangle06
Posts: 32
Joined: Sat May 23, 2015 5:54 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by CircularTriangle06 »

I've been working since then, and this issue you're seeing may or may not have been fixed already.
The problem is fixed.
Encountered a BSOD trying to ping 127.0.0.1 (log https://yadi.sk/i/fUWaF2LsuMvGo )
if changed the tag there, ping localhost works - but ping network address fails (log https://yadi.sk/i/jZcEQ7LUuMxE9 )
saulius2
Posts: 18
Joined: Sat Jan 17, 2009 4:01 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by saulius2 »

zhu48 wrote:Regarding mTCP being a user library...
[...] and I'm sure you've learned a few things from all the other posts people have made here.
Definitelly yes. My main omission, I guess, was the surrounding layers of networking stack -- these below TCP/IP (NDIS) and ones above (AFD, TDI).

Then differences in the conditions of acting in kernel-mode gave even more fundamentals to me. Thanks, zhu48. I'm glad and sad at the same time to see your GSoC session is over : )

Congrats!
goldsimon
Posts: 1
Joined: Sat Sep 16, 2017 9:20 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by goldsimon »

Guys, sorry to respond to such an old thread (I hope someone reads this at all).

Has anyone of you bothered to ask the lwIP developers for help in integrating lwIP into reactos? I know lwIP started as "lightweight" in the sense of small microprocessors in 2003. But nowadays, multithreading is used in microprocessors, too. As such, I could definitively imagine support for multithreading and parallelism built into the stack...

lwIP is BSD licensed, but that doesn't mean you *have* to copy it without speaking to the authors/maintainers ;-)

Cheers,
Simon (maintainer of lwIP)
cb88
Posts: 73
Joined: Fri Oct 06, 2017 3:26 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by cb88 »

https://github.com/m-labs/smoltcp

Seeing as Mozilla has seen massive improvements in their ability to effectively write threaded code in Firefox with release 57 (it's way faster on the lowish end machines I've tried) and they designed Rust to live seamlessly with C code.

Why not use a TCP/IP layer written in Rust... from what I can tell memory access violations are one of the biggest annoyances of ReactOS. It probably wouldn't be a bad idea to port the RedoxOS memory managment as well... sometimes C is a bad tool for the job relative to it's modern peers even, and yes especially at the system level (ie as soon as you start doing dynamic allocation). It's the whole reason MISRA C exists and that's a pain to have to deal with... to attain a good level of nearly guaranteed stability... it's better to just use a tool that does more of that for you, and leaves you able to be more creative with your code than having to spend time figuring out where hard to track down problems are manually.
Last edited by cb88 on Fri Oct 06, 2017 7:08 pm, edited 2 times in total.
User avatar
dizt3mp3r
Posts: 1874
Joined: Mon Jun 14, 2010 5:54 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by dizt3mp3r »

Prepare for incoming fire...
Skillset: VMS,DOS,Windows Sysadmin from 1985, fault-tolerance, VaxCluster, Alpha,Sparc. DCL,QB,VBDOS- VB6,.NET, PHP,NODE.JS, Graphic Design, Project Manager, CMS, Quad Electronics. classic cars & m'bikes. Artist in water & oils. Historian.
milon
Posts: 969
Joined: Sat Sep 05, 2009 9:26 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by milon »

Pro Tip: Anyone unknown to the ReactOS project (ie. people with post count less than, say, 100) really should attempt to provide references etc when making claims that the project should move in an unexpected direction.
cb88
Posts: 73
Joined: Fri Oct 06, 2017 3:26 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by cb88 »

IwIP is an embedded library... sure its great but as ReactOS developers themselves have already said it isn't ideal to build a fully featured IP stack around as it is lacking in a few minor but difficult to change areas.

Also, how many times has memory management and several other large features been rewritten... a language that statically verifies safety at compile time would be a godsend in those areas. I don't think you are going to find ANYONE that would dispute that... even though they may not like Rust itself.

ProTip: nobody likes "ProTips" , as far as incoming fire... whatever I have no interest in anyone else or even my own ego... its just a suggestion after all. Also, it is something that did not exist as an option when the lwIP work started, my post count may be one here.... but I'm in the top 12% on the reverse engineering stack overflow and have commented for many years on several other forums so whatever I've also been reading the blog posts since around 3.1 or so I just never bothered to post as I didn't have the free time to get involved.
PurpleGurl
Posts: 1790
Joined: Fri Aug 07, 2009 5:11 am
Location: USA

Re: Blog: Google SoC lwIP Report Week 10

Post by PurpleGurl »

I agree that LwIP needs rewritten as it is not inherently thread safe, at least not in all functions. What we have now is some sort of semaphore system to control the gating into the function. At least it works, but it's not ideal. Really, the functions should have all that safety themselves, not rely on wrappers to do so. But as long as it works and nobody is volunteering to rewrite them, then we should be satisfied for now.

As for Rust and other languages, that has already been discussed before. Now, if you can get up a team willing to maintain your Rust functions, then you'd have a greater chance of moving language targets. However, C and C++ tends to be the most portable, so unless you have Rust compilers for other architectures, then such a arrangement could prevent your code from compiling on ARM and other non-PC platforms.
Last edited by PurpleGurl on Sat Oct 07, 2017 4:19 pm, edited 1 time in total.
middings
Posts: 1073
Joined: Tue May 07, 2013 9:18 pm
Location: California, USA

Re: Blog: Google SoC lwIP Report Week 10

Post by middings »

I like Pro Tips.
cb88
Posts: 73
Joined: Fri Oct 06, 2017 3:26 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by cb88 »

PurpleGurl wrote:I agree that LwIP needs rewritten as it is not inherently thread safe, at least not in all functions. What we have now is some sort of semaphore system to control the gating into the function. At least it works, but it not ideal. Really, the functions should have all that safety themselves, not rely on wrappers to do so. But as long as it works and nobody is volunteering to rewrite them, then we should be satisfied for now.

As for Rust and other languages, that has already been discussed before. Now, if you can get up a team willing to maintain your Rust functions, then you'd have a greater chance of moving language targets. However, C and C++ tends to be the most portable, so unless you have Rust compilers for other architectures, then such a arrangement could prevent your code from compiling on ARM and other non-PC platforms.
I agree on your first point code that works now shouldn't really have a lynch mob going after it especially with it being a good improvement over the previous code.... I was merely pointing out that Rust is becoming a viable road to go down. I wouldn't be surprised if 3 or 4 years down the road this code gets rewritten and maybe again after all that as such is the nature of code that mostly works but tickles an itch. Personally I think it would be a good investment of time to investigate a method that would probably be fun for someone to work on and would more concretely fix the problems. "The Problems" mostly being the very things Rust makes easier.

As far as Rust for other architectures I'm not sure what you mean... Rust isn't any less portable that C/C++ and is slowly acreteing support for more oddball architectures say like Sparc... x86 and ARM though are basically first class architectures so its like a non issue? And ReactOS is a PC architecture... so you basically get the compilers for free. I imagine this was discussed awhile back before rust had a stable release... probably valid points then but basically those issues have been surpassed now. Also, as far as Rust code running on ReactOS that has to happen if it doesn't already work as Firefox 57 has a very large amount of Rust code in it (the entire CSS engine and then some). This probably means that ReactOS needs to move toward targeting Window 7 compatibility but that is probably a lot of work but again its pretty much a given that this must happen someday.
PurpleGurl
Posts: 1790
Joined: Fri Aug 07, 2009 5:11 am
Location: USA

Re: Blog: Google SoC lwIP Report Week 10

Post by PurpleGurl »

cb88 wrote:As far as Rust for other architectures I'm not sure what you mean... Rust isn't any less portable that C/C++ and is slowly acreteing support for more oddball architectures say like Sparc... x86 and ARM though are basically first class architectures so its like a non issue? And ReactOS is a PC architecture... so you basically get the compilers for free. I imagine this was discussed awhile back before rust had a stable release... probably valid points then but basically those issues have been surpassed now. Also, as far as Rust code running on ReactOS that has to happen if it doesn't already work as Firefox 57 has a very large amount of Rust code in it (the entire CSS engine and then some). This probably means that ReactOS needs to move toward targeting Window 7 compatibility but that is probably a lot of work but again its pretty much a given that this must happen someday.
Okay, I didn't know it was that popular. I mention other architectures since ROS targets more than x86, though most of the effort is on x86. There is ARM, x64, and Xbox support too. So you don't want to change to something that might not build on the others.

As for discussions, I mean even earlier this year. Most of the coders code only in c, C++, and/or assembly. Plus you can't have just one person using some oddball language in an area of codel If they leave, then who will maintain it? And it would be labor intensive to convert it to C or C++ by hand (and the team won't accept machine generated code).
val
Posts: 69
Joined: Fri Feb 10, 2017 5:22 am

Re: Blog: Google SoC lwIP Report Week 10

Post by val »

what to answer when someone shows up and begins a "very interesting" lecture on replacement of C with Rust/Dust/javascript/yetanotherFirefoxtoy or whatever "new superior cool thing(R)"?
Emmm.
[ external image ]
middings
Posts: 1073
Joined: Tue May 07, 2013 9:18 pm
Location: California, USA

Re: Blog: Google SoC lwIP Report Week 10

Post by middings »

The talk about the Rust language is moving this forum thread off topic. A discussion about experimentally re-implementing some ReactOS modules in Rust deserves its own topic.

"Talk is cheap. Show me the code."--Linus Torvalds
cb88
Posts: 73
Joined: Fri Oct 06, 2017 3:26 pm

Re: Blog: Google SoC lwIP Report Week 10

Post by cb88 »

@middings quite right the intention was not to ruffle feathers or derail the topic. But such is life.

As far as to the point that most programmers interested only know asm, C and C++... That's a disservice to themselves as sticking to just those languages means alot of the time you are using the wrong tool for the job.. And honestly C++ is too complex to really know well... People that think they know it well are like the sysadmin that thinks he doesn't need backups and can do production on a live system. Which is part of why Rust does away with alot of the things that make C++ complex it's a middle ground language that gives you some up front pain so you don't shoot your foot off etc...

You mention rust code being harder to find maintainers for I would counter that with, is there even any intention to maintain this code or is it just a stop gap untill something better can replace it...as that is historically the case on reactos. I realize that as a reverse engineered is this is inevitable possibly even most of the time but it is still something nto be aware of and to try to avoid.
Post Reply

Who is online

Users browsing this forum: No registered users and 43 guests