[ros-kernel] Kernel Debugger doesn't correctly identify all types of calls

[Gregor Anich]

Hi!

Today I have tested KDB a bit and it works well but I have seen it 
stepping over indirect calls like the following code:
FF 15 F4 68 87 69

This is the code which is checking wether the current instruction is a 
call to decide wether to step over it (break after the instruction and 
continue) or single-step:

  /* Check if the current instruction is a call. */
  while (Eip[0] == 0x66 || Eip[0] == 0x67)
  {
    Eip++;
  }
  if (Eip[0] == 0xE8 || Eip[0] == 0x9A || Eip[0] == 0xF2 || Eip[0] == 
0xF3 ||
      (Eip[0] == 0xFF && (Eip[1] & 0x3C) == 0x10))
  ...

Reading the Intel instruction reference I found out that 0xFF is the 
primary opcode,
0x15 is the ModR/M byte and F4 68 87 69 is the address (0x698768F4)
The ModR/M byte consists of the following parts:
bits 0..2  R/M
bits 3..5 Opcode/Reg
bits 6..7 Mod

 From the Intel Manual: "The mod field combines with the r/m field to 
form 32 possible values: eight registers and
24 addressing modes."

So for the previously mentioned instruction the ModR/M byte is 
0b00010101/0x15 (Mod = 0, Opcode = 2, R/M = 5)
I think the only thing relevant for the debugger to identify wether this 
is a call or not is the Opcode part of  the ModR/M byte but the check 
(Eip[1] & 0x3C) == 0x10 masks bits 2..5 (0x3C = 0b00111100) which 
includes the Opcode and the highest bit of R/M - I think it should be 
0x38/0b00111000

The code is in ntoskrnl/dbg/kdb.c, line 677

If anybody who knows x86 machine code can confirm this it would be nice 
if someone changes the code.

 --blight