Design

Segmentation is one of the easiest ways to trap memory accesses. The majority of instructions which reference memory implicitly use either the DS or ES segment registers to do so. The one exception to this rule are instructions that deal with the stack. These instructions implicitly use the SS segment register. There are a few different ways one can go about causing a general protection fault when accessing an address relative to a segment selector, but one of the easiest is to take advantage of the null selector. The null selector, 0x0, is a special segment selector that will always cause a general protection fault when using it to reference memory. By loading the null selector into DS, for example, the mov [eax], 0x1 instruction would cause a general protection fault when executed. Using the null selector solves the problem of being able to intercept memory accesses, but there still needs to be some mechanism to allow the application to execute normally after intercepting the memory access.

When a general protection fault occurs in user-mode, the kernel generates an access violation exception and passes it off to the user-mode exception dispatcher in much the same way as was described in 2.2. Registering a custom exception handler makes it possible to catch this exception and handle it gracefully. To handle this exception, the custom exception handler must restore DS and ES segment registers to valid segment selectors by updating the thread context record associated with the exception. On 32-bit versions of Windows, the segment registers should be restored to 0x23. Once the the segment registers have been updated, the exception dispatcher can be told to continue execution. However, before this happens, there is an additional step that must be taken.

It is not enough to simply restore the segment registers and then continue execution. This would lead to subsequent reads and writes being missed as a result of the DS and ES segment registers no longer pointing to the null selector. To address this, the custom exception handler should toggle the trap flag in the context record prior to continuing execution. Setting the trap flag will cause the processor to generate a single step exception after the instruction that generated the general protection fault executes. This single step exception can then be processed by the custom exception handler to reset the DS and ES segment registers to the null selector. After the segment registers have been updated, the trap flag can be disabled and execution can be allowed to continue. By following these steps, the application is able to make forward progress while also making it possible to trap all memory reads and writes that use the DS and ES segment registers.