|Informative Information for the Uninformed|
In order to combine Mach and BSD into one kernel (xnu), syscall numbers are divided into different tables. On a PowerPC system, when the ``sc'' instruction is executed, the syscall number is stored in r0 and used to determine which syscall to execute. Positive syscall numbers (smaller than 0x6000) are treated as FreeBSD syscalls. In this case the sysent table is offset and the appropriate function pointer is used.
In cases where the syscall number is greater than 0x6000, PPC specific syscalls are used and the ``PPCcalls'' table is offset. However, in the case of a negative syscall number, the mach_trap_table is indexed and used.
The code below is taken from the xnu source and shows this process.
oris r15,r15,SAVsyscall >> 16 ; Mark that it this is a ; syscall cmplwi r10,0x6000 ; Is it the special ppc-only ; guy? stw r15,SAVflags(r30) ; Save syscall marker beq-- cr6,exitFromVM ; It is time to exit from ; alternate context... beq-- ppcscall ; Call the ppc-only system ; call handler... mr. r0,r0 ; What kind is it? mtmsr r11 ; Enable interruptions blt-- .L_kernel_syscall ; System call number if ; negative, this is a mach call... lwz r8,ACT_TASK(r13) ; Get our task cmpwi cr0,r0,0x7FFA ; Special blue box call? beq-- .L_notify_interrupt_syscall ; Yeah, call it...
On an Intel system, things are a little different. The ``int 0x81'' (cd 81) instruction is used to call Mach traps. The ``sysenter'' instruction is used for the BSD syscalls. However, the syscall number convention remains the same. The eax register is used to store the syscall number in either case.
It seems that most people developing shellcode on Mac OS X stick to using the FreeBSD syscalls. This may be due to lack of familiarity with Mach traps, so hopefully this paper is useful in re-mediating that. I have extracted a list of the Mach traps in the mach_trap_table from the xnu kernel. (792.6.22).