Debugging the Crash

One of the many benefits of remote kernel debugging is the ability to view a stack back trace with symbol information. The vulnerability described in the previous chapter showed crashes in many different functions such as sta_add, ath_copy_scan_results, and sta_update_not_seen.

Googling these function names will reveal that many of them are present in the open source Madwifi project for Atheros based wireless hardware. They are also present in the FreeBSD net80211 project. Apple based their driver on these open-source projects. Since these projects use the BSD open-source license, Apple is not required to open their source code modifications.

While the Apple Atheros driver does not exactly match the open source projects, they match close enough to make reverse engineering much easier. The source tree for the Apple Airport driver and Madwifi are so close that the same debug flags work. Using sysctl to set the debug options on either debug.net80211 or debug.athdriver will cause a flood of diagnostic information to fill /var/log/system.log.

TestBox:~ root# sysctl debug
debug.bpf_bufsize: 4096
debug.bpf_maxbufsize: 524288
debug.bpf_maxdevices: 256
debug.iokit: 0
debug.net80211: 0 0
debug.athdriver: 0 0
TestBox:~ root# sysctl -w debug.net80211=0xffffffff
debug.net80211: 0 0 -> 2147483647 2147483647
TestBox:~ root#
TestBox:~ root# tail /var/log/system.log
Aug 5 18:07:12 TestBox kernel[0]: [en:00:1c:10:0b:d0:a1] discard 
[en:00:13:46:a8:73:c4] discard received beacon from 00:1c:10:0b:d0:a1 rssi 33
Aug 5 18:07:12 TestBox kernel[0]: [en:00:1c:10:0b:d0:a1] discard 
[en:00:13:46:a8:73:c4] discard received beacon from 00:1c:10:0b:d0:a1 rssi 33
Aug 5 18:07:12 TestBox kernel[0]: [en:00:1c:10:0b:d0:a1] discard 
[en:00:13:46:a8:73:c4] discard received beacon from 00:1c:10:0b:d0:a1 rssi 32
Aug 5 18:07:12 TestBox kernel[0]: [en:00:1c:10:0b:d0:a1] discard 
[en:00:13:46:a8:73:c4] discard received beacon from 00:1c:10:0b:d0:a1 rssi 32
Aug 5 18:07:12 TestBox kernel[0]: [en:00:1c:10:0b:d0:a1] discard 
[en:00:13:46:a8:73:c4] discard received beacon from 00:1c:10:0b:d0:a1 rssi 31
Aug 5 18:07:12 TestBox kernel[0]: [en:00:1c:10:0b:d0:a1] discard 
[en:00:13:46:a8:73:c4] discard received beacon from 00:1c:10:0b:d0:a1 rssi 32
Aug 5 18:07:12 TestBox kernel[0]: [en:00:1c:10:0b:d0:a1] discard 
[en:00:13:46:a8:73:c4] discard received beacon from 00:1c:10:0b:d0:a1 rssi 32
Aug 5 18:07:12 TestBox kernel[0]: [en:00:1c:10:0b:d0:a1] discard 
[en:00:13:46:a8:73:c4] discard received beacon from 00:1c:10:0b:d0:a1 rssi 31
Aug 5 18:07:12 TestBox kernel[0]: [en:00:1c:10:0b:d0:a1] discard 
[en:00:13:46:a8:73:c4] discard received beacon from 00:1c:10:0b:d0:a1 rssi 31
TestBox:~ root#

One can read what each bit does and how they can be set using the debug tools found in the tools directory of the Madwifi source tree. The open-source 80211debug.c file corresponds to Apple's debug.net80211 module and athdebug.c corresponds to debug.athdriver. An enum found at the top of each debug source file defines the bit mask and what functionality it enables. You can activate all debugging functionality by setting the bit field to 0xffffffff. However, when doing this, a problem arises due to the large amount of data written to the log file. The function that performs the logging, IOLog, cannot always keep up with the flood of messages and does not know or care if a write is unsuccessful. For this reason, targeting a specific function may give more information and help to ensure that it is not buried under a wave of data. For instance, the following command will only show debug messages that involve the scanning code where this vulnerability occurs.

sysctl –w debug.net80211=0x00200000

If one does not want to remember the bit fields, the Madwifi tools required only minor tweaks to work with OS X, and the source is in the accompanying tar ball with other examples for this paper.

The task of kernel debugging ultimately rests with gdb which is not well-suited for the job. Those people who learned kernel hacking with SoftICE will be unhappy with gdb. It lacks basic debugger functionality such as the ability to search through memory. Tracepoints do not work nor do hardware breakpoints. However, it makes up for the lack of built-in functionality with the ability to script and the ability to set commands to execute after a breakpoint is reached. Stringing a lot of these features together makes it possible to hack together tools that help to supplement missing features. A short list of helpful tricks discovered during the use of gdb are included in the following sections.

Subsections