Keyed Encoders

Many encoders utilize encoding techniques which require a key value. The Call+4 Dword XOR encoder[8] and the Shikata Ga Nai polymorphic XOR additive feedback encoder[9] are examples of keyed encoders.

Key Selection

Encoders which make use of key data during their encoding process have traditionally used either random or static data chosen at the time of encoding, or data that is tied to the encoding process itself[10], such as the index value of the current position in the buffer being operated on, or a value relative to that index.

Example: Metasploit Single-byte XOR Countdown Encoder (x86)

The Metasploit Single-byte XOR Countdown Encoder[10] uses the length of the remaining payload to be operated upon as a position-dependent encoder key. The benefit that this provides is a smaller decoder stub, as the decoder stub does not need to contain any static keying information. Instead, it tracks the length property of the payload as it decodes and uses that information as the key.

Weaknesses

The most significant weakness of most keyed encoders available today is that the keying information that is used is either observable directly or constructable from the observed decoder stub. Either the static key information is transmitted within the exploit as part of the decoder stub itself, or the key information is reproducible once the encoding algorithm is known. Knowledge of the encoding algorithm is usually obtainable by recognizing known decoder stubs or analyzing unknown decoder stubs instructions in detail.

The expected inherent functionality of the decoder stub also introduces a weakness. Modern payload encoders rely upon the decoder stub's ability to properly decode the payload at run-time. It is feasible that an active observer may exploit this inherent functionality to decode a suspected payload within a sandbox environment in real-time[5, 3] in order to inspect the contents of the payload and make a control decision about the network traffic it was found in. Because the decoder stub requires only that it is being executed by a processor that will understand its instruction-set, producing such a sandbox is trivial.

Unfortunately, all of the aforementioned keyed encoders include the static key value directly in their decoder stubs and are thus vulnerable to the weaknesses described here. This allows an observer of the encoded payload in transit to potentially decode the payload and inspect it's content. Fortunately, all of the keyed encoders previously mentioned could potentially be improved to use contextual keying as is described in the following chapter.