Honeynet Reverse Engineering Challenge

Posted by Bagpuss on August 04, 2012
Tags: honeynet, digital forensics, reverse engineering

Recently, I've succeeded in coming joint first in Honeynet challenge 11: Dive Into Exploit by Georg Wicherski. Both of the winning answers are worthwhile reading as they supply highly complementary analyses (Ruud rightly pipped myself at the post here as he managed to succeed in getting gdlog to behave).

Presented with a network packet capture file, that appeared to be the encrypted contents of a secure telnet session, we were tasked with the problem of extracting and then reverse engineering the malicious code actions that had created this network traffic.

In this instance, the malicious code was delivered via a series of stages designed to frustrate and hinder analysts. Reversing stage 2 of the malware's delivery, provided evidence of its armoured capabilities. Namely:

  • stages 3 and 4 had network communications, at least from the compromised machine to the delivery server, encrypted using the Rabbit stream cipher
  • the Rabbit stream cipher was initialised using data fed (via a Diffie-Hellman like key exchange) to the compromised machine via the delivery server.

So, beyond an infeasible brute forcing attempt against the Rabbit stream cipher, the only realistic way to crack these communication layers was to reverse engineer the initialisation data structure for the Rabbit stream cipher. As this data structure was being read from the delivery server, and this information was being communicated using a Diffie-Hellman like encryption scheme, our problem now reduced to calculating discrete logarithms in a 384 bit prime multiplicative group.

The Wikipedia article Discrete Logarithm Records, soon betrayed how computationally difficult this problem would be! Read my full answer writeup to Challenge11 in order to learn more about how I fared with this problem.

It is also worth noting that Georg Wicherski plans to talk more about this malware at T2 (see Secure Exploit Payload Staging).

Finally, thanks again to all the hard work from everyone at the Honeynet Project in producing such an interesting and stimulating challenge.