One of the most effective techniques used in CTF is the usage of various exploits, written with the help of well-known tools or even manually during the game. Experience in CTF participation shows that the mechanism for detecting such exploits is able to significantly increase the defense level of the team.
In this presentation we propose an approach and hybrid shellcode detection method, aimed at early detection and filtering of unknown 0-day exploits at the network level. The proposed approach allows us to summarize capabilities of shellcode detection algorithms developed over recent ten years into optimal classifiers. The proposed approach allows us to reduce the total fp rate almost to 0, provides full coverage of shellcode classes detected by individual classifiers and significantly increases total throughput of detectors. Evaluation with shellcode datasets, including Metasploit Framework 4.3 plain-text, encrypted and obfuscated shellcodes, benign Win32 and Linux ELF executables, random data and multimedia shows that hybrid data-flow classifier significantly boosts analysis throughput for benign data - up to 45 times faster than linear combination of classifiers, and almost 1.5 times faster for shellcode only datasets.