One of the most effective techniques used in CTF is the usage of various exploits, written with the help of well-known tools or even manually during the game. Experience in CTF participation shows that the mechanism for detecting such exploits is able to significantly increase the defense level of the team.
In this presentation we propose an approach and hybrid shellcode detection method, aimed at early detection and filtering of unknown 0-day exploits at the network level. The proposed approach allows us to summarize capabilities of shellcode detection algorithms developed over recent ten years into optimal classifiers. The proposed approach allows us to reduce the total fp rate almost to 0, provides full coverage of shellcode classes detected by individual classifiers and significantly increases total throughput of detectors. Evaluation with shellcode datasets, including Metasploit Framework 4.3 plain-text, encrypted and obfuscated shellcodes, benign Win32 and Linux ELF executables, random data and multimedia shows that hybrid data-flow classifier significantly boosts analysis throughput for benign data - up to 45 times faster than linear combination of classifiers, and almost 1.5 times faster for shellcode only datasets.
Svetlana Gaivoronski is a PhD student at Computer Systems Lab, Computer Science Dept. of Moscow State University, Russia. Svetlana is a member of the Bushwhackers CTF team which shows the following results in recent years: 2nd place in Deutsche Post Security Cup 2010, 6th place in the final of ruCTF 2012 (8th at qualification), 12th place at ruCTF Europe 2011, 4th place in the final of ruCTF 2011 (and 1st at qualification), etc. Svetlana works at Redsecure project (experimental IDS/IPS) at Moscow State University. Her primary interests are network worms propagation detection and filtering, shellcode detection, static and runtime analysis of malware. Twitter:@SadieSV lvk.cs.msu.su/~sadie
Dennis Gamayunov holds a PhD and works as Senior Researcher at Computer Systems Lab, Computer Science Dept. of Moscow State University, Russia. Dennis is the leader of the small network security research group in MSU, project lead of the experimental event-driven and natively multicore Redsecure IDS/IPS, founder of Bushwhackers CTF team, with primary research and practical interests in network level malcode detection, high-speed traffic processing (including FPGA-based), and OS security with fine-grained privilege separation, SELinux and beyond. Twitter: @jamadharma http://redsecure.ru/team/denis-gamayunov