Windows Defender Antivirus's mpengine.dll implements the core of Defender's functionality in an enormous ~11 MB, 30,000+ function DLL.
In this presentation, we'll look at Defender's emulator for analysis of potentially malicious Windows binaries on the endpoint. To the best of my knowledge, there has never been a conference talk or publication on reverse engineering any antivirus binary emulator before.
We'll cover a range of topics including emulator internals—machine code to intermediate language translation and execution; memory management; Windows API emulation; NT kernel emulation; file system and registry emulation; integration with Defender's antivirus features; the virtual environment; etc.—building custom tooling for instrumenting the emulator; tricks that binaries can use to evade or subvert analysis; and attack surface within the emulator.
Attendees will leave with an understanding of how modern antivirus software conducts emulation-based dynamic analysis on the endpoint, and how attackers might go about subverting or attacking these systems. I'll publish code for a binary for exploring the emulator from within, patches that I developed for instrumenting Defender built on top of Tavis Ormandy's loadlibrary project, and IDA scripts to help with analyzing mpengine.dll and Defender's "VDLLs
Alexei Bulazel (@0xAlexei) is a security researcher at ForAllSecure. He also provides expertise on reverse engineering and cyber policy at River Loop Security. Alexei has previously presented his research at venues such as Black Hat, REcon, and ShmooCon, among many others, and has published scholarly work at USENIX WOOT and ROOTS. Alexei is a proud alumnus of RPISEC. @0xAlexei