Windows Defender Antivirus's mpengine.dll implements the core of Defender's functionality in an enormous ~11 MB, 30,000+ function DLL.
In this presentation, we'll look at Defender's emulator for analysis of potentially malicious Windows binaries on the endpoint. To the best of my knowledge, there has never been a conference talk or publication on reverse engineering any antivirus binary emulator before.
We'll cover a range of topics including emulator internals—machine code to intermediate language translation and execution; memory management; Windows API emulation; NT kernel emulation; file system and registry emulation; integration with Defender's antivirus features; the virtual environment; etc.—building custom tooling for instrumenting the emulator; tricks that binaries can use to evade or subvert analysis; and attack surface within the emulator.
Attendees will leave with an understanding of how modern antivirus software conducts emulation-based dynamic analysis on the endpoint, and how attackers might go about subverting or attacking these systems. I'll publish code for a binary for exploring the emulator from within, patches that I developed for instrumenting Defender built on top of Tavis Ormandy's loadlibrary project, and IDA scripts to help with analyzing mpengine.dll and Defender's "VDLLs