In most academic and industry conferences, we get to learn about the success stories of security analytics systems but we rarely explore what to do when these systems don’t work as intended. This talk address the gap: we share the lessons learned from deploying failed machine learning intrusion detection systems and more importantly, how to fix them. Specifically, we will focus on the unsuccessful experiments when attempting to solve two important security problems: detecting lateral movement in the cloud environment and identifying geo login anomalies. This talk has three parts wherein in each section we first explain the security problem, present the approach that failed to produce the desired outcome, followed by a deep dive into why the system failed and finally conclude with how we fixed the machine learning system. The goal of this lecture is to emphasize that it is important to recognize that security data science systems are imperfect, and share different options to proceed when security analytics experiments don’t go as expected. This talk represents the work from Microsoft Research, Azure Security Data Science and the Microsoft’s Cyber Defense Operations Center.