Networks are complex systems and too often, despite their best effort, no one knows everything about what’s going on. And most of the knowledge about the network is about typical activity. But what about the atypical activity?
There are many reasons to want to find unusual behavior in your network. The biggest reason is that it may be a sign of something new and unexpected—rather than the usual stuff—driving the activity. This doesn’t necessarily imply that a network intrusion in underway. There are many other possibilities, both innocuous and dangerous. In any case, though, unusual behavior is probably something you want to know.
There are a variety of tools related to “anomaly detection” or “outlier detection,” and this talk isn’t about any of them. Instead, this talk is an introduction to writing your own tools for detecting unusual network events. We’ll use Python, with some easily available pip installations, and look at some simple approaches to the problem that answer some interesting questions and scale well.
John O’Neil is the Data Scientist at Edgewise Networks. He writes and designs software for data analysis and analytics, search engines, natural language processing and machine learning. He has a PhD in linguistics from Harvard University, is the author of more than twenty papers in Computer Science, Linguistics, and associated fields, and has given talks at numerous professional and academic conferences.