Anomaly Detection in System Event Logs

Abstract

In this work, we explore approaches for detecting anomalies in system event logs. We define the system log anomaly detection problem and research existing methods. We apply the methods to a practical task of detecting anomalous events in logs of file behavior analysis sandbox. To validate results and compare methods we calculate quality metrics on a manually labeled dataset. First, we try an approach based on calculating event document frequency and use it as a baseline. We improve it by creating an event normalization algorithm and significantly reducing the number of false positives. After that, we implement a different approach that involves extracting event features and training random forest and logistic regression models to model a probability of an event belonging to a clean or anomalous log. Finally, we create a sequence model based on a recurrent neural network and use it to detect anomalies in event sequences.