Detection of Application-Layer Tunnels with Rules and Machine Learning

Abstract

Application-layer tunnels are often used to construct covert channels in order to transmit secret data, which is often applied to raise network threats in recent years. Detection of application-layer tunnels can assist identifying a variety of network threats, thus has high research significance. In this paper, we explore application-layer tunnel detection and propose a generic detection method by applying both rules and machine learning. Our detection method mainly consists of two parts: rule-based domain name filtering for Domain Generation Algorithm (DGA) based on a trigram model and a machine learning model based on our proposed generic feature extraction framework for tunnel detection. The rule-based DGA domain name filtering can eliminate some obvious tunnels in order to reduce the amount of data processed by machine learning-based detection, thereby, the detection efficiency can be improved. The generic feature extraction framework comprehensively integrates previous research results by combining multiple detection methods, supporting multiple layers and performing multiple feature extraction. We take the three most common application-layer tunnels, i.e., DNS tunnel, HTTP tunnel and HTTPS tunnel as examples to analyze and test our detection method. The experimental results show that the proposed method is generic and efficient, compared with other existing approaches.