SB-Net: A Novel Spam Botnet Detection Scheme With Two-Stage Cascade Learner and Ensemble Feature Selection

A botnet is a type of malware that infects multiple devices and operates under the control of a botmaster to carry out malicious activities, including spamming. Detecting botnets, especially distinguishing between normal traffic, non-spam botnets, and spam botnets, remains a critical challenge in cybersecurity. Previous research has primarily focused on classifying network traffic as benign or botnet-related. However, there has been limited exploration of multiclass classification, particularly in distinguishing spam botnets, with relatively few in-depth studies on this subject. This research proposes a two-stage cascade learner classification framework combined with ensemble feature selection using rank aggregation to enhance detection accuracy. The ensemble feature selection method integrates multiple techniques, including SelectKBest (Chi-Squared, ANOVA F-Test, Mutual Information), Variance Threshold, Backwards Elimination, Recursive Feature Elimination, and SelectFromModel (tree-based), with rankings aggregated using the Borda count method. The classification process follows a two-stage approach: Stage 1 differentiates between normal and botnet traffic, while stage 2 further classifies botnet traffic into non-spam and spam botnets. The model was evaluated on three widely used datasets: CTU-13, NCC, and NCC-2. Experimental results show that using Random Forest (RF) in both classification stages and the top three selected features yields exceptional performance. This proposed method achieves an average macro precision of 99.81%, recall of 99.78%, F1-score of 99.79%, and F2-score of 99.79%, with 99.99% accuracy. Compared to previous studies, SB-Net demonstrates state-of-the-art performance, particularly in spam botnet detection. Moreover, due to its relatively high detection accuracy and low inference time, the method is well-suited for post-incident forensic analysis, enabling rapid investigation and attribution of botnet-driven spam activities in compromised networks.