Enhancing Anomaly Classification Over Log Files through Topic Modeling and Ensemble Methods

Log files play a vital role in monitoring system changes, and their usage is rapidly increasing in cloud computing. To effectively address this challenge, text processing is an essential step in extracting information from unstructured text data, specifically log files, and transforming it into identifiable patterns using specific methods. Given the brevity of most log file text, this study focuses on the application of short text topic modeling and ensemble methods to classify log files andextract meaningful insights. The findings demonstrate that anomaly detection using short-text topic modeling and ensemble methods surpasses the benchmark method of Latent Dirichlet Allocation (LDA) topic modeling. Notably, the classification approach utilizing GSDMM,in combination with experiments involving XGBoost, achieves the highest performance when compared to other ensemble methods such as Random Forest, Gradient Boosting, andAdaBoost. To further optimize the performance of the XGBoost method in anomaly detection classification, hyperparameter tuning is conducted using Optuna. This approach effectively identifies the most optimal hyperparameters for XGBoost, leading to enhanced performance. Overall, this research illustrates that the utilization of short text topic modeling and ensemble methods, along with hyperparameter optimization, significantly improves the accuracy and effectiveness of anomaly detection in log file classification.