Enhancing Breast Cancer Detection: Optimizing YOLOv8's Performance Through Hyperparameter Tuning

Abnormality recognition and precise location designation are crucial to determine the precision value achieved in breast cancer detection. Our research aimed to improve the performance of the YOLOv8 Model by optimizing the best hyperparameters in detecting masses in breast cancer. Mean Average Precision (mAP) was used to measure the effectiveness of the model. This research develops three optimization methods namely Adaptive Moment Estimation (Adam), Stochastic Gradient Descent (SGD), and Root Mean Square Propagation (RMSprop) as the main focuses of YOLOv8 optimization. For the experimental stage, we used the Cancer Imaging Archive (TCIA) Public Access Digital Database of Mammography (CDD-CESM) data with 2,085 images after the data augmentation process. The dataset was divided into three categories: benign, malignant, and normal. Our experimental results show that the SGD optimizer outperforms the other optimizers with the shortest training time of about 2 hours and 35 minutes. The highest mAP value for detecting normal is 0.939, benign is 0.762, and malignant is 0.911. From the results obtained, the proposed model can detect breast cancer with a good level of accuracy and efficiency in training time. The contribution of this research is that the detection results obtained are expected to help radiologists in making a diagnosis.