A Comparative Study of Statistical and Machine Learning Models for Price Prediction: A Case Study for Agricultural Dataset

Traditional econometric models, such as the Fixed Effects Model (FEM) and Random Effects Model (REM), effectively address structured datasets by capturing fixed and inter-group variability but face limitations in handling non-linear relationships, multicollinearity, and temporal dynamics. This study evaluates the performance of FEM and REM against advanced machine learning (ML) models like XGBoost and CatBoost and deep learning (DL) models, including Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), on agricultural panel datasets. Performance is assessed using metrics like Root Mean Square Error (RMSE), Mean Square Error (MSE), and Mean Absolute Error (MAE). The results highlight FEM's robustness for datasets with simpler or moderately complex structures, such as those associated with crops like Soya and Corn. However, ML models demonstrated better capacity to manage non-linearities, albeit requiring substantial feature engineering to leverage cross-sectional and temporal attributes. DL models showed promise in capturing temporal dependencies but faced challenges with cross-sectional heterogeneity. The study underscores the importance of aligning model selection with dataset characteristics. While FEM remains a reliable choice for panel data, ML and DL methods hold transformative potential when coupled with advanced architectures like hierarchical models or attention mechanisms. Future research could explore hybrid approaches that integrate statistical and advanced learning techniques to address the diverse complexities in agricultural datasets, paving the way for more accurate, scalable, and efficient predictive solutions in precision agriculture.