Deep Learning Deployment on Big Data Infrastructure Using Apache Spark (Case Study: COVID-19 Detection Using X-Ray Images)

It is possible to use GPU (Graphic Processing Unit) to increase deep learning performance. This requires us to invest in separate GPUs, which can be relatively expensive. However, if we already have big data infrastructures, it is possible to deploy deep learning on top of them. We utilize the BigDL library on the Apache Spark cluster to run deep learning tasks. BigDL is different from traditional deep learning as it implements distributed and parallel processing. This allows for horizontal scaling of workers using BigDL, resulting in faster training times. Simulation testing on the Apache Spark cluster can use deep learning applications with the transfer learning method, leveraging pre-existing models such as InceptionVl. Deep learning can be developed using the BigDL framework. We use a case study of medical image classification for COVID19 detection. Based on the experiments, the deployment model using BigDL on the Apache Spark infrastructure achieved an average accuracy of 92%, and the average running time is 2 hours, 23 minutes, and 28 seconds.