Web scraping for automated water quality monitoring system: A case study of PDAM Surabaya

The need for a better online water quality monitoring system to control and provide drinking water treatment processes is proven. Water utility company in Surabaya that called as PDAM Surabaya has few reservoirs in their water supply system which are being monitored by WTW IQ SensorNet 2020 XT. However, on that sensor devices while it can provide some of the water quality parameters value information but the sensor is passive and the internal data is still stored in the sensor itself. To solve the problem, we proposed an application of data logger to manage data collections online water quality monitoring system by using web scraping. This application built by using Python language, the application is able to collect data from the sensor by using web scraping. We have utilized Beautiful Soap library and store the data in SQL. In these research, our experiment shows the performance of the application by measuring the accuracy of data that has been scraped from sensors are around 99%, the error rate less than 1%, and MSE (Mean Square Error) around 0.35%. We also calculated the growth of the data size that is gradually increasing and also measured the correlation between the data size with the delay which shows the value of our data is around 0.0000002739. Its means that our application is able to work in real-time and the delay is not affected by the size of data.