Integrated Data Flow System Using Google Cloud Services in Case Study of Open Data Bus Rapid Transit Indonesia

Bus Rapid Transit (BRT) is an integrated bus system with high passenger capacity and frequency as an efficient alternative to congestion in urban areas. The increasing number of BRT passengers results in a surge in the volume of data processed by the system. Unfortunately, only a few cities make BRT data publicly available. In addition, the structure and format of the data vary from city to city. In this research, the author tries to simplify the process by using the method of extracting data from BRT data provider websites, transforming the data, and loading it in a centralized and more secure storage location using the Airflow data orchestration tool. Airflow is run with 2 options to analyze, namely on a virtual machine and Cloud Composer with the best comparison of convenience and cost on a virtual machine with a total of $0.05 in just one month by utilizing scheduling runs of 1 hour because the average integration process only takes one minute and system setup for 15 minutes. Compared to Cloud Composer which costs $288 per month while the Airflow process time is only used once a month due to the availability of files in the data source. In addition, Google Storage in this research is used for dirty data storage, and BigQuery for an online centralized data warehouse. Overall, this research presents an efficient and cost-effective solution for managing BRT data across cities in Indonesia.