Scalable Teacher-Forcing Networks under Spark Environments for Large-Scale Streaming Problems

Large-scale data streams remains an open issue in the existing literature. It features a never ending information flow, mostly going beyond the capacity of a single processing node. Nonetheless, algorithmic development of large-scale streaming algorithms under distributed platforms faces major challenge due to the scalability issue. The network complexity exponentially grows with the increase of data batches, leading to an accuracy loss if the model fusion phase is not properly designed. A largescale streaming algorithm, namely Scalable Teacher Forcing Network (ScatterNet), is proposed here. ScatterNet has an elastic structure to handle the concept drift in the local scale within the data batch or in the global scale across batches. It is built upon the teacher forcing concept providing a short-term memory aptitude. ScatterNet features the data-free model fusion approach which consists of the zero-shot merging mechanism and the online model selection. Our numerical study demonstrates the moderate improvement of prediction accuracy by ScatterNet while gaining competitive advantage in terms of the execution time compared to its counterpart.