Hybrid SciBERT + Multi-LLM Ensemble for Automated Citation Validation: A Two-Stage Approach

Citation validation in scientific writing is labor-intensive, requiring researchers to manually verify the relevance and appropriateness of references. We propose a two-stage architecture that combines SciBERT semantic filtering with a heterogeneous five-model LLM ensemble for automated citation validation. Stage 1 uses SciBERT to pre-filter high-similarity candidates from 128 IEEE papers, reducing 6.89 million potential pairs to 252 candidates (99.9963% reduction). Stage 2 validates candidates through majority voting across five architecturally diverse LLMs (LLaMA3-8B, Qwen2.5-14B, Gemma2-27B, Mixtral-8x7B, Dolphin-Mixtral-8x7B), achieving a 62.70% validation rate. The ensemble demonstrated substantial inter-rater agreement (Fleiss' κ=0.698, Cohen's κ=0.7125) and high stability (94.76% leave-one-out consistency), with 81.01% of decisions receiving strong majority support (≥4/5 votes). A human validation study with four domain experts (n = 100) revealed that inter-LLM agreement (κ= 0.698) exceeded inter-human agreement (κ= 0.4653), whereas human-LLM alignment reached 70.8% (mean κ= 0.2025). Notably, adding human raters decreased the measured agreement (Fleiss' κ: 0.5436→0.4653), highlighting the inherent subjectivity in citation validation. Ablation studies validate the design choices for voting thresholds and ensemble sizes. This architecture offers a rigorous approach to automated citation validation, with applications as decision-support tools in manuscript preparation systems, complementing rather than replacing human expert judgment. It is important to note that inter-model agreement indicates consistency in assessment rather than ground-truth correctness of citations.