Automatic detection of inconsistencies between MOOC text reviews and ratings based on BERT+RNN model

Guoxia Dong; Tingmei Wang; Yunfei Xu

doi:10.63313/IJED.9092

Authors

Guoxia Dong Beijing Union University, Beijing 100101, China Author
Tingmei Wang Beijing Union University, Beijing 100101, China Author
Yunfei Xu Beijing Union University, Beijing 100101, China Author

DOI:

https://doi.org/10.63313/IJED.9092

Keywords:

MOOCs, Text Sentiment Analysis, BERT Model, Inconsistency Detection

Abstract

To address the inconsistency between text reviews and star ratings on Chinese university MOOC platforms, this paper proposes a BERT+RNN-based automatic detection and correction model. The study collected 343519 course reviews and constructed a five-category sentiment labeling system (class 1–class 5) through manual annotation. The performance of three deep learning models, namely, a CNN, an RNN, and a BERT+RNN, was compared. Experimental results show that the BERT+RNN model achieved an accuracy of 93.7% on the test set, outperforming other models. The model was used to automatically correct the ratings of all reviews, with a total of 27794 reviews (8.1% of the total), representing different ratings between the original and model-generated annotations. Subject and course satisfaction rankings were analyzed based on the corrected ratings. This study provides an effective technical approach for improving the reliability of MOOC evaluation data and platform governance, and has strong practical application value.

References

[1] Papadakis S. MOOCs 2012-2022: an overview[J]. Advances in Mobile Learning Educational Research, 2023, 3(1): 682-693.

[2] Moore R L, Blackmon S J. From the learner's perspective: A systematic review of MOOC learner experiences (2008–2021)[J]. Computers & Education, 2022, 190: 104596.

[3] Salama R, Hinton T. Online higher education: Current landscape and future trends[J]. Journal of Further and Higher Education, 2023, 47(7): 913-924.

[4] Huang H, Qi D. Is MOOC really effective? Exploring the outcomes of MOOC adoption and its influencing factors in a higher educational institution in China[J]. Plos one, 2025, 20(2): e0317701.

[5] Cheng J, Yuen A H K, Chiu D K W. Systematic review of MOOC research in mainland China[J]. Library Hi Tech, 2023, 41(5): 1476-1497.

[6] Alturkistani A, Lam C, Foley K, et al. Massive open online course evaluation methods: systematic review[J]. Journal of medical Internet research, 2020, 22(4): e13851.

[7] Mishra M, Dash M K, Sudarsan D, et al. Assessment of trend and current pattern of open educational resources: A bibliometric analysis[J]. The Journal of Academic Librarianship, 2022, 48(3): 102520.

[8] Su P Y, Guo J H, Shao Q G. Construction of the quality evaluation index system of MOOC platforms based on the user perspective[J]. Sustainability, 2021, 13(20): 11163.

[9] Mehta P, Pandya S. A review on sentiment analysis methodologies, practices and applications[J]. International Journal of Scientific and Technology Research, 2020, 9(2): 601-609.

[10] Liu, J., Tan, J. Q., Zhang, J. H., et al. Research on Text Analysis of Emotional Dictionaries for Mathematics Courses Based on MOOC Platforms[J]. Science & Education Guide, 2021(19): 78-80. DOI: 10.16400/j.cnki.kjdk.2021.19.026.

[11] Graves A. Long short-term memory[J]. Supervised sequence labelling with recurrent neural networks, 2012: 37-45.

[12] Cho K, Van Merriënboer B, Gulcehre C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[J]. arXiv preprint arXiv:1406.1078, 2014.

[13] Sun, S. Q. Research on Learners’ Emotional Tendency Analysis Based on Course Evaluation in the MOOC Platform[D]. Liaoning Normal University, 2023. DOI: 10.27212/d.cnki.glnsu.2023.000229.

[14] Chen X, Zou D, Cheng G, et al. Deep neural networks for the automatic understanding of the semantic content of online course reviews[J]. Education and Information Technologies, 2024, 29(4): 3953-3991.

[15] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.

[16] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.

[17] Cavalcanti A P, Mello R F, Gašević D, et al. Towards explainable prediction feedback messages using BERT[J]. International Journal of Artificial Intelligence in Education, 2024, 34(3): 1046-1071.

[18] Cavalcanti A P, Mello R F, Miranda P, et al. Utilização de recursos linguísticos para classificação automática de mensagens de feedback[C]//Simpósio Brasileiro de Informática na Educação (SBIE). SBC, 2021: 861-872.

[19] Wei, X. C., Yu, L. Construction and Application of Emotional Recognition Corpus for Chinese MOOC Reviews[J]. Journal of Chongqing University of Technology (Natural Science), 2023, 37(04): 174–181.

[20] Liu, S. N. Y., Peng, X., Liu, Z., et al. Research on Learner Topic Mining for MOOC Course Reviews[J]. Journal of Educational Technology, 2017, 38(10): 30–36. DOI: 10.13811/j.cnki.eer.2017.10.005.

[21] Zhang, X. X., Duan, Y. H. MOOC Quality Evaluation Based on Learners’ Online Review Texts: A Case Study of “Chinese University MOOC” Reviews[J]. Modern Educational Technology, 2020, 30(09): 56–63. DOI: CNKI:SUN:XJJS.0.2020-09-009.

[22] Chen X, Zou D, Xie H, et al. Automatic Classification of Online Learner Reviews Via Fine-Tuned BERTs[J]. International Review of Research in Open and Distributed Learning, 2025, 26(1): 57-79.