Journal Home > Volume 4 , Issue 1
Journal of Social Computing 2023, 4(1): 12-29 https://doi.org/10.23919/JSC.2023.0007
Open Access | Issue | Published: 30 March 2023

Prediction of Academic Performance of Students in Online Live Classroom Interactions —An Analysis Using Natural Language Processing and Deep Learning Methods

Show Author's Information Yuanyi Zhen1,  Jar-Der Luo1( ),  Hui Chen2
Laboratory of Computational Social Science and National Governance, Tsinghua University, Beijing 100084, China
School of Chinese Language and Literature, Beijing Foreign Studies University, Beijing 100089, China
Keywords:
natural language processing, deep learning, academic performance prediction, live classroom dialogue, emotional expression, interactive type
Cite this article:
Zhen Y, Luo J-D, Chen H. Prediction of Academic Performance of Students in Online Live Classroom Interactions —An Analysis Using Natural Language Processing and Deep Learning Methods. Journal of Social Computing, 2023, 4(1): 12-29. https://doi.org/10.23919/JSC.2023.0007
Download citation
EndNote(RIS)
BibTeX

655

Views

37

Downloads

Citations

0

Crossref

N/A

WoS

0

Scopus

N/A

CSCD

Abstract Full text About this article

Prior studies have shown the importance of classroom dialogue in academic performance, through which knowledge construction and social interaction among students take place. However, most of them were based on small scale or qualitative data, and few has explored the availability and potential of big data collected from online classrooms. To address this issue, this paper analyzes dialogues in live classrooms of a large online learning platform in China based on natural language processing techniques. The features of interactive types and emotional expression are extracted from classroom dialogues. We then develop neural network models based on these features to predict high- and low-academic performing students, and employ interpretable AI (artificial intelligence) techniques to determine the most important predictors in the prediction models. In both STEM (science, technology, engineering, mathematics) and non-STEM courses, it is found that high-performing students consistently exhibit more positive emotion, cognition and off-topic dialogues in all stages of the lesson than low-performing students. However, while the metacognitive dialogue illustrates its importance in non-STEM courses, this effect cannot be found in STEM courses. While high-performing students in non-STEM courses show negative emotion in the last stage of lessons, STEM students show positive emotion.


menu
Abstract
Full text
Outline
About this article

Prediction of Academic Performance of Students in Online Live Classroom Interactions —An Analysis Using Natural Language Processing and Deep Learning Methods

Show Author's information Yuanyi Zhen1, Jar-Der Luo1( ), Hui Chen2
Laboratory of Computational Social Science and National Governance, Tsinghua University, Beijing 100084, China
School of Chinese Language and Literature, Beijing Foreign Studies University, Beijing 100089, China

Abstract

Prior studies have shown the importance of classroom dialogue in academic performance, through which knowledge construction and social interaction among students take place. However, most of them were based on small scale or qualitative data, and few has explored the availability and potential of big data collected from online classrooms. To address this issue, this paper analyzes dialogues in live classrooms of a large online learning platform in China based on natural language processing techniques. The features of interactive types and emotional expression are extracted from classroom dialogues. We then develop neural network models based on these features to predict high- and low-academic performing students, and employ interpretable AI (artificial intelligence) techniques to determine the most important predictors in the prediction models. In both STEM (science, technology, engineering, mathematics) and non-STEM courses, it is found that high-performing students consistently exhibit more positive emotion, cognition and off-topic dialogues in all stages of the lesson than low-performing students. However, while the metacognitive dialogue illustrates its importance in non-STEM courses, this effect cannot be found in STEM courses. While high-performing students in non-STEM courses show negative emotion in the last stage of lessons, STEM students show positive emotion.

Keywords: natural language processing, deep learning, academic performance prediction, live classroom dialogue, emotional expression, interactive type

References(63)

[1]

J. E. Knowles, Of needles and haystacks: Building an accurate statewide dropout early warning system in wisconsin, J. Educ. Data Min., vol. 7, no. 3, pp. 18–67, 2015.
Google Scholar
[2]

S. Lee and J. Y. Chung, The machine learning-based dropout early warning system for improving the performance of dropout prediction, Appl. Sci., vol. 9, no. 15, p. 3093, 2019.
DOI Google Scholar
[3]

L. Yan, A. Whitelock-Wainwright, Q. Guan, G. Wen, D. Gašević, and G. Chen, Students’ experience of online learning during the COVID-19 pandemic: A province-wide survey study, Br. J. Educ. Technol., vol. 52, no. 5, pp. 2038–2057, 2021.
DOI Google Scholar
[4]
H. Li, W. Ding, and Z. Liu, Identifying at-risk K-12 students in multimodal online environments: A machine learning approach, arXiv preprint arXiv: 2003.09670, 2020.
[5]

S. Hennessy, S. Rojas-Drummond, R. Higham, A. M. Márquez, F. Maine, R. M. Ríos, R. García-Carrión, O. Torreblanca, and M. J. Barrera, Developing a coding scheme for analysing classroom dialogue across educational contexts, Learn. Cult. Soc. Interact., vol. 9, pp. 16–44, 2016.
DOI Google Scholar
[6]

M. Zembylas, Adult learners’ emotions in online learning, Distance Educ., vol. 29, no. 1, pp. 71–87, 2008.
DOI Google Scholar
[7]
Z. Liu, and Y. Zhang, A semantic role mining and learning performance prediction method in MOOCs, presented at Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint Int. Conf. Web and Big Data, Macau, China, 2018, pp. 259–269.
DOI
[8]

W. Zou, X. Hu, Z. Pan, C. Li, Y. Cai, and M. Liu, Exploring the relationship between social presence and learners’ prestige in MOOC discussion forums using automated content analysis and social network analysis, Comput. Hum. Behav., vol. 115, p. 106582, 2021.
DOI Google Scholar
[9]

C. Howe and M. Abedin, Classroom dialogue: A systematic review across four decades of research, Camb. J. Educ., vol. 43, no. 3, pp. 325–356, 2013.
DOI Google Scholar
[10]
R. J. Alexander, Essays on Pedagogy. New York, NY, USA: Routledge, 2008.
[11]
J. G. Greeno, Classroom talk sequences and learning, in Socializing Intelligence Through Academic Talk and Dialogue, L. B. Resnick, C. S. Asterhan, and S. N. Clarke, eds. New York, NY, USA: American Educational Research Association, 2015, pp. 255–262.
DOI
[12]

H. Muhonen, E. Pakarinen, A. M. Poikkeus, M. K. Lerkkanen, and H. Rasku-Puttonen, Quality of educational dialogue and association with students’ academic performance, Learn. Instr., vol. 55, pp. 67–79, 2018.
DOI Google Scholar
[13]
M. Baker, S. Järvelä, J. Andriessen, Affective Learning Together: Social and Emotional Dimensions of Collaborative Learning. New York, NY, USA: Routledge, 2013.
DOI
[14]

G. Zhu, W. Xing, S. Costa, M. Scardamalia, and B. Pei, Exploring emotional and cognitive dynamics of Knowledge Building in grades 1 and 2, User Model. User Adapt. Interact., vol. 29, no. 4, pp. 789–820, 2019.
DOI Google Scholar
[15]
P. R. Kleinginna Jr and A. M. Kleinginna, A categorized list of emotion definitions, with suggestions for a consensual definition, Motiv. Emot., vol. 5, no. 4, pp. 345–379, 1981.
DOI
[16]

K. R. Scherer, What are emotions? And how can they be measured, Soc. Sci. Inf., vol. 44, no. 4, pp. 695–729, 2005.
DOI Google Scholar
[17]

A. Bakhtiar, E. A. Webster, and A. F. Hadwin, Regulation and socio-emotional interactions in a positive and a negative group climate, Metacognition Learn., vol. 13, no. 1, pp. 57–90, 2018.
DOI Google Scholar
[18]

M. Cleveland-Innes and P. Campbell, Emotional presence, learning, and the online learning environment, Int. Rev. Res. Open Distrib. Learn., vol. 13, no. 4, p. 269, 2012.
DOI Google Scholar
[19]

A. R. Damasio, Descartes’ error and the future of human life, Sci. Am., vol. 271, no. 4, p. 144, 1994.
DOI Google Scholar
[20]
R. Pekrun and L. Linnenbink-Garcia, Academic emotions and student engagement, in Handbook of Research on Student Engagement, S. L. Christenson, A. L. Reschly, and C. Wylie, eds. New York, NY, USA: Springer, 2012, pp. 259–282.
DOI
[21]

L. Yorks and E. Kasl, Toward a theory and practice for whole-person learning: Reconceptualizing experience and the role of affect, Adult Educ. Q., vol. 52, no. 3, pp. 176–192, 2002.
DOI Google Scholar
[22]
R. Pekrun, A social cognitive, control–value theory of achievement emotions, in Motivational Psychology of Human Development, J. Heckhausen, ed. Oxford, UK: Elsevier, pp. 143–163, 2000.
DOI
[23]

R. Pekrun, S. Lichtenfeld, H. W. Marsh, K. Murayama, and T. Goetz, Achievement emotions and academic performance: Longitudinal models of reciprocal effects, Child Dev., vol. 88, no. 5, pp. 1653–1670, 2017.
DOI Google Scholar
[24]

R. S. J. D. Baker, S. K. D'Mello, M. M. T. Rodrigo, and A. C. Graesser, Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments, Int. J. Hum. Comput. Stud., vol. 68, no. 4, pp. 223–241, 2010.
DOI Google Scholar
[25]
Z. A. Pardos, R. S. Baker, M. O. San Pedro, S. M. Gowda, and S. M. Gowda, Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes, in Proc. 3rd Int. Conf. learning analytics and knowledge, Leuven, Belgium, 2013, pp. 117–124.
DOI
[26]

S. D’Mello, B. Lehman, R. Pekrun, and A. Graesser, Confusion can be beneficial for learning, Learn. Instr., vol. 29, pp. 153–170, 2014.
DOI Google Scholar
[27]
M. Worsley, P. Blikstein, Using learning analytics to study cognitive disequilibrium in a complex learning environment, in Proc. 5th Int. Conf. Learning Analytics and Knowledge, Poughkeepsie, NY, USA, 2015, pp. 426–427.
DOI
[28]

L. Zheng, J. Niu, and L. Zhong, Effects of a learning analytics-based real-time feedback approach on knowledge elaboration, knowledge convergence, interactive relationships and group performance in CSCL, Br. J. Educ. Technol., vol. 53, no. 1, pp. 130–149, 2022.
DOI Google Scholar
[29]

L. Przybilla, K. Klinker, M. Kauschinger, and H. Krcmar, Stray off topic to stay on topic: Preserving interaction and team morale in a highly collaborative course while at a distance, Commun. Assoc. Inf. Syst., vol. 48, no. 1, pp. 177–184, 2021.
DOI Google Scholar
[30]

J. H. Flavell, Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry, Am. Psychol., vol. 34, no. 10, pp. 906–911, 1979.
DOI Google Scholar
[31]

M. E. Martinez, What is metacognition, Phi Delta Kappan, vol. 87, no. 9, pp. 696–699, 2006.
DOI Google Scholar
[32]

C. D. Zepeda, C. O. Hlutkowsky, A. C. Partika, and T. J. Nokes-Malach, Identifying teachers’ supports of metacognition through classroom talk and its relation to growth in conceptual learning, J. Educ. Psychol., vol. 111, no. 3, pp. 522–541, 2019.
DOI Google Scholar
[33]

P. H. Winne and J. C. Nesbit, The psychology of academic achievement, Annu. Rev. Psychol., vol. 61, pp. 653–678, 2010.
DOI Google Scholar
[34]

A. Cadamuro, E. Bisagno, G. A. Di Bernardo, L. Vezzali, and A. Versari, Making the school Smart: the interactive whiteboard against disparities in children stemming from low metacognitive skills, J. E-Learn. Knowl. Soc., vol. 16, no. 1, pp. 33–43, 2020.
Google Scholar
[35]

C. M. Roebers, S. S. Krebs, and T. Roderer, Metacognitive monitoring and control in elementary school children: Their interrelations and their role for test performance, Learn. Individ. Differ., vol. 29, pp. 141–149, 2014.
DOI Google Scholar
[36]

H. L. Swanson, Influence of metacognitive knowledge and aptitude on problem solving, J. Educ. Psychol., vol. 82, no. 2, pp. 306–314, 1990.
DOI Google Scholar
[37]

M. V. J. Veenman, R. Kok, and A. W. Blöte, The relation between intellectual and metacognitive skills in early adolescence, Instr. Sci., vol. 33, no. 3, pp. 193–211, 2005.
DOI Google Scholar
[38]

Z. R. Mevarech and C. Amrany, Immediate and delayed effects of meta-cognitive instruction on regulation of cognition and mathematics achievement, Metacognition Learn., vol. 3, no. 2, pp. 147–157, 2008.
DOI Google Scholar
[39]
R. J. Yinger, A study of teacher planning, Elem. Sch. J. , vol. 80, no. 3, pp. 107–127, 1980.
DOI
[40]
M. Hunter, Knowing, teaching and supervising, in Using What We Know About Reading, P. Hosford, ed. Alexandria, Egypt: Association for Supervision and Curriculum Development, 1984, pp. 169–203.
[41]

A. P. Johnson, It’s time for madeline hunter to go: A new look at lesson plan design, Action Teach. Educ., vol. 22, no. 1, pp. 72–78, 2000.
DOI Google Scholar
[42]
T. S. Farrell, Lesson planning, in Methodology in Language Teaching: An Anthology of Current Practice, J. C. Richards and W. A. Renandya, Eds. New York, NY, USA: Cambridge University Press, 2002, pp. 30–39.
DOI
[43]

G. Tonguç and B. O. Ozkara, Automatic recognition of student emotions from facial expressions during a lecture, Comput. Educ., vol. 148, p. 103797, 2020.
DOI Google Scholar
[44]
M. Ford, C. T. Baer, D. Xu, U. Yapanel, and S. Gray, The LENATM language environment analysis system: Audio specifications of the DLP-0121, Technical report, LENA Foundation, Boulder, CO, USA, 2008.
[45]

Z. Wang, K. Miller, and K. Cortina, Using the LENA in teacher training: Promoting student involvement through automated feedback, Unterrichtswissenschaft, vol. 4, pp. 290–305, 2013.
Google Scholar
[46]

Z. Wang, X. Pan, K. F. Miller, and K. S. Cortina, Automatic classification of activities in classroom discourse, Comput. Educ., vol. 78, pp. 115–123, 2014.
DOI Google Scholar
[47]
D. Jiang, Y. Chen, and A. Garg, A hybrid method for overlapping speech detection in classroom environment, Comput. Appl. Eng. Educ., vol. 26, no. 1, pp. 171–180, 2018.
DOI
[48]

Y. Song, S. Lei, T. Hao, Z. Lan, and Y. Ding, Automatic classification of semantic content of classroom dialogue, J. Educ. Comput. Res., vol. 59, no. 3, pp. 496–521, 2021.
DOI Google Scholar
[49]
P. M. Moreno-Marcos, P. J. Muñoz-Merino, C. Alario-Hoyos, I. Estévez-Ayres, and C. Delgado Kloos, Analysing the predictive power for anticipating assignment grades in a massive open online course, Behav. Inf. Technol., vol. 37, nos. 10–11, pp. 1021–1036, 2018.
DOI
[50]
K. Mrhar, L. Benhiba, S. Bourekkache, and M. Abik, A Bayesian CNN-LSTM model for sentiment analysis in massive open online courses MOOCs, Int. J. Emerg. Technol. Learn. , vol. 16, no. 23, pp. 216–232, 2021.
DOI
[51]

S. Wachter, B. Mittelstadt, C. Russell, Counterfactual explanations without opening the black box: Automated decisions and the GDPR, SSRN Electron. J., vol. 31, no. 2, pp. 841–887, 2017.
DOI Google Scholar
[52]
A. Chatzimparmpas, R. M. Martins, I. Jusufi, and A. Kerren, A survey of surveys on the use of visualization for interpreting machine learning models, Inf. Vis. , vol. 19, no. 3, pp. 207–233, 2020.
DOI
[53]

G. Plumb, D. Molitor, A. S. Talwalkar, Model Agnostic Supervised Local Explanations, NIPS, vol. 31, pp. 2520–2529, 2018.
Google Scholar
[54]
M. T. Ribeiro, S. Singh, C. Guestrin, Why should I trust you?: Explaining the predictions of any classifier, in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, San Francisco, CA, USA, 2016, pp. 1135–1144.
DOI
[55]
S. M. Lundberg, S. I. Lee, A unified approach to interpreting model predictions, NIPS, vol. 30, pp. 4768–4777, 2017.
[56]

S. Karlos, G. Kostopoulos, and S. Kotsiantis, Predicting and interpreting students’ grades in distance higher education through a semi-regression method, Appl. Sci., vol. 10, no. 23, p. 8413, 2020.
DOI Google Scholar
[57]
S. Kim, W. Kim, Y. Jang, S. Choi, H. Jung, and H. Kim, Student knowledge prediction for teacher-student interaction, in Proc. AAAI Conf. Artif. Intell. , vol. 35, no. 17, pp. 15560–15568, 2021.
DOI
[58]
J. Devlin, M. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 1810.04805, 2018.
[59]

J. Hoth, G. Kaiser, A. Busse, M. Doehrmann, J. Koenig, and S. Blömeke, Professional competences of teachers for fostering creativity and supporting high-achieving students, ZDM Math. Educ., vol. 49, no. 1, pp. 107–120, 2017.
DOI Google Scholar
[60]
J. T. Guthrie and A. Wigfield, Engagement and motivation in reading, in Handbook of Reading Research, P. D. Barr, ed. New York, NY, USA: Lawrence Erlbaum Associates, 2000, pp. 403–422.
[61]

X. Xu, J. Wang, H. Peng, and R. Wu, Prediction of academic performance associated with Internet usage behaviors using machine learning algorithms, Comput. Hum. Behav., vol. 98, pp. 166–173, 2019.
DOI Google Scholar
[62]

C. F. Rodríguez-Hernández, M. Musso, E. Kyndt, and E. Cascallar, Artificial neural networks in academic performance prediction: Systematic implementation and predictor evaluation, Comput. Educ., vol. 2, p. 100018, 2021.
DOI Google Scholar
[63]

L. Zheng, M. Long, J. Niu, and L. Zhong, An automated group learning engagement analysis and feedback approach to promoting collaborative knowledge building, group performance, and socially shared regulation in CSCL, Int. J. Comput. Support. Collab. Learn., vol. 18, no. 1, pp. 101–133, 2023.
DOI Google Scholar
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 13 February 2023
Revised: 01 June 2023
Accepted: 06 June 2023
Published: 30 March 2023
Issue date: March 2023

Copyright

© The author(s) 2023.

Acknowledgements

Acknowledgment

This work was supported by the Center for Social Network Research of Tsinghua University, Tsinghua’s Research Project (No. 2016THZWYY03), and the Project of Tencent Social Research Center (No. 20162001703).

Rights and permissions

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Return