Abstracts
Abstract
As large-scale, sophisticated open and distance learning environments expand in higher education globally, so does the need to support learning at scale in real time. Valid, reliable rubrics of critical discourse are an essential foundation for developing artificial intelligence tools that automatically analyse learning in educator-student dialogue. This article reports on a validation study where discussion transcripts from a target massive open online course (MOOC) were categorised into phases of cognitive presence to cross validate the use of an adapted rubric with a larger dataset and with more coders involved. Our results indicate that the adapted rubric remains stable for categorising the target MOOC discussion transcripts to some extent. However, the proportion of disagreements between the coders increased compared to the previous experimental study with fewer data and coders. The informal writing styles in MOOC discussions, which are not as prevalent in for-credit courses, caused ambiguities for the coders. We also found most of the disagreements appeared at adjacent phases of cognitive presence, especially in the middle phases. The results suggest additional phases may exist adjacent to current categories of cognitive presence when the educational context changes from traditional, smaller-scale courses to MOOCs. Other researchers can use these findings to build automatic analysis applications to support online teaching and learning for broader educational contexts in open and distance learning. We propose refinements to methods of cognitive presence and suggest adaptations to certain elements of the Community of Inquiry (CoI) framework when it is used in the context of MOOCs.
Keywords:
- cognitive presence,
- MOOC,
- text classification,
- online discussion,
- content analysis
Appendices
Bibliography
- Alario-Hoyos, C., Estévez-Ayres, I., Pérez-Sanagustín, M., Kloos, C. D., & Fernández-Panadero, C. (2017). Understanding learners’ motivation and learning strategies in MOOCs. The International Review of Research in Open and Distributed Learning, 18(3), 119-137. https://doi.org/10.19173/irrodl.v18i3.2996
- Amemado, D., & Manca, S. (2017). Learning from decades of online distance education: MOOCs and the community of inquiry framework. Journal of E-Learning and Knowledge Society, 13(2), 21-32. http://www.je-lks.org/ojs/index.php/Je-LKS_EN/article/view/137/75
- Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanovic, V., & Gaševic, D. (2020). Towards automatic cross-language classification of cognitive presence in online discussions. In C. Rensing & H. Drachsler (Chairs), LAK’20: Proceedings of the tenth international conference on learning analytics and knowledge (pp. 605-614). ACM. https://doi.org/10.1145/3375462.3375496
- Buchem, I., Amenduni, F., Poce, A., Michaescu, V., Andone, D., Tur, G., Urbina, S., & Šmitek, B. (2020). Integrating mini-moocs into study programs in higher education during COVID-19. Five pilot case studies in context of the open virtual mobility project. In EDEN 2020 annual conference: Human and artificial intelligence for the society of the future (pp. 299-310). https://doi.org/10.38069/edenconf-2020-ac0028
- Cha, H., & So, H.-J. (2020). Integration of formal, non-formal and informal learning through MOOCs. In D. Burgos (Ed.), Radical solutions and open science. Lecture notes in educational technology (pp. 135-158). Springer Singapore. https://doi.org/10.1007/978-981-15-4276-3_9
- Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219-243. https://doi.org/10.1080/00461520.2014.965823
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104
- Corich, S., Kinshuk, & Hunt, L. (2006). Measuring critical thinking within discussion forums using a computerised content analysis tool. In S. Banks, V. Hodgson, C. Jones, B. Kemp, D. McConnell, & C. Smith (Eds.), Proceedings of the 5th international conference on networked learning, 2(1), 1-8. http://www.lancaster.ac.uk/fss/organisations/netlc/past/nlc2006/abstracts/pdfs/P07 Corich.pdf
- Dewey, J. (1933). How we think: A restatement of the relation of reflective thinking to the educative process (revised ed.). D. C. Heath.
- Dillahunt, T., Wang, Z., & Teasley, S. D. (2014). Democratizing higher education: Exploring MOOC use among those who cannot afford a formal education. The International Review of Research in Open and Distributed Learning, 15(5), 177-196. https://doi.org/10.19173/irrodl.v15i5.1841
- Farrow, E., Moore, J., & Gašević, D. (2021). A network analytic approach to integrating multiple quality measures for asynchronous online discussions. In M. Scheffel, N. Dowell, S. Joksimovic, & G. Siemens (Chairs), LAK’21: Proceedings of the 11th international conference on learning analytics and knowledge (pp. 248-258). ACM. https://doi.org/10.1145/3448139.3448163
- Fleiss, J., Levin, B., & Paik, M. (2003). Statistical methods for rates and proportions (3rd ed.). John Wiley & Sons. https://onlinelibrary.wiley.com/doi/book/10.1002/0471445428
- Garrison, D. R. (2007). Online community of inquiry review: Social, cognitive, and teaching presence issues. Journal of Asynchronous Learning Networks, 11(1), 61-72.
- Garrison, D. R., & Anderson, T. (2011). E-learning in the 21st century: A framework for research and practice (2nd ed.). Routledge.
- Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2), 87-105. https://doi.org/10.1016/S1096-7516(00)00016-6
- Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23. https://doi.org/10.1080/08923640109527071
- Garrison, D. R., Anderson, T., & Archer, W. (2010). The first decade of the community of inquiry framework: A retrospective. The Internet and Higher Education, 13(1-2), 5-9. https://doi.org/10.1016/j.iheduc.2009.10.003
- Henri, F., & Lundgren-Cayrol, K. (2005). Apprentissage collaboratif à distance: Pour comprendre et concevoir les environnements d’apprentissage virtuels [Collaborative distance learning: Understanding and conceptualizing virtual learning environments]. Presses de l’Université du Québec.
- Hu, Y., Donald, C., Giacaman, N., & Zhu, Z. (2020). Towards automated analysis of cognitive presence in MOOC discussions: a manual classification study. In C. Rensing & H. Drachsler (Chairs), LAK’20 : Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp.135-140). ACM. https://doi.org/10.1145/3375462.3375473
- Jézégou, A. (2010). Community of inquiry in e-learning: A critical analysis of the Garrison and Anderson model. The Journal of Distance Education / Revue de l’éducation à Distance, 24(3), 1-18. http://www.ijede.ca/index.php/jde/article/view/707
- Kanuka, H., Rourke, L., & Laflamme, E. (2007). The influence of instructional methods on the quality of online discussion. British Journal of Educational Technology, 38(2), 260-271. https://doi.org/10.1111/j.1467-8535.2006.00620.x
- Kaul, M., Aksela, M., & Wu, X. (2018). Dynamics of the community of inquiry (CoI) within a massive open online course (MOOC) for in-service teachers in environmental education. Education Sciences, 8(2), Article 40. https://doi.org/10.3390/educsci8020040
- Kovanović, V., Joksimović, S., Gašević, D., & Hatala, M. (2014). Automated cognitive presence detection in online discussion transcripts. In K. Yacef & H. Drachster (Eds.), Workshop proceedings of LAK 2014. Sun SITE Central Europe (CEUR). http://ceur-ws.org/Vol-1137/LA_machinelearning_submission_1.pdf
- Kovanović, V., Joksimović, S., Poquet, O., Hennis, T., Čukić, I., De Vries, P., Hatala, M., Dawson, S., Siemens, G., & Gašević, D. (2018). Exploring communities of inquiry in massive open online courses. Computers & Education, 119, 44-58. https://doi.org/10.1016/j.compedu.2017.11.010
- Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards automated content analysis of discussion transcripts. In D. Gašević & G. Lynch (Eds.), LAK’16: Proceedings of the sixth international conference on learning analytics and knowledge (pp. 15-24). ACM. https://doi.org/10.1145/2883851.2883950
- Liu, C.-J., & Yang, S. C. (2014). Using the community of inquiry model to investigate students’ knowledge construction in asynchronous online discussions. Journal of Educational Computing Research, 51(3), 327-354. https://doi.org/10.2190/EC.51.3.d
- Mayer-Schönberger, V., & Cukier, K. (2013). Big data: A revolution that will transform how we live, work, and think. Houghton Mifflin Harcourt.
- McKlin, T. (2004). Analyzing cognitive presence in online courses using an artificial neural network [Doctoral dissertation, Georgia State University]. ScholarWorks @ Georgia State University. https://scholarworks.gsu.edu/msit_diss/1/
- McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press. https://doi.org/10.1017/CBO9780511894664
- Mladenić, D. (2010). Feature selection in text mining. In C. Sammut & G. I. Webb (Eds.), Encyclopedia of Machine Learning (pp. 406-410). Springer US. https://doi.org/10.1007/978-0-387-30164-8_307
- Neto, V., Rolim, V., Ferreira, R., Kovanović, V., Gašević, D., Dueire Lins, R., & Lins, R. (2018). Automated analysis of cognitive presence in online discussions written in Portuguese. In V. Pammer-Schindler, M. Pérez-Sanagustín, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), EC-TEL 2018: Lifelong technology-enhanced learning. Lecture notes in computer science, vol. 11082 (pp. 245-261). Springer International. https://doi.org/10.1007/978-3-319-98572-5_19
- Park, C. L. (2009). Replicating the use of a cognitive presence measurement tool. Journal of Interactive Online Learning, 8(2), 140-155. http://www.ncolr.org/jiol/issues/pdf/8.2.3.pdf
- Phan, T., McNeil, S. G., & Robin, B. R. (2016). Students’ patterns of engagement and course performance in a massive open online course. Computers and Education, 95, 36-44. https://doi.org/10.1016/j.compedu.2015.11.015
- Rourke, L., & Kanuka, H. (2009). Learning in communities of inquiry: A review of the literature. Journal of Distance Education, 23(1), 19-48. http://www.ijede.ca/index.php/jde/article/view/474/815
- Sadaf, A., & Olesova, L. (2017). Enhancing cognitive presence in online case discussions with questions based on the practical inquiry model. American Journal of Distance Education, 31(1), 56-69. https://doi.org/10.1080/08923647.2017.1267525
- Siemens, G. (2013). Massive open online courses: Innovation in education? In R. McGreal, W. Kinuthia, & S. Marshall (Eds.), Open educational resources: Innovation, research and practice (pp. 5-16). Commonwealth of Learning and Athabasca University. https://www.oerknowledgecloud.org/archive/pub_PS_OER-IRP_web.pdf
- Ullmann, T. D. (2019). Automated analysis of reflection in writing: Validating machine learning approaches. International Journal of Artificial Intelligence in Education, 29(2), 217-257. https://doi.org/10.1007/s40593-019-00174-2
- University of Auckland. (n.d.) Logical and critical thinking. FutureLearn. https://www.futurelearn.com/courses/logical-and-critical-thinking
- Xin, C. (2012). A critique of the community of inquiry framework. The Journal of Distance Education, 26(1), 1-7. http://www.ijede.ca/index.php/jde/article/download/755/1333?inline=1