Abstracts
Abstract
This study aims to examine the relationship between the productive knowledge of some lexical and phraseological indices and the quality of English as a Foreign Language (EFL) learners’ writing. A sample of 120 expository essays, written by semesters 1 and 5 university students in a less proficient EFL context, are rated by human evaluators and automatically examined for the target indices. The results show that, unlike the index of lexical diversity, both indices of content word frequency and range could significantly discriminate between different proficiency levels. For the phraseological indices, both the proportions of rare and frequent bigrams yielded between-group differences, with higher proficiency students performing significantly better in both categories. Using a regression analysis, the results show that the use of rare and contextually restricted content words and the production of larger proportions of rare and frequent bigrams could be considered indicators of better writing proficiency. The study suggests implications for the teaching of EFL.
Keywords:
- EFL,
- bigrams,
- frequency,
- range,
- diversity
Résumé
Cette étude vise à examiner la relation entre les compétences lexicales/phraséologiques et la qualité de la production écrite des étudiants de l’anglais/langue étrangère. Un échantillon de 120 essais explicatifs écrits par des étudiants des semestres 1 et 5 dans un contexte EFL moins compétent, sont notés par des évaluateurs humains et examinés pour les indices cibles en utilisant un traitement automatique. Les résultats montrent que, contrairement à l’indice de diversité lexicale, la fréquence et la distribution contextuelle de mots de contenu peuvent distinguer entre les deux différents niveaux. Pour les indices phraséologiques, les proportions de bigrammes rares et fréquents ont produit des différences entre les deux groupes : les étudiants de S5 sont les plus performants avec de meilleurs résultats dans les deux catégories. À l’aide d’une régression linéaire, les résultats montrent que l’utilisation de mots de contenu rares et contextuellement restreints et aussi une plus grande proportion de bigrammes rares et fréquents pourraient être considérés comme des indicateurs d’une meilleure compétence en production écrite. L’étude suggère des implications pour l’enseignement et l’apprentissage de l’anglais en tant que langue étrangère.
Appendices
Bibliography
- Bestgen, Y. (2016a). Using collocational features to improve automated scoring of EFL texts. Proceedings of the 12th workshop on multiword expressions, 84-90.
- Bestgen, Y. (2016b). Evaluation automatique de textes : Validation interne et externe d'indices phraséologiques pour l'évaluation automatique de textes rédigés en anglais langue étrangère. Traitement automatique des langues, 57(3), 91-115.
- Bestgen, Y. (2019). Évaluation de textes en anglais langue étrangère et séries phraséologiques : comparaison de deux procédures automatiques librement accessibles. Revue française de linguistique appliquée, XXIV, 81-94. https://doi.org/10.3917/rfla.241.0081
- Bestgen, Y., & Granger, S. (2014). Quantifying the development of phraseological competence in L2 English writing: An automated approach. Journal of Second Language Writing, 26, 28-41. https://doi.org/10.1016/j.jslw.2014.09.004
- Cohen, J. (1969). Statistical power analysis for the behavioral sciences. Academic Press.
- Covington, M. A., & McFall, J. D. (2010). Cutting the Gordian Knot: The moving average type token ratio (MATTR). Journal of Quantitative Linguistics, 17(2), 94-100. https://doi.org/10.1080/09296171003643098
- Cowie, A. P. (1994). Phraseology. In R. E. Asher (Ed.), The encyclopedia of language and linguistics (pp. 3168-3171). Oxford University Press.
- Cowie, A. P. (Ed.). (1998). Phraseology: Theory, analysis, and applications. Oxford University Press.
- Crossley, S. A., & Kyle, K. (2018). Assessing writing with the tool for the automatic analysis of lexical sophistication (TAALES). Assessing Writing, 38, 46-50. https://doi.org/10.1016/j.asw.2018.06.004
- Crossley, S. A., & McNamara, D. S. (2012). Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading. 35(2). 115-135. https://doi.org/10.1111/j.1467-9817.2010.01449.x
- Crossley, S. A., Cai, Z., & McNamara, D. (2012). Syntagmatic, paradigmatic, and automatic N-gram approaches to assessing essay quality. Proceedings of the 25th International Florida Artificial Intelligence Research Society Conference, FLAIRS-25 (pp. 214-219).
- Crossley, S. A., Salsbury, T., & McNamara, D. (2010). The development of polysemy and frequency use in English second language speakers. Language Learning, 60, 573-605. https://doi.org/10.1111/j.1467-9922.2010.00568.x
- Crossley, S. A., Salsbury, T., & McNamara D. S. (2014). Assessing lexical proficiency using analytic ratings: A case for collocation accuracy. Applied Linguistics, 36(5), 570-590, https://doi.org/10.1093/applin/amt056
- Crossley, S. A., Subtirelu, N., & Salsbury, T. (2013). Frequency effects or context effects in second language word learning: What predicts early lexical production? Studies in Second Language Acquisition, 35(4), 727-755. http://doi.org/10.1017/S0272263113000375
- Crossley, S. A., Kyle, K., Allen, L., Guo, L., & McNamara, D. S. (2014). Linguistic microfeatures to predict L2 writing proficiency: A case study in automated writing evaluation. Journal of Writing Assessment, 7(1). https://escholarship.org/uc/item/06n1v820
- Daller, H., Milton, J., & Treffers-Daller, J. (Eds.). (2007). Modelling and assessing vocabulary knowledge. Cambridge University Press. doi:10.1017/CBO9780511667268
- Durrant, P. (2014). Corpus frequency and second language learners’ knowledge of collocations: A meta-analysis. International Journal of Corpus Linguistics, 19(4), 443-477. https://doi.org/10.1075/ijcl.19.4.01dur
- Durrant, P., & Schmitt, N. (2009). To what extent do native and non-native writers make use of collocations? International Review of Applied Linguistics in Language Teaching (IRAL), 47,157-177. https://doi.org/10.1515/iral.2009.007
- Ellis, N. C. (2002a). Frequency effects in language processing: A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition, 24(2), 143-188. https://doi.org/10.1017/S0272263102002024
- Ellis, N. (2002b). Reflections on frequency effects in language processing. Studies in Second Language Acquisition, 24(2), 297-339. http://doi.org/10.1017/S0272263102002140
- Evert, S. (2005). The statistics of word cooccurrences: Words pairs and collocations [Unpublished doctoral dissertation]. Universität Stuttgart.
- Evert, S. (2009). Corpora and collocations. In A. Lüdeling & M. Kytö (Eds.), Corpus linguistics: An international handbook (pp. 1211-1248). Mouton de Gruyter.
- Fergadiotis, G., Wright, H. H., & West, T. M. (2013). Measuring lexical diversity in narrative discourse of people with aphasia. American Journal of Speech-Language Pathology, 22(2), S397-S408. 10.1044/1058-0360(2013/12-0083)
- Fergadiotis, G., Wright, H. H., & Green, S. B. (2015). Psychometric evaluation of lexical diversity indices: Assessing length effects. Journal of Speech, Language, and Hearing Research: JSLHR, 58(3), 840-852. https://doi.org/10.1044/2015_JSLHR-L-14-0280
- Firth, J. R. (1957). A synopsis of linguistic theory, 1930-1955. In Studies in linguistic analysis (pp. 1-32). Blackwell.
- Gablasova, D., Brezina, V., & McEnery, T. (2017). Collocations in corpus-based language learning research: Identifying, comparing, and interpreting the evidence. Language Learning, 67, 155-179. https://doi.org/10.1111/lang.12225
- González, M. C. (2017). The contribution of lexical diversity to college-level writing. TESOL Journal, 8(4), 899-919. https://doi.org/10.1002/tesj.342
- Granger, S., & Bestgen, Y. (2014). The use of collocations by intermediate vs. advanced non-native writers: A bigram-based study. International Review of Applied Linguistics in Language Teaching (IRAL), 52(3), 229-252. https://doi.org/10.1515/iral-2014-0011
- Granger, S. & Bestgen, Y. (2017). Using collgrams to assess L2 phraseological development: A replication study. In P. de Haan, S. van Vuuren & R. de Vries (Eds.), Language, learners and levels: Progression and variation. corpora and language in use Proceedings 3 (pp. 385-408). Presses Universitaires de Louvain.
- Gries, S. T. (2008). Dispersions and adjusted frequencies in corpora. International Journal of Corpus Linguistics, 13, 403-437. https://doi.org/10.1075/ijcl.13.4.02gri
- Gries, S. T. (2010). Dispersions and adjusted frequencies in corpora: Further explorations. In Corpus-linguistic applications (pp. 197–212). Brill Rodopi. https://doi.org/10.1163/9789042028012_014
- Gries, S. T., & Ellis, N. (2015). Statistical measures for usage-based linguistics. Language Learning, 65(1), 228-255. https://doi.org/10.1111/lang.12119
- Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218-238. https://doi.org/10.1016/j.asw.2013.05.002
- Halliday, M. A. K. (1966). Lexis as a linguistic level. In C. Bazell, J. Catford, M. A. K. Halliday, & R. Robins (Eds.), In memory of J. R. Firth (pp. 148-162). Longman.
- Howarth, P. A. (1996). Phraseology in English academic writing. Max Niemeyer Verlag. https://doi.org/10.1515/9783110937923
- Howarth, P. A. (1998). Phraseology and second language proficiency. Applied Linguistics, 19(1), 24-44. https://doi.org/10.1093/applin/19.1.24
- Hoey, M. (2005). Lexical priming: A new theory of words and language. Routledge. https://doi.org/10.4324/9780203327630
- Jansen, T., Vögelin, C., Machts, N., Keller, S., Köller, O., & Möller, J. (2021). Judgment accuracy in experienced versus student teachers: Assessing essays in English as a foreign language. Teaching and Teacher Education, 97, 103216. https://doi.org/10.1016/j.tate.2020.103216
- Jarvis, S. (2013). Defining and measuring lexical diversity. In S. Jarvis & M. Daller (Eds.). Vocabulary knowledge: Human ratings and automated measures (pp. 13-44). John Benjamins Publishing. https://doi.org/10.1075/sibil.47.03ch1
- Jarvis, S. (2017). Grounding lexical diversity in human judgments. Language Testing, 34(4), 537-553. https://doi.org/10.1177/0265532217710632
- Jarvis, S. & Hashimoto, B. J., (2021). How operationalizations of word types affect measures of lexical diversity. International Journal of Learner Corpus Research, 7(1),163-194. https://doi.org/10.1075/ijlcr.20004.jar
- Johnson, W. (1944). Studies in language behavior I: A program of research. Psychological Monographs, 56(2), 1-15. https://doi.org/10.1037/h0093508
- Jung, Y. J., Crossley, S. A., & McNamara, D. (2019). Predicting second language writing proficiency in learner texts using computational tools. Journal of Asia TEFL, 16(1), 37-52. https://doi.org/10.18823/asiatefl.2019.16.1.3.37
- Kim, M., Crossley, S.A. & Kyle, K. (2018). Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. The Modern Language Journal,102, 120-141. https://doi.org/10.1111/modl.12447
- Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49(4), 757-786. https://doi.org/10.1002/tesq.194
- Kyle, K., & Crossley, S. A. (2016). The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, 34, 12-24. https://doi.org/10.1016/j.jslw.2016.10.003
- Kyle, K., Crossley, S., & Berger, C. (2018). The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behavior Research Methods, 50(3), 1030-1046. https://doi.org/10.3758/s13428-017-0924-4
- Kyle, K., Crossley, S. A., & Jarvis, S. (2020). Assessing the validity of lexical diversity indices using direct judgements. Language Assessment Quarterly, 0(0), 1-17. https://doi.org/10.1080/15434303.2020.1844205
- Kyle, K., Allen, L., Guo, L., & McNamara, D. (2014). Linguistic microfeatures to predict L2 writing proficiency: A case study in automated writing evaluation. Journal of Writing Assessment, 7(1), 1-16.
- Laufer, B. (1994). The lexical profile of second language writing: Does it change over time? RELC Journal, 25(2), 21-33. https://doi.org/10.1177/003368829402500202
- Laufer, B., & Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16(3), 307-322. https://doi.org/10.1093/applin/16.3.307
- Malvern, D., Richards, B., Chipere, N., & Durán, P. (2004). Lexical diversity and language development. Palgrave. https://doi.org/10.1057/9780230511804
- McCarthy, P. M. (2005). An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity [Unpublished doctoral dissertation]. The University of Memphis. https://www.proquest.com/docview/305349212
- McCarthy, P. M., & Jarvis, S. (2007). Vocd: A theoretical and empirical evaluation. Language Testing, 24(4), 459-488. https://doi.org/10.1177/0265532207080767
- McCarthy, P. M., & Jarvis, S. (2010). MTLD, Vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381–392. https://doi.org/10.3758/BRM.42.2.381
- McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27(1), 57-86. https://doi.org/10.1177/0741088309351547
- Mel'cuk, I. (1995). Phrasemes in language and phraseology in linguistics. In M. Everaert, E.-J. van der Linden, A. Schenk & R. Schreuder (Eds.), Idiom: Structural and psychological perspectives (pp. 167-232). Lawrence Erlbaum Associates. https://doi.org/10.4324/9781315806501
- Mel'cuk, I. (1998). Collocations and lexical functions. In A.P. Cowie (Ed.), Phraseology: Theory, analysis, and applications (pp. 23-53). Clarendon Press.
- Monteiro, K. R., Crossley, S. A., & Kyle, K. (2018). In search of new benchmarks: Using L2 lexical frequency and contextual diversity indices to assess second language writing. Applied Linguistics, 41(2), 280-300. https://doi.org/10.1093/applin/amy056
- Paquot, M., & Granger, S. (2012). Formulaic language in learner corpora. Annual Review of Applied Linguistics, 32, 130-149. http://doi.org/10.1017/S0267190512000098
- Qian, Y. (2019). Dynamism of collocation in L2 English writing: A bigram-based study. International Review of Applied Linguistics in Language Teaching,0(0). https://doi.org/10.1515/iral-2019-0012
- Read, J. (2000). Assessing vocabulary. Cambridge University Press. https://doi.org/10.1017/CBO9780511732942
- Römer, U. (2009). The inseparability of lexis and grammar: Corpus linguistic perspectives. Annual Review of Cognitive Linguistics, 7(1), 140-162. https://doi.org/10.1075/arcl.7.06rom
- Salsbury, T., Crossley, S. A., & McNamara, D. S. (2011). Psycholinguistic word information in second language oral discourse. Second Language Research 27(3), 343-360. https://doi.org/10.1177/026765831039585
- Schmitt, N. (2010). Researching vocabulary: A vocabulary research manual. Palgrave Macmillan. http://dx.doi.org/10.1057/9780230293977
- Sinclair, J. M. (1987). Collocation: A progress report. In R. Steele & T. Threadgold (Eds.), Language topics (pp. 319-331). Benjamins.
- Sinclair, J. (1991). Corpus, concordance, collocation. Oxford University Press.
- Sinclair, J. (1996). The search for units of meaning. Textus, 9(1), 75-106.
- Stefanowitsch, A., & Gries, S. T. (2003). Collostructions: Investigating the interaction of words and constructions. International Journal of Corpus Linguistics, 8(2), 209-243. http://doi.org/10.1075/ijcl.8.2.03ste
- Treffers-Daller, J. (2013). Measuring lexical diversity among L2 learners of French: an exploration of the validity of D, MTLD and HD-D as measures of language ability. S. Jarvis & M. Daller (Eds.). Vocabulary knowledge: Human ratings and automated measures (pp. 79-104). John Benjamins Publishing. https://doi.org/10.1075/sibil.47.05ch3
- Vidal, K., & Jarvis, S. (2018). Effects of English-medium instruction on Spanish students’ proficiency and lexical diversity in English. Language Teaching Research, 24(5), 568-587. https://doi.org/10.1177/1362168818817945
- Wolk, K., Wolk, A. & Marasek, K. (2017). Unsupervised tool for quantification of progress in L2 English phraseological. Proceedings of the 2017 Federated Conference on Computer Science and Information Systems, 383-388.
- Wray, A. (2000). Formulaic sequences in second language teaching: Principles and practice. Applied Linguistics, 21(4), 463-489. https://doi.org/10.1093/applin/21.4.463
- Yoon, Hyung-Jo. (2018). The development of ESL writing quality and lexical proficiency: Suggestions for assessing writing achievement. Language Assessment Quarterly, 15(4), 387-405. https://doi.org/10.1080/15434303.2018.1536756
- Zenker, F., & Kyle, K. (2021). Investigating minimum text lengths for lexical diversity indices. Assessing Writing, 47, 100505. https://doi.org/10.1016/j.asw.2020.100505