The Validity of TOEFL iBT Reading Section: Reassessment and Evaluation
DOI:
https://doi.org/10.31185/eduj.Vol56.Iss2.3977Keywords:
TOEFL iBT, Validity, Topic Effect, Individual differencesAbstract
The TOEFL iBT, which is a reputable test designed to measure such skills as listening, reading, speaking, and writing, has, for over a decade, caused a heated debate among scholars. The debate revolving around the reading section of this test is whether or not it accurately and effectively gauges what it claims to measure. This study examined the reading section of this test to determine if it adequately covers the materials that students encounter in an academic setting. Furthermore, it investigated whether the reading section properly assesses what it is designed to gauge or if it involves other irrelevant constructs, which consequently might overshadow the individual differences. To accomplish this, the study analyzed 209 authentic passages drawn from five different books published in different years in an attempt to evaluate whether or not the reading section is in possession of enough content and construct validity to be a reliable indicator of examinees’ reading proficiency. Although the study discovered that this test leans toward hard science subjects, which may obscure the true and diverse intellectual abilities among testees, it also found some positive aspects. These encompass the omission of low-frequency words in both the passages and vocabulary questions, as well as the careful design of multiple-choice questions.
Downloads
References
Ahmadjavaheri, Z., & Zeraatpishe, M. (2020). The impact of construct irrelevant factors on the validity of reading comprehension test. International Journal of Language Testing, 10(1), 1-10.
• Akyol, P., Key, J., & Krishna, K. (2022). Hit or miss? Test taking behavior in multiple choice exams. Annals of Economics and Statistics, 147, 3–50. https://doi.org/10.2307/48684785
• Alavi, S. M., & Bordbar, S. (2012). A Closer look at reading strategy use in reading section of TOEFL iBT. Theory and Practice in Language Studies (2)3, 450-460. doi:10.4304/tpls.2.3.450-460
• Alderson, J.C. (2000). Assessing reading. Cambridge: Cambridge University Press.
• Alderson, J. C. (2009). Test review: Test of English as a Foreign LanguageTM: Internet-based Test (TOEFL iBT®). Language Testing, 26(4), 621–631. https://doi.org/10.1177/0265532209346371.
• Alderson, J. C., & Urquhart, A. H. (1983). The effect of student background discipline on comprehension: A pilot study. In A. Hughes & D. Porter (Ed.), Current developments in language testing (pp. 121–127). London: Academic Press.
• Alderson, J. C., & Urquhart, A. H. (1984). Student discipline and reading comprehension. In T. Culhane (Ed.), Practice and problems in language testing. University of Essex Occasional Papers
• Alderson, J. C., & Urquhart, A. H. (1985). The effect of students’ academic discipline on their performance on ESP reading tests. Language Testing, 2(2), 192–204. https://doi.org/10.1177/026553228500200207
• Al-Faris, E. A., Alorainy, I. A., Abdel-Hameed, A. A., & Al-Rukban, M. O. (2010). A practical discussion to avoid common pitfalls when constructing multiple choice questions items. J Family Community Med, 17(2), 96-102. Doi: 10.4103/1319-1683.71992. PMID: 21359033; PMCID: PMC3045096
• Allanson, P. E., & Nortar, C. E. (2019). Writing multiple choice questions that are reliable and valid. American International Journal of Humanities and Social Science, 5(3), 1-9.
• Al-Musawi, N. (2001). The validity of scores on TOEFL and FCE for predicting students' success at the university. Dirasat: Educational Science, 28(1).
• Alshammari, H. A. M. (2012). Effects of time constraint on second language reading comprehension [MA Thesis, Southern Illinois University/Carbondale]. https://opensiuc.lib.siu.edu/theses/1071/
• Amiryousefi, M., & Tavakoli, M. (2011). The relationship between test anxiety, motivation and MI and the TOEFL iBT reading, listening and writing scores. Procedia - Social and Behavioral Sciences, 15, 210-214.
• Anderson, A., & Lynch, T. 2000. Listening. Oxford: Oxford University Press.
• Anderson, R.C., & Pearson, P.D (1984). A schematic-theoretic view of basic processes in reading. In P.D. Pearson, M. Kamil, R.Barr, & P.Mosenthal (Ed.), Handbook of reading research (pp.255-291). New York: Longman.
• Anderson, R. C., Pichert, J. W., & Shirey, L. L. (1979). Effects of the reader’s schema at different point in time. Urbana-Champaign, IL: University of Illinois, Center for the Study of Reading.
• Axe, et al. (2020). Validity evidence supporting the interpretation and use of TOEFL iBT scores. (TOEFL iBT Research Report Vo. 4). Princeton, NJ: Educational Testing Service.
• Asrida, R., & Fitrawati, F. (2019). The Difficulties of English department students
at Universitas Negeri Padang in answering reading section of TOEFL.
Journal of English Language Teaching, 8(4), 496-503–503.
https://doi.org/10.24036/jelt.v8i4.106497
• Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press
• Bachman, L., Davidson, F., Ryan, K., & Chol, Inn. (1995). An investigation into the comparability of two tests of english as a foreign language. Cambridge, Cambridge University Press.
• Baghaei, P., & Amrahi, N. (2011). The effects of the number of options on the psychometric characteristics of multiple-choice items. Psychological Test and Assessment Modeling, 53(2), 192-211.
• Barnitz, J. G. (1985). Reading development of nonnative speakers of English. Orlando, FL: Harcourt Brace Jovanovich.
• Begum, T. (2012). A guideline on developing effective multiple-choice questions and construction of single best answer format. Journal of Bangladesh College of Physicians and Surgeons, 30(3), 159-166.
• Bever, T.G. (1970). The cognitive basis for linguistic structures. In J. R. Hayes (Ed.), Cognition and the development of language. (pp. 279-362). New York: Wiley.
• Brame, C. (2013). Writing good multiple choice test questions. https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/
• Bridgeman, B., Cho, Y., & DiPietro, S. (2015). Predicting grades from an English language assessment: The importance of peeling the onion. Language Testing, 33(3), 307–318. https://doi.org/10.1177/0265532215583066
• Brookhart, S. M. (2015). Making the most of multiple choice. Educational Leadership, 73(1), 36-39.
• Carrell, P., Devine, J., & Eskey, D. (1988). Interactive approaches to second language reading. Cambridge: Cambridge University Press.
• Carter, C. (2011, September 21). The Case against standardized tests. NEBRASKAMATH. https://news.unl.edu/newsrooms/csmce/article/how-valid-are-standardized-tests-not-much-test-makers-want-us-think/
• Cheng, L., Klinger, D., Fox, J., Doe, C., Jin, Y., & Wu, J. (2014). Motivation and test anxiety in test performance across three testing contexts: The CAEL, CET, and GEPT. TESOL Quarterly, 48(2), 300–330. https://doi.org/10.1002/tesq.105
• Chiang, C.C., & Dunkel, P. (1992). The Effect of speech modification, prior knowledge and listening proficiency on EFL lecture learning. TESOL Quartely. 26 (2), 345- 374.
• Cho, Y., & Bridgeman, B. (2012). Relationship of TOEFL iBT® scores to academic performance: Some evidence from American universities. Language Testing, 29(3), 421–442. https://doi.org/10.1177/0265532211430368
• Clapham, C. (1998). The effect of language proficiency and background knowledge on EAP students’ reading comprehension. In A. J. Kunnan (Ed.), Validation in language assessment (pp. 141–168). Mahwah, NJ: Lawrence Erlbaum
•
• Cohen, A., Glasman, H., Rosenbaum-Cohen, P. R., Ferrara, J., & Fine, J. (1979). Reading English for specialized purposes: Discourse analysis and the use of student informants. TESOL Quarterly, 13, 551–564.
• Cohen, A. D. & Upton, T. A. (2006). Strategies in responding to the new TOEFL reading tasks. TOEFL Monograph Series, MS – 33. Princeton, NJ: Educational Testing Service.
• Collins and Sorrenson. (2014). McGraw-Hill education TOEFL iBT. New York: McGraw-Hill Education.
• Collins English Dictionary. (2023). Available from: https://www.collinsdictionary.com/ [Accessed 2 May 2023]
• Daneman, M., & Carpenter, P. A. (1980). Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior, 19(4), 450-466. http://dx.doi.org/10.1016/S0022-5371(80)90312-6
• Dechant, E. (1991). Understanding and teaching reading: An interactive model. Hillsdale, NJ: Lawrence Erlbaum.
• Dornyei, Z. (2009). The Psychology of second language acquisition. Oxford: Oxford University Press.
• Dörnyei, Z., & Skehan, Z. (2003). Individual differences in second language learning. In C. J. Doughty, & M. H. Long (Eds.), The handbook of second language acquisition (pp. 589-630). Oxford: Blackwell Publishing Ltd.
https://doi.org/10.1002/9780470756492.ch18
• Educational Testing Service. (July 1, 2023). TOEFL iBT Scores.
https://www.ets.org/toefl/test-takers/ibt/scores.html
• Educational Testing Service. (July 10, 2023). TOEFL iBT reading section. https://www.ets.org/toefl/test-takers/ibt/about/content/reading.html
• Esfandiari, M. R., Riasati, M. J., Vaezian, H., & Rahimi, F. (2018). A quantitative analysis of TOEFL iBT using an interpretive model of test validity. Language Testing in Asia 8(1), 1-13. DOI: 10.1186/s40468-018-0062-7
• Fahim, M., Bagherkazemi, M., & Alemi, M. (2010). The relationship between test takers' multiple intelligences and their performance on the reading sections of TOEFL and IELTS. Broad Research in Artificial Intelligence and Neuroscience, 1 (3), 0-14
• Feast, V. (2002). The impact of IELTS scores on performance at university. International Education Journal, 3(4), 70–85.
• Forster, K. I. (1976). Accessing the mental lexicon. In F. Wales & E. Walker (Eds). New approaches to language mechanisms (p. 257-287). Amsterdam: North Holland.
• Fox, J. (2004). Test decisions over time: tracking validity. Language Testing, 21(4), 437–465. https://doi.org/10.1191/0265532204lt292oa
• Fries, C. C. (1945). Teaching and learning English as a foreign language. Ann Arbor: University of Michigan Press.
• Fries, C. C. (1963). Linguistics and reading. New York: Holt, Rinehart and Winston.
• Fries, C. C. (1972). Learning to read English as part of the oral approach. In K. Croft (Ed.), Reading on English as a second language: For teachers and teacher-trainers (pp.168- 173). Cambridge: Winthrop Publishers.
• Gebhard, J. (2000). Teaching English as a foreign or second language: A teacher self –development and methodology guide. United States of America: The University of Michigan Press
• Golder, K., Reeder, K., & Fleming, S. (2009). Determination of appropriate IELTS band score for admission into a program at a Canadian post-secondary polytechnic institution. IELTS Research Reports, 10, 69–94.
• Goodman, K. S. (1967). Reading: A psycholinguistic game. Journal of the Reading Specialist, 6(1), 126-135.
• Gough, P. B. (1984). Word recognition. In Pearson P. D. (Ed.), Handbook of reading research (pp. 225–254). New York: Longman.
• Grabe, W. (2009). Reading in a second language: Moving from theory to practice. New York: Cambridge University Press.
• Grandt, J. (1987). Characteristics of examinees who leave questions unanswered on the GRE general test rights-only scoring (ETS Research Report 87- 83). Princton, NJ: Educational Testing Service.
• Hammad, E. A. (2021). Palestinian EFL university students' problems with the reading sections of the TOEFL internet-based test and the revised TOEFL paper-delivered Test. Arab World English Journal, 12 (3), 51-65. DOI: https://dx.doi.org/10.24093/awej/vol12no3.4
• Harsch, C., Ushioda, E., & Ladroue, C. (2017). Investigating the predictive validity of TOEFL iBT®test scores and their use in informing policy in a United Kingdom university setting (TOEFL iBT Research Report No. 30). Princeton, NJ: Educational Testing Service. https://doi.org/10.1002/ets2.12167
• Hill, K., Storch, N., & Lynch, B. (1999). A comparison of IELTS and TOEFL as predictors of academic success. IELTS Research Reports, 2, 52–63.
• Hill, Y. Z., & Liu, O. L. (2012). Is there any interaction between background knowledge and language proficiency that affects TOEFL iBT Reading performance? (TOEFL Research Report No. 18). Princeton, NJ: Educational Testing Service.
• Harris, T. L., & Hodges, R. E. (Eds.). (1995). The literacy dictionary: The vocabulary of reading and writing. Newark, DE: International reading Association.
• Hughes, A. (2003). Testing for language teachers. Cambridge: Cambridge University Press.
• Kim, Y., & Jang, E. E. (2009). Differential functioning of reading subskills on the OSSLT for L1 and ELL students: A multidimensionality model-based DBF/DIF approach. Language Learning, 59(4), 825-865. https://doi.org/10.1111/j.1467-9922.2009.00527.x
• Khonamri, F., & Kojidi, M. E. (2011). Metacognitive awareness and comprehension monitoring in reading ability of Iranian EFL learners. PROFILE, 13(2), 99-111.
• Krashen, S. (1982). Principles and practice in second language acquistion. Oxford: Pergamon Press
• Kubat, U. (2018). Identifying the individual differences among students during learning and teaching process by science teachers. International Journal of Research in Educational and Science, 4(1), 30-38. DOI:10.21890/ijres.369746
• Lee, J. Y. (2011). Second language reading topic familiarity and test score: Test-taking strategies for multiple-choice comprehension questions. [PhD Dissertation, University of Iowa]. https://mobt3ath.com/pdf.php?ext=pdf&id=23628&tit=Second_language_reading_topic_familiarity_and_test_score-_test-taking_strategies_for_multiple-choice_comprehension_questions_%D8%B1%D8%B3%D8%A7%D9%84%D8%A9_%D8%AF%D9%83%D8%AA%D9%88%D8%B1%D8%A7%D9%87
• Lee, Y. J., & Greene, J. (2007). The predictive validity of an ESL placement test: A mixed methods approach. Journal of Mixed Methods Research, 1(4), 366–389. https://doi.org/10.1177/1558689807306148
• Li, Y. (2018) A comparison of TOEFL iBT and IELTS reading tests. Open Journal of Social Sciences, 6(8), 283-309. https://doi: 10.4236/jss.2018.68023.
• Liu, O. L. (2011). Do major field of study and cultural familiarity affect TOEFL ® iBT reading performance? A confirmatory approach to differential item functioning. Applied Measurement in Education, 24(3), 235-255.
• Liu, O, L., Schedl, M., Malloy, J., & Kong, N. (2009). Does content knowledge affect TOEFL iBT reading performance? A confirmatory approach to differential item functioning. TOEFL iBT research report. Princeton New Jersey, Educational Testing Service.
• Maizarah, M. (2019). Analysis of the students’ common difficulties in TOEFL
reading comprehension at the islamic university of indragiri tembilahan.
EJI (English Journal of Indragiri) : Studies in Edzarnisucation, Literature, and
Linguistics, 3(2), 99–106. https://doi.org/10.32520/eji.v3i2.561
• Manganello, M. (2011). Correlations in the new TOEFL era: An investigation of the statistical relationships between IBT scores, placement test performance, and academic success of international students at Iowa State University [doctoral dissertation, Iowa State University]. https://dr.lib.iastate.edu/entities/publication/0c621172-310b-47cf-a25f-a4375c42954b
• Messick, S. (1996). Validity and washback in language testing. Language Testing, 13(3), 241–256. https://doi.org/10.1177/026553229601300302
• Miller, G. (2017). Metacognitive awareness and reading strategy use: Investigating the intermediate level ESL students’ awareness of metacognitive reading strategies. [Master Thesis, St. Cloud State University] https://repository.stcloudstate.edu/do/search/?q=Metacognitive%20Awareness%20and%20Reading%20Strategy%20use%3A%20Investigating%20the%20Intermediate%20Level%20ESL%20Students%E2%80%99%20Awareness%20of%20Metacognitive%20Reading%20Strategies&start=0&context=2177325&facet=
• Morton, J. (1979). Facilitation in word recognition: Experiments causing change in the logogen model. In: Kolers, P. A., Wrolstad, M. E., Bouma, H. (eds) Processing of visible language (pp. 259-268). Nato Conference Series, 13. Springer, Boston, MA. https://doi.org/10.1007/978-1-4684-0994-9_15
• Ng, J. N. (2007). Test of English as a foreign language (TOEFL): Good indicator for student success at community college? [Mater Thesis, Oregon State University]. https://ir.library.oregonstate.edu/downloads/0p096931h
• Nguyen, T. T. T. (2012). The Impact of background knowledge and time constraint on reading comprehension of Vietnamese learners of English as a second Language. [Master Thesis, Southern Illinois University/ Carbondale]. https://opensiuc.lib.siu.edu/theses/931/
• O’Dwyer, J., Kantarcioglu, E., & Thomas, C. (2018). An investigation of the predictive validity of the TOEFL iBT® Test at an English-medium university in turkey. (TOEFL iBT Research Report No. 83). Princeton, NJ: Educational Testing Service.
• Ovilia, R. (2018). The relationship of topic familiarity and listening comprehension. Proceedings of the Sixth International Conference of English Language and Teaching (ICOELT 1028) - Advances in Social Science, Education and Humanities Research, 276, 182-186. https://dx.doi.org/10.2991/icoelt-18.2019.29
• Othman, J., & Vanathas, C. (2017). Topic familiarity and its influence on listening comprehension. The English Teacher, XXXIV, 19-32
• Panahi, A. (2014). Threats to validity: construct-irrelevant variances contributing to performance under-representation on Graduate Record Exam (GRE). Journal of Education & Human Development, 3(1), 327-346.
• Phillips, D. (2015). Longman preparation course for the TOEFL iBT test. Ney York: Pearson Education, Inc.
• Priebe, S.J., Keenan, J. M., & Miller, A.C. (2012). How prior knowledge affects word identification and comprehension. Read and Writing, 25, 131-149.
• Putlack, M. A., Poirier, S., & Jacobs, A. C. (2020). Decoding the TOEFL ® iBT: Actual test, reading 2. Korea: Darakwon.
• Rivers, W. M. (1968). Teaching foreign language skills. Chicago: University of Chicago Press.
• Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13. https://doi.org/10.1111/j.1745-3992.2005.00006.x
• Rogers, B. (2007). The complete guide to the TOEFL test: iBT edition. Boston: Thomson Heinle.
• Sadighi, F., & Zare, S. (2006). Is listening comprehension influenced by the background knowledge of the learners? A case study of iranian EFL learners. The Journal of Linguistics 1(3), 110-126.
• Schmidt-Rinehart, B. (1994). The Effects of topic familiarity on second language listening comprehension. The Modern Language Journal 78 (2), 179-198
• Siqueira, M. T., Gonçalves, J. P., Mendonça, V. S., Kobayasi, R., Arantes-Costa, F. M., Tempski, P. Z., & Martins, M. A. (2020). Relationship between metacognitive awareness and motivation to learn in medical students. BMC Medical Education, 20(1). https://doi.org/10.1186/s12909-020-02318-8
• Sun, M. (2021). Validity and fairness of TOEFL iBT reading test. Learning & Education, 10(8), 141-142.
• Syarif, H. (2018). Lexical density vs grammatical intricacy: How are they related? Advances in Social Science, Education and Humanities Research, 276, 16-22
• Takala, S., & Kaftandjieva, F. (2000). Test fairness: a DIF analysis of an L2 vocabulary test. Language Testing, 17(3), 323–340. https://doi.org/10.1177/026553220001700303
• ETS. (2017). The official guide to the TOEFL® test. New York, NY :McGraw-Hill.
• Toker, D. (2019). Topic familiarity matters: A critical analysis of TOEFL iBT reading section. TESL-EJ: The Electronic Journal for Teaching English as a Second Language, 23(1), 1-9.
• Torres, C., Lopes, A. P., Babo, L., & Azevedo, J. (2011). Improving multiple-choice questions. US-China Education Review, 8(1), 1-11
• Vu, L. T., & Vu, P. H. (2013). Is the TOEFL score a reliable indicator of international graduate students’ academic achievement in american higher education? International Journal on Studies in English Language and Literature (IJSELL), 1(1), 11-19.
• Wait, I., & Gressel, J. (2009). Relationship between TOEFL score and academic success for international engineering students. Journal of Engineering Education, 98(4), 389–398. https://doi.org/10.1002/j.2168-9830.2009.tb01035.x
• Woodrow, L. (2006). Academic success of international postgraduate education students and the role of english proficiency. University of Sydney Papers in TESOL, 1, 51–70.
• Young, D. J. (1991). Creating a low-anxiety classroom environment: What does the language anxiety research suggest? Modern Language Journal, 75(4), 426-437. https://doi.org/10.1111/j.1540-4781.1991.tb05378.x
• Zalha, F.B., Alfiatunnur, A., & Kamil, C. A. T. (2020). Strategies in dealing with the reading section of ‘TOEFL prediction’: A case of aceh EFL learners. Indonesian Journal of English Education, 7(2), 159-171.
• Zarnis, Y. (2020). Analysis of English education department students’ difficulties
in reading comprehension text of TOEFL [Diploma, UIN SMH BANTEN].
http://repository.uinbanten.ac.id/5553/
• Zhao, X., & Zhu, L. (2012). Schema theory and college English reading teaching. English Language Teaching, (5)1, 111-117.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 م.م. ژیار عثمان حمد امین
This work is licensed under a Creative Commons Attribution 4.0 International License.
