Striking the Balance between Validity and Reliability of a Listening Test in Turkish as a Second Language

Emel Tozlu, Aylin Ünaldı

Research output: Contribution to journalArticle

Abstract

Evidence on the efficacy of an assessment tool is necessary in order to justify the decisions we make based on the scores from it. Validity evidence can be collected from several sources such as the stages before and after test administration. In the present research study, validity evidence of several types on a Turkish as a Second Language (TSL) Academic Listening Test is presented in order to establish the efficacy of it. This paper presents cognitive, contextual and scoring validity (reliability) evidence from the first and second versions of the test and investigates whether the modifications made after the first administration have had a positive effect on the quality of the test. The study concludes that although the changes made in the first version of the test strengthened the validity claims in terms of cognitive and contextual requirements, the reliability scores of the test worsened in the second version. This reminded us that although it is necessary to build the foundations of a test firmly by operationalizing the necessary contextual features and cognitive processes, this will not thoroughly guarantee the technical quality of the items. Scoring validity should be established carefully as well. This study exemplifies a thorough attempt in establishing the validity of a TSL test from multiple perspectives and aims to be an exemplary study for further test development in TSL.
Original languageEnglish
Article number627635
Pages (from-to)49-73
Number of pages25
JournalBogazici University Journal of Education
Volume34
Issue number1
Publication statusPublished - 17 Dec 2018
Externally publishedYes

Fingerprint

language
evidence
guarantee

Cite this

@article{d6e43d3a942a43dea4050bafb2678984,
title = "Striking the Balance between Validity and Reliability of a Listening Test in Turkish as a Second Language",
abstract = "Evidence on the efficacy of an assessment tool is necessary in order to justify the decisions we make based on the scores from it. Validity evidence can be collected from several sources such as the stages before and after test administration. In the present research study, validity evidence of several types on a Turkish as a Second Language (TSL) Academic Listening Test is presented in order to establish the efficacy of it. This paper presents cognitive, contextual and scoring validity (reliability) evidence from the first and second versions of the test and investigates whether the modifications made after the first administration have had a positive effect on the quality of the test. The study concludes that although the changes made in the first version of the test strengthened the validity claims in terms of cognitive and contextual requirements, the reliability scores of the test worsened in the second version. This reminded us that although it is necessary to build the foundations of a test firmly by operationalizing the necessary contextual features and cognitive processes, this will not thoroughly guarantee the technical quality of the items. Scoring validity should be established carefully as well. This study exemplifies a thorough attempt in establishing the validity of a TSL test from multiple perspectives and aims to be an exemplary study for further test development in TSL.",
keywords = "Cognitive validity, contextual validity, scoring validity, assessment of listening in Turkish as a second language",
author = "Emel Tozlu and Aylin {\"U}naldı",
year = "2018",
month = "12",
day = "17",
language = "English",
volume = "34",
pages = "49--73",
journal = "Bogazici University Journal of Education",
issn = "1300-9567",
number = "1",

}

Striking the Balance between Validity and Reliability of a Listening Test in Turkish as a Second Language. / Tozlu, Emel ; Ünaldı, Aylin.

In: Bogazici University Journal of Education, Vol. 34, No. 1, 627635, 17.12.2018, p. 49-73.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Striking the Balance between Validity and Reliability of a Listening Test in Turkish as a Second Language

AU - Tozlu, Emel

AU - Ünaldı, Aylin

PY - 2018/12/17

Y1 - 2018/12/17

N2 - Evidence on the efficacy of an assessment tool is necessary in order to justify the decisions we make based on the scores from it. Validity evidence can be collected from several sources such as the stages before and after test administration. In the present research study, validity evidence of several types on a Turkish as a Second Language (TSL) Academic Listening Test is presented in order to establish the efficacy of it. This paper presents cognitive, contextual and scoring validity (reliability) evidence from the first and second versions of the test and investigates whether the modifications made after the first administration have had a positive effect on the quality of the test. The study concludes that although the changes made in the first version of the test strengthened the validity claims in terms of cognitive and contextual requirements, the reliability scores of the test worsened in the second version. This reminded us that although it is necessary to build the foundations of a test firmly by operationalizing the necessary contextual features and cognitive processes, this will not thoroughly guarantee the technical quality of the items. Scoring validity should be established carefully as well. This study exemplifies a thorough attempt in establishing the validity of a TSL test from multiple perspectives and aims to be an exemplary study for further test development in TSL.

AB - Evidence on the efficacy of an assessment tool is necessary in order to justify the decisions we make based on the scores from it. Validity evidence can be collected from several sources such as the stages before and after test administration. In the present research study, validity evidence of several types on a Turkish as a Second Language (TSL) Academic Listening Test is presented in order to establish the efficacy of it. This paper presents cognitive, contextual and scoring validity (reliability) evidence from the first and second versions of the test and investigates whether the modifications made after the first administration have had a positive effect on the quality of the test. The study concludes that although the changes made in the first version of the test strengthened the validity claims in terms of cognitive and contextual requirements, the reliability scores of the test worsened in the second version. This reminded us that although it is necessary to build the foundations of a test firmly by operationalizing the necessary contextual features and cognitive processes, this will not thoroughly guarantee the technical quality of the items. Scoring validity should be established carefully as well. This study exemplifies a thorough attempt in establishing the validity of a TSL test from multiple perspectives and aims to be an exemplary study for further test development in TSL.

KW - Cognitive validity

KW - contextual validity

KW - scoring validity

KW - assessment of listening in Turkish as a second language

M3 - Article

VL - 34

SP - 49

EP - 73

JO - Bogazici University Journal of Education

JF - Bogazici University Journal of Education

SN - 1300-9567

IS - 1

M1 - 627635

ER -