Describe one use of the WCTT that you believe is not justified, given the information provided in the manual. In order for the WCTT to be used in this way (or for this purpose), what information would you like to see in the manual?Practical Application of Psychometric Basics
DUE DATE: See Blackboard for due date.
Format: Submit to Blackboard, Assignments tab. Naming format of the file should be the following: First name Last Name_Module 6 Integrative Assignment. Assignments without this naming format will have 5 points deducted.
Instructions: You are given two pages of a mock test manual that presents information for a fabricated 12-item test called the Wagner’s Critical Thinking Test, or WCTT. The test manual contains information about how the measure was developed and norms/basic statistics. Further, the manual provides brief summaries of several reliability and validity studies, as well as a factor analysis table. Importantly, these summaries provide only the methods and findings of the studies, but do not specify their purposes (i.e., that the study is designed to provide evidence for internal consistency, or predictive criterion validity, or discriminant construct validity, etc.). You must evaluate the measure based on the information provided. Each question is worth 5 points.
After you have read the Abbreviated Test Manual for the WCTT (the next pages of this assignment), answer the questions presented below. Your responses should include both a definition of the measurement concept reflected in the question (e.g., reliability, criterion-related validity) and reference to specific pieces of information contained in the manual.
1. Fully describe the evidence in the test manual that supports the WCTT’s reliability. How well does the evidence support the reliability of the measure?
2. Fully describe the evidence in the test manual that supports the measure’s content-related validity. How well does the evidence support the content-related validity of the measure?
3. Fully describe the evidence in the test manual that supports the measure’s criterion-related validity. How well does the evidence support the criterion-related validity of the measure?
4. Fully describe the evidence in the test manual that supports the measure’s construct-related validity. How well does the evidence support the construct-related validity of the measure?
5. Describe one use of the WCTT that you believe is justified, given the information provided in the manual.
6. Describe one use of the WCTT that you believe is not justified, given the information provided in the manual. In order for the WCTT to be used in this way (or for this purpose), what information would you like to see in the manual?
WAGNER CRITICAL THINKING TEST (WCTT)*
Abbreviated Test Manual
* The Wagner Critical Thinking Test and the Abbreviated WCTT Manual are not actual test materials
Brief Introduction
Critical thinking is an essential tool of inquiry. Although critical thinking has been conceptualized in various ways, virtually all scholars agree that this way of thinking involves cohesive and logical reasoning about assumptions, arguments, evidence, and conclusions.
The Wagner Critical Thinking Test (WCTT) was designed to assess students’ critical thinking abilities. The WCTT’s brevity, low cost, and ease of administration and scoring set it apart from other published measures of critical thinking. Potential uses of the WCTT include prediction of students’ success in both undergraduate and graduate academic programs; assessment of students’ progress in applying critical thinking skills, and evaluation of the effectiveness of academic courses designed to develop students’ critical thinking skills.
Development of the WCTT
In developing the WCTT, Wagner conducted an exhaustive search and review of the scholarly literature on critical thinking. Based on his research, Wagner concluded that there are four core elements of critical thinking:
(1) Understanding Meaning (UM) : the ability to understand the meaning and significance of a statement or argument
(2) Recognizing Assumptions (RA) : the ability to recognize unstated assumptions and values in a statement or argument
(3) Identifying and Evaluating Arguments (IEA) : the ability to identify central arguments and pieces of evidence, and to evaluate their strength and relevance
(4) Assessing Inferences and Conclusions (AIC): the ability to discern whether inferences and conclusions follow logically from evidence
Wagner then began to construct his measure of critical thinking. First, he wrote a short passage on a specific topic (the death penalty as a deterrent to crime) which contained unstated assumptions, various pieces of data/evidence, and several inferences. Because Wagner’s goal was to develop a measure of critical thinking skills, not verbal ability, he attempted to write the passage using relatively elementary vocabulary, understandable terms, and simple sentence structure.
Wagner generated four sets of eight multiple-choice items related to the passage. Each set of items measured one of the four core elements of critical thinking (UM, RA, IEA, and AC), yielding a total of 32 multiple-choice items. Unlike traditional multiple-choice items in which one response alternative is correct and the others are incorrect, Wagner’s multiple choice items included two correct/acceptable response options and two incorrect/unacceptable response options. Items could be scored as 2 (best response), 1 (acceptable response), or 0 (incorrect response).
The written passage and the 32 multiple choice items were administered to 236 students (M age=20.1 years, SD=1.52) at a large Midwestern university. An item analysis was performed on the 32 original items, and those with item-total correlations of less than .30 were discarded. Twelve items (three items for each of the four critical thinking elements) remained, forming the final Wagner Critical Thinking Test (WCTT).
WCTT Norms
Normative data are based on a combined sample of 1524 undergraduate students (863 females and 661 males; M age=19.5 years, SD=2.01) attending five universities in the United States. No significant differences in scores between gender, regional, or ethnic groups were found. Range of WCTT scores= 3-23; Mean score=15.2 (SD=3), Median score=15.
Studies on the WCTT
Study #1: The WCTT was administered to 332 undergraduate students enrolled in general education courses. Sixteen days later, the same students took the WCTT again. The correlation between the two sets of scores was .89.
Study #2: A sample of 475 college students completed the WCTT, along with several other measures. Correlations between the WCTT and these measures were as follows: the Watson-Glaser Critical Thinking Appraisal (r=.61, p<.001), the Peak Verbal Ability Test (r=.43, p<.001) and the Need for Achievement Inventory (r=.06, ns).
Study #3: A sample of 300 college-bound students completed the WCTT during their senior year of high school. One year later, these students were contacted and asked to participate in a follow-up study. Of the original 300 students, 282 agreed to provide official transcripts from their first year in college. The correlation between participants’ WCTT scores and first-year college grade point average was .30 (p<.001).
Study #4: Analysis of WCTT scores obtained from a sample of 640 students at a large Midwestern university yielded a Cronbach’s alpha coefficient of .78.
Study #5: The WCTT was administered to all graduating seniors (N=315) of a liberal arts college in the northeast. The mean WCTT score of students graduating with honors (cum laude, magna cum laude, and summa cum laude) was 18.4 and the mean score of the other graduating students was 15.9. This difference was statistically significant (p<.01).
Study #6: A confirmatory factor analysis was performed on WCTT scores obtained from a sample of 1524 undergraduate students. The results of the factor analysis are shown in the table on the next page.
Table 1: Factor Analysis of Wagner Critical Thinking Test
Critical Thinking Element FACTOR LOADINGS
(& items written for element)
I II III IV
Understanding Meaning (UM)
UM item 1 .54 .30 .09 -.01
UM item 2 .61 .32 .02 .06
UM item 3 .52 .20 .12 .05
Recognizing Assumptions (RA)
RA item 1 .39 .49 .09 .14
RA item 2 .28 .62 .11 .15
RA item 3 .31 .50 .06 .09
Identifying and Evaluating Arguments (IEA)
IEA item 1 .12 -.04 .25 .46
IEA item 2 .02 .09 .40 .19
IEA item 3 .09 .02 .51 .32
Assessing Inferences and Conclusions (AIC)
AIC item 1 .10 .07 .21 .50
AIC item 2 .16 .12 .31 .29
AIC item 3 . .04 .10 .44 .34
____________________________________________________
