The data has been collected at an assessment center where four raters rated 100 managerial candidates on their performance in a series of situational exercises and interviews. One hundred subjects were rated on 5-point Likert type scales on 17 items. Based on judgments by a content expert, the 17 items were divided into 4 dimensions, where the expert believes that the items in the same dimension share a common trait. Items are grouped into the 4 dimensions as follows:
The main goal is to collect evidence to support the content expert’s proposed 4-dimensional structure of the 17 rating items. In the data set, the four sets of 17 variables are the four rater’s assessment of the 100 subjects on each of the 17 items. Each variable is identified by rater and item. For examples, the variable “r1i1” indicates the rating from the first rater on the first item, “r3i16” indicates the ratting from the third rater on the 16th item, and so on.
We need to find the answers to the following questions and hypotheticals about the reliability of this 17-item rating scale. First though you will need to create 17 new variables, where each variable represents the mean of the 4 raters for each of the 17 items.
- Perform a confirmatory factor analysis on the 17 items by Mplus. Use the 17 new variables created. First, you need to save the SPSS data file as a tab-delimited file, which contains only the aggregated 17 variables. Do not include variable names when you create a tab-delimited file. When you run Mplus, request standardized coefficients.
a. First, assume a 1-factor model. Examine standardized factor loadings and comment on their magnitudes. Are they generally high? Are there any items with considerably lower loading than others?
b. What is the chi-square to degrees-of-freedom ratio (c2/df)?
c. What are the CFI and TLI?
d. What are the RMSEA and its 90% confidence interval?
- Now, perform a confirmatory factor analysis on the 17 items, by assuming a 4-factor model based on the suggested 4-dimensional structure.
a. Examine standardized factor loadings and comment on their magnitudes. Are they generally high? Are there any items with negative or near zero loading than others?
b. Examine fit indices: chi-square to degrees-of-freedom ratio (c2/df), CFI and TLI, RMSEA and its 90% CI, Do these model-fit indices indicate better fit of the 4dimensional structure, compared to the fit of the 1-dimensional structure?
c. Compute convergent validity coefficient for each of the 4 dimensions. Use unstandardized and unstandardized 2 for computations.
d. Compute discriminant validity coefficients. Note that there are 12 discriminant validity coefficient in this case.
e. Summarize the results in questions 2-c and 2-d as a convergent-discriminant validity matrix.
f. Do results support the construct evidence of the validity for the “expert’s judgments”? Why? Why not? 3. Based on a series of analyses you have conducted, do you agree with the 4-dimensional structure of the 17 rating items? Why? Why not? Discuss both from your analysis results and the content of the rating items.
- Comment on what is the best way to report the scores. The total of 17 items, or separate 4 scores for the 4 dimensions? Something else?