I ran across this very helpful article recently that does with reading and listening comprehension skills that I have been doing with reading-writing subskills.
Song, M-Y. (2009). Do divisible subskills exists in second language (L2) comprehension? A structural equation modeling approach. Language Testing, 25 (4), 435-464.
In essence, the author asks: to what degree does student performance on a test suggest that reading, listening, and 2-3 identified subskills exist as separate constructs (as evidenced by structural equation modeling)? In comparison, I have been trying to identify the degree to which reading comprehension, writing ability, synthesis comprehension, and paraphrase writing ability are all subskills of reading-to-write tasks. So this article provided quite a bit of theoretical framing and technical analysis that I need to further refine my own study.
Song starts the article by pointing out that the concept of “subskills” is a controversial one. Not only do researchers not agree on how many subskills (if any) exist for a particular language skill (i.e. reading), but they also do not agree on the value of identifying subskills. For example, I recently read (and commented on this blog about) Grabe’s 2009 book on L2 Reading. He provides about 6 subskills for reading, but other researchers claim there are really only 3, whereas others identify over 30.
These divergent views on the existence and number of subskills can seem confusing: are there or aren’t there a certain number of subskills? In truth, I think that it is all relative to the context. As Song points out at the end of the article, “the validity… is significantly affected by the characteristics of he specific dataset explored in this study” (p. 459). In other words, evidence for or against subskills is dependent on the meauring instrument (does it cause users to access those subskills) and on the proficiency of the specific group of leaners (are they at such a level that they is a measureable difference among their subskill use). Song found evidence for 3 listening subskills but only 2 reading skills. This could be a result of the test construction (that gave users more time to reading and hence eliminated the differentiation between subtle subskills) and a result of user ability (the test takers may simply be better readers holistically, so they perform all test items well, thus eliminating any statistical difference among the subskill specific items).
This is why it is important that my study (and my measurement instrument) be used with the target audience of new or in-coming university-level English for Academic Purposes students. Learners who are below this level (such as nearly arrived ESL students in the novice range) would perform poorly on all aspects of my test, and users well above this target level (such the Western European students in the law school class that I teach) would perform extremely well on all parts of the test. For both groups, there would be little evidence of subskills for either reading or writing, since all test items would have similar scores for each group. However, the group in the middle (which is the TOEFL target group and those in their first year post-TOEFL) would take the test and provide much evidence that seperate subskills do exists when completing reading-to-write tasks. And since it is this group that the diagnostic testing instrument was designed for, then this works perfectly. This is the group of learners that need help to develop the more advanced subskills that my study highlights.
Remaining thoughts:
- I need to show how my study addresses the 3 implications that Song offers on page 460
- subskills help us to test for needed language abilities
- subskills should be reported as seperate scores to students and instructors
- researchers need to investigate how users accomplish test tasks that claim to measure specific subskills (in order to learn whether those subskills are being used, or whether something else is happening)
- I need to more clearly identify the subskills that current research identifies as part of the reading-to-write task
- I need to explictly state my models for SEM
- Test for unitary trait (literacy)
- Test for two skills (reading versus writing)
- Test for two abilities (basic literacy versis advanced literacy)
- Test for multiple subskills (reading, writing, synthesis, paraphrasing)
- I need to define all of these constructs
- I need to justify that my test takers are in the target proficiency group
- I need to have the right software to analyze my data
