Summary of WHP Research 2021-22
SUMMARY OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING 2021/22 SCHOOL YEARRachel S. Phillips, PhD and Meghan Oster, PhD
2SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARTable of ContentsIntroduction .........................................................................................................................................................3Data Collection ...........................................................................................................................................3Study 1—WHP Origins and WHP 1750 ............................................................................................................4Participants ..................................................................................................................................................4Findings ........................................................................................................................................................5Analysis of Student Writing ...............................................................................................................5Scores by Writing Variable ..................................................................................................................6Discussion of Study 1 .................................................................................................................................9Study 2—WHP 1200 .........................................................................................................................................10Participants ................................................................................................................................................10Findings ......................................................................................................................................................10Analysis of Student Writing .............................................................................................................10Scores by Writing Variable .................................................................................................................11Discussion of Study 2 ..............................................................................................................................14Conclusions ................................................................................................................................................14Cover: The sultan of the Ottoman Empire Selim III receiving dignitaries at the Gate of Felicity in Istanbul, painting by an unknown artist. Turkey, 18th century. Istanbul, Topkapi Museum. © DeAgostini / Getty Images.
3SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARIntroductionEvery year OER Project engages in research to examine factors related to student learning and teacher and student perceptions and satisfaction with our courses. This research report focuses on student growth in writing in our World History Project (WHP) courses. Improving students’ ability to write coherent, logical essays that effectively use evidence and apply disciplinary concepts is one of WHP’s most important goals. Writing is essentially thinking on paper. Developing students’ capacity to write and communicate well is among the most important educational outcomes. WHP is a well-scaffolded, writing-intensive course that includes ample professional development activities. Students are exposed to lessons designed to develop their historical thinking and reasoning skills while being given many opportunities to refine their writing ability. Teachers receive tools that enable them to extend their students’ capacity to use a range of texts as evidence and develop coherent and sophisticated arguments. Although it is the wealth of resources related to historical content in the course that draws the most attention, in many ways we are proudest of the growth we see in student thinking as displayed in their writing growth. The 2021/22 school year was the second year that WHP was widely available to teachers, and the second year we have studied student growth in writing over the course of the school year. As with the first year, most students who participated in the study realized writing growth, with an average gain of .75 points on a 4-point scale. Given that this year’s study involved more participants than the 2020/21 school year did, and it involved more teachers who were using our curriculum for the first time, we feel confident that these results are both valid and replicable for students who take our WHP courses. Data Collection For the WHP study, 1,445 students in 42 classrooms submitted both the baseline Wave 1 and the post essay Wave 2. Note that the essays for Wave 1 and Wave 2 are identical DBQs, which gives us a true comparison between the two waves. The DBQ question for both Wave 1 and Wave 2 is: “Evaluate the extent to which globalization since the Second World War has benefited everyone.”To complete each DBQ, students were required to read, analyze, question, and corroborate seven primary source texts, data charts and tables, images, and infographics, and then apply WHP concepts to construct an evidence-based essay to the prompt. Teachers were instructed to provide students with approximately 50 minutes to complete the essay in class, and they were instructed to submit these essays through Revision Assistant, an online platform that has the ability to provide students with in-the-moment feedback on their writing and also score the essays. The Revision Assistant rubric is made up of four criteria: claim and focus, analysis and evidence, organization, and language and style. According to Revision Assistant, their rubric is standards-aligned. Students can earn a score of emerging, developing, advanced, or proficient in each category. The WHP course has three different versions: WHP Origins, which starts at the beginning of time; WHP 1200, which starts circa 1200; and WHP 1750, which begins circa 1750. All courses run through present day. Although these three courses do differ from one another to some extent (the content for each aligns to a different set of state standards and a different periodization scheme), their approaches to skills development are nearly identical. Given those similarities, and the fact that this study is focused on skills related to writing, we had intended to conduct one analysis across
4SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARall three courses. However, due to a technical issue, we are not able to combine the results from WHP Origins and 1750 with WHP 1200. The issue was this: Revision Assistant has a feature called “signal checking,” which, when turned on, provides feedback to students while they write. Typically, “signal checking” is disabled for the pre and post assessment; however, signal checking was turned on this year for the WHP 1200 course. We conducted preliminary analysis to determine if the scores from the WHP 1200 course were significantly different from the scores in the WHP Origins and 1750 courses and found that they were. Therefore, this report is broken into two sections. Study 1 includes students who were in the Origins or 1750 course and did not have signal checking at pre or post assessment; Study 2 examines writing growth for those students in the 1200 course who didhave the benefit of signal checking for both assessments. Study 1—WHP Origins and WHP 1750ParticipantsTable 1 shows the number and percentage of WHP teachers and students who had reported scores by school type for the Origins and 1750 courses. To qualify to participate in the study, teachers had to teach WHP as a yearlong course and had to report that they were teaching WHP as a course (as opposed to only supplementing another course with our materials). In the total teacher population (n=23), one (4%) teaches at an independent school, one (4%) teaches at public non-Title I1 school, and 21 (91%) teach at public Title I schools. In the student population (n=962), 18 (2%) attend an independent school, 22 (2%) attend a public non-Title I school, and 922 (96%) attend public Title I schools. School TypeWHP Origins and 1750TeachersStudentsN%N%Independent14%182%Public non-Title I14%222%Public Title I2191%92296%Total23100%962100%Table 1. Population used for WHP growth analysis, 2021/22. Given the limited number of students in the sample in independent and public non-Title I schools, we will draw some comparisons between student writing growth for students in public Title I schools who participated in the study in 2020/21 school year to those who participated in the 2021/22 1 Title I, Part A, of the Elementary and Secondary Education Act, as amended, provides financial assistance to local educational agencies and schools with high numbers or high percentages of children from low-income families to help ensure that all children meet challenging state academic standards. A public school qualifies as Title I if 40% or more of the student population is eligible for free or reduced-price lunch.
5SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARschool year. The study designs for the two years were identical. In the 2020/21 school year, 229 students in 6 public Title I classrooms were included in the study. While drawing comparisons, it is important to do so with caution given the overall differences in sample sizes. Comparing WHP Participants from Title I SchoolsN2020/212021/22Teachers 621Students229922Table 2. Population used for WHP comparisons.FindingsAnalysis of Student Writing In the 2021/22 school year, 88% of students earned a score of emerging or developing for Wave 1 (see Figure 1). This figure declined to 55% at Wave 2, a 33 percentage-point decline. In the 2020/21 school year, students saw a decline of 34 percentage points, which is nearly identical year-over-year. Put differently, in the 2021/22 school year, 12% of students of students obtained an advanced or proficient score in Wave 1. This figure increased to 48% of students at Wave 2, a change of 36 percentage points. Comparatively, in the 2020/21 school year, the gain was 45 percentage points for those in the proficient or advanced category. While there is a 9-point difference between the two school years, given the small sample size in 2020/21, the differences in growth should be viewed as similar. Figure 1. Comparing overall growth for Title I schools year-over-year.
6SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARWhile we don’t have school-type comparisons this year, it is notable that in the 2020/21 school year public Title I schools experienced more growth than both public non-Title I schools and independent schools. Given the similarity of our Title I trends from this year to last, we would anticipate a similar result if we had the data to compare to the other school types.Scores by Writing VariableOverall scores can often mask the degree to which students improve on the four distinctive features of writing and thinking that WHP measures: claim and focus, analysis and evidence, organization, and language and style. Therefore, we examine each of the variables independently. Table 3 depicts the average growth in each subscore by year. Growth was similar across all categories between the two years, with the 2021/22 growth being slightly higher across all categories with the exception of Claim and Focus, where growth was nearly identical. While differences exist, these should not be interpreted as noticeably different. However, confidence in this year’s results should generally be higher given the overall increase in sample size, pointing to positive results overall. School TypeTotalClaim and FocusAnalysis and EvidenceOrganizationLanguage and StylePublic Title I – 2020/210.670.650.560.460.52Public Title I – 2021/220.750.620.640.620.64Table 3. Comparing year-over-year growth in scores from Wave 1 to Wave 2.Claim and Focus: In the 2021/22 school year, 6% of student essays were scored as proficient or advanced, whereas in Wave 2, this number climbed to 32%, a difference of 26 percentage points (see Figure 2). Comparatively, in the 2020/21 school year, the difference was 29 percentage points from Wave 1 to Wave 2. While slightly more growth was realized in the 2020/21 school year, a three-percentage-point difference is insignificant given the difference in sample sizes. Seen differently, at Wave 1, 94% of students were scored as emerging or developing, whereas at Wave 2, this number dropped to 68%, a decrease of 26 percentage points. In the 2020/21 school year, the decrease was 32 percentage points. While the decrease was larger for 2020/22, again, given the small sample size this should not be taken as a significant difference.
7SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARFigure 2. Comparing year-over-year growth for Claim and Focus.Analysis and Evidence: In 2021/22, 13% of student essays were scored as proficient or advanced at Wave 1, whereas for Wave 2, this number climbed to 47%, a difference of 34 percentage points (see Figure 3). This increase is nearly identical to that seen in 2020/21, which was a difference of 35 percentage points. While these are remarkably similar, in 2021/22 there were fewer students who were scored as proficient or advanced at baseline, which is why the comparative overall growth for this category is higher in the 2021/22 school year.Figure 3. Comparing year-over-year growth for Analysis and Evidence.
8SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEAROrganization: The organization results are somewhat similar to those seen for Analysis and Evidence. Overall, in the 2020/21 school year 47% of students scored proficient or advanced at Wave 1, as compared to only 33% of students of students in 2021/22, showing that at baseline, students from the 2021/22 school year were not performing as well (see Figure 4). However, those same students grew 39 percentage points from Wave 1 to Wave 2 while students in 2020/21 only grew 35 percentage points. This difference is small and likely due to inconsistent sample sizes; however, as mentioned, it is reasonable this year’s results given the larger pool of participants involved in the study.Figure 4. Comparing year-over-year growth for Organization.Language and Style: Language and style, which largely looks at writing mechanics, again saw students perform lower at baseline but realize more growth in the 2021/22 school year than in the 2020/21 school year. In the 2020/21 school year, 45% of students were scored as proficient or advanced, while only 28% of students from 2021/22 fell into this category at Wave 1 (see Figure 5). While more students from 2020/21 were scored as proficient or advanced at Wave 2 (78%), their overall growth in this category was 33 percentage points while students from the 2021/22 school year realized growth of 38 percentage points. Again, while these numbers are not comparatively all that different, the positive trend and larger sample size for the 2021/22 school year is encouraging.
9SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARFigure 5. Comparing year-over-year growth for Language and Style.Discussion of Study 1Overall, the findings for Study 1 are positive. Across the board, students showed growth in all areas of writing as measured by our rubrics, and as compared to the 2020/21 school year, students overall showed more growth in the 2021/22 school year. At the end of the school year across all categories except Claim and Focus, nearly 50% or more of all students were scored in the proficient or advanced categories. Given the larger sample size this year, we have more confidence in our findings and in the WHP course more generally as one that fosters student growth in writing and disciplinary thinking. In addition, many of the teachers who participated in the study last year were those who were part of our initial WHP Pilot cohort, so we can assume they have more familiarity with the curriculum than those who joined the study in the 2021/22 school year. This is encouraging in that it appears that new adoptions of the WHP curriculum can still foster great gains in student learning. Taking on a new curriculum can be a significant challenge for teachers. There is often a learning curve on which the sweet spot may not be found for a few years. Even those teachers new to WHP had students who showed impressive gains in their writing. While we would have liked to have had a larger population of students and teachers from which to draw our conclusions so we could compare across school types, findings are still positive.
10SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARStudy 2—WHP 1200ParticipantsTable 4 shows the number and percentage of WHP teachers and students who had reported scores by school type. In the total teacher population (n=19), two (11%) teach at independent schools, two (11%) teach at public non-Title I schools, and 15 (79%) teach at public Title I schools. In the student population (n=483), 51 (11%) attend an independent school, 38 (8%) attend a public non-Title I school, and 394 (82%) attend a public Title I school. This is the first year of full implementation for WHP 1200. In coming years, we expect the number of teachers and students participating to increase. School TypeWHP 1200TeachersStudentsN%N%Independent211%5111%Public non-Title I211%388%Public Title I1579%39482%Total19100%483100%Table 4. WHP population used for growth analysis 2021/22.FindingsAnalysis of Student Writing Overall, the findings for WHP 1200 are positive, yet should be taken as directional only. Given the small sample sizes for both independent and public non-Title I schools, we don’t have much basis for comparison. Additionally, we don’t have prior years’ scores to which we can compare. While we expect students to see growth in writing over the course of a school year, it’s difficult to determine how strong this growth is without a method for comparison. Caveats aside, growth is positive for the WHP 1200 course. Growth is strongest for public Title I schools, which mimics our findings from all OER Project courses that study growth in student writing, including those from the Big History Project and the World History Project Origins and 1200 courses. In terms of proficiency, at baseline, only 18% of students were considered proficient or advanced, while at the end of the year, almost half of students were scored as proficient or advanced (see Figure 6).
11SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARFigure 6. WHP 1200 total score at Wave 1 and Wave 2, 2021/22.Scores by Writing VariableTable 5 depicts the average growth in each subscore by school type for WHP 1200. Across the board, growth was highest for public Title I schools, followed by independent schools, with public non-Title I schools showing the least growth. Again, given the small sample sizes for both independent and non-Title I schools, we caution any direct comparisons across the groups. However, we do feel confident that growth for public Title I schools will be replicated in future years. In this section, growth by subscore will be examined more closely. Note that these include all school types involved in the study this year. School TypeTotalClaim and FocusAnalysis and EvidenceOrganizationLanguage and StylePublic non-Title I0.470.370.450.320.34Public Title I0.690.580.590.590.58Independent0.570.530.530.430.49Table 5. Average growth in subscore from Wave 1 to Wave 2 by school type, 2021/22.Claim and Focus: For Wave 1, only 12% of student essays were scored as proficient or advanced, whereas in Wave 2, this number climbed to 35%, an increase of 23 percentage points (see Figure 7). Or, seen differently, at Wave 1, 88% of students were scored as emerging or developing, whereas at for Wave 2, this number dropped to 65%. Overall, this category had the fewest number of students move into the proficient or advanced category at the end of the year.
12SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARFigure 7. WHP 1200 writing growth for Claim and Focus category of rubric, 2021/22.Analysis and Evidence: For Wave 1, 17% of student essays were scored as proficient or advanced, whereas in Wave 2, this number climbed to 47%, a difference of 30 percentage points (see Figure 8). Or, seen differently, at Wave 1, 83% of students were scored as emerging or developing, whereas at for Wave 2, this number dropped to 53%. While this is a big decrease, we would like to see a higher percentage of students in the proficient or advanced categories at the end of the school year.Figure 8. WHP 1200 writing growth for Analysis and Evidence category of rubric, 2021/22.
13SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEAROrganization: Of all the subcategories, Organization had the most students move out of the emerging or developing categories and into the proficient category. At baseline, more than half the students (57%) were scored as emerging or developing, but at Wave 2, only 24% of students met these criteria, with just over three-quarters of students (76%) being scored as proficient or advanced (see Figure 9). Figure 9. WHP 1200 writing growth for the Organization category of rubric, 2021/22.Language and Style: Language and style, which focuses heavily on writing mechanics, also saw marked improvement. For Wave 1, 42% of student essays were scored as proficient or advanced, whereas in Wave 2, this number climbed to 74%, a difference of 32 percentage points (see Figure 10). Or, seen differently, at Wave 1, over half of students (57%) were scored as emerging or developing, whereas at Wave 2, this number dropped to one-quarter of students (26%). In other words, nearly three-quarters of students were scored as proficient or advanced at the end of the year.
14SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARFigure 10. WHP 1200 writing growth for the Language and Style category of rubric, 2021/22.Discussion of Study 2While the directional results for the WHP 1200 course are positive, we would like to see more students in the proficient or advanced categories at the end of the school year. It is possible that growth was somewhat masked by the use of signal checking and automated feedback. We do know that at baseline, students who have access to signal checking versus those who do not earn scores that are statistically significantly higher than those who do not, which could mask the overall effect of growth. While you might assume this advantage would also boost students at Wave 2, our internal analysis shows that there are eventually diminishing returns when it comes to using Revision Assistant. When students use signal checking one to five times per essay, compared to six to ten times, there are statistically significant changes in their scores. However, over time, there is more of a plateau effect, so baseline scores may have been artificially high. In future years and studies, students will not have access to this feature at Wave 1 or Wave 2, allowing us to get a truer measure of growth. ConclusionsWriting growth is strong, showing that use of the WHP curriculum may dramatically improve student writing over the course of the year. WHP is reading- and writing-intensive. Students encounter a wide range of informational texts across multiple disciplines and are asked to write frequently and in a variety of writing genres, including informal writing, narratives, explanations, and arguments. In short, WHP teachers must do more than teach historical content. They also must help students develop and enhance their capacity to read, evaluate, use, and create such texts.While writing growth is strong, we would like to see more students finish the year in the proficient or advanced categories, particularly in the area of claim and focus. We have recently added extra
15SUMMARY OF OF WHP STUDENT GROWTH IN DISCIPLINARY WRITING2021/22 SCHOOL YEARwriting supports and writing tools to WHP Origins, 1200, and 1750, which we hope will help students further develop their writing skills. These tools, taken from our extension course, Project Score, were created by a literacy expert, and although we have not studied their effectiveness, teacher reports have pointed to their usefulness in the classroom. This is an area we will watch closely in the 2022/23 school year. We are especially proud of the growth from this year and last given the overall decline in academic progress in the United States since the start of the COVID-19 pandemic. The most recent reading and math results from the National Assessment of Educational Progress (NAEP) show that two decades of growth in these areas were essentially wiped out by the pandemic. This decline has been seen in other assessments more recently administered by the National Center for Educational Statistics (NCES).2 We are grateful not to be part of this trend. As always, it is important to note the limitations of this study. In addition to the sample-size related caveats mentioned throughout the report, we have no additional data on WHP students, such as GPAs or reading scores, and we do not know the degree to which the baseline performance is an accurate representation of student writing or the degree to which it correlates with other factors. In addition, we do not know how teachers presented each of the assessments or how faithful they were to the course as designed. To attempt to ameliorate some of these issues, in the 2022/23 school year we are recruiting a much larger sample of participants so that we can have even more certainty in our findings. In addition, we’re raising the bar for participation, and all teachers must commit to teaching 50% or more of the curriculum, which will ensure high fidelity to WHP practices. This commitment will include teachers agreeing to use a mix of our skills progressions as well as activities, articles, and videos. This will increase our confidence when pointing to the curriculum as having a positive impact on students. At OER Project, we feel it’s important to measure the impact of our courses on student learning. We invest effort in this research year over year to assess if we are reaching our goals so that we can celebrate our successes, but also so we can change direction as needed to ensure we are doing what we can to improve student and teacher learning and perceptions. Although we are quite pleased with the gains students are making in their writing skills, students in the 2021/22 school year still suffered from disruptions related to COVID-19. However, even with those disruptions—and many recent studies show that students suffered academically due to those disruptions3 —we were able to replicate the positive findings from prior years (including pilot years), showing the strength and resilience of WHP even when students and teachers are faced with major disruptions and challenges. WHP’s attention to disciplinary literacy, its curriculum—which is grounded in research and in practice—and the support the WHP Project team provides and gets from its teachers help explain our strong growth findings. We expect to continue to see this type of growth moving forward. 2 Peggy G. Carr, Commissioner’s Statement on the Upcoming Release of NAEP Long-Term Trend Assessment Results (August 31, 2022).3 Laura Moscoviz and David K. Evans, “Learning Loss and Student Dropouts during the COVID-19 Pandemic: A Review of the Evidence Two Years after Schools Shut Down” (Center for Global Development, Working Paper 609, March 2022).