Zoom

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEARRachel S. Phillips, PhD and Meghan Oster, PhD

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR2Table of ContentsIntroduction.........................................................................................................................................................3Section 1: OER Project Schools and Data Collection Instruments............................................................3Section 2: Teacher Satisfaction with and Perceptions of Social Studies Courses................................5Study 1 – Sector Survey Comparison......................................................................................................5Data Collection and Participants.......................................................................................................5Findings.................................................................................................................................................6Study 2 – BHP Teacher Satisfaction and Perceptions..........................................................................9Data Collection and Participants.......................................................................................................9Findings.................................................................................................................................................9Study 3 –WHP Teacher Satisfaction and Perceptions.........................................................................11Data Collection and Participants ....................................................................................................12Findings...............................................................................................................................................12Discussion of Perception and Satisfaction Survey Findings.............................................................13Section 3: Comparing Social Studies Teacher Classroom Practices......................................................13Data Collection and Participants ...........................................................................................................14Findings...............................................................................................................................................14Section 4: Student Growth in Disciplinary Writing ....................................................................................18Study 1 – BHP Student Writing Growth.................................................................................................19Data Collection ...................................................................................................................................19Participants.........................................................................................................................................19Findings..............................................................................................................................................20Study 2 – WHP Student Writing Growth................................................................................................26Data Collection ...................................................................................................................................26Participants.........................................................................................................................................27Findings...............................................................................................................................................27Discussion of OER Project Writing Findings........................................................................................34Conclusions.......................................................................................................................................................35Cover: Calligraphic Galleon, dated A.H. 1180 / A.D. 1766-67, Turkey, Ink and gold on paper, H. 19 in. (48.3 cm), Codices, The hull of this sailing ship comprises the names of the Seven Sleepers and their dog. The tale, found in pre-Islamic Christian sources, concerns a group of men who sleep for centuries within a cave, protected by God from religious persecution. © Sepia Times/Universal Images Group via Getty Images.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR3IntroductionOver the 2020/21 school year, the OER Project conducted a few studies to measure student learning and teacher perceptions of the Big History Project (BHP) and World History Project (WHP) courses. We conduct these studies annually in the interest of overall program evaluation and improvement. All research findings point to favorable results related to the quality and rigor of the courses—especially in writing—for students who participated in the BHP curriculum in public Title I schools.1 Overall teacher perceptions of the OER Project as a whole were high More OER teachers responded that they presented their students with more opportunities to engage in geography and emphasized historical thinking, compared to the national sample of teachers. This year’s results should be taken with caution given the myriad educational disruptions that most of our teachers and students experienced throughout the school year due to the COVID-19 pandemic. Loss of instructional time, technology access issues, frequently changing schedules, and of course, anxiety and illness were just some of the challenges our students and teachers faced. A paper published in the summer of 2021 by Angela Duckworth and colleagues in the esteemed journal Educational Researcher reported that of the more than 6,500 students surveyed, those who attended school remotely suffered socially, emotionally, and academically. To avoid further reduction of valuable instructional time, we minimized our data-collection efforts. This report is divided into four sections. The first offers an overview of the courses OER offers, the schools it serves, and the instruments it employs to collect data. The second section describes the findings from teacher perception data, including a comparison of OER Project and other social studies teachers. The third section features the findings from a comparison study of OER Project teachers’ reported teaching practices and a national sample of social studies teachers. The fourth section examines student writing growth in both BHP and WHP. Note that throughout this report, when we refer to OER Project courses, we are speaking collectively about BHP and WHP data and findings. Otherwise, we refer to them separately. Section 1: OER Project Schools and Data Collection InstrumentsThe OER Project is a coalition of educators and historians solely focused on boosting student engagement and achievement through transformational social studies programs. Currently, the OER Project offers four courses—Big History Project, World History Project, Project X, and Project Score—all of which are completely free, online, and adaptable to different standards and classroom needs. Unlike textbooks, lesson websites, and other commercial products, everything has been purposely built to truly empower teachers and leave traditional history courses in the past. The Big History Project (BHP) has been reporting on students’ writing and participants’ perceptions of the course for nine years. This is the first year the World History Project (WHP) is reporting on non-pilot data; however, with COVID disruptions, the corpus of data collected is not as complete as we had hoped. 1 Title I, Part A, of the Elementary and Secondary Education Act, as amended, provides financial assistance to local educational agencies and schools with high numbers or high percentages of children from low-income families to help ensure that all children meet challenging state academic standards. A public school qualifies as Title I if 40% or more of the student population is eligible for free or reduced-price lunch.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR4In the 2020/21 school year, we had an active OER Project program of just over 2,200 schools in the United States, which was a modest increase compared to prior years (see Figure 1).2 Although the OER Project serves schools on almost every continent, this report only focuses on schools in the United States. Figure 1. Number of schools with an active OER Project program.In addition to the number of participating schools and teachers, the OER team has tracked four outcomes over the 2020/21 school year:1.Comparison of OER Project teachers’ perceptions and other social studies teachers’ perceptions2.Comparison of OER Project teachers’ reported teaching practices and a national sample of social studies teachers’ 3.OER Project teachers’ perceptions of its courses and teacher professional development (PD) opportunities4.BHP and WHP student growth in writing and making arguments2 “Active use” is defined either behaviorally or via self-reporting. Teachers who visit the site three times for three consecutive months, or those who report that more than 25% of their course materials are from the BHP website, are considered “active.” Additionally, a teacher who reports that they are teaching the course and drawing more than 25% of their course materials from the BHP site is considered to be actively teaching the course.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR5To gather data in these areas, OER used the following procedures and instruments:1.Teacher satisfaction and perception surveys 2.Text-based assessments of student writingTypically, we collect student perception data and conduct interviews with OER Project student alumni to develop longitudinal retrospective case studies. However, we were unable to do so this year due to COVID-19 disruptions. Section 2: Teacher Satisfaction with and Perceptions of Social Studies CoursesThe OER Project seeks to understand how teachers perceive its courses. In addition to gathering in-depth information regarding perceptions of OER Project courses, we also examine how OER Project teachers’ perceptions compare to how other social studies teachers perceive the curricula they use. Generally, the OER Project finds that teachers have a high level of satisfaction with the course, and more often than not, will recommend this course to others. When looking at OER Project courses compared to other social studies courses, perceptions are generally higher for OER Project courses. In the following three sections, we describe outcomes from survey studies. The first study, coined the “Sector Survey Comparison,” describes how OER Project teachers compare to other social studies teachers when reporting on their perceptions of the course they teach. The second and third studies examine teachers’ satisfaction with and perceptions of the BHP and WHP courses, respectively. Study 1 – Sector Survey ComparisonIn this section, perception and satisfaction survey findings comparing OER Project teachers and other demographically matched social studies teachers are shared. Data Collection and ParticipantsIn the spring of 2021, the OER team collected data on teacher perceptions using online surveys. In these surveys, teachers answered a series of questions to determine their likes, dislikes, and attitudes about how the courses were working. Teachers, for example, responded to questions regarding their confidence in teaching historical content and the required reading and writing practices, and were also asked to assess the effectiveness of course materials and the courses’ perceived impact on students. One of the surveys was sent to demographically similar social studies teachers around the United States. This was done to enable a comparison to be drawn between OER Project teachers and a sector of social studies teachers who use other curricula.3 Note: The same sample of OER teachers was used across all three survey studies. 3 PSB Insight is a consultancy company that conducts research studies. PBS Insight reviewed instruments, administered the sector survey, and analyzed the sector survey results that are discussed in this section of the report.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR6All active OER Project teachers who reported using at least 25% of our courses in a yearlong implementation were eligible for participation in the perception surveys. OER teacher participation in the perception surveys was a bit higher than prior years. This was expected, given a new cadre of WHP teachers being added to the sample. In total, 165 OER Project teachers took our survey. For the sector survey comparison recruitment, PSB Insight, a consultancy company, used a series of screeners to determine if respondents met the qualifications to take the survey. As stated by PSB: “The sample was recruited from Dynata’s open market sample, who are a quality provider that is pre-approved by organizations like Microsoft, Gates Ventures, and the US Census.” In total, PSB recruited 175 teachers; however, we eliminated 26 teachers from the findings upon discovering they were, in fact, BHP teachers. Rather than include PSB-recruited BHP teacher responses to our sample of OER Project teachers, we eliminated them entirely. It should be noted that the sector survey participants seem less affected by the COVID-19 pandemic than OER teachers who completed the survey. As shown in the findings below, just 5% of the sector survey respondents indicated that they couldn’t make comparisons to previous years because of COVID-19 disruptions while 20% of the OER sample said that they could not make such a comparison. As a result, we were concerned about selection bias in our sample. Looking into this further, we found that a higher proportion of the sector survey teachers had been teaching for fewer years than OER teachers, suggesting that they might be younger teachers who perhaps could make the transition to online or hybrid teaching more easily than teachers who had been in the classroom longer. Also, we hypothesize that newer teachers may have the advantage of not being as wedded to their classroom practices as those teachers who are more senior, an attitude that might make transitions to online and hybrid schooling simpler. Forty percent of the sector survey had been teaching 4 to 7 years while 57% of the OER sample had been teaching for more than 11 years. FindingsOverall, we don’t see major differences in responses from OER teachers and our sample of other social studies teachers. Given the small size of the samples, a difference of a few percent might not actually be a true difference. For example, Figure 2 shows that OER teachers were slightly less satisfied (93%) compared to the sector teachers (96%).

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR7Figure 2. Overall satisfaction comparison.However, we do not believe this is a meaningful difference given the responses to a question that asks teachers if they would recommend the course they are teaching to a student. OER Project teachers were much more likely to “definitely” recommend the course to a student than were other social studies teachers (see Figure 3). In fact, 73% of OER teachers reported they would definitely recommend the course, as compared to only 45% of social studies teachers, a difference of 28%. This question is a version of a net promoter score question, which tends to be a great measure of overall happiness and satisfaction related to a product or experience, while other satisfaction measures tend to be better for examining the component parts of satisfaction. Therefore, we suspect that OER teachers are generally more satisfied than other social studies teachers. Figure 3. Likelihood to recommend course to a student comparison.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR8We also asked teachers to report on their confidence and preparation for teaching the course, how the course has impacted their own teaching, and whether they think the course prepares students for future studies. As shown in Table 1, OER Project teachers reported feeling less confident teaching reading and more confident teaching writing than their counterparts who teach other social studies courses. OER Project courses have a lot of writing supports for teachers and scaffolds for students, and OER Project courses also include Score, our writing platform, which provides automated feedback and scores to students as they write, and which may account for this difference in part. That said, the difference is small and might not be meaningful.A smaller percentage of OER teachers reported feeling prepared to teach this course. What is difficult to tease out here if the difficulty was in teaching the course or in teaching the course at a distance. About the same number of teachers felt that the course prepares students for future studies and that their teaching style had changed. Again, this year it’s difficult to know if respondents’ teaching style changed due to using their respective curricula, or if it was due to shifts they had to make related to various virtual school situations. Course PerceptionsPercent of teachers who “strongly agree” + “somewhat agree” with the following statements.Social studies teachers OER Project teachersn = 146 n = 165I feel confident teaching reading9282I feel confident teaching writing8487I was prepared to teach this course9183Prepares students for future studies8988My teaching style has changed8581Table 1. Course perception comparisons.In addition to wanting to know about overall teacher satisfaction and course perceptions, we were curious to know teachers’ thoughts regarding student engagement in their current courses. We asked teachers to report if their students were more engaged than they usually are in their courses. Teachers using OER Project courses were much more likely to indicate that courses are not comparable this year because of COVID-19. A very small percentage of the sector surveys indicated that they could not compare, again suggesting that the sector survey might be subject to some selection bias.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR9Figure 4. Perceived student engagement comparison.Ultimately, the OER Project would like to understand if BHP and WHP are having more of a positive impact on both teachers and students than other social studies curricula available on the market. The findings from this comparison survey study show that while OER Project teachers more likely recommend the curriculum, their level of satisfaction and confidence slightly declined this year relative to the sector teachers. OER teachers were much more likely to say their classes were subject to COVID disruptions in one of questions compared to the sector teachers, suggesting that the sector teachers may have had different teaching environment this year than the OER teachers. Study 2 – BHP Teacher Satisfaction and PerceptionsWe also examine our own teacher perceptions longitudinally to ensure we are serving our teachers and their students in the best ways possible year over year. We compare findings to the same or similar data that has been collected in prior years. However, given how unusual the year was due to COVID disruptions, all comparisons to prior years should be examined with those differences in mind. Note that for this year, we had about half the number of respondents than we’ve had in prior years and this limitation should be kept in mind when interpreting findings. Data Collection and ParticipantsPlease see Study 1 in this section for more detailed information on data collection and participants.FindingsWhen comparing 2021 results to those of prior years, one of the biggest positive differences is that 85% of teachers agreed or strongly agreed that BHP prepares students for future studies and this year, the percentage of teachers that recommend the course went up dramatically (95% in 2021 compared to 87% the previous year). Anecdotally we heard from many teachers that they felt our courses were much simpler to transition to online learning than others that they were teaching, but overall, while we are delighted to see this improvement, we aren’t entirely sure of its origin.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR10While there has been modest variation from year to year, the overwhelming majority of teachers over the six years reported are satisfied with the course, recommend the course, feel that the class prepares students for the future, and that students stay engaged. Table 2 shows that at the end of the year, 92% of the teachers reported being satisfied with BHP and 95% would recommend it to students. A majority of teachers (65%) reported that the course materials kept students more engaged than those of other courses, while 7% did not think the BHP course was more engaging than other courses for their students. While reported engagement has declined this year, significantly fewer teachers responded to the question this year, and 14 respondents felt they couldn’t compare to prior years either due to COVID or because they were first-year teachers, putting these results further into question. This year, a smaller proportion of teachers indicate that the BHP curriculum changed their teaching, that they liked the teaching materials and website, and that they were satisfied with program communications. When looking more closely at the responses about changes to teaching practices, the biggest shift was from the affirmative (agree or strongly agree) to the neutral category. There were only three responses that indicated that BHP did NOT change their teaching. While the decline might not be as marked as it seems when making a surface comparison, it’s still important to try to understand the decrease. Some reasons for this may be: 1) Teachers likely spent more time focusing on how to teach online rather than how the BHP course and OER approaches to teaching could impact their practices. 2) We had fewer teachers take our online teaching courses than in prior years, which may suggest that they didn’t have the time or attention to devote to this. 3) More teachers who have been using BHP for multiple years are taking the survey and it is likely that their teaching is changing less and less each year as they become more comfortable and familiar with BHP curriculum. Over 50% of the survey sample has taught BHP more than three years. In regard to the decrease in website satisfaction, a new site was launched in the 2020/21 school year, which frustrated many teachers, especially those who were used to navigating the old site. One teacher wrote about their experience4:I realize I sound like an old man who doesn’t like change, but ... the old BHP website was easier to navigate and use. Also, please add a digital display for PDFs online that’s easier to use. I couldn’t just share screen from the website because it wasn’t easy to show the text. —BHP teacherThe PDF viewer issue this BHP teacher mentions has been resolved (it came up repeatedly throughout the year). There is a scrolling issue that persists but is being addressed. OER Project will continue to inquire about issues teachers are facing with the website and will continue to make improvements where possible. The OER Project Online Teacher Community platform was also revamped over the summer as a result of teacher feedback about difficulty navigating that part of the site. A slightly smaller percentage of survey respondents indicated that they were confident in teaching and writing than previous years. As with other questions, it’s difficult to know if this was due to teaching at a distance or due to simply feeling less confident, and it’s also difficult to determine if the difference is big enough to be meaningful. We tend to think that overall it has little to do with actually feeling less confident, given that more than 50% of teachers in our sample have been 4 All verbatim quotes in this report are taken from responses to the question, “Is there anything else you’d like us to know about your experience with the OER Project?”

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR11teaching the course for three or more years, and a large percentage of our research participants have engaged with the program for multiple years and have reported higher confidence in the past. To us, if the difference is meaningful, it suggests the nature of online school decreased teacher confidence rather than their confidence simply declining from the prior year. Our biggest decline this year was with program communications. It’s not clear to us why satisfaction dropped this much, so we have been reaching out to participants to try to discover the source of dissatisfaction. To date, only one teacher has responded to our follow-up, suggesting that we overcommunicated. BHP Teacher Course Satisfaction and PerceptionsSpring 2018Spring 2019Spring 2020Spring 2021n = 68n = 108n = 117n = 62Course satisfaction91%91%93%92%Would recommend91%96%87%95%Prepares students for future79%79%86%85%Students stay engaged574%70%72%65%Changed my teaching87%83%90%84%Course content91%86%87%94%Course structure94%90%90%94%Program communications91%91%95%80%Teacher materials91%87%94%85%OER Project Site93%88%94%80%Confidence teaching reading83%87%87%84%Confidence teaching writing82%88%88%84%Table 2. BHP teacher course satisfaction and perceptions, spring 2018 to spring 2021.Study 3 –WHP Teacher Satisfaction and PerceptionsThe OER Project’s WHP course was developed and piloted over three years, starting in the 2017/18 school year, in collaboration with historians, world history teachers and professors, and researchers. It fills the void created by an absence of high-quality, non-Eurocentric, comprehensive, high-school world history curricula that is focused on developing historical thinking skills. WHP has in many ways been built on both the lessons learned and successes of BHP. The 2020/21 school year was the first year of full implementation in schools. 5 Teachers were able to respond that they couldn’t compare to other courses because of COVID-19 disruptions or because they were first-year teachers. Nine respondents indicated that COVID-19 had affected their ability to make comparisons, and fewer than five indicated that they were first-year teachers.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR12As with BHP, instead of using a traditional textbook, students read articles written by historians from around the world. These articles are available to students at multiple reading levels and in audio format. The curriculum provides sets of videos, filmed on-location around the world, to provide a more multimedia approach to learning. The course also includes a series of scaffolding activity progressions that help students develop their historical thinking practices. For teachers, we have professional development opportunities and a strong, active teacher community. Also, as in BHP, the course has a driving narrative—actually, it has three. The course employs the concept of frames, or lenses, for students to look through to help them construct historical narratives. These WHP course frames are community, production and distribution, and networks. Data Collection and Participants Please see Study 1 in this section for more detailed information on data collection and participants.FindingsOverall, the results were very positive for this year (see Table 3). This year, 103 WHP teachers responded, relative to the 28 teachers in the pilot year. Bearing in mind that the pilot teachers were an exceptional group who came highly recommended by their colleagues or other people in the global world history community for being outstanding teachers, the teachers who responded this year viewed WHP as extremely favorable compared to the pilot teachers. WHP Teacher Course Satisfaction and PerceptionsSpring 2020Spring 2021n = 28n = 103Course satisfaction89%94%Would recommend100%95%Prepares students for future89%89%Students stay engaged661%55%Changed my teaching89%79%Course content71%92%Course structure86%87%Program communications93%83%Teacher materials100%88%OER Project SiteN/A80% Confidence teaching reading82%80%Confidence teaching writing89%82%Table 3. Comparing WHP teacher course perceptions, spring 2020 (pilot year) and spring 2021.6 Teachers were able to respond that they couldn’t compare to other courses because of COVID-19 disruptions or because they were first-year teachers. Twenty-four respondents indicated that COVID-19 had affected their ability to make comparisons, and fewer than five indicated that they were first-year teachers..

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR13The majority of teachers indicated that they were satisfied with the course, would recommend it, and indicated that it prepares students for the future. They also indicated a high satisfaction with course content, course structure, program communications, and teacher materials. A little more than half (55%) of teachers said that students were more engaged with WHP than their other classes, although 24 teachers indicated that they couldn’t compare to other courses because of COVID-19 disruptions. We hope and suspect that for the most part, these positive findings are due to having created a course that works well for teachers and their students, and does indeed fill the gaps we mentioned earlier. However, it’s also possible that some of the positive findings are a result of having a somewhat incomplete course during the pilot years, and the finding therefore might reflect teachers’ excitement over finally having a full curriculum to use. Many of the pilot teachers were part of this year’s sample. Discussion of Perception and Satisfaction Survey FindingsOverall, we are happy to report that even during what many teachers say has been the hardest year of their careers teachers still remain satisfied with our curricula and are very likely to recommend it to other teachers. While all of the findings this year should be looked at with an extra level of scrutiny, we still feel they are directionally very strong, suggesting that we continue to offer courses that support our teachers and students well. While the results are strong, there are still some areas for us to better understand. First, we will look to unpack differences in satisfaction versus teachers being willing to recommend our course in order to gain a qualitative understanding of why those numbers vary. We will continue to inquire about website usability as well as program communications and use any findings to make improvements. Section 3: Comparing Social Studies Teacher Classroom PracticesIn the 2020/21 school year, we added a battery of questions from the NAEP’s (National Assessment of Educational Progress) 2018 U.S. History, Geography, and Civics at Grade 8 teacher questionnaire to our own teacher satisfaction and perception surveys. This allows us to compare our teachers’ responses to a national representative sample of teachers. While the grade level of the students is not perfectly matched to the entire OER student population, the focus is more on teacher practice, their approaches to historical thinking, and the opportunities they create for students to critically engage in social studies. These historical thinking skills apply to all social studies subjects such as historical causation or comparison—we did not include items that were about specific time periods or subjects in history. Rather, the questions are largely about implementation and therefore give us a sense of OER Project students’ opportunities to learn as compared to students in all classrooms in the United States. Survey responses from our teachers also help give us a sense of teachers’ fidelity to the curriculum and overall approaches. This section of the report helps us answer the following questions: 1.Do OER Project teachers present their students with more opportunities to engage in historical thinking practices than the average teacher in the United States?

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR142.How do our teachers both use and supplement our curriculum as compared to the average teacher in the United States?Overall, we found that more often than not, OER students are provided with more opportunities to engage in specific historical thinking practices than in an average classroom. We also found that OER teachers are less likely to supplement their curriculum with outside resources, pointing to overall satisfaction with the materials provided. Data Collection and Participants Please see Study 1 in Section 2 for more detailed information on how OER teachers were sampled for this study. For the comparison data set, 42,000 students from about 780 schools across the nation participated in the NAEP. This data set represents the responses from the teachers of those students. Please note that the results are all shared in terms of the percentage of students whose teachers report each result. FindingsWe asked our teachers three questions from the NAEP exam, with a total of 15 items across the three questions. The first set of questions is about reading and research, the second is about issues related to geography, and the final set is about historical thinking practices.Reading and ResearchThe first set of questions asks teachers about how often they have students read from their own textbook, from texts outside their curriculum, how often they ask students to conduct research, and how often they use primary source materials. Overall, we found that OER Project teachers used their own textbook7 on a regular basis, and did not seek outside resources nearly as often as the NAEP teachers. Eighty-six percent of OER teachers have students read from the textbook every day or almost every day, or once or twice per week, while only 59% of NAEP teachers reported the same (see Figure 5), a marked difference. Note that throughout this section, we often report the top two options ( every day or almost every day, and once or twice a week). Because of the education disruptions caused by the pandemic, we assume that many schools switched to hybrid schedules where students and teachers only interacted once or twice a week. Because once or twice a week may have been the only option for many teachers, we wanted to make sure that we grouped the top two options together.7 Because the OER Project does not have a traditional textbook, we defined it for them in the survey in the following way: “OER curriculum/website, including videos, count as a social studies textbook.”

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR15Figure 5. How often students read material from their textbook.It’s important to note that this was not because reading wasn’t being assigned; it’s that the NAEP teachers were using materials outside their textbooks for students to read. Only 28% of OER Project teachers have students read extra material outside of the textbook once a week or more, while 75% of the national sample report using outside material more than once a week (see Figure 6).Figure 6. How often teachers have students read material about social studies that is not included in the textbook.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR16Another notable difference between the groups was in the use of primary source material (see Figure 7). NAEP teachers reported using primary source materials far more often than WHP teachers. Overall, 58% of NAEP teachers reported using primary source materials one or more times per week while only 38% of OER Project teachers reported using primary source materials one or more times per week. If we look more closely, this difference is more pronounced for BHP. This is to be expected, as BHP does not incorporate much primary source material into the course. However, WHP teachers lag considerably behind NAEP teachers in how often they use primary sources. This is not a surprising finding, as our teachers mentioned this in their feedback throughout the school year. As a result, we have curated primary source collections for each of the eras/units in WHP. We will examine findings closely next year to see if WHP teachers report using primary sources more now that they are more readily available.Figure 7. How often teachers report having students use primary sources.GeographyAcross the board, we assume that OER Project teachers spend a lot more time emphasizing topics in geography than NAEP teachers, largely due to assumed differences in the curricula. Our curricula do attend to topics in geography more often than we’ve seen in other popular social studies curricula, and our attention to those topics goes beyond what we see in states standards. We also include geographical concepts in some of our progression activities and tools, such as contextualization. Findings from these questions reinforce the notion that our teachers are likely implementing our curriculum with fidelity. There were four specific questions that asked how much teachers emphasize specific concepts in their social studies course, and OER teachers reported emphasizing these far more than NAEP teachers (see Table 4).

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR17NAEP TeachersOER Project TeachersCountries and cultures34%59%Environmental issues21%66%Physical and human geography42%58%Variation among regions46%87%Table 4. Percent of teachers who report emphasizing topics in geography a lot or quite a bit. Historical Thinking PracticesWe also included seven questions about how much teachers emphasize common historical thinking practices. We did this not only to give us a sense of whether teachers are teaching our courses with fidelity, but to learn how often OER Project students have opportunities to engage in these skills as compared to a national sample of students. For each question, teachers were asked how often they emphasize each skill in their teaching, with five response options ranging from “Not at all” to “A lot.” For the most part, OER Project teachers emphasize the historical thinking practices asked about far more often than the national sample of NAEP teachers (see Table 5). The topics that OER teachers emphasize more include: examining the causes and effects of events important events in history, judging whether information from a source is accurate, examining how time periods are similar or different, analyzing the relationship between two historical events, and comparing and evaluating different points of view about the past. These findings are encouraging for a few reasons. First, they suggest our teachers are implementing the OER Project curriculum with high fidelity. Our courses have a strong emphasis on historical thinking practices and include spiralizing sequences of activities that address each historical thinking practice. They also suggest that OER Project teachers are generally emphasizing these historical thinking practices more often than the national sample. One surprising finding is related to sourcing. As noted in Figure 7, NAEP teachers report spending more time using primary source materials than OER Project teachers; however, OER Project teachers report emphasizing judging whether information from a source is accurate. These results suggest that while NAEP teachers are incorporating primary sources more often into their instruction, they may not be focusing on the actual skills related to sourcing as often as their OER Project counterparts. In terms of how much teachers emphasized comparing and evaluating different points of view about the past and coming up with research questions about the past, OER Project teachers lagged behind NAEP teachers. OER Project courses do not give students many opportunities to come up with research questions about the past, so this result is not particularly shocking. However, the ability to ask good, researchable historical questions is an important skill and one we will consider building into the curricula more deliberately. It was surprising that OER Project teachers reported emphasizing taking and defending a position on a historical issue less often the NAEP teachers. When we examine the numbers more closely, WHP teacher actually emphasize this more often than NAEP or BHP teachers. However, we were still puzzled by the BHP findings. One of the skills that is focused on throughout the course is what

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR18is referred to as claim testing, which is the idea of using intuition, logic, authority, and evidence as a framework for determining whether claims are defensible or not. So, we would assume teachers would focus on taking and defending positions more often. One possibility is that the first half of the course is about prehuman history, so perhaps teachers do not see that early history as containing “historical issues,” and therefore report emphasizing this less. It could also be that they feel that claims and issues are very different things. Either way, we will continue to monitor this because we do feel that taking and defending positions about historical issues is an important skill to hone. NAEP TeachersOER Project TeachersBHP TeachersWHP TeachersCausation91%99%99%99%Sourcing59%66%70%71%Periodization 71%85%77%94%Analyzing relationships between historical events66%76%77%75%Comparing different points of view about the past70%73%75%72%Coming up with research questions about the past46%34%29%37%Taking and defending a position about a historical issue48%43%28%56%Table 5. Percentage of teachers reporting they emphasize these topics in history “Quite a bit” or “A lot.” Section 4: Student Growth in Disciplinary Writing Improving students’ ability to write coherent, logical essays that effectively use evidence and apply disciplinary concepts is one of the OER Project’s most important goals. Writing is essentially thinking on paper. Developing students’ capacity to write and communicate well is among the most important educational outcomes. BHP and WHP provide many opportunities for students to write, and they support both students and teachers through scaffolds and lessons designed to develop historical thinking, reasoning, and writing skills. BHP and WHP are writing-intensive courses that include curriculum and professional development activities for teachers that enable them to extend their students’ capacity to use a range of texts as evidence and develop coherent and sophisticated arguments. Although it is the wealth of resources related to historical contents in the course that draws the most attention, in many ways we are proudest of the growth we see in student thinking as displayed in their writing growth, which occurs year after year, including the 2020/21 school year. In examining this year’s growth, it is first important to note some major differences in data collection from years prior to COVID-19 disruptions.

SUMMARY OF OER PROJECT RESEARCH 2020/21 SCHOOL YEAR191.No human scoring. Typically, we report on essays scored by humans. This was not possible this year, so for this analysis, essays that were machine scored by Revision Assistant were analyzed. 2.Different rubric used for scoring. The writing rubric typically used by the human scorers is different from the rubric used for machine scoring. The machine scoring rubric does not include the category Applying BHP Concepts, and it is also on a four-point scale instead of a five-point scale. In what follows, we describe the data collection, analysis, and results of OER Project’s study of student writing. Study 1 refers to BHP student writing; Study 2 applies to WHP student writing.Study 1 – BHP Student Writing GrowthIn past years, the analysis of BHP students’ writing showed growth from the beginning to the end of the course. This improvement held true across all the schools and teachers studied, with the strongest growth occurring in Title I schools. On average, BHP students have consistently demonstrated improvement in their capacity to tackle a big question, analyze between eight and ten different sources or texts, reason toward a conclusion, and then construct a coherent, well-structured, well-supported, and well-written essay.Data Collection For the BHP study, 484 students submitted both the baseline (Wave 1 - Investigation 0) and the post essay (Wave 2 - Investigation 9). The essays and prompts were: 1.Wave 1– Investigation 0: “How and why do individuals change their minds?” 2.Wave 2 – Investigation 9: “To what extent has the Modern Revolution been a positive or negative force?” To complete each Investigation, students were required to read, analyze, question, and corroborate a variety of informational and historical texts—including primary and secondary sources, data charts and tables, images, and infographics—and apply BHP concepts before constructing an evidence-based essay to answer the Investigation question. Teachers were instructed to provide students with approximately 45 minutes to complete the essay in class, and they were instructed to submit these essays through Revision Assistant, an online platform that can provide students with in-the-moment feedback on their writing and that can also score the essays. The Revision assistant rubric is made up of four criteria: claim and focus, analysis and evidence, organization, and language and style. According to Revision Assistant, their rubric is standards-aligned. Students can earn a score of emerging, developing, advanced, or proficient in each category. ParticipantsThere were 13 BHP teachers who had at least 10 students who completed the baseline and end-of-year essays and a total of 484 students. This year, as in the past, BHP used school type as a proxy to get at differences in context in which learning occurs. Public Title I schools, public non-Title I schools, and independent schools is a frequently used categorization for differentiating school type. In the past, BHP aimed for samples stratified with 45% of the students assessed attending Title I schools, 45% attending non-Title I schools, and 10% attending independent schools. This year, given the disruptions from COVID-19, we simply used all possible scored essays.

Cookie Policy

Summary of OER Project Research 2020-21