Background
COVID-19 had significant impacts on the field of education and, in turn, on school-based research. For our purposes, school-based research refers to research conducted within the school setting. During this unprecedented time in our recent history, schools needed to close unexpectedly, resulting in loss of learning time as educators shifted curricula to a virtual format. The shift to remote learning needed to happen quickly, but schools needed time, effort, and funding to source technology and to train both staff and students to use it. Although the use of technology (i.e., Chromebooks/laptops and internet connectivity) increased and these specific devices are now more widely accessible than before COVID-19, a “digital divide” still exists (Darling-Hammond et al., 2020). Furthermore, districts and schools report staffing concerns such as extreme staff turnover rates and the inability to fill vacated positions (Institute of Education Sciences, 2023). Addressing the lasting effects brought about by school closures, learning loss, increased use of technology in instruction, and staffing issues are major challenges that education leaders and decision-makers face in the post–COVID-19 education climate. Consequently, education leaders indicate that the burden of addressing these challenges has limited their willingness or ability to participate in external research initiatives (Feedback from Education Leaders Focus Groups to Inform Recruitment Efforts for the High School and Beyond Longitudinal Study of 2022 and Other NCES Studies, 2021).
The RTI International school recruitment team hosted a series of five focus groups in May and June 2021 to engage with key education leaders at the state, district, and school levels. Education leaders (e.g., state education agency officials; district superintendents, assessment directors, and research personnel; and school principals representing public, private, charter, and virtual schools) from 30 states shared insight on the perceived barriers and benefits to participation in school-based research. These individuals were asked questions about their research priorities and decision-making processes for participation in research initiatives. As leaders openly shared their perspectives throughout the focus groups, the following summarizes their sentiments about the additional challenges to school-based research after COVID-19: "the burden placed on educators as schools are beginning to recover from COVID-related disruptions and are trying to address student learning loss is widespread. Participation in additional research activities, especially that of voluntary studies, may be considered an untimely request." Looking ahead to school-based research in a post-pandemic world, education leaders conveyed that there would be additional challenges that include the loss of instructional time, increased teacher workloads, and restriction to visitor access, which could impact in-person data collection (Feedback from Education Leaders Focus Groups to Inform Recruitment Efforts for the High School and Beyond Longitudinal Study of 2022 and Other NCES Studies, 2021).
The post-pandemic challenges faced by educators are further highlighted in Institute of Education Sciences (IES) School Pulse Panel results. This survey collected information on the impact of the COVID-19 pandemic from a national sample of elementary, middle, and high schools. Surveys were conducted during the 2021–22 and 2022–23 school years and provide parallel data relative to what was summarized in the RTI focus group report discussed previously. Even 2 years after initial COVID-19 lockdowns, public schools reported increased student and teacher absenteeism in May 2022 compared with prior school years (Institute of Education Sciences, 2023). In the August 2022 Pulse Survey, 53% of public schools reported that they felt understaffed for the upcoming 2022–23 school year, with Midwest and West census regions in the United States reporting the largest number of teacher shortages because of COVID-19. Additionally, 89% of public schools reported that teachers were concerned about students meeting academic standards in the 2021–22 school year, and about half of schools said their students were behind grade level at the beginning of the 2022–2023 school year.
When seeking participation in research studies, it is important that school-based researchers be mindful of these and other challenges facing schools after the return to in-person instruction. The need to prioritize these challenges may lead to nonparticipation by a subset of schools, which can introduce bias into the data or inhibit reliable estimates. School principals and leaders are faced with balancing the demands of participation in an increased number of state and district assessments while knowing that these activities create a loss of instructional time and can increase burden on staff, which could potentially lead to even more staff turnover. In the post-pandemic era, researchers must develop mitigation strategies to contend with education decision-makers’ reluctance to participate in external research.
With the demands placed on schools and educators, it is important to consider the value of education research. Dating back to a congressional mandate of 1867, the collection of statistics relative to education in the United States has been a key component to reflect on the state of our nation’s education system (NCES Blog, 2023). Although the styles, scope, and nature of the research has evolved since then, education research continues to document trends, outcomes, and comparisons that are key to our students’ future successes. For example, since 1972, the National Center for Education Statistics (NCES) has conducted a series of longitudinal studies covering cohorts of students’ whole school career. NCES data provide important information about student growth and achievement, learning experiences, and transitions through each school level and out of the secondary education system and into postsecondary education or the workforce. Data from NCES studies like these provide educators and policymakers vital information needed to inform policy and action in schools (National Center for Education Statistics, n.d.). To understand what is working or not working in terms of educating students and preparing them for the labor force, we must have data on what our students know, the teaching practices and supports available to them, and their outcome data in terms of level of education and labor force success. This information can then be used to adapt the education provided as the world of work changes.
Even before COVID-19, school participation in these large education studies had been declining over the last decade. For example, participation in the early 2010s was around 55% (Ingels et al., 2011) and fell to approximately 45% just before the COVID-19 pandemic (Russell & Zhang, 2023). However, it should be noted that the in-school administration of these studies allows for a nationally representative sample of students that is more challenging to achieve with other study designs. School-based researchers are already applying strategies to gain participation in their research, many of which are designed to counteract the stress education leaders experience (Roschelle et al., 2014). The COVID-19 pandemic intensified the challenges faced by educators; therefore, school-based researchers must also amplify their efforts to gain participation in their research to ensure the validity and representativeness of their data.
Methods and Results
To examine the impacts of COVID-19 on schools’ willingness to participate in school-based research, we reviewed data from the school contacting systems from four large-scale, nationwide school-based research studies that occurred during the 2022–23 academic year. A school contacting system is a centralized, web-based application used to maintain a record of contact data at all levels and information about study participation. Each study spanned the United States and collected data from students in public and private elementary, middle, and high schools. Three of the four studies’ student component lasted a minimum of 1.5 hours, had a school staff survey component, and employed research staff to facilitate the student piece. The fourth study had a significantly shorter in-school student data collection (less than 0.5 hours) and had no staff survey but was administered by classroom teachers.
School Refusal Reason Review
While recruiting districts and schools for these large-scale research studies, schools sometimes provided reasons when they declined to participate. In each study, recruiters were trained to probe for clarifying information when a refusal was given and when no reason was given. The school contacting system was used to document these reasons for refusal. The system captured 717 refusal reasons and an additional 429 refusals where no reason was given. Our team reviewed the 717 reasons cited and coded responses based on select “key terms” previously identified in the focus groups conducted in 2020 and 2021. Table 1 breaks down the categories across studies of this coding.
In many instances where schools did not provide a reason for declining participation, most simply stated that they were not interested and did not respond to attempts to clarify their reason for refusal. This unresponsiveness makes the refusal reasons that were provided even more telling and valuable when considering how researchers can design their recruitment efforts to better encourage schools’ participation in the future.
Although few school leaders directly cited COVID-19 as a reason for refusing (~3%), school staff gave reasons for refusal that may be associated with the stress COVID-19 placed on schools, namely learning loss and teacher shortages. For instance, one educator said, “the pandemic had a significant disruption on our students’ time on learning and consistency of their schedules.”
School leaders specifically cited loss of instruction time as one of the reasons for nonparticipation (~8%). However, an additional ~42% cited having too much other testing or ongoing surveys as a refusal reason. Although respondents did not mention this category in the 2021 virtual focus groups, we included it in the refusal reason coding because schools also lose a significant amount of instructional time to required federal, state, and local assessments, and school leaders frequently cited this as a refusal reason. Given the impact of COVID-19 and the subsequent learning loss, schools prioritize using classroom time for instruction rather than additional voluntary, and often burdensome, assessments. One school noted the following:
Due to the enormous loss of contact hours with our students due to the Covid [sic] impact either directly from sickness or quarantine time away from the classroom, we have to place our students’ academic needs at the top of our list as we make our future calendar plans. As we already have grade-level testing scheduled throughout the year that is essential to help us view and meet benchmarks, we have to decline participation in this study at this time.
Schools strive to maximize every minute of instructional time and are working intentionally to minimize interruptions to provide a flow and continuity of learning that will accelerate student growth.
On average, respondents cited staffing-related issues as a reason for refusal ~8% of the time, including cases where substantial leadership changes occurred in the prior year. One school leader commented, "At this time, we are going to opt out of participating as our school has undergone significant changes for both staff, students and leadership over the course of the last two years." Staffing issues can pose significant challenges for conducting school-based research. School-based research often involves collaborating with school staff members—such as teachers or administrators—who play a crucial role in facilitating the study and ensuring its successful implementation. When these school staff members are already stretched thin because of unfilled positions or other responsibilities, it can create several obstacles such as limited availability and reduced support and cooperation for the research project. Some schools also noted that they did not have enough teaching staff to accommodate a research study: “Currently we do not have the manpower to facilitate a school coordinator for the optional study… As you are aware, the unprecedented times have put a strain on the demands of our teachers and staff and we are working closely together to ensure the success of our students.”
Schools did provide other non–COVID-19 reasons for their refusal in many instances (~39%):
-
The study was not mandatory, or there was a school, district, or state policy to not participate in outside research.
-
Some schools claimed that their student population is unique in some way (e.g., other language, boarding schools, student with disabilities) and should thus not be included, although it was explained that the study design does allow for these uniquities.
-
Schools noted time constraints (not mentioning testing or surveys).
-
Staff capacity issues where staff turnover or teacher shortage was also mentioned.
However, because of the narrow focus of this brief, we will not delve into these other reasons in further detail.
Study 4
Although study 4 was also a nationwide study conducted during the same period as studies 1 through 3, schools that declined to participate did not cite COVID-19, staffing shortages, or loss of instructional time as refusal reasons nearly as often as in studies 1 through 3. For study 4, schools that declined to participate provided no reason at all for their refusal 1.5 times more than schools that did provide a reason; of the schools that did provide a reason for refusing, the majority (~54%) cited participation in other testing or surveys as the primary reason they declined to participate. Table 2 summarizes the research method differences among studies 1 through 4.
Future Considerations
Nonparticipation of a subset of schools causes a gap in data. This gap has implications for the accuracy and representativeness of the data collected. Our review of the IES Pulse Survey data in conjunction with RTI recruitment experiences after COVID-19 reveals that schools are even less likely to participate, and the reasons for this vary. The finding from the IES Pulse survey that Midwest and West census regions reported the largest number of teacher shortages because of COVID-19 should be considered as it can further skew participation in nationally representative studies. The nonparticipation of schools in these regions can introduce biases and affect the generalizability of the study’s results. Additional research and correlating study data are needed to determine the statistical significance of this finding and to explore opportunities to mitigate the effects of nonparticipation in particular census regions. One possible mitigation strategy to overcome staffing shortages would be for the study to offer to pay for a substitute teacher to offset the time it may take for school staff to prepare for and administer the survey. Another mitigation strategy includes oversampling, where regions known to be underrepresented are oversampled so participation results can still be nationally representative.
Schools that refused studies 1 through 3 were also concerned about research activities interfering with instructional time; however, this was not cited in study 4. The study 4 design was significantly shorter in length than the other studies (less than 30 minutes) and could be completed entirely online, without external research staff coming into the building and disrupting routines. To limit research activities interfering with instructional time, researchers should consider (1) whether their research questions could be effectively answered with a study design in which research staff do not need to visit the school campus and (2) the possibility of using instruments that can be completed within one class period (i.e., typically less than 45 minutes). Such study designs have, in part, been made more feasible as a result of increased student and school access to computers and internet. Although an alternate study design may be considered for future studies (like study 4), research teams must consider whether schools are still able to provide devices for web-based research.
Beginning with the planning phase of any study, researchers should start thinking through mitigation strategies in the planning stages and consider the substantial time and effort required for recruiting schools into voluntary research and budget accordingly. Allocating an extended timeframe for recruitment efforts can significantly enhance the prospects of school participation. A longer timeframe allows for the development of robust relationships with educational institutions, nurturing trust and understanding about a study’s potential impact and allowing time to consult with district staff on the development of research application packages, which a growing number of educational institutions now require. Time for a more in-depth engagement strategy enables recruitment staff to develop materials and communication strategies tailored to articulate the value of the particular research study to each intended audience member—district and school leaders, teachers, students, and the broader field of education. This extended recruitment duration not only provides flexibility but also allows for multiple touchpoints and interactions to address concerns, provide clarifications, account for the challenge of working with school staff who are overburdened, and build a sense of collective ownership of the research endeavor among all impacted audiences. An extended recruitment timeline offers the opportunity to adapt strategies in response to evolving situations, fostering a better chance of garnering enthusiasm and commitment from schools that may initially be hesitant. Ultimately, investing additional time and resources in the recruitment process is a proactive approach that can significantly increase the likelihood of school involvement and overall research success.
Conclusion
As we have seen in the past few years, COVID-19 has created a new set of obstacles to overcome in the field of school-based research. These barriers to school-based research, including those that have arisen more prominently since COVID-19, will continue to persist. However, the value of conducting school-based research studies remains. With this in mind, those who design these studies, as well as those who work with the schools to participate in them, must prepare to face these additional challenges. Continued in-depth examination of these barriers will help both study designers and recruiters devise agile solutions aimed at specifically addressing school’s concerns with participation. School-based recruiters genuinely believe in the value of these research studies and will continue to seek ways to alleviate concerns to gain maximum cooperation from schools and districts. As we continue to articulate the value of these data, we will continue to be mindful of common refusal themes and trends, document these trends, and work to develop mitigation strategies.
Acknowledgments
We would like to thank Elizabeth Glennie for providing mentorship and guidance during the writing process. Your insight and feedback were greatly appreciated and helped bring our paper to life. Also, a special thank you to Debbie Herget who has supported and advocated for us and encouraged us to write about our school-based recruitment experiences to share with the education community.
We would also like to thank Kimrey Millar who has led much of our recruitment work and developed many of the team’s strategies and methods. Also, thank you to RTI Recruitment members: Susan Fleming, Therese Hansen, Alisa Debnam, Monica Funari, and Jennifer Krause. The experiences within this paper are a culmination of your heartfelt outreach to education leaders in our nation and how we can work together on critical research topics in our schools. A special thank you to the RTI programming team that supports and maintains our school contacting systems, including Christopher Alexander, Courtney Waterman, and Wesley Toliver, among others.
RTI Press Associate Editor: Amber Gove