Challenges Facing CAHPS Surveys and Opportunities for Modernization

Introduction

Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys are widely used tools for obtaining patient assessments of health plans, health care facilities, and health care providers. The Centers for Medicare & Medicaid Services (CMS) requires certain providers and facilities who participate in some programs to participate in CAHPS surveys. These surveys measure care experiences from the patient perspective and cover general aspects of care as well as specific conditions. General care questions include communication with medical and clinic staff or facility cleanliness. Specific condition questions can ask about pain from procedures or treatment options presented in the facility. CAHPS survey analysts share results with CMS, providers, and facilities to stimulate improvement in the US health care system which benefit all concerned because:

• Providers learn which aspects of patient experience they can improve.

• Providers receive financial bonuses from CMS for above-average care experience scores.

• Providers are incentivized to keep care experience metrics high, so patients may be more likely to have a better experience with the health care system. Research findings associate better care experiences with higher adherence to treatment protocols, better health outcomes, and reduced costs (Anhang Price et al., 2014).

• The data from publicly available CAHPS surveys help consumers make informed choices about health care (e.g., dialysis facilities, home health agencies, hospices, hospitals) for themselves or their family members. Payers also evaluate the data when making business decisions.

Despite the value of these surveys, there are challenges to conducting them and sharing their results. This paper discusses six main challenges:

1. Declining response rates

2. Speed of delivering survey results

3. Cost of administration

4. Adoption of newer modes of administration

5. Retention of measures which are “topped out” and difficulty introducing new measures

6. Questionnaire length

Then, we propose several opportunities for modernization to ameliorate those challenges:

1. Addressing nonresponse though survey and statistical methods

2. Improving speed and reducing cost by applying innovative sampling methods

3. Implementing newer survey modes or combinations of survey modes

4. Developing new questions

5. Increasing stakeholder feedback

6. Reducing questionnaire length

7. Providing incentives to respondents

8. Marketing CAHPS results to the public

We encourage CMS, the CAHPS Consortium, survey researchers, and other interested organizations to consider ways to implement these suggestions. Some of these suggestions may also apply to other areas of health care or survey research.

What Are CAHPS Surveys?

Medicare has surveyed Part C (Medicare Advantage) enrollees about their care experiences since 1997 and traditional Medicare enrollees since 1998 (Agency for Healthcare Research and Quality, 2021). CAHPS surveys have evolved into the standard survey tool for obtaining patient assessments of health plans, hospitals, physicians, and other health care providers. CAHPS is a program of AHRQ, US Department of Health and Human Services. Different versions of CAHPS surveys designed to measure care experiences in a variety of health care settings have been developed and approved by AHRQ. Many results are reported to the public, while others are used within the government for payment, quality assessments and improvement purposes.

It is telling that CAHPS has the word “consumer” in its name. The original survey development stemmed from the movement toward health care consumerization (Zakkar, 2019). This involved framing the person using health care services as a consumer to be satisfied, rather than a patient at the receiving end of whatever doctors thought best.

What was formerly called “patient satisfaction” has now been reframed as “care experiences” (Martino et al., 2009). Instead of using a consumerization framework, the industry has begun thinking about person-centered and person-directed care as critical to health care quality (Lines et al., 2015). CAHPS surveys, ahead of this trend, have always focused on measures that can only be accurately reported by the person receiving care (or their close family, or both).

Instead of asking about satisfaction, which inherently varies from person to person, CAHPS surveys ask about experiences with provider behaviors that are hallmarks of high-quality care. Core CAHPS questions, for example, ask how often providers explain things in a way the patients can understand, spend enough time with the patient, and answer questions during and after office hours as soon as the patient needs them answered. The questions are clear and unambiguous; no patient idiosyncrasies nor expectations of how satisfied the patient feels they should be are baked into the responses. The questions are actual evaluations of the experience the provider gives the patient, and this is how we know they are valid for comparing providers. This is the strength of CAHPS measurement science.

Why CAHPS Data Matter

Positive patient care experience can profoundly and positively affect health care quality, costs, and outcomes. One of the most significant publications in this area is a meta-analysis of more than 40 articles summarizing how care experience positively affects clinical outcomes, safety, utilization, and cost (Anhang Price et al., 2014). Positive care experience is associated with higher levels of patient adherence (Haskard Zolnierek & Dimatteo, 2009; Lee & Lin, 2009; Ratanawongsa et al., 2013), improved clinical outcomes (Boulding et al., 2011; Meterko et al., 2010), and lower utilization of inpatient and emergency department services (Anhang Price et al., 2014). In primary care settings, positive care experience is associated with high-quality care (Cook et al., 2015); in hospital settings, positive care experience is associated with improved patient safety and lower costs (Aleccia & Bailey, 2017). Improving care experiences spurs improvements in these other vital areas.

How people are treated matters. Apart from how positive care experiences affect cost and health care quality and outcomes, treating people respectfully is a value in our health care system. Our society should have the maturity and compassion to treat any person coming for needed services with respect, especially people who may be in precarious states because of illness, age, or disability.

CAHPS surveys are increasingly tied to provider payment from Medicare. CMS, as the largest insurer in the United States, was quick to recognize the quality and financial benefits of positive health care experience. These benefits are CMS’s motivation for mandating or encouraging CAHPS surveys by many providers in a variety of health care settings. The results from CAHPS surveys as well as clinical outcomes determine how much a provider is paid in CMS’s value-based programs (Anhang Price et al., 2014). Tying payment to clinical outcomes and CAHPS results financially incentivizes providers to try to improve both care experiences and clinical outcomes. The shift away from volume toward quality will continue to increase, with nearly all CMS payments expected to be tied to value-based contracts by 2025 (Haskard Zolnierek & Dimatteo, 2009). This shift to paying providers for value rather than volume will make it increasingly important to measure the quality patients receive from their health care providers, and CMS views CAHPS as the measurement gold standard for determining quality of patient experience.

CAHPS surveys provide accountability in the health care system. Poor care, thankfully, is the exception and not the rule in the United States. When there is a problem with a facility or provider, complaints from patients and fellow providers may eventually expose the issue to authorities. CAHPS surveys, however, act as an early-warning system. A provider or facility frequently or consistently receiving far below average scores on its CAHPS alerts CMS to these problems, allowing it to devote the appropriate resources to investigating and acting before matters escalate. It can also point to systemic problems. For example, according to the CAHPS Hospice Survey, which is one of the newer CAHPS surveys in operation, 1 in 5 respondents said their hospice agency did not always show up when they needed help (Aleccia & Bailey, 2017). This is clearly an area that needs work on the part of these providers.

Challenges Facing CAHPS Surveys

Declining Response Rates

Response rate trends show declining response rates. Since the CAHPS program’s inception in the mid-1990s, AHRQ and the CAHPS Consortium have focused on producing results that represent the target enrollee or patient population, accurately reflect people’s experiences with health care, and are comparable across health care groups and organizations (AHRQ, 2019). Response rates are frequently used as an indicator of the representativeness of collected data. Higher response rates lend validity to survey results, increase the likelihood that results can be generalized to the target population, and decrease the potential of nonresponse bias. Declining response rates have been a concern for survey researchers over the past few decades, and this concern is echoed by CAHPS survey users, stakeholders, sponsors, clinicians, and accreditation organizations.

A recent Mathematica report compiling trends for eight federally funded national surveys sponsored by the Department of Health and Human Services showed declines in response rates for all eight surveys in recent years. The National Health Interview Survey showed a dramatic decline in the household module, falling 18 percentage points from 1997 (92 percent) to 2014 (74 percent), while the initial interview for each new panel for The Medicare Beneficiary Survey went from around 83 percent in 1995 to about 73 percent in 2013, a 10-percentage point decrease (Czajka & Beyler, 2016).

Other national surveys have seen similar declines. The Survey of Consumer Attitudes experienced an average yearly decline of 1.50 percentage points from 1997 to 2003, while response rates to the General Social Survey varied between 73.5 percent and 82.4 percent from 1975 to 1998 but dropped to 70.0 percent and 70.1 percent in 2000 and 2002 (Curtin et al., 2005). Possible reasons for the decline in survey response rates include an increase in the number of two-worker households, an increasing prevalence of caller ID and cell phone–only households, and growth in the number of federal surveys, political polling, and telephone solicitations (Kim & Lepkowski, 2002; Link et al., 2006; O’Toole et al., 2008).

Response rates from population subgroups should be inclusive. In addition to overall concerns about declining response rates, ensuring representation across the sample is important since researchers want to implement positive changes for all members of the relevant population. When subgroups of the population are missed, there is an increased risk to data quality because of nonresponse bias. Researchers can address some of the nonresponse bias with statistical solutions such as weighting. However, it is possible to reach a point where weights are not enough to make up for a lack of specific respondents. In a later section, we discuss adjustments to existing methodology that can result in increased response from previously hard-to-reach populations.

Speed of Delivering Survey Results

Current CAHPS surveys have a relatively long time from the beginning of the survey period to the public reporting of results. Some surveys have a detailed sampling process which adds to the overall time of the survey. Others have complex analytical adjustments which must be completed before data are compiled and reported on CMS websites. The overall length of the survey period means there is a lag between when the data were collected and when either patients or providers can see if a problem exists. Some CAHPS surveys have public reporting requirements to present a rolling 12-months of response data. This protects facilities against fluctuations in care experience and response rates but means there is an additional lag time for the public to see and use survey results when a program is beginning.

Cost of Administration

As response rates continue to decline, the cost of getting an individual completed survey from a patient increases. CAHPS provides useful insights to CMS, providers, and patients, but that must be balanced with the cost of the survey. In some surveys, the cost is borne by the hospital, agency, or facility. In other programs, CMS pays for the survey. No matter the payer, the desired outcome is the same: accurate estimates of care experience. Sample size is a large driver of CAHPS cost for mail and telephone administration, and the decreasing response rates requires ever-larger sample sizes. Many CAHPS researchers and data users adhere to old-fashioned brute force methods that demand a high number of completed surveys per health care provider. These are not effective in balancing cost and accuracy.

Updated sampling and estimation approaches may enable smaller sample sizes. It may be possible to obtain accurate estimates without ever-increasing sample sizes by using prior information to reduce variance. RTI conducted preliminary research in 2019 on a patient survey in the primary care setting. We measured the change in key estimates between completed mail surveys from the full mail protocol and the completed mail surveys from only the first month of fielding. The estimates for the national and larger regions were nearly identical in the first month only versus the full mail protocol. This demonstrates that larger health care organizations, which have high sample sizes under the current methods, could reduce sample size without sacrificing precision.

We acknowledge limitations to this preliminary analysis—primarily, other research indicates there are differences between early and late responders, which may extend to care experience (Peytchev et al., 2009). This area of research should be of interest to CMS and CAHPS researchers.

In addition, some CAHPS surveys have been conducted for many years, so the annual variation of key measures between and within entities is known and minimal. Researchers should take advantage of that known variation to adjust the sample sizes.

Adoption of Newer Modes of Administration

Minimizing the costs and increasing the coverage of CAHPS surveys have been continued focuses of stakeholders and users. A possible way to achieve these goals is by incorporating newer modes of survey administration, such as web or text-message surveys.

Web surveys are widely accepted. In the past decade, researchers have investigated whether web-based surveys are a viable alternative to traditional survey modes. Web surveys are increasingly appealing because of the cost savings over other modes of data collection such as postal surveys and telephone interviews. One study found that collecting data by mail or telephone can cost around $10 per completed survey, whereas the cost per web-based survey was less than $2 (Bergeson et al., 2013). Web-based surveys can be completed faster, leading to a shorter data collection period than mail modes. Because data are captured directly and automatically in an electronic format, analysis times are faster than those of mail questionnaires, which take time to complete, return, and process (Wyatt, 2000). Additionally, web surveys offer further reach than traditional modes of data collection, as 93 percent of American adults use the Internet, allowing more respondents to be included than in postal or phone surveys (Pew Research Internet Project, 2021). In 2019, the US Census Bureau began asking respondents to complete the American Community Survey on the Internet (https://consumer.ftc.gov/consumer-alerts/2019/08/american-community-survey-legit), signaling that web surveys gained widespread acceptance as a legitimate mode of collection by the federal government.

The potential for text message–based surveys is growing. As with the number of Internet users, the number of mobile phone users has increased worldwide. Of all Americans, 97 percent own cell phones and 79 percent say they use text messaging on their phone (Pew Research Internet Project, 2021). Thus, text-message surveys—either conducting the survey by texting the questions/answers or by using the text to direct the respondent to the online survey—have become an attractive tool to survey researchers as they offer the ability to contact a large and diverse number of people faster than traditional modes of data collection. Text-message surveys are estimated at 1/10th the cost of a random-digit dial (RDD) phone survey and responses are collected immediately (Hoe & Grunwald, 2015). In one study, researchers found that 88 percent of responses were recorded on the same day as the initial or reminder text, and 47 percent of all responses came within the first hour of the invitation or reminder text (Hoe & Grunwald, 2015).

Coverage presents a challenge to implementing newer modes. Web-based surveys are accompanied by concerns about nonresponse bias and the ability to fully represent the target survey population. Although Internet usage is increasing, a high proportion of US households still lack Internet access or do not access the Internet frequently. Further, people who do not have email addresses may be left out of web-based surveys. Researchers have found that older Americans (65 and older) are less likely to have email addresses and are more likely to respond to mail surveys compared with web surveys. In one study using the CAHPS Clinician and Group Survey, researchers found that the mail-only protocol had a 43 percent response rate, while email-only had a response rate of only 20 percent (Fowler et al., 2019). Using the same survey, another study found a 14 percent response rate for web survey administration compared with a 33 percent response rate for mailed surveys (Bergeson et al., 2013).

Given that this demographic of older Americans comprises such a large proportion of CAHPS survey respondents, there has been considerable reluctance to promote web-based CAHPS surveys for fear of hurting response rates. In addition, Internet usage is lower among adults who are poorer, less educated, Hispanic, or African American (Perrin & Duggan, 2015).

Researchers have experimented with supplementing the modes of mail and phone with a web option to gain some of the advantages of web (e.g., cost, speed, representativeness from younger groups) without inhibiting the response rate of older respondents. In a recent mode experiment, conducted by RTI International, of the Medicare and non-Medicare population, we explored and compared five modes: mail only, phone only, web only, web with mail follow-up, and web with phone follow-up. Web alone was not successful, but web with phone follow-up and web with mail follow-up of contact were very successful in that they boosted response from mail-alone or telephone-alone modes. This study relied on patients providing their email addresses to their health care providers. As health care providers and electronic health record systems make the collection of email addresses more common, this barrier to response may be reduced.

Another recent RTI study found that an approach using a web survey invitation followed by mail nonresponse follow-up, and followed by telephone nonresponse follow-up yielded approximately a 34 percent response rate—equivalent to the response rate of mail with telephone nonresponse follow-up.

Regardless of whether web surveys yield higher response rates compared with traditional survey approaches, we assert that the inclusion of a web mode in CAHPS is essential for lower costs and improving coverage given the changing demographics and willingness to use web-modes. Overall response rates matter, but we know response is not universal. Response varies based on individual characteristics and differential response rates mean a survey may be subject to coverage bias. The addition of modes to reach groups who are less likely to respond may have value, even with a low overall response rate in that mode, because the final responders may better reflect the target population. As more CAHPS studies explore the use of web as a survey mode—on its own or as part of a combined mode—CMS and researchers should share their successes and lessons learned so that the use of newer modes can help mitigate the issue of coverage bias in existing CAHPS surveys.

Retention of Measures Which Are “Topped Out” and Difficulty Introducing New Measures

What is a “topped out” measure? Some core CAHPS measures include “Overall Rating of Provider” (one survey question), “How Well Providers Communicate” (four survey questions), and “Access to Care and Getting Timely Care, Appointments and Information” (three survey questions) (https://www.ahrq.gov/cahps/surveys-guidance/cg/about/survey-measures.html). A measure is considered “topped out” when a high percentage of providers find a large percentage of their patients answering these questions in the most positive way (e.g., answering Yes, definitely or Always to questions about desired provider behavior). Because so many patient responses are positive across most providers, the construct measures have little variance. The consequences are an inability for providers to grow and improve (as they are measured on concepts at which they already excel) and uncertainty for consumers in distinguishing providers with good performance from bad.

Are measures topped out? Some measures in CAHPS are certainly topped out. On the CAHPS for Merit-based Incentive Payment Systems (MIPS) and Accountable Care Organization (ACO) CAHPS programs, for example, the number of patients who gave their provider a rating of 9 or 10 on a scale of 0 to 10 was disproportionate, as the 90th percentile was 94 percent and the 30th percentile was 91 percent. There is tremendous clustering of high scores, as about 2.5 percentage points separate the majority 30th and 90th percentiles. There is also significant clustering of scores in the high 80s and mid 90s (out of a maximum score of 100) for “How Well your Providers Communicate,” “Getting Timely Care, Appointments and Information,” and other measures used in other CAHPS surveys. In one study of linked Medicare CAHPS and Surveillance, Epidemiology and End Results (SEER) data, the median on each composite score was 100 (out of 100), and for the single-item global rating of personal doctor and specialist, the median value was 10 (out of 10) (Lines et al., 2019).

These questions may be a victim of their own success. CAHPS’s 25-year focus on these important elements of care experience has imbued these good practices of patient care in provider interactions. If patients do not have good experiences with them, we would be concerned. Therefore, analytic focus may need to shift to focus only on those providers scoring unusually low on these measures, as is this an early warning system to signal problematic providers.

Many other CAHPS measures are not topped out. “Questions about Access to Specialists,” “Preparation for Surgery,” “Stewardship of Patient Resources,” “Health Promotion and Education,” and “Shared Decision Making” are a few examples of measures where there remains room for improvement and growth.

Introducing new measures can be difficult. At the same time, it is important for CAHPS to stay relevant with new questions that align with aspects of care currently important to patients and providers. Some aspects include the following:

• Experience with the telephone triage system (e.g., whether people can only speak with a service center to leave messages or are able to speak with clinicians)

• Whether patients are comfortable with the provider and are able to be honest about their concerns

• Patients’ ability to take care of their own health

Any new questions should follow CAHPS measurement science principles: be clear and unambiguous and measure the extent to which respondents experienced providers exhibiting certain behaviors without allowing expectations and idiosyncrasies to affect answers. Crafting questions to do this is not easy. Another challenge with new questions is that it is hard to remove questions from surveys. CMS often uses the benchmarking process to compare survey results from one program to another program; therefore, retiring questions is usually not possible. New questions, when written, need to be added to the survey and this causes the questionnaire to become lengthier, which is not desirable.

Questionnaire Length

Long surveys may dampen response rates. In recent years, key stakeholders have focused on the impact of survey length on the efficacy and costs of data collection and its correlation to overall response rates on CAHPS surveys (AHRQ, 2019). End-stage renal disease experts and stakeholders have expressed concerns about survey burden due to the length of the In-Center Hemodialysis CAHPS (ICH CAHPS) Survey, which is 59–62 questions long, depending on mode of administration, and surveys nearly all patients in a dialysis facility twice each year. Similarly, the Clinician & Group CAHPS (CG-CAHPS) survey, which is 33 questions long, is perceived as too lengthy by the health care organizations that administer the instrument (Stucky et al., 2016). Individuals who have more health difficulties may be less able or less likely to complete a lengthy survey. Sponsors are interested in investigating whether shortened versions of CAHPS surveys would reduce both patient and administrator burden and increase response rates.

Results from research on the association between survey length and response rates have varied. Some researchers have found that reduced survey length can improve response rates. For example, when examining response rates for a one-page survey compared with a two-page survey, researchers found that the response rate for participants who received the one-page survey was 8 percentage points higher than those who received the two-page survey (Olmstead et al., 2005). However, other researchers have found that survey length has little to no effect on response rates. For instance, researchers studying differences in the response rates of a two-page double-sided questionnaire compared with a four-page double-sided questionnaire found that response rates were nearly equal: 40.7 percent for the two-page double-sided questionnaire compared with 40.9 percent for the four-page double-sided questionnaire (Bolt et al., 2014).

There is additional concern about the effect of survey length on response rates based on mode—such as whether response rates are lower for the telephone mode because the length of time to complete the survey is stated at the beginning of the interview. The length of time to complete CAHPS surveys via phone varies. For example, the Home Health Care CAHPS (HHCAHPS) Survey takes about 12 minutes to complete, while the CG-CAHPS Survey takes 15 minutes, and the ICH CAHPS Survey takes about 16 minutes to complete. Some research has shown that when the stated length of a survey is shorter, participants are more likely to participate. For instance, researchers testing the stated survey length in web surveys found that participants who were told that the survey would last 10 minutes were 13 percent more likely to start the survey compared with those who were told the survey would last 30 minutes and 10 percent more likely to start compared with those who were told the survey would last 20 minutes (Galesic & Bosnjak, 2009).

Opportunities for Modernization

CAHPS surveys and dissemination of their results can be advanced in many ways. CAHPS contractors should propose innovative approaches to help CMS meet their goals. The wider research community should publish on the improvements of CAHPS and the relevance of the measures.

Summary of Recommendations

To keep CAHPS surveys relevant for health care providers and patients, future surveys should modernize their approaches. Examples from this paper include:

• Addressing nonresponse through survey and statistical methods

• Improving speed and reducing cost by applying innovative sampling methods

• Implementing newer survey modes or combinations of survey modes

• Developing new questions

• Increasing stakeholder feedback

• Reducing questionnaire length

• Providing incentives to respondents

• Marketing CAHPS results to the public.

Address Nonresponse Through Survey and Statistical Methods

“Survey methods” is a broad term used to describe the knowledge and skills needed for the collection and analysis of quantitative data suitable for statistical inference to large populations. Components of survey methodology include questionnaire design, measurement error, usability testing, and applying the advantages and disadvantages of various data collection modes. Statistical methods include sample design, nonresponse bias adjustment, patient-mix adjustment, and statistical analysis.

We previously noted the challenge of declining response rates. This section focuses on methods for improving response rates. It also describes statistically adjusting for nonresponse bias which may, but not always, emerge from low response rates.

Use targeted methods to better reach subgroups. Researchers conducting the Kidney Disease Quality of Life (KDQOL) survey as part of an alternative payment model project within the Center for Medicare & Medicaid Innovation found that survey response was lagging among African American, Hispanic, and younger respondents (Djangali et al., 2019). End-stage renal disease disproportionately impacts African American and Hispanic populations, so the lack of response from these two groups was particularly concerning. The KDQOL survey follows a mail-telephone sequential mode design. To counteract subgroup response rate decline, the researchers made small, data-driven changes to the study’s existing methodology. These changes included beginning telephone follow-up with these groups earlier in the data collection period, offering their preferred mode to previous responders and developing a model to identify likely Spanish-speaking respondents and to mail those respondents both English and Spanish materials. As a result of these changes, the researchers saw statistically significant response rate gains for each targeted subgroup and overall. The researchers maintained these changes to the methodology for the following year and saw consistently higher response rates when compared with the methods employed previously.

In a similar methodological experiment with a CAHPS population, researchers conducting the Fee-for-Service (FFS) CAHPS tracked declining response among Hispanic beneficiaries over several years (Djangali et al., 2020). To address this issue, the researchers drew from the above-referenced KDQOL survey results and work developed by the RAND Corporation on the use of predictive modeling to detect likely Spanish language preference. The FFS CAHPS also employs a mail-telephone sequential mode design. The FFS CAHPS researchers worked with RAND to identify 11,286 sample members more likely to prefer Spanish materials as identified by RAND’s Medicare Bayesian Improved Surname Geocoding model. Researchers randomly selected 7,000 sample members for the treatment group and the remaining 4,286 served as the control. Treatment group members received both English and Spanish materials throughout the mail phase. The control group received English materials with instructions to call the study helpline to request materials in Spanish. Using logistic regression to analyze the results, researchers found that respondents in the treatment group were 17 percent more likely to respond to the survey than the control group; a statistically significant result. When reviewing completion language, 53 percent of Hispanic respondents completed their survey in Spanish. However, Spanish completion for members of the treatment group was 34 percent compared with 11 percent for the control group.

Help providers’ offices convey the importance of the survey. Some CMS programs are addressing the problem of response rates by distributing written material (e.g., flyers or posters) to patients in the provider’s office. These materials concisely and simply explain the basic survey purpose and legitimacy. Office staff also become familiar with the material so they can answer patients’ questions with ease, increasing confidence and support of the survey. An endorsement from a trusted provider conveys that this is a legitimate survey, which can be very meaningful in a patient’s decision whether to complete a survey.

Reduce requests from multiple surveys. The growth in the number of surveys has been implicated as a cause of declining response rates, and while this may be a trend that is here to stay, there are ways to blunt some of this impact. CMS is starting to take deliberate steps in the sampling stage to reduce the number of times a single individual can be sampled for a survey. In CAHPS surveys of the home health, outpatient surgery, and dialysis settings, CMS specifies that a person can be sampled no more than twice in a year. (https://homehealthcahps.org, https://oascahps.org, https://ichcahps.org). CAHPS surveys in primary care settings have taken the step of removing patients who were recently sampled for other CAHPS surveys (https://gpdccahps.org).

Analyze data for nonresponse bias and conduct adjustments where bias is present. While we tackle the problem of declining response rates, it is important to remember that higher response rates do not always equate to good representation. Good representation emerges when members of the subgroups that comprise the population participate in the survey, so it is important to continue efforts to reach subgroups with lower response rates.

Nonresponse analysis is an important part of identifying subgroups that may be left behind, and nonresponse bias analysis is a tool that identifies whether the missing responses are causing bias. When bias is present, nonresponse weighting is a good method for addressing this. CMS typically uses the patient-mix adjustment method. This method recognizes patients in some subgroups have a propensity to respond to care experience questions in a way that is significantly different from other groups, after controlling for other factors. For example, people with a college education tend to give their facility higher scores on communication than patients with less than a high school education. Thus, a facility’s mix of types of patients can impact the facility’s score. Patient-mix adjustment measures the differences between groups and applies correction factors that statistically level the playing field between facilities. After the patient-mix adjustment process is completed, the survey results no longer have nonresponse bias, which has been shown in some CAHPS surveys (Eicheldinger et al., 2018; Zuckerbraun et al., 2020) These statistical methods should be used in conjunction with survey methods that increase response from underrepresented groups. Increasing response from lagging subgroups can improve overall response rates and data quality.

Improve Speed and Reduce Cost by Applying Innovative Sampling Methods

In the section on challenges, we noted that CAHPS Surveys are expensive and their administration can take months before results are released to providers and the public. This is problematic because CAHPS surveys are supposed to act as an “early warning system” and that feature could be considered fragile if the speed is not improved. One reason for the high cost of the CAHPS surveys is the number of completed surveys required for statistical reliability and precision.

A Bayesian approach can be used to reduce sample size. One option that permits a smaller sample is a Bayesian approach where researchers would use the most recent survey’s information to inform the next year’s design. Starting with the past year’s score and using it as the prediction for the current year will allow a smaller sample size than starting with a blank slate for that entity’s score. Ignoring several years of already collected data when planning for the next year is wasteful. CAHPS questionnaires and patient populations tend to be consistent from year to year, which makes this approach reasonable for reducing cost.

A variance-based approach can also be used to reduce sample size. Another method of reducing sample size is variance-based. Hospitals or practices that consistently perform well on measures and have low variation will need less sample to get an accurate result. This means that patients typically agree the hospital performs well and there is low value in getting additional respondents who will provide the same information. However, a hospital that has high variation in patient responses will need more respondents to ensure a clear view of care experience. Survey organizations and CMS should consider the within-facility variance when planning and allocating samples.

Adaptive sampling design may offer some cost-savings and increase representativeness of the sample. Ideally, with adaptive sampling, adjustments could be made to the sample size in real time during data collection. This approach is used to improve the representativeness of the sample based on already completed interviews, at that point in data collection. Therefore, health care facilities or their vendors can concentrate more effort on capturing diverse opinions from their patients. Additionally, adaptive sampling is used to monitor sampling targets, and sampling ceases after the targets are met. Using targets helps protect against some of the known variation between early and late responders. This results in reduced data collection costs compared with a fixed sample strategy. This approach may also speed the data collection and enable faster public reporting.

Implement Web Modes

In recent years, CMS and RTI have received feedback from facilities and vendors indicating that they now collect email addresses and many are using web administration for other surveys. Although the siren call of web surveys is strong, CMS is proceeding cautiously given that response by web trails response by other modes among the most populous demographic group in CAHPS surveys, older Americans.

Research evidence shows the success of web mode in conjunction with other modes. CMS has investigated (Outpatient and Ambulatory Surgery [OAS] CAHPS, Hospital CAHPS, Comprehensive Primary Care First) or is in the process of investigating (ICH CAHPS, HHCAHPS, Hospice CAHPS) how web-based survey modes could be introduced into existing surveys. The 2019 OAS CAHPS mode experiment sampled patients receiving surgical services in Medicare-certified hospital outpatient departments and ambulatory surgery centers and included the following objectives:

• Testing whether three experimental modes (web only, web with mail follow-up, and web with telephone follow-up) would be viable options for the OAS CAHPS Survey

• Assessing the effectiveness of the use of email contacts to facilitate the new web modes

• Determining whether a mode adjustment was needed

• Identifying factors not in the control of OAS facilities that affect OAS CAHPS scores

The experiment included five modes: mail only, telephone only, web only, web with mail follow-up, and web with a telephone follow-up. Results revealed that all five modes were viable options: The response rates by mode were 35 percent for mail only, 19 percent for telephone only, 29 percent for web only, 39 percent for web with mail follow-up, and 35 percent for web with telephone follow-up. The survey mode did not have a significant impact on survey estimates, and no mode effect adjustments were needed. Given the low availability of email addresses on the sampling frames, a web-only approach was not authorized for national implementation, because access to email addresses is critical to allow easy access to the web survey via a link in an emailed invitation. Various web-based mixed modes of administration are viable options for the OAS CAHPS patient population. There were small, yet statistically significant differences in responses attributable to patient-mix characteristics, including (1) surgery type; (2) overall health; (3) overall mental health; (4) age; (5) education; and (6) elapsed time between date of service and the survey.

Based on the outcome of this mode experiment effort, CMS introduced two new web-based survey administration modes (web with mail follow-up and web with telephone follow-up) in national implementation in calendar year 2022 (Federal Register, 2021).

To highlight another CMS CAHPS Survey, CMS contracted with RAND to design and conduct a mode experiment for the Emergency Department CAHPS Survey in 2018 to test “several mixed-mode protocols that focus on a push-to-web approach for survey administration.” Nine experimental protocol arms were tested. The objectives of this mode experiment included

• Determining whether any of the experimental protocol arms with a web survey component could obtain a higher response rate than a standard mixed-mode protocol (mail with telephone follow-up)

• Identifying which methods of survey invitation (email, text, and hardcopy letter) were most beneficial with improving the likelihood of response (Parast et al., 2019)

Results revealed that the response rate to the protocol arm involving email, mail, and telephone was the highest (27.3 percent) “but was not significantly higher than the standard mixed mode (mail with telephone follow-up (25.5 percent). The response rates for all other arms were significantly lower than the standard mixed mode.” This mode experiment demonstrates that multiple modes involving web, mail, and telephone and multiple methods of web survey invitation will likely obtain the highest response rates in the emergency department population (Parast et al., 2019).

The outcome of each of these mode experiments informed or will inform CMS’s decision to roll out web-based mode(s) as part of their CAHPS national implementation.

Develop New Questions

Suggest new questions about relevant patient experiences. Researchers should explore and develop new questions for CAHPS because it is important the CAHPS questions stay relevant to current issues in patient experience. We believe the following areas are important and timely, and they are currently uncovered by CAHPS:

• How do patients experience new technological aspects of health care such as remote communications, patient portals, and virtual visits?

• What is a more granular way to capture certain patient demographics such as gender and disability status?

• Do patients experience providers treating them differently due to their racial, ethnic, religious, sexual, gender, disability, or other identity?

• Do patients feel their provider cares about them as a person?

• Does the provider have an encouraging effect on the patient?

• Does the provider engage with and help patients with nonmedical needs such as behavioral health (e.g., stress, depression, substance use) or basic needs that impact health (e.g., housing, transportation, food)?

As discussed previously, wording questions on these topics in a way that focuses on provider behavior as observed by the patient, while being agnostic to patient expectations and preferences, is a difficult job. In contrast to some of the existing CAHPS measures which are “topped out,” we expect more variation in these newer questions. Increased variability will provide more information for consumers to distinguish between care providers.

Patients are increasingly interacting with health care technology. As the diffusion of web-based communication technologies continues to spread in the United States, more health care providers recognize the advantages of incorporating web-based systems into care strategies. Broadly termed “telehealth,” these web-based services include virtual patient portals where users can access patient-specific health care records and insurance documentation, update data, and track prescriptions. Telehealth services also include technologies that allow providers to conduct virtual office visits with patients via audio or video. In the context of CAHPS, virtual office visits are applicable for many types of care. Exceptions include dialysis care, hospice care, and hands-on activities that would require in-person visits. The experience of care can go beyond the hands-on work of the provider, and the use of technology for communication may be most applicable across the CAHPS family of surveys.

Proponents of telehealth assert that the introduction of virtual visits and associated services allows for increased communication with patients, efficiency in the dissemination of critical diagnostic and care information or records, and a decrease in cost and risk of exposure to communicable diseases through lower volumes of in-person visits. However, patients have inequitable access to telehealth services stemming from varying root causes. Researchers broadly refer to the inaccessibility of web-technologies as a “digital divide” and point to factors such as digital literacy or comfort, infrastructure development in rural areas, and economic barriers as key contributors to lower adoption among some populations. Telehealth was originally developed to improve access to care in rural and underserved populations. The CMS Medicare Telemedicine Snapshot showed 53 percent of Medicare users experienced a telemedicine visit between March 1, 2020, and February 28, 2021, and of these, 44 percent were in rural areas (CMS, 2021).

Even if the services are feasible due to the “digital divide,” the next issue is whether these meet the needs and expectations of patients. New CAHPS measures could explore the availability, effectiveness, quality, timeliness, or other aspects of this type of care. The CAHPS Consortium is working to build out this area and has released question wording that is inclusive of in-person, video, and telephone-based doctors’ visits (Agency for Healthcare Research and Quality, 2022).

Up-to-date demographic categories (ethnic, racial, gender, and disability status) are important to patients. Health care research is clear that demographic factors such as a person’s age, racial identity, ethnicity, gender, or socioeconomic status can have a statistically significant influence on their experience of care (Martino et al., 2019). CMS and its contractors recognize the need to account for these factors in the analysis and public reporting of care experience survey results. We account for the impact of patients’ identity characteristics on their experience of care by using self-reported survey responses to “About You” questions to develop risk adjustments. Further, the way these questions are asked, and the response options provided must be carefully constructed to be inclusive and not discourage a respondent from sharing their care experience.

Sex and gender identity questions are one area undergoing rapid change. Within the health care industry, patient sex and gender identity data are undergoing transformation. As health care data managers and researchers work with patients to better capture gender identity data, data analysts have been confronted with a lack of standardization in language used to categorize patients. This issue arose because gender identity does not always correspond with the sex of the patient, as defined by standard medical records such as birth certificates. One option is to use the response categories and questions in the established in the United States Core Data for Interoperability (USDCI).

Data also need to be able to capture disability status more effectively. In 2016, an estimated 25 percent of Americans, or 61 million people, were living with a disability (Okoro et al., 2018). Even though disabled people face greater barriers in receiving health care, the CAHPS surveys “About You” sections currently do not ask about patients’ disability status. The most common type of disability is mobility disability (Iezzoni, Rao, Ressalam, Bolcic-Jankovic, Donelan, et al., 2021), yet research shows that most physicians do not use accessible medical equipment like scales and examination tables, despite those items being more comfortable and safer for patients. Further, Iezzoni and colleagues (2021) surveyed providers about their perceptions of disabled patients and their confidence in treating disabled people. They found large disparities in welcoming disabled people into their practices and problematic perceptions of quality of life for disabled people. The Iezzoni et al. research is conducted at the provider-level and indicates a potential for physician or provider bias, and therefore the possibility of actual differences in how disabled and abled people would be treated. However more research is needed to understand whether there are also care experience preferences or expectations that are specific to disabled patients. Gaining clarity on this issue may help inform future risk-adjustment.

Specific measures of disability equity should be explored and developed to understand the experience of care with the CAHPS-related providers. Disability equity has not been fully measured in CAHPS surveys, and in the existing research, there appears be room for improvement. This can be a distinguishing factor when the public reviews CAHPS scores to select a provider.

Studying equity will help develop meaningful questions for CAHPS. Recently, members of the Patient Experience Policy Forum Workgroup on Measurement and Reporting petitioned CMS to “consider adjusting for race and ethnicity while retaining transparency in disparities” (Fullam et al., 2021). They agree that appropriate patient-mix adjustments should remain in place for organization-level public reporting scores but advocate for the publication of additional resources on how these factors correlate to patient-level response data, which may be of significant interest to stakeholders and the public.

RTI researchers, among others, have analyzed CAHPS data to determine whether statistical evidence points to racial, ethnic, or other disparities in patient experience. This has been examined for the home health setting (Smith et al., 2015) and in primary care (Greene et al., 2022). This research found differences in reported experience, with Black, non-white, and Hispanic respondents reporting lower experience of care than their nonminority counterparts.

In our review of similar data collection efforts, we find few health care research entities responding to the call for more direct measures of health equity in a standardized way. RTI aims to be among those early leaders developing clear, meaningful questions addressing these issues in a way that is faithful to the CAHPS measurement science (i.e., does not measure patient personal feelings but rather provider behavior). The British Columbia 2018 Emergency Department Patient Survey (R. A. Malatest & Associates, 2019) provides questions regarding trust, confidence, spiritual needs, and other topics that may be relevant in the health care context, and some CMS formative research projects are crafting new questions about whether patients have experienced providers treating them differently due to their racial, ethnic, religious, or other identity (source: RTI project information).

Increase Stakeholder Feedback

Input received from the public, stakeholders, industry experts, and providers is an essential and integral part of AHRQ ensuring that the CAHPS Surveys continue to serve their original intended purpose: “To advance our scientific understanding of patient experience with healthcare” (Anhang Price et al., 2014). Public comment and feedback could help AHRQ leaders better understand the public’s view of health issues that affect them and their loved ones personally. This act of participation could also help citizens better understand the critical research policy issues that affect them. This is of special importance for vulnerable patient populations and those with serious health problems. Public comments can also help government agencies identify whether any existing regulations should be candidates for review.

At this time, anyone in the public can submit formal comments on documents posted to the Federal Register (final rules, proposed rules, and notices) via its integration with Regulations.gov; however, these documents can be difficult for the lay person to decipher, and the comment submission process is not always very intuitive. Perhaps it is time to think more broadly about how we can obtain public input regarding these important surveys, especially from the interested public and providers. To do this, we must consider not only our reasons for requesting this public feedback but also our reciprocal actions. What will be done with the input received? Can we provide a guarantee to the public that all input will be respected and reviewed, even if not implemented? There is much to be considered and discussed regarding this enterprise.

CMS or their contractors may consider various methods to obtain public feedback. Those could include:

• Existing AHRQ methods like the CAHPS Blog or webcasts

• CAHPS Public Meetings or Conferences, with a virtual option

• CAHPS Public Workshops or Panel Meetings, with a virtual option

• Stakeholder focus groups or advisory groups, with a virtual option

• Creating a virtual space for public expression using the Electronic Democracy model

The government currently has a blog devoted to all things CAHPS (https://www.ahrq.gov/cahps/news-and-events/events/index.html); however, it does not appear that users can comment on the blog posts. The posts that summarize the proposed and final rules may be especially helpful to patients and providers who have difficulty understanding the official documents found on the Federal Register. There may be a benefit in allowing users who do not feel comfortable commenting on the Federal Register to instead comment on these more easily digestible blog posts. CAHPS contractors could notify their registered users of these proposed and final rule blog posts on their websites in addition to notifying their users of the official documents posted on the Federal Register.

AHRQ also periodically sponsors webcasts and other events to provide information about CAHPS. AHRQ could consider presenting a yearly webcast on each of the CAHPS Surveys while also allowing for public comment during the webcast.

CMS could create a space for public expression using the Electronic Democracy model. This model aims to engage more members of the public in expressing their opinions on a website, via email, or through other electronic communications options to influence planning and decision-making; creates a virtual public space where people can interact, discuss issues, and share ideas; allows citizens to participate at their own convenience; can reach very large audiences with relative ease and little cost; facilitates interactive communication; and disseminates large amounts of information effectively and without distortion.

The success of the engagement depends on the awareness of methods like the CAHPS blogs, the webcasts, and Electronic Democracy. Therefore, we will need to use a variety of methods to notify our audience that we want to hear from them. These could include:

• Advertising our request via newspapers, magazines, social media websites, listservs, and other media outlets (such as those that often cover the announcement of a proposed or final rule and the public comment period)

• Using the approved CAHPS vendors by requesting that they notify their clients and all users of the request for public feedback, including adding an announcement or notice on their website

• Mailing or emailing a notice of the request for public feedback to patients who have been sampled for a CAHPS survey during a particular period

• Asking that public interest organizations inform citizens of the request for feedback by posting information on their websites

• Requesting that senior administration officials who are capable of generating significant interest publicize the request for input

To deliver the highest quality of health care to patients, we may need to reconsider whether the voices of the individuals directly affected by these services are currently being heard in a meaningful and impactful way.

Reduce Questionnaire Length

In the challenges section, we discussed how questionnaire length may dampen response rates. CAHPS stakeholders continue to express worry about the length of certain CAHPS surveys (Agency for Healthcare Research and Quality, 2019; Stucky et al., 2016). Additionally, research indicates that respondents may be more likely to participate in a survey if it is short (Olmstead et al., 2005). Reducing questionnaire length is one potential solution to the current response rate challenges the CAHPS surveys are experiencing, however, because CAHPS survey results are used by consumers to evaluate health care options, and by providers to evaluate the quality of care provided, it is important that the measures maintain standards of reliability. During the process of refining the HHCAHPS and ICH CAHPS Surveys, CMS focused on shortening the surveys while maintaining psychometric integrity of composites and considering survey items and topics that are important to people receiving care and the health care organizations providing this care.

There are several options for investigating the importance of survey content, including:

• Convening Technical Expert Panels (TEPs) during which subject matter experts are asked to provide input on which survey items should be kept or removed and preferred methods for survey item removal

• Holding focus groups with people receiving care to discuss which topics are most important to them in terms of the quality of care they receive

• Conducting cognitive interviews with people receiving care to access their opinions of survey questions and question wording

• Continually engaging with CMS, stakeholders, patient advocacy groups, and subject matter experts to discuss how to effectively shorten and refine CAHPS surveys

These types of feedback take advantage of individual and group settings, but also context of the person with the survey data. A TEP will typically have more researchers and data users, whereas focus groups of people who receive care are interested in the care for themselves and other personal aspects of the survey (e.g., whether it is easy to understand and feels worth their time). A mix of perspectives will be helpful in exploring reductions to survey instruments.

To ensure that potential revisions do not negatively impact the statistical properties of the domains and measures in CAHPS surveys, it is important to determine which items can be removed while still maintaining reliability. This can be done through various statistical and psychometric analyses. In one study investigating item removal in subsets of the CG CAHPS Survey, researchers concluded that the survey could be reduced from 34 to 23 items, which was estimated to reduce response burden by 25 percent (Stucky et al., 2016). Similar analyses on the HHCAHPS Survey identified six items that made minimal contributions to the survey composites (Bann et al., 2017). Psychometric analyses on the ICH CAHPS Survey identified 22 items that were not included in any composite or were unnecessary for the psychometric functioning of the composite (RTI International, 2021).

Another example of a successful effort to keep a survey short is the survey created by researchers working with the American Health Care Association. They developed a short, four-item survey measuring patient experience of care at skilled nursing facilities. The four-question survey produces valid and reliable data for distinguishing between facilities and is endorsed by the National Quality Forum (Castle et al., 2021).

Provide Incentives to Respondents

Currently, CMS does not allow survey vendors or health care providers to offer incentives of any kind for participation in CMS-sponsored CAHPS surveys (CMS, 2017). However, survey research literature shows a deep history of success stories where both monetary and nonmonetary token incentives (such as a bookmark) were used to increase response rates (Singer & Ye, 2013). Although monetary incentives tend to have more influence over response rate than other token incentives, the use of any incentive can increase response rates when compared with no incentive at all (Flanigan et al., 2019).

The timing of the incentive delivery matters, studies show. In the broad survey research literature, prepaid incentives are generally more effective than postpaid incentives. For example, one study found that a prepaid check improved response rates by 12 percentage points compared with the postpaid condition (Wiant et al., 2018). Increased response rates to prepaid incentives can be explained by concepts rooted in social exchange theory. The researcher has given something to the participant, and the participant, in turn, completes the questionnaire to return this favor, even though doing so is not required (Dillman et al., 2014).

There is no one-size-fits-all model for selecting and using incentives. Decisions about incentive value and type are often determined after considering factors related to the target population and practical limitations or constraints for the survey sponsor. Past research has shown that even small monetary incentives (less than $5) can increase participation and response rates (Halpern et al., 2002). For example, in one study the introduction of a $2 prepaid cash incentive increased response rates by 7 percentage points (Noel & Huang, 2019). Researchers have also found that offering good-quality pens led to increased response rates; in one study the response rate for sample members who received a pen was 4 percentage points higher compared with those who did not receive one (Beatty & Jamoom, 2013; Stanley et al., 2016). However, some researchers deemed pens too bulky and found they created practical problems during mailing, such as torn envelopes, crushed pens, or difficulty using some automated processes; they found that using a flatter, lighter incentive (sticky note pads) was just as effective in gaining participant cooperation (Torcasso et al., 2019).

The use of incentives can also reduce the need for follow-up contact, which can offset the increase in overall survey administration costs. In one study, 69 percent of respondents in the prepaid incentive groups responded before the second survey mailing, compared with 57 percent of respondents in the postpaid condition (Wiant et al., 2018). In another study, researchers found that 89 percent of all prepaid checks were cashed by individuals who completed the survey, while only 4 percent were cashed without a returned survey, indicating that there is a small financial risk associated to the use of a prepaid incentive (Hogan, 2007; Wiant et al., 2018).

Although providing even a small prepaid monetary or nonmonetary incentive has been shown to improve response rates, more research is needed to determine the effects of incentives on CAHPS surveys. While the CAHPS surveys are not exclusively given to an older population, there should be some attention on what types of incentives are well-received for a population that skews older and possibly less healthy than the standard US population. Investigations into the effects of different incentive types and incentive amounts and the potential for bias in survey results are needed to determine whether the use of incentives would be beneficial for CAHPS surveys.

Market CAHPS Results to the Public

We encourage CMS and other stakeholders to perform a literature review and conduct focus groups to understand consumer awareness of, attitudes about, and use of quality information displayed on Medicare’s Care Compare website. Care Compare is the consumer or public-facing website with tools, maps, and filters to help identify providers and share their rating information. Key research questions may include the following: What are the barriers to consumer awareness and use of this information? If possible, how does this vary by patient/family characteristics? In addition to patient focus groups, it may be beneficial to conduct focus groups with hospital discharge planners, as they are a primary conduit to people receiving home health care.

Haines et al. (2016), looked at similar questions for hospice quality data, publishing the results in a blog post. This post focused on awareness, knowledge, attitude, and behavior in using survey results for patient care decisions. RTI staff has also analyzed a similar topic related to nursing homes, which appeared in the journal Health Affairs (Lepore & Leland, 2015).

Conclusion

CAHPS surveys, measures, scores, and publicly reported information serve to improve the US health care system. Taken as a whole, this information drives better care experiences, encourages better care delivery, and allows consumers to make more informed choices about their health care. The current challenges facing CAHPS surveys include declining response rates, slow adoption of newer response modes which could yield cost savings, retention of “topped-out” measures, difficulty developing and adding new and relevant questions, and questionnaire length. Finding ways to overcome these challenges using some of the methods outlined here will help keep the surveys relevant so these benefits can continue. Consumers and health care providers benefit from the improvements in care catalyzed by the CAHPS surveys. A future of surveys where we use survey and statistical methods to address nonresponse, apply innovative sampling methods to improve cost and speed, implement newer survey modes or combinations of survey modes, develop new questions, increase stakeholder feedback, reduce questionnaire length, provide incentives to respondents, and improve marketing of CAHPS results to the public will enable us to better measure and drive progress in care experiences and health care quality.

Acknowledgments

The authors thank Sarah With and Sara Yarnell for their contributions to this paper.

RTI Associate Editor: Anne Deutsch