RELEASE OF HEDIS® 3.0 DATA FOR REPORTING YEAR 1996

I. INTRODUCTION

In 1997, the Health Care Financing Administration (HCFA) began collecting information on the quality of health care provided in Medicare managed care plans. Plans were required to report a subset of performance measures from the Health Plan Employer Data and Information Set (HEDIS®)[1] for health care provided in calendar year 1996. Therefore the “reporting year” for this dataset is 1996, and this report refers to this first collection of HEDIS data as “HEDIS data for reporting year 1996.”

Initiating the collection of HEDIS was an unprecedented collaborative effort among HCFA, the nation’s largest purchaser of care, managed care plans, and the National Committee for Quality Assurance (NCQA), which sponsors and maintains the HEDIS performance measure set. Performance measures, such as the HEDIS measures, provide feedback about care provided by managed care plans to government entities, corporations, other organizations, and individuals that purchase the care. The HEDIS measures cannot detail all of the health care provided by a plan, but they can broadly describe important aspects of care. HCFA envisions three uses for HEDIS data: 1) provide consumers with information for comparison of plans; 2) inform internal quality improvement; and 3) facilitate HCFA oversight of Medicare managed care plans. HCFA also calculates performance measures for its fee-for-service (FFS) population and envisions similar uses: quality improvement and comparison of quality between FFS and managed care.

HCFA has consistently maintained that Medicare HEDIS data, like commercial HEDIS data, will be made available to the public, and this release of the HEDIS data for reporting year 1996 fulfills that commitment. The data included in this release is unaudited, although HCFA did obtain some idea of the data’s reliability through a post-submission audit.[2] Because HCFA’s audit occurred after plans had submitted their HEDIS data and only assessed a sample of contract markets in detail for five clinical measures, HCFA did not allow plans to correct any errors uncovered during the audit.

Although unaudited, public reporting is appropriate because public dollars purchased the health care described by the HEDIS measures and because public reporting begins an important process of both rewarding health plans for providing quality care and motivating them to achieve even better outcomes in the future. This report documents HCFA’s first experience collecting HEDIS data. It consists of two sections: (1) a description of HCFA’s collection and post-submission audit process and (2) a discussion of limitations in the data and HCFA’s approach to analyzing and using the data given these limitations. Ultimately, HCFA has concluded that the HEDIS data for reporting year 1996 can be useful for some purposes, such as quality improvement, but always should be recognized as the result of a learning experience for plans, NCQA, HCFA, and auditors.

II. COLLECTION AND AUDIT OF THE HEDIS DATA FOR REPORTING YEAR 1996

A. What are the HEDIS measures and which measures did HCFA collect?

Health Plan Employer Data and Information Set (HEDIS) is a set of standardized performance measures designed to collect data needed for purchasers and consumers to be able to reliably compare the performance of managed care plans. In conjunction with public and private purchasers, health plans, researchers, and consumer advocates, NCQA sponsors and maintains HEDIS, updating HEDIS by identifying and testing new measures and refining older measures and reporting guidelines. Development of the HEDIS measures represents major progress in performance measurement and in the effort to compare the performance of managed care plans.

The delivery of health care is multifaceted, integrating many different processes to produce the service of “health care.” For this reason, HEDIS provides information about eight general areas (or domains): Effectiveness of Care (clinical outcomes); Use of Services (utilization); Access/Availability of Care; Satisfaction with Experience of Care; Health Plan Stability; Cost of Care; Informed Health Care Choices; and Health Plan Descriptive Information. Considering all of these measures on different aspects of care combine to create a more complete picture of health care and services provided by a particular managed care plan. The usefulness of any single measure for evaluation is likely to be less than the usefulness of the combined set.

Table 1. Required HEDIS Measures All plans with members in 1996 reported the following measures by health plan (legal entity): Domain: Health Plan Stability Indicators of Financial Stability Plans reported the following measure by contract. Domain: Cost of Care High-Occurrence/High-Cost DRGs Rate Trends Domain: Health Plan Descriptive Information Provider Compensation Total Enrollment Enrollment by Payer Plans reported the following measure by contract market. Domain: Effectiveness of Care Breast Cancer Screening Beta Blocker Treatment After Heart Attack Eye Exams for People with Diabetes Follow-up After Hospitalization for Mental Illness The Health of Seniors Domain: Access to/Availability of Care Adults’ Access to Prevention/Ambulatory Health Services Availability of Primary Care Providers Availability of Mental Health/Chemical Dependency Providers Availability of Language Interpretation Services, Part II Domain: Health Plan Stability Years in Business/Total Membership Disenrollment Provider Turnover Domain: Use of Services Frequency of Selected Procedures Inpatient Utilization - General Hospital/Acute Care Ambulatory Care Inpatient Utilization - Non-Acute Care Mental Health Utilization - Inpatient Discharges and Average Length of Stay Mental Health Utilization - Percentage of Members Receiving Inpatient, Day/Night and Ambulatory Services Readmission for Specified Mental Health Disorders Chemical Dependency Utilization - Inpatient Discharges and Average Length of Stay Chemical Dependency Utilization - Percentage of Members Receiving Inpatient, Day/Night and Ambulatory Services Readmission for Specified Chemical Dependency Disorders Outpatient Drug Utilizations (for those with a Drug Benefit) Domain: Informed Health Care Choices Language Translation Services Domain: Health Plan Descriptive Information Board Certification/Residency Completion Preventive Care and Health Promotion

HCFA required reporting of most, but not all, of the HEDIS measures pertinent to the Medicare population for health care provided in reporting year (calendar year) 1996 (Table 1). A description of each measure is available in the documentation accompanying the HEDIS data for reporting year 1996. One HEDIS measure specific just to the Medicare population, the Health of Seniors measure, is a survey of the functional status of health plan members. HCFA first administered the Health of Seniors survey in 1998, and these data are not included in the HEDIS release. HCFA also collected additional information on enrollee satisfaction through the Consumer Assessment of Health Plan Study (CAHPS) survey beginning in 1998. HCFA will report on the results of these surveys as they become available.

B. What is the HEDIS reporting unit for Medicare?

HCFA collected 288 HEDIS submissions from Medicare managed care plans for care provided in 1996. The reporting unit for HCFA is a “contract market.” HCFA signs contracts with managed care plans to provide services for an area of contiguous counties that the health plan can support with its provider network. For this reason, a health plan may sign several contracts with HCFA to provide service in different areas. Usually, the geographic service area for a contract is of moderate size and enrollment. However, some contract service areas are large, encompassing more than one distinct city. For example, a health plan could have a single contract that covers most of Southern California, which includes many counties and several large cities. In such instances, HCFA believes that one HEDIS submission for the entire service area would not be useful to beneficiaries making choices among health plans because one large HEDIS submission might not accurately represent the quality of care being provided in their local area. To solve this problem, HCFA divided contracts covering large distinct geographic areas and high enrollment into smaller “market areas” containing at least 5,000 beneficiaries. Therefore, the 288 submissions by contract market represent only 266 contracts.

C. Why did HCFA decide to audit the HEDIS data, and what was the scope of the audit process?

The accuracy of collected data determines whether HCFA can use the results of HEDIS measures to support quality improvement, contract monitoring, and consumer choice. Each goal requires different levels of accuracy. If reported data for a given measure has a high error rate, that measure may still be useful for internal improvement by broadly indicating areas for attention. However, unless the exact error rates are known and can be corrected for all plans, use of the data to compare plans could very easily lead to incorrect conclusions that care is better or worse depending on the direction and extent of the errors.

HCFA suspected that the rates might contain some error because the state of the art of performance measurement is still in its infancy and because many managed care plans did not have a lot of experience collecting and reporting performance measures. For this reason, HCFA wished to obtain a rough estimate of the reliability (accuracy and completeness) of the rates reported by plans to (1) determine the usefulness of submitted data, (2) identify areas for quality improvement, and (3) inform the 1997 data collection and audit processes. In order to obtain such an estimate, HCFA decided to conduct a post-submission audit of health plans. In general, a HEDIS audit would occur prior to data submission. However, HCFA’s audit occurred after plans had submitted their HEDIS data and only assessed a sample of contract markets in detail. For this reason, HCFA did not allow plans to correct any errors uncovered during the audit process. Therefore, the HEDIS data for reporting year 1996 were not audited prior to submission and were not corrected based on audit findings after submission.

HCFA chose the Island Peer Review Organization (IPRO), a contractor with significant audit experience and with involvement in the development of NCQA audit standards, to lead development and implementation of the audit. HCFA and IPRO developed a detailed audit based largely on NCQA audit standards with some important additional expansions. The post-submission audit included a preliminary assessment of data systems processes and plan experience with HEDIS calculation and auditing for almost all (284) contract markets and an on-site audit of 79 contract markets.[3] HCFA chose the 79 contract markets for review through weighted random sampling based on enrollment, which gave plans with higher enrollment a greater probability of inclusion in the audit. The 79 contract markets audited served about 65% of all beneficiaries enrolled in managed care. During an on-site visit, auditors spent several days reviewing the health plan’s data collection and reporting systems used for HEDIS and verifying, in great detail, the preparation of five clinical HEDIS measures:

C breast cancer screening,

C beta blocker treatment after a heart attack,

C eye exams for people with diabetes,

C follow-up after hospitalization for mental illness, and

C the frequency of eleven selected procedures.

The Medicare post-submission audit resulted in many valuable conclusions about auditing HEDIS data, the process of HEDIS data collection, and the HEDIS measures themselves. IPRO found problems with inadequate information systems and processes and in the interpretation and application of HEDIS specifications that undermine the reliability and accuracy of reported measures. The findings of the audit are discussed briefly in the section on data limitations below and summarized in the executive summary of IPRO’s report to HCFA, Audit of 1996 Medicare HEDIS Data: Findings, which has been included with this release.[4]

Both the content of the post-submission audit and its narrow scope limited the ability of the audit to fully verify the accuracy of reported rates. With regard to content, the audit did not examine 1) plan review of medical records; 2) completeness of administrative (electronically stored) databases; or 3) the accuracy of physician coding at the point of service. In over-sampling larger plans, the audit may not have caught the data problems specific to some smaller plans. Larger, more sophisticated plans presumably have better data collection systems and processes in place. Further, the audit provided little insight into possible error in the interpretation and calculation of non-clinical measures.

In summary, this post-submission audit of the Medicare HEDIS data identified a significant number of problems in the reported HEDIS measures that limit the usefulness of this first HEDIS collection. However, the audit also identified the most significant problem areas in data collection and reporting processes that will inform HCFA’s 1998 audit of HEDIS data for reporting year 1997 and that HCFA will consider for quality improvement activities.

D. How does HCFA interpret its experience collecting and auditing the HEDIS measures for Reporting Year 1996?

With this first mandatory reporting of HEDIS measures, HCFA and the managed care plans have made major strides in the right direction. HCFA is impressed with the vigor and rigor with which most plans have approached this very complex task and has noted the difficulties plans have encountered. Lessons learned for all participants: plans, HCFA, NCQA, and IPRO, have been invaluable, and HCFA will use information gathered from all facets of HEDIS collection for reporting year 1996 and audit process to guide future HEDIS collection efforts.

III. DATA LIMITATIONS OF THE HEDIS DATA FOR REPORTING YEAR 1996

HCFA’s post-submission audit combined with knowledge of measure design and internal analysis of the data suggest that the HEDIS data for reporting year 1996 contain more than minor error. Performance data cannot support plan-to-plan comparison if error in the collected data obscures the actual rate for some plans. The presence of error placed HCFA in the uncomfortable position of releasing the HEDIS data to meet public obligations, knowing that the data contained error and that individuals and organizations might use the data for plan-to-plan comparison. To address these concerns, HCFA chose to release the data with the following detailed discussion of the data’s limitations and of HCFA’s attempt to identify the most useful and highest quality data that lends some insight into comparative plan quality. HCFA hopes that researchers and analysts using this data set will strongly consider the following discussion when conducting their own analysis.

In using this early HEDIS data, HCFA will be careful for the following reasons: 1) the data contains error as noted above, and 2) the HEDIS measures will be an important subset of HCFA’s eventual performance measure system. In this last regard, HCFA cautions against relying completely on Medicare HEDIS measures to portray fully plan quality. HEDIS is only one set of indicators about health care quality and services, and individuals should consider many aspects of a health plan when comparing plans. To this end, HCFA collects additional information on Medicare managed care plans, such as beneficiary satisfaction, health outcomes, and disenrollment measures in addition to the one collected through HEDIS. Together these measures provide a more robust picture of health care quality and services and invite more informed explanations of health outcomes. HCFA hopes to make more information available for consideration with reported HEDIS measures in the future.

A. What are the limitations in the HEDIS data for reporting year 1996?

Anyone using the HEDIS data should be aware of possible error in the reported HEDIS data. This is especially true for comparison among plans, as error in reported rates will misrepresent plan quality and bias any conclusions drawn from plan-to-plan comparisons. As noted above, the database of HEDIS data for reporting year 1996 contains unaudited HEDIS submissions. Error arises from two sources, those internal to the measure collection process, including design and reporting of measures, and those external to the measure collection process. Internal sources include measurement reliability and validity, as well as the use of sampling, possible submission error, and the presence of missing values. External sources of error are influences outside the HEDIS design and collection process that may result in observed differences among plans that do not reflect actual differences in plan quality. Variation in the composition of health plan demographics is one example. Differences in the extent of risk assumed by a plan also can introduce error, i.e. cost vs.risk plans.

B. What are possible internal sources of error?

1. Measurement Validity and Reliability

A measure is valid to the extent that it measures what it is intended to measure, and a measure is reliable to the extent that, in a given situation, it produces the same results repeatedly. Measurement validity and reliability are essential for comparison of plan quality, as plans should be compared only on valid measures of health care quality that are reliable across health plans. HCFA’s audit did not attempt to address measure validity directly because the HEDIS Committee on Performance Measurement (CPM) considers validity issues when it specifies the measures for the HEDIS performance measure set. However, a measure cannot be valid if it is not reliable. Measures that suffer from measurement error are not reliable. Measurement error is the variation in reported results that occur from calculation to calculation due to differences in the measurement situation.

In HCFA’s post-submission audit, IPRO identified more than minor measurement error in three of the five clinical HEDIS measures they reviewed in detail. Specifically, HCFA’s post-submission audit expressly considered problems with reliability by examining (1) the ability of health plan information systems and processes to accurately capture and supply data for HEDIS reporting and (2) plan application of the HEDIS technical specifications to calculate individual measures. Audit findings suggest that poor information systems and processes and improper interpretation and application of HEDIS specifications introduced measurement error and reduced the reliability of reported HEDIS rates. In general, poor reliability obscures true differences in health care quality and services among plans, unless there is systematic bias. HCFA’s audit did uncover a tendency for measurement error to favor the plan, however, IPRO did not identify any intentionally biased reporting activity. Although, the results of the audit do not represent completely the entire population of HCFA plans because IPRO employed a sampling methodology that over-sampled larger plans, HCFA has decided to broadly employ audit findings to guide analysis of error in the data.

(a) Information Systems and Processes

Information systems and processes directly impact the extent of measurement error in the data used to calculate HEDIS measures. The diverse and changing character of the managed care industry leads to reliance on multiple and varied information systems to maintain medical, provider, membership, and pharmacy data. Most information systems have been developed primarily for billing and claims payment, rather than reporting performance measurement. HCFA’s audit found limitations in the ability of health plan information systems and processes to accurately capture and supply data for HEDIS reporting. Broadly, problems included inaccurate and poor capture of data; inadequate or incompatible information systems; numerous information systems; poorly monitored processing procedures; and poor documentation of system integration, staffing, and system protocols. HCFA’s audit also found that encounter data were not complete. Incomplete data, i.e. not obtaining data from all sites of service in a plan, could greatly influence reported rates. If sites with significantly better or worse rates did not send encounter data to the health plan, the reported rate would not reflect care provided in the contract market. Further, some sites may not report complete encounter data. During their post-submission audit, IPRO often observed data missing from 20% of sites. This last finding questions the reliability of measures based solely on administrative encounter data. In their review of clinical measures, IPRO determined that 25% to 28% of the procedures in the frequency of selected procedures measure were in significant non-compliance with reporting requirements, primarily due to incomplete data. IPRO has recommended that HCFA not use this measure for plan-to-plan comparison. By inference, most, if not all, of the utilization measures also suffer from incomplete data problems.

To address some of the known deficiencies in health plan information systems, HEDIS was designed to draw from a variety of sources in producing measures. Multiple sources can minimize the impact of poor information systems. Specifically, plans that believe their electronic medical encounter data are not complete can supplement this administrative data with a sample of medical records for the four clinical effectiveness measures (breast cancer screening, beta blocker treatment after a heart attack, eye exams for diabetics, and follow-up after hospitalization for mental illness). For these measures, about 65% of contract markets used medical records. HCFA does not have an assessment of possible error introduced by using medical record information on reported rates because the audit did not address the accuracy of medical record abstraction and interpretation.

(b) Implementing HEDIS Specifications

Often, reported rates can contain measurement error because plans fail to implement the HEDIS specifications exactly. Much of a plan’s ability to implement HEDIS specifications is associated with the capabilities of their information systems and processes to provide appropriate data. However, measurement error also occurs because plans misinterpret calculation specifications. HCFA’s post-submission audit found such discrepancies for the clinical measures reviewed in detail. For example, the eye exams for diabetics measure specifies that an ophthalmologist or optometrist provide the eye exam. If a plan counted eye exams provided by any provider, the plan was not compliant with HEDIS specifications and the rate would contain error.

In the post-submission audit, IPRO identified significant deviations from specifications in 6% of plans for breast cancer screening, in 5% of plans for beta blocker use, in 22% of plans for eye exams for people with diabetes, and in 22% of plans for follow-up after hospitalization for mental illness. Exact application of measure specifications are necessary for meaningful plan-to-plan comparison. For this reason, IPRO has recommended that HCFA not use the eye exams for diabetics and follow-up after hospitalization for mental illness measures for plan-to-plan comparison. The extent of deviation from specifications in other measures is unknown, although the clinical measures IPRO reviewed are some of the more difficult measures to calculate. For the sample of plans for which IPRO conducted a site visit during the post-submission audit, those with experience reporting HEDIS data deviated less frequently from the HEDIS specifications than plans with less experience. NCQA also believes that the quality of reported data improves with experience reporting HEDIS.

2. Other Sources of Error

In addition to measurement validity and reliability, other sources of error remain in the data, including sampling error, submission error, differences in coverage between cost and risk plans, and missing information.

(a) Sampling Error

Most HEDIS measures are calculated using electronic administrative data for the entire population of a contract market’s members or providers. However, health plans have the option of calculating the four effectiveness of care measures with a hybrid of data from medical records and administrative data for a sample of members rather than relying completely on administrative information systems. This approach allows plans to compensate for incomplete electronic encounter data. However, the reported sample rate will represent, but probably will not match exactly, the plan’s actual rate. These differences are due to chance and do not introduce bias into the rate.

Table 2. Comparing Plans on Sample Measures
Plan | Lower 95% Confidence Interval | Breast Cancer Screening | Upper 95% Confidence Interval
Plan A | 55.9 | 61.1 | 65.8
Plan B | 69.6 | 72.9 | 76.2
Plan C | 68.1 | 76 | 83.9
In the example above, it appears that Plan A is different from Plan B and Plan C because the lower 95% confidence interval value for both Plan B and Plan C, 69.6 and 68.1% respectively, are higher than the upper 95% confidence interval value for plan A, 65.8%. In addition, the reported rate for plan A, 61.1%, is at least 10 percentage points lower than Plan B, 72.9%, and Plan C, 76%. Plan B and Plan C are not different because their confidence intervals overlap. Making multiple comparisons ( A to B, B to C, C to A) in this manner does create a higher probability of detecting a difference between plans when one does not truly exist. This probability increases with the number of comparisons.

A rate based on a sample will be within a certain margin of error of the actual rate. For HEDIS measures based on a sample, plans with denominators smaller than 100 calculate a confidence interval, or range of rates with a 95% probability of capturing the actual plan rate. When comparing two plans, it is important to identify differences between plans larger than the difference indicated by the two confidence intervals. That is, the confidence intervals should not overlap (Table 2). The larger the sample size and the further expected reported rates move from 50%, the smaller the margin of error becomes. If a plan finds fewer eligible members than required minimum sample size, the plan must calculate the rate for all members.[5]

In Appendix 1, HCFA has calculated confidence intervals for all plans for the two clinical effectiveness measures that can rely on a sample: breast cancer screening and beta blocker treatment after a heart attack. To prevent comparisons among plans that have large margins of error around their sample rate, NCQA advocates that sample measures based on fewer than 100 members not be used for comparisons among health plans. Appendix 1 follows this guideline. For plans reporting measures based on the minimum required sample size, observed differences of 10 percent represent true difference in plan rates regardless of sampling error.

(b) Submission Errors

A review of the reported data reveals that a small percentage of the HEDIS rates have been reported incorrectly. A reported rate falling outside possible values is an obvious submission error. For example, for percentage measures, such as provider turnover, the reported rates should not exceed 100% or drop below 0%. However, some plans have reported rates outside acceptable ranges. HCFA views these rates as data entry errors, and such rates should not be considered. It also is possible that some reported rates are submission errors, even when they do not fall outside of acceptable ranges. For example, a 0% residency completion for specialists is most likely a data entry error. The prevalence of these two types of errors is small because HCFA’s data collection process afforded plans the opportunity to review their initial HEDIS submission and correct any data entry errors.

(c) Missing Information

The HEDIS guidelines distinguish between two different types of missing values: Not Applicable (NA) and Not Reported (NR). Plans report NA rather than a rate when they do not have enough members to calculate a representative rate (n<30) or when they are not eligible for a measure, e.g. a plan cannot calculate outpatient drug utilization if it does not offer an outpatient drug benefit. Plans report NR (or leave a measure blank) when they choose not to calculate and report a rate. In theory, the two missing value designations convey different information about a plan’s compliance with HEDIS reporting requirements. A plan should not be held responsible for reporting NA instead of a rate, because the plan has no control over the number of members eligible for any given measure. NR, on the other hand, represents the failure of a plan to report a Medicare required measure. However, the criteria for NA vs. NR reporting were not consistently applied in reporting year 1996. This undermines the ability of missing value designations to accurately represent the appropriate reason for failing to report. Although HCFA did attempt to “clean” the report designations in the measures it considered (see Appendix 1), no missing value report designations should be considered definitive.

C. What are possible external sources of error?

(a) Error Introduced by Differences in Health Plan Composition

Often, processes external to a health plan can confound reported rates, making comparisons among plans invalid on some measures, especially utilization (use of services) measures. For example, a health plan cannot control the various demographic factors, e.g. race, gender, age, and socioeconomic status, of its beneficiaries. Differences in the composition of health plans’ members may lead to reported differences among plans that are not related to the quality of care administered by the plan. For example, a plan with older members may report higher inpatient utilization than plans with younger members.

(b) Error in Comparing Cost Plans with Risk Plans

Managed care plans can serve Medicare beneficiaries through risk contracts and cost contracts. HEDIS reporting is a HCFA contract requirement for all 1876 contracts, risk or cost. Unlike risk plans, cost plans’ members may seek care outside of the plan’s affiliated delivery system and, therefore, reported HEDIS rates may not reflect all the health care a member received during the reporting year. Cost plans are identified in the HEDIS data with an indicator variable.

Risk plans are paid a per capita premium set at 95% of the projected average expenses for fee‑for‑service beneficiaries in a plan’s service area. Risk plans assume full financial risk for all care provided to Medicare beneficiaries. Risk plans must provide all Medicare‑covered services, and most plans offer additional services, such as prescription drugs and eyeglasses. With the exception of emergency and out‑of‑area urgent care, members of risk plans must receive all of their care through the plan. However, as of January 1, 1996, risk plans can provide an out‑of‑network option that, subject to certain conditions, allows beneficiaries to go to providers who are not part of the plan.

Cost plans also are paid a pre‑determined monthly amount per beneficiary based on a total estimated budget, however, adjustments to that payment are made at the end of the year for any variations from the budget up to a maximum amount. Cost plans must provide all Medicare‑covered services but may not provide the additional services that some risk plans offer. Further, beneficiaries also can obtain Medicare‑covered services outside the plan without limitation. When a beneficiary goes outside the plan, Medicare pays its traditional share of those costs and the beneficiary pays the coinsurance and deductibles stipulated by Medicare.

IV. HCFA USE OF HEDIS DATA FOR REPORTING YEAR 1996 GIVEN LIMITATIONS

Faced with all of the possible sources of error in plan rates, HCFA wished to narrow the scope of its review to HEDIS data that was most likely to be credible and useful. To achieve this goal, HCFA decided to consider a limited number of contract markets and measures. HCFA explored several methods of narrowing the number of contract markets reporting data and ultimately limited its review to those contracts held by plans with experience reporting HEDIS data. HCFA also reviewed the range of HEDIS measures required for Medicare reporting and identified a subset of measures that are both useful for contract monitoring and quality improvement and that HCFA believes deviate the least from the plan’s actual rate.

A. Choice of Plans

To identify those contract markets for plans generally reporting the most credible data, HCFA carefully considered several plan characteristics that would indicate familiarity with managed care and with HEDIS reporting. Both NCQA and IPRO have indicated that plan experience reporting HEDIS improves the quality of HEDIS data. Greater managed care and HEDIS experience should correlate with improved information systems, application of HEDIS specifications and submission processes, reducing the number of missing values and submission errors in measure reporting. These plans probably are more established and are located in an area with high managed care penetration. These plans also should hold contracts with higher enrollment. HCFA specifically considered the following plan characteristics: the number of Medicare managed care plans reporting in a county, commercial managed care experience, enrollment, and years of experience reporting HEDIS data.

In the end, HCFA decided only to consider data from the 152 contract markets administered by plans that had been reporting HEDIS data for four or more years, where four is both the mean and the median of plan experience reporting HEDIS data based on IPRO’s preliminary assessment of all plans. A frequency distribution for this variable appears below. HCFA chose this criterion because it reflects NCQA’s and IPRO’s experience that greater experience reporting HEDIS results in better quality data and because HCFA felt that the use of a single criterion would be easy to apply and communicate. The proportion of plans not reporting a rate was significantly lower for plans with four or more years experience reporting HEDIS data than those with less experience reporting HEDIS data for most measures. Plans with four or more years reporting HEDIS data are identified in the database of HEDIS data for reporting year 1996 with an indicator variable.

Table 3. Years Experience Reporting HEDIS YEARS REPORTING CUMULATIVE CUMULATIVE HEDIS FREQUENCY PERCENT FREQUENCY PERCENT ------------------------------------------------------ 0 4 1.4 4 1.4 1 38 13.3 42 14.7 2 24 8.4 66 23.2 3 67 23.5 133 46.7 4 132 46.3 265 93.0 5 20 7.0 285 100.0 Totals do not add to 288 because three plans did not complete a baseline assessment for HCFA's audit, and one baseline assessment collected by IPRO covered two contract markets.

Profiles of the 152 contract markets confirmed previous assumptions. These contract markets are located in areas with a large number of managed care plans and have greater commercial experience and higher enrollment than the remaining plans. Not unexpectedly, most of the contract markets administered by plans with four or more years experience reporting HEDIS data are located in areas with high managed care penetration. California, Florida, New York, Texas, and Oregon contain 58% of the contract markets. In addition, these contract markets demonstrated a significantly higher average number of commercial years in business for HMO and POS products than plans with less than four years experience reporting HEDIS data. These plans also reported significantly higher total enrollment for their Medicare contracts at the end of 1996 than plans with less than four years reporting HEDIS data. In theory, this last observation makes HCFA’s audit findings more generalizable to these 152 contract markets, as contract markets with higher enrollment had a greater probability of being selected to participate in the post-submission audit site-visit.

Some users of the HEDIS data for reporting year 1996 will notice that contract markets of plans with four or more years experience reporting HEDIS data demonstrate significantly higher reported rates for the sample measures: breast cancer screening and beta blocker after heart attack, but not for the non-sample measures. However, such differences in sample measures should be considered as suggestive and not definitive because the use of a sample by some plans implies that some of the reported rates could be attributed to chance and not to actual plan differences.

B. Choice of Measures

HCFA also identified a subset of measures that may provide, for reporting year 1996, the best insight into comparative plan quality in light of known data limitations. The chosen measures strike a balance between measures that do not contain much error and measures that HCFA estimates contain some error but complement more reliable measures by providing information from different areas of health care. These measures include:

C breast cancer screening,

C beta blocker treatment after a heart attack,

C adults’ access to preventive/ambulatory health services,

C provider turnover,

C indicators of financial stability (operating profit margin and ratio of cash to claims payable), and

C board certification/residency completion.

Through HCFA’s post-submission audit, IPRO determined that HCFA could report breast cancer screening and beta blocker treatment after a heart attack if HCFA chose to release the HEDIS data for reporting year 1996. IPRO also determined that eye exams for diabetics, follow-up after hospitalization for mental illness, and frequency of selected procedures should not be reported. HCFA did not consider any utilization measures both because these measures are difficult to interpret by themselves and because the audit identified completeness problems in the encounter data supporting these measures. Although the audit is not generalizable to the population of contract markets due to sampling methodology, it is the best information about reliability available for the clinical measures examined. For this reason, HCFA has decided to include only breast cancer screening and beta blocker treatment after a heart attack in its list of useful and relatively reliable measures.

Some of the measures should not contain sizeable error because their calculation is simple and because they draw on fairly reliable sources of information. For the indicators of financial stability, plans submit information from their independently audited financial statements. While all financial indicators provide some insight into plan stability, HCFA believes that operating profit margin and ratio of cash to claims payable are two of the more important reported rates.[6]

Looking at reported results for many HEDIS measures depicts a more robust picture of plan quality than one or two measures. For this reason, HCFA identified the remaining measures (provider turnover, adults’ access, and board certification) because they complement the fairly reliable clinical and stability measures identified above. HCFA does not know the extent of error in these measures, and IPRO advised HCFA to use unaudited measures cautiously. Adults’ access poses the greatest risk for error, even among plans that have had experience reporting HEDIS data, because it relies on claims/encounter data that HCFA’s post-submission sample audit determined to be incomplete.

V. CONCLUSIONS FROM ANALYSIS OF SUBSET OF PLANS AND MEASURES

These data roughly depict the performance of Medicare managed care plans in 1996 and establish some aggregate baselines against which to consider future HEDIS reporting. On the whole, this first HEDIS report demonstrated the delivery of effective care, but also highlighted areas for concern and improvement. In the 152 health plans reviewed, 73% of eligible women in each plan received a mammogram and 91% of beneficiaries over 65 had at least one preventive care visit. On average, physicians prescribed a beta blocker following heart attack for 63% of eligible beneficiaries. These numbers suggest that plans have done fairly well in implementing preventive care and treating acute illness, but that room for improvement remains. Health plans demonstrated acceptable board certification rates, reporting a median 78% for primary care physicians and 81% for specialists. Although health plans appeared to have stable provider networks, reporting a median turnover of 5%, they demonstrated steadily declining operating profit margins over the 1994 to 1996 period. Median operating profit margin dropped from 3.7% in 1994 to 0.3% in 1996. The ratio of cash to claims payable held steady at just under 2.5% for the 1994 to 1996 period. The large number of NRs appearing in the data, may reflect problems reporting NA vs. NR in the first collection of HEDIS data, but, in part, they also identify areas needing greater attention to data collection and storage, such as tracking residency completion, geriatricians, provider turnover for non-physician primary care providers.

Please see Appendix 1 for a detailed discussion of each measure identified above, a national distribution, and the individual rate for the 152 plans with four or more years experience reporting HEDIS data.

VI. COMPARISON OF MANAGED CARE AND FEE-FOR-SERVICE

In addition to collecting performance measures of health care quality and services on its managed care population, HCFA also measures performance for its FFS population. Two of these performance measures are mammography and beta blocker prescription after AMI. While the intent of these clinical measures are the same as the breast cancer screening and beta blocker after a heart attack HEDIS measures: preventive care and treatment of acute illness, the calculations do differ. Further, HCFA cannot control for differences in demographics or geographic influences, which may influence access and utilization, between managed care and FFS. As a result, reported rates for FFS will vary from those reported in managed care due to the confounding influences just discussed, as well as true differences in care. On the other hand, managed care numbers presumably should appear higher because managed care plans coordinate the care of their enrollees.

HCFA calculates biennial mammography rates for Medicare beneficiaries from 100% Medicare claims and enrollment data. For 1995-1996 HCFA reports a biennial mammography rate of 46.4% for women 50 to 64 years and 54.6% for women 65 to 69 years. The biennial rate covers services provided between January 1, 1995 through December 31, 1996. Individuals younger than 65 and covered by Medicare are disabled. The denominator for these rates are all female Medicare beneficiaries who had part A and B coverage, including the institutionalized; were not enrolled in an HMO; and did not die during the time period. Calculation of this measure is similar to the HEDIS measure, but some differences exist. The FFS rate may be lower than HEDIS rates because HCFA does not attempt to identify and exclude women who have had a bilateral mastectomy, an option offered in the HEDIS specifications. In FFS, a mammogram is subject to deductibles and co-insurance that may inhibit some beneficiaries from seeking a mammogram. In addition, the age range for inclusion and specifications for identifying a mammogram varies slightly. HCFA also speculates that FFS rates are low because they do not reflect free mammography services or mammograms paid by insurance other than Medicare and because they include the poorest and sickest Medicare beneficiaries. Reported rates for FFS (46.4% and 54.6%) are well below the HEDIS average rate reported by managed care of 73%.

As part of the Cooperative Cardiovascular Project, a HCFA quality improvement project, Krumholz et al. report that during 1994-1995, 50% of patients 65 and older discharged alive after admission with diagnosis for acute myocardial infarction (AMI) received a prescription for beta blocker treatment.[7] This rate is based on 45,308 Medicare beneficiaries considered ideal for beta blocker treatment under study protocol and identified from 100% claims during a four to eight month period that varied by state. Patients transferred to another facility or contraindicated for beta blocker therapy were not included.

Calculation of the CCP beta blocker measure differs from the HEDIS beta blocker measure. Contraindications for exclusion used in the CCP measure vary somewhat from the those strongly recommended, but not required, for exclusion in the HEDIS specifications. The specifications to identify AMI vary. The HEDIS measure includes patients older than 35 whereas the FFS measure includes patients older than 65, although the disabled younger than 65 comprise only a small percentage of Medicare beneficiaries. Medicare managed care plans can use pharmacy data in addition to medical record notations to identify a beta blocker prescription. Further, the HEDIS measure counts all beta blocker prescriptions between 30 days before and 7 days following AMI, whereas the CCP measure only used a beta blocker discharge prescription noted in the medical record. The HEDIS measure specifically covers reporting year 1996, while FFS numbers reflect discharges during an eight month period between 1994 and 1995 that varies from state to state. The frequency of beta blocker prescription may have improved by 1996. The FFS rate for CCP beta blocker prescription at discharge for AMI is much lower than the 63% average beta blocker treatment reported rate for managed care.

VII. CURRENT AND FUTURE PERFORMANCE MEASURE EFFORTS

HCFA is conducting and/or planning a number of activities which should have a positive impact on the ability to use performance measurement effectively for quality improvement, consumer information, and purchaser oversight. HCFA requires plans to report HEDIS measures in 1998, reflecting services provided in 1997. HCFA expects to present audited 1997 data on its Medicare.gov web site.

HCFA’s 1998 audit included major changes: 1) on-site review has been expanded to include all plans; 2) audited measures, both clinical and non-clinical, cover more HEDIS areas; 3) re-abstraction of medical record reviews will occur; 4) the impact of incomplete data is being assessed; and 5) the audit is taking place prior to final submission of the data so that plans may correct errors identified in the audit prior to submission of their data to HCFA. This has increased our certainty about reported HEDIS rates and improves their usefulness.

As additional performance measures become available, HCFA plans to combine HEDIS quality and services information on each plan with information on beneficiary satisfaction, additional disenrollment measures, appeals and grievances, physician contracting, and plan contract information to better understand how different aspects of health care combine to reflect the quality of care and services provided by an individual plan.

HCFA believes that the careful use of performance measurement that is being required for the managed care plans is valuable for Medicare’s FFS program as well. Currently, HCFA is exploring the feasibility of calculating the effectiveness of care HEDIS measures, other than breast cancer screening, for its FFS population through a contract with Health Economics Research, Inc. HCFA developed other cardiovascular measures in addition to beta blocker prescription upon discharge after admission for AMI as part of the CCP quality improvement project, and projects similar to CCP are underway. HCFA created the Outcome Assessment Information Set (OASIS) of outcome quality indicators for home health care. HCFA also has begun developing measures of performance for nursing home and rehabilitations hospital stays drawing on the Minimum Data Set (MDS) and for renal dialysis facilities.

HCFA will continue to work with NCQA and other organizations to develop performance measures which reflect important aspects of care for the Medicare population. Currently, HCFA is involved actively in development of performance measures in geriatrics, diabetes, and medication use at the national level with NCQA, the Joint Commission on Accreditation of Health Care Organizations, the American Medical Association, and others. As the science of performance measurement, data collection, and reporting improves, we can look forward to having valid, reliable measures of quality that HCFA can use for all of its intended objectives of internal plan improvement, consumer information, and purchaser oversight.

[1]HEDIS® is a registered trademark of the National Committee for Quality Assurance (“NCQA”), and a copyright for HEDIS® 3.0 is held by the National Committee for Quality Assurance, 2000 L Street, NW, Suite 500, Washington, DC 20036. All rights reserved.

[2]Interested individuals can obtain a full report of findings on HCFA’s web site (http://www.hcfa.gov/quality/qlty‑3e.htm) and IPRO’s web site (http://www.ipro.org).

[3]Medicare received 288 submissions. Three contracts did not complete a preliminary assessment for HCFA's audit, and one preliminary assessment collected by IPRO covered two contract markets.

[4]Interested individuals can obtain a complete copy of findings from this report on HCFA’s web site (http://www.hcfa.gov/quality/qlty‑3e.htm ) and IPRO’s web site (http://www.ipro.org).

5 Individuals interested in greater detail about minimum required sample size and power calculations should contact HCFA through the e-mail available at this web site or consult NCQA’s HEDIS 3.0 Volume 2 Technical Specifications.

[6]Medicare managed care plans will furnish plan audited financial statements upon request.

[7] Krumholz HM, Radford MJ, Wang Y, Chen J, Heiat A, Marciniak TA. National use and effectiveness of -blockers for the treatment of the elderly patients after acute myocardial infarction. Journal of the American Medical Association. 1998; 280: 623-629.