Skip to Main Content
Main Content

Testing


Overview of the Testing Phase of Measure Development

Measure testing assesses the suitability of the quality measure’s technical specifications and acquires empirical evidence to help assess the strengths and weaknesses of a measure and its:

  • Importance to measure and report—including analysis of opportunities for improvement such as reducing variability in comparison groups or disparities in healthcare related to race, ethnicity, age, or other classifications.
  • Scientific acceptability—including analysis of reliability, validity, and exclusion appropriateness.
    • Reliability—Description of reliability statistics and assessment of adequacy in terms of norms for the tests, and the rationale for analysis approach. Demonstrates that measure results are repeatable and the measurement error is acceptable, producing the same results a high proportion of the time when assessed in the same population in the same time period.
    • Validity—Specific analyses and findings related to any changes observed relative to analyses reported during the prior assessment/endorsement process, or changes observed based on revisions to the measure. These may include assessment of adequacy in terms of norms for the tests conducted, panel consensus findings, and rationale for analysis approach. Refers to the degree to which evidence, clinical judgment, and theory support the interpretations of a measure score. Indicates the ability of a measure to record or quantify what it purports to measure
  • Feasibility—including evaluation of reported costs or perceived burden, frequency of missing data, and description of data availability.
  • Usability—including planned analyses to demonstrate that the measure is meaningful and useful to the target audience. This may be accomplished by the TEP reviewing the measure results such as means and detectable differences, dispersion of comparison groups, etc. More formal testing, if requested by CMS, may require assessment via structured surveys or focus groups to evaluate the usability of the measure (e.g., clinical impact of detectable differences, evaluation of the variability among groups).

Measure testing includes testing the components of the quality measures, such as the data elements, the scales (and the items in the scales if applicable), and the performance score.

For electronic clinical quality measures (eCQMs), the systems and tools used include the Measure Authoring Tool (MAT), the Value Set Authority Center (VSAC), and testing tools (i.e., Bonnie or Cypress, as well as JIRA

TestingImage

The steps for measure testing include: 

  • Develop the Measure Testing Plan
  • Submit the plan and obtain approval
  • Implement the plan
  • Analyze the test results
  • Refine the measure
  • Retest the refined measure
  • Compile and submit deliverables for approvals
  • Participate in the endorsement process as needed (e.g., be available to answer questions)

Measure testing in iterative—you may go through the cycle multiple time as you refine it based on testing results and stakeholder input. During this phase, input is sought during alpha (for example regarding face validity) and beta (for example the feasibility and assessment of data collection burden) testing as well as to review overall results. The input can be obtained through a Technical Expert Panel (TEP), consultations with subject matter experts, outreach to professional associations or patient advocacy groups and through the public comments process.

Resources:

Develop the testing plan:

CMS MMS Webinar BP101-Measure Testing

Measures Management Up Close: Testing

Implement the plan and conduct Alpha and Beta Testing:

CMS Blueprint (Section 3. In-Depth Topics – 19 Alpha and Beta Testing)

Modify measure specifications, data collection instructions, and calculation of measure results:

CMS Blueprint (Section 3. In-Depth Topics – 18.5 Refine the Measure)

Retest the refined measure:

CMS Blueprint (Section 3. In-Depth Topics – 18.6 Refine the Measure)

Support CMS during NQF endorsement process:

CMS Blueprint (Section 3. In-Depth Topics – 18.8 Support CMS during NQF endorsement process)

Electronic Clinical Quality Measures Testing Tools

Tools:

Develop the testing plan:

CMS Blueprint (Section 3. In-Depth Topics – 18.1 Develop the Testing Work Plan)

Complete measure testing summary and submit to CMS for approval:

CMS Blueprint (Section 3. In-Depth Topics – 18.7 Compile and Submit Deliverables to CMS)

Update the MIF and MJF:

CMS Blueprint (Measure Information Form Instructions)

CMS Blueprint (Blank Measure Justification Form Template)

Complete the measure evaluation report:

Measure Evaluation Report

eCQM Standards Based Guidance and Tools

Next Phase

Return to Measure Development by Phase

Implementation

.