Overview of the Testing Phase of Measure Development
Measure testing assesses the suitability of the quality measure’s technical specifications and acquires empirical evidence to help assess the strengths and weaknesses of a measure and its:
- Importance to measure and report—including analysis of opportunities for improvement such as reducing variability in comparison groups or disparities in healthcare related to race, ethnicity, age, or other classifications.
- Scientific acceptability—including analysis of reliability, validity, and exclusion appropriateness.
- Reliability—Description of reliability statistics and assessment of adequacy in terms of norms for the tests, and the rationale for analysis approach. Demonstrates that measure results are repeatable and the measurement error is acceptable, producing the same results a high proportion of the time when assessed in the same population in the same time period.
- Validity—Specific analyses and findings related to any changes observed relative to analyses reported during the prior assessment/endorsement process, or changes observed based on revisions to the measure. These may include assessment of adequacy in terms of norms for the tests conducted, panel consensus findings, and rationale for analysis approach. Refers to the degree to which evidence, clinical judgment, and theory support the interpretations of a measure score. Indicates the ability of a measure to record or quantify what it purports to measure
- Feasibility—including evaluation of reported costs or perceived burden, frequency of missing data, and description of data availability.
- Usability—including planned analyses to demonstrate that the measure is meaningful and useful to the target audience. This may be accomplished by the TEP reviewing the measure results such as means and detectable differences, dispersion of comparison groups, etc. More formal testing, if requested by CMS, may require assessment via structured surveys or focus groups to evaluate the usability of the measure (e.g., clinical impact of detectable differences, evaluation of the variability among groups).
Measure testing includes testing the components of the quality measures, such as the data elements, the scales (and the items in the scales if applicable), and the performance score.
For electronic clinical quality measures (eCQMs), the systems and tools used include the Measure Authoring Tool (MAT), the Value Set Authority Center (VSAC), and testing tools (i.e., Bonnie or Cypress, as well as JIRA.
The steps for measure testing include:
- Develop the Measure Testing Plan
- Submit the plan and obtain approval
- Implement the plan
- Analyze the test results
- Refine the measure
- Retest the refined measure
- Compile and submit deliverables for approvals
- Participate in the endorsement process as needed (e.g., be available to answer questions)
Measure testing in iterative—you may go through the cycle multiple time as you refine it based on testing results and stakeholder input. During this phase, input is sought during alpha (for example regarding face validity) and beta (for example the feasibility and assessment of data collection burden) testing as well as to review overall results. The input can be obtained through a Technical Expert Panel (TEP), consultations with subject matter experts, outreach to professional associations or patient advocacy groups and through the public comments process.
Develop the testing plan:
Implement the plan and conduct Alpha and Beta Testing:
Modify measure specifications, data collection instructions, and calculation of measure results:
Retest the refined measure:
Support CMS during NQF endorsement process:
Develop the testing plan:
Complete measure testing summary and submit to CMS for approval:
Update the MIF and MJF:
Complete the measure evaluation report:
- Page last Modified: 02/14/2018 10:22 AM
- Help with File Formats and Plug-Ins