To: Administrative File: (CAG-00022R)
Reconsideration of Ultrasound Stimulation for Nonunion Fracture Healing
From: Steve Phurrough, MD, MPA
Director
Coverage and Analysis Group
Marcel Salive, MD, MPH
Director
Division of Medical and Surgical Services
Shamiram Feinglass, MD, MPH
Medical Officer
Division of Items and Devices
Niccole Corbin, DPM
Health Policy Analyst
Division of Medical and Surgical Services
Subject: Proposed Coverage Decision Memorandum for Ultrasound Stimulation for Nonunion Fracture Healing
Date: January 27, 2005
I. Proposed Decision
The Centers for Medicare & Medicaid Services (CMS) proposes the following:
The evidence is not adequate to conclude that non-invasive ultrasound stimulation for the treatment of nonunion bone fractures prior to surgical intervention is reasonable and necessary absent safeguards that would be present in formal, protocol-driven clinical investigations. These trials must compare patients who are randomized to ultrasound stimulation or surgical intervention and have as the goals, monitoring, evaluating, and improving clinical outcomes. The trials must:
- Meet the requirements of Food and Drug Administration (FDA) category B investigational device exemption (IDE); or
- Meet the following basic criteria:
- Written protocol on file;
- Institutional Review Board review and approval;
- Scientific review and approval by two or more qualified individuals who are not part of the research team;
- Certification that investigators have not been disqualified.
For purposes of this coverage decision, CMS will determine whether specific clinical trials meet these criteria.
CMS is requesting public comments on this proposed decision memorandum pursuant to Section 731 of the Medicare Modernization Act. After considering the public comments, we will issue a final decision memorandum.
II. Background
On July 27, 2004, CMS began a national coverage determination (NCD) for reconsideration of the requirement that patients must have failed at least one surgical intervention before treatment of a nonunion bone fracture within ultrasound stimulation.
Normally, the body is able to effectively heal fractures with conservative medical treatment within 8 to 12 weeks. Like most biological processes, normal fracture healing is complex and involves the integrated action of numerous cells, genes, and extracellular matrix. Healing of a fracture is the re-uniting of disrupted bone caused by a break in continuity. The process can be divided into three main stages – inflammatory, reparative, and remodeling. These factors are documented in detail with the original decision memo on ultrasound stimulation for nonunion fractures. 1
Many factors are involved in the healing of a fracture. The severity of the fracture, its location, the degree of bone loss, the extent of immobilization, nature of the blood supply, the amount of soft tissue damage, the involvement of a tumor, the use of radiation therapy, or the presence of infection can all retard the successful union of a fracture. Health considerations such as diabetes, vascular insufficiency, osteoporosis, nutritional deficiency, anemia, hormonal deficiency, smoking, advanced age, or certain medications can also affect the quality and rate of fracture healing. Co-morbidities associated with aging and the aforementioned health considerations may predispose the Medicare population to delayed or impaired healing. Additionally, certain patients with chronic diseases may be ineligible for surgery due to the risk of complications.
Fractures that do not heal within usual time frames are known as delayed unions and nonunions. Nonunions occur when there is no indication of healing for at least 3 months.2 A nonunion is defined clinically as the point when bone healing is stopped and will not proceed without some type of intervention. A nonunion is clinically established when there is no visible progression of the healing process. A delayed union is defined when healing has not advanced at the “average” rate for the location and type of fracture.3
Treatment options for nonunion fractures range from non-surgical treatments to various surgical techniques. Non-surgical treatments include osteogenic stimulation (involving pulsed electromagnetic fields, capacitive coupling, direct electrical stimulation or ultrasound), immobilization, and/or casting. Surgical techniques may involve external fixation and internal fixation that includes pins, nails, screws, wires, intramedullary rods, compression plates, and/or bone grafts. Successful treatment of a nonunion often depends on appropriate reduction of the fracture, bone grafting if necessary, and stabilization (internal or external fixation).
Ultrasound stimulation treatments have been advocated for nonunion fractures. It is a non-invasive external treatment and may be used alone, in conjunction with surgical stabilization of the fracture, or after surgical failure.
Scientific literature provides evidence that both mechanical and electrical stimuli can send regulatory signals to the bone causing physical remodeling of tissue. Ultrasound stimulation is a form of mechanical energy that is transmitted into tissue as high frequency acoustical pressure waves. The acoustical waves promote fracture healing by producing a thermal effect. 4 Specific physiological effects that have been attributed to ultrasound stimulation include increased signaling pathways in osteoblasts, increased release of growth factors, increased enzymatic activity, increased calcium absorption, increased blood flow to the fracture site and increased callus formation.
III. History of Medicare Coverage
In August 1996, the Technology Advisory Committee (TAC) reviewed available data relating to ultrasound treatment and found insufficient evidence for effectiveness in the Medicare population. Therefore, the Health Care Financing Administration (HCFA, now CMS) revised the Coverage Issues Manual section 35-48 to issue a national noncoverage policy of ultrasound for all indications.
In November 1998, HCFA reviewed ultrasound for the treatment of fresh fractures again and national noncoverage continued.
In July 2000, Medicare completed a review on ultrasound stimulation for nonunion fractures but not for fresh fracture or delayed unions. Following this review, CMS modified its policy to allow ultrasound stimulation for nonunion when the following criteria are met:
A minimum of two sets of radiographs obtained prior to starting treatment with the osteogenic stimulator, separated by a minimum of 90 days. Each radiograph must include multiple views of the fracture site accompanied with a written interpretation by a physician stating that there has been no clinically significant evidence of fracture healing between the two sets of radiographs. Also, indications that the patient failed at least one surgical intervention for the treatment of the fracture. 5
On July 1, 2004, CMS received a formal request for reconsideration of the previous national coverage determination (NCD) on ultrasound treatment of nonunion fractures from Smith & Nephew, Inc., the manufacturer of an ultrasound bone healing system. The requestor asked specifically for reconsideration of the national coverage decision requiring a surgical intervention for the Medicare patient prior to utilization of the ultrasound bone stimulator.
Benefit Category Determination
For an item or service to be covered by the Medicare program, it must meet one of the statutorily defined benefit categories outlined in the Social Security Act. CMS’s Center for Medicare Management (CMM) has determined that the application of non-invasive ultrasound stimulation for the treatment of nonunion fractures is included in the following benefit category:
§1861 (n) Durable Medical Equipment (DME), 42 U.S.C § 1395x(n)
IV. Timeline of Recent Activities
| Date |
Action |
| May 24, 2004 |
Representatives from Smith & Nephew, Inc. met with CMS staff to discuss the process for coverage reconsideration. |
| July 27, 2004 | CMS accepted the formal request for national coverage reconsideration of ultrasound bone stimulation for the treatment of long bone fractures. CMS also began its standard, initial 30-day comment period on this NCD to obtain public and scientific input relevant to the issue under consideration. |
| Sept. 20, 2004 | CMS met with Dr. Mark Dollard, a practicing podiatrist, to discuss the biological effects of ultrasound stimulation on the bone healing process. |
| Sept. 24, 2004 | Initial public comments posted to CMS website at http://www.cms.hhs.gov/mcd/viewtrackingsheet.asp?id=135 |
V. Food and Drug Administration (FDA) Status
On February 22, 2000, FDA approved the premarket approval application (PMA) for the low intensity ultrasound fracture treatment system, Exogen 2000®, indicated for the non-invasive treatment of established nonunions, excluding skull and vertebra. In addition, the device is indicated for accelerating the time to a healed fracture for fresh, closed, posteriorly displaced distal radius fractures and fresh, closed or grade 1 open tibial diaphysis in skeletally mature individuals when these fractures are orthopedically managed by closed reduction and cast immobilization. A nonunion is considered to be established when the fracture site shows no visibly progressive signs of healing.6
VI. General Methodological Principles of Study Design
When making NCDs, CMS evaluates relevant clinical evidence to determine whether or not the evidence is of sufficient quality to support a finding that an item or service is reasonable and necessary. The critical appraisal of the evidence enables us to determine to what degree we are confident that: 1) the specific assessment questions can be answered conclusively; and 2) the intervention will improve net health outcomes for patients. Evidence may consist of external technology assessments, internal review of published studies, recommendations from the Medicare Coverage Advisory Committee (MCAC), evidence-based guidelines, professional society position statements, expert opinion, and public comments (as appropriate).
A detailed account of the methodological principles of study design the agency staff utilizes to assess the relevant literature on a therapeutic or diagnostic item or service for specific conditions follows the conclusion and references for this proposed decision memorandum (see Appendix B).
VII. Evidence
A. Introduction
We are providing a summary of evidence we considered before the public comment period. We will, of course, consider additional evidence submitted through the public comment period. This summary represents the body of evidence describing use of ultrasound stimulation for nonunion fractures without prior surgical intervention. The health outcomes of interest to CMS include healing rate and proportion with complete fracture healing.
In analyzing the evidence, CMS focused upon the following question, “Is there adequate evidence to conclude that non-invasive ultrasound stimulation improves net health outcomes in the treatment of nonunion bone fractures without prior surgical intervention in the Medicare population?”
B. Discussion of Evidence Reviewed
The evidence reviewed includes CMS’s 1998 and 2000 ultrasound stimulation decision memoranda, CMS’s internal technology assessment of new or reconsidered evidence, as well as any professional society position statements and expert opinion.
1. Internal Technology Assessment
Literature Search
CMS extensively searched PubMed (1990 to present) for new randomized controlled trials (RCTs) and systemic reviews evaluating the use of non-invasive ultrasound stimulation for the treatment of nonunion fractures. The literature search was limited to the English language and specific to the human population. The search was limited to those articles published in the last 10 years, but included studies conducted in all countries, including the United States (see evidence tables in Appendix A). In addition, CMS reviewed all articles submitted by the requestor and incorporated studies previously reviewed in earlier decision memoranda if they included patients treated with ultrasound stimulation without prior surgical intervention.
New Evidence
Since the 2000 decision, only two new sources of evidence are available, a case series with 18 subjects and a meta-analysis (see Appendix A –Evidence tables). Both have subgroups of patients with nonunion fractures without prior surgery and both find healing improved with the use of self-administered ultrasound bone stimulation for 20 minutes per session.
Lerner, et al (2004), reported a small (n=18) case series of severe compound high-energy long bone fractures of from 1997-2001. Subjects were aged 19 through 63. Sixteen of 18 patients had surgical stabilization prior to ultrasound stimulation. Authors found that 16/18 nonunion fractures treated with ultrasound united within one year after starting treatment (median time 26 weeks, 89% heal rate). There were no side effects of the procedure reported. Authors recommend ultrasound as an “adjunct modality in the treatment of severe high-energy injuries.”
Busse, et al (2002) conducted a meta-analysis of ultrasound treatment and found three articles meeting their criteria with only one addressing nonunion fractures (Mayr 2000, see below). This one article was considered in the precious NCA. Time to fracture healing was shorter with ultrasound (64 days shorter on average). The authors conclude that ultrasound may be beneficial to fracture healing, and that further clinical trials are needed.
Review of Evidence Previously Considered
Three studies that include results of ultrasound treatment in patients with nonunion fractures without prior surgery were identified from the literature reviewed for the previous decision memoranda.
Mayr, et al (2000), in a retrospective case series from a registry, found 153 nonunion patients without surgery prior to ultrasound stimulation. These patients had a success rate of 86% (132/153) and an average heal time of 140 days after beginning ultrasound therapy. These results are similar to those nonunion cases with surgery prior to ultrasound stimulation (success rate 85%, average heal time of 169 days).
Nolte, et al (2001) examined 41 cases of nonunion fractures in Dutch patients 18-90 years old from 1995-1997. Of the 29/39 cases that completed the study, the average fracture age was 1.2 years. Only eight cases had no prior surgery. Of these eight, 7 cases healed following ultrasound and the average heal time was 157 days which is similar to those who had surgery (152 days). In this self-paired analysis of pooled data, heal rates for cases with prior surgery (86%) and no surgery (87.5%) were similar.
Gebauer et al (2000), reported a case series of 67 nonunion fracture patients, including only 10 non-surgical nonunion cases treated with ultrasound. Of these cases, 7 healed following ultrasound.
2. Medicare Coverage Advisory Committee (MCAC)
This issue was not referred to the MCAC.
3. Evidence-Based Guidelines
No evidence-based guidelines were identified.
4. Professional Society Position Statements
CMS received no statements from professional societies pertaining specifically to ultrasound stimulation for nonunion fractures. However, the American Association of Orthopaedic Surgeons and American Academy of Orthopaedic Surgeons (AAOS) published a position statement entitled, “Crendentialing in the Use of Specialized Instrumentation on Orthopaedics.” The statement indicated that use of specialized instrumentation should be based on educational exposure to the instruments and an understanding of their indications and contraindications. It further suggested that clinical privileges should be based on a surgeon’s knowledge, skills and experience, and should be the responsibility of local hospitals. AAOS’s position statement is available electronically at http://www.aaos.org/wordhtml/papers/position/1105.htm
5. Expert Opinion
CMS received no expert opinions.
6. Public Comments
During the initial comment period, July 27, 2004 to August 27, 2004, CMS received 14 public comments supporting the expansion of coverage for ultrasound osteogenic stimulation in the treatment of nonunion fractures.
VIII. CMS Analysis
National coverage determinations (NCDs) are determinations by the Secretary with respect to whether or not a particular item or service is covered nationally under title XVIII of the Social Security Act § 1869(f)(1)(B). In order to be covered by Medicare, an item or service must fall within one or more benefit categories contained within Part A or Part B, and must not be otherwise excluded from coverage. Moreover, with limited exceptions, the expenses incurred for items or services must be “reasonable and necessary for the diagnosis or treatment of illness or injury or to improve the functioning of a malformed body member.” § 1862(a)(1)(A).
There is a paucity of new evidence for this reconsideration and the evidence that was considered was severely limited by the small sample sizes and the nature of the studies (case series). Thus, the quality of the studies is not adequate to evaluate the effects of ultrasound stimulation treatment for nonunion fractures prior to surgical intervention.
As found previously, this body of literature is limited by the use of retrospective case series, registry data, the use of overlapping cases, and small studies (as discussed in Appendix B).
One noticeable problem with the use of registry data is the overlap between studies, which weakens the strength of the overall body of evidence. Sequential reports from a given registry often analyze some of the same cases. For instance, two studies contain cases entered in the registry between October 17, 1994 and October 17, 1996. 7,8
One strength of the body of evidence is the applicability of the study procedures beyond the research setting. The majority of the patients in these studies were treated at home under the instruction of their own physician. Therefore, the studies conducted on the ultrasound treatment of nonunion are relevant to the everyday clinical setting.
The articles reviewed for this reconsideration are fairly consistent both in terms of design and outcome. All the studies pertaining to nonunion fractures use an outcome of healed versus failure to heal and they all utilize ultrasound treatment without concomitant treatment. Though the numbers of patients without previous surgery is small, they were consistent with the overall body of evidence. However, no paper solely addresses the ability of this technology to work in patients without previous surgery.
The previous decision had minimally acceptable evidence. However, for those patients who were post surgery, no other options were available and a high quality randomized trial was not feasible. Those limitations remain. Now, with limited new evidence, the current NCD request is for the removal of the surgical requirement prior to utilizing ultrasound therapy. Other clinical options (i.e. surgery) are available and comparative studies with the level of evidence we accept today are feasible and easily accomplished. We would expect a trial to compare patients who are randomized to ultrasound stimulation or surgical intervention and have as its goal to monitor, evaluate, and improve clinical outcomes. CMS finds the evidence is not adequate to evaluate the effects of ultrasound stimulation treatment for nonunion fractures prior to surgical intervention.
Ultrasound stimulation for non-union fracture healing is no longer an experimental device. Our previous decision reviewed literature demonstrating the positive clinical outcomes of ultrasound stimulation for non-union fractures post surgery. However, the new evidence was insufficient to reach a conclusion that ultrasound stimulation for non-union fracture healing prior to surgical intervention is reasonable and necessary in all instances. More evidence of net health benefit is needed for patients without previous surgical intervention who undergo ultrasound stimulation for nonunion fractures. A sufficient inference of benefit, however, can be drawn to support limited coverage if certain safeguards for patients are provided. This inference is based on the evidence discussed previously regarding the benefits of ultrasound stimulation in the small number of patients treated prior to surgery. We believe that patient protection can be provided by requiring, as part of a clinical study, data collection on patients receiving ultrasound stimulation prior to surgery. This is consistent with the general application of therapeutic interventions. Therapeutic interventions provide results that are used to influence patient management but the conclusions are reevaluated as additional data are obtained. The additional data may alter the clinical management and potentially improve health outcomes. The effective and accurate use of ultrasound stimulation for non-union fractures prior to surgical intervention can only be ensured by this data collection.
In addition, patient care provided under clinical protocols is typically associated with a higher quality of care. Protocols include patient selection criteria, guidelines for test administration, and recommended treatments. These benefits offer safeguards for patients to ensure appropriate evaluation and use of ultrasound stimulation for non-union fracture healing.
Finally, it is important to have a means of assessing the quality of patient care over time to ensure that positive outcomes are maintained or improved. Data from prospective clinical studies can be an invaluable aid in ongoing assessment of the quality of care provided to patients.
CMS considers acceptable any one of the following types of prospective clinical studies:
- A clinical trial of ultrasound stimulation for non-union fracture healing prior to surgical intervention that meets the requirements of Food and Drug Administration (FDA) category B investigational device exemption (42 CFR 405.201); or
- A clinical study of ultrasound stimulation for non-union fracture healing prior to surgical intervention that is designed to collect additional information at the time of the procedure to assist in patient management.
The clinical study must ensure that:
- Specific hypotheses are identified prospectively;
- Hospitals and providers are qualified to provide to and educate the patient on the proper use of the ultrasound stimulation device for non-union fracture healing prior to surgical intervention;
- Participating hospitals and providers report data on all enrolled patients undergoing ultrasound stimulation for non-union fracture healing prior to surgical intervention;
- The data to be collected includes:
- Baseline patient characteristics
- Type of ultrasound stimulator used
- Results of all imaging studies used to assess healing outcomes
- Facility and provider characteristics
- Long-term patient outcomes, such as no pain on ambulation, adequate fracture healing (a minimum of three or four cortices bridged), and disease management changes
- Data collection is prospective and patients are randomized to ultrasound stimulation or surgical intervention with the goal to monitor, evaluate, and improve clinical outcomes;
- Compliance with all applicable patient confidentiality, privacy, and other Federal laws is met, including the Standards for Privacy of Individually Identifiable Health Information (Privacy Rule).
Further refinement of the clinical study design is expected to occur based on further discussion with clinicians, methodology experts, and stakeholders.
During implementation of its current NCD on clinical trials (NCD Manual 310.1), CMS asked AHRQ to consult with a multi-agency panel in order to develop a set of criteria CMS could use to identify clinical trials that should receive Medicare coverage. AHRQ convened a panel composed of representatives from the FDA, National Institutes of Health, Centers for Disease Control and Prevention, Department of Defense, Veteran's Administration (VA), and the DHHS Office for Human Research and Protection. This panel held several meetings, including two public meetings in which interested parties were given the opportunity to provide comments. The panel recommended that clinical trials should be limited to trials meeting specific criteria. We propose to apply the same criteria to determining which trials would potentially be eligible for Medicare payment of experimental costs (Appendix C). In addition, all patient confidentiality, privacy and HIPAA requirements will be maintained.
Furthermore, to provide greater clarity about how CMS will choose which studies to cover, the CMS Council on Technology and Innovation will now begin to develop a draft guidance document on this policy approach in order to make the process more systematic, predictable and transparent. We will shortly announce an open door forum and separately convene an expert panel. Comments on this policy can be submitted through the CTI website at http://www.cms.hhs.gov/providers/cti. An initial draft guidance will be issued in March 2005, at which time additional public feedback will be solicited.
Conclusion:
CMS is proposing the following decision:
The evidence is not adequate to conclude that non-invasive ultrasound stimulation for the treatment of nonunion bone fractures prior to surgical intervention is reasonable and necessary absent safeguards that would be present in formal, protocol-driven clinical investigations. These trials must compare patients who are randomized to ultrasound stimulation or surgical intervention and have as the goals, monitoring, evaluating, and improving clinical outcomes. The trials must:
- Meet the requirements of Food and Drug Administration (FDA) category B investigational device exemption (IDE); or
- Meet the following basic criteria:
- Written protocol on file;
- Institutional Review Board review and approval;
- Scientific review and approval by two or more qualified individuals who are not part of the research team;
- Certification that investigators have not been disqualified.
For purposes of this coverage decision, CMS will determine whether specific clinical trials meet these criteria.
CMS is requesting public comments on this proposed decision memorandum pursuant to Section 731 of the Medicare Modernization Act. After considering the public comments, we will issue a final decision memorandum.
APPENDIX A
Evidence tables from the previous decision memorandum published on July 31, 2000 (including evidence review from Gebauer et al, Mayr et al. and Nolte et al.) are available electronically at http://www.cms.hhs.gov/coverage/download/id76.pdf?orgin=globalsearch&page=/mcd/viewdecisionmemo.asp&id=76 [PDF, 129KB].
Evidence Tables for proposed decision:
| Author/Year |
Study Design/Purpose |
Intervention/Outcomes |
Demographics |
Results |
| Lerner A, et al 2004 |
Case series Effect of low energy ultrasound (US) in healing severe compound high-energy limb injuries; supplemental to surgical skeletal stabilization (“combined treatment”) |
ultrasound stimulation for 20 minutes applied by patient at home treatment continued until clinical and radiographic signs of solid bone union. |
N=18 injuries in 17 patients. Data from 1997-2001 14 male,3 female age range: (19-63) 2 without prior surgical stabilization severe compound high energy limb injuries, various modes of injury, various sites:femur, tibia, forearm and humerus 16 with surgical stabilization (wires/screws) |
16/18 fractures united with solid bone union within 13-52 weeks (89%). Conclude useful as adjunct modality in severe injuries. |
| Busse JW, et al, 2002 |
Meta-analysis Evaluate effect of low intensity ultrasound (US) on time to fracture healing |
2 independent searchers identified relevant RCTs from 5 major databases; 2 independent readers |
Baseline characteristics: 20 minute sessions. 138 studies, 6 met inclusion criteria and reviewed, 3 met final analysis criteria. |
Time to fracture heal: significantly shorter with ultrasound-64 days shorter on average. Ultrasound may be beneficial to fracture healing They call for further clinical trials |
APPENDIX B
General Methodological Principles
When making national coverage decisions, CMS evaluates relevant clinical evidence to determine whether or not the evidence is of sufficient quality to support a finding of reasonable and necessary. The evidence may consist of external technology assessments, internal review of published and unpublished studies, recommendations from the Medicare Coverage Advisory Committee, evidence-based guidelines, professional society position statements, expert opinion, and public comments.
The overall objective for the critical appraisal of the evidence is to determine to what degree we are confident that: 1) specific clinical questions relevant to the coverage request can be answered conclusively; and 2) the intervention will improve net health outcomes for patients.
We divide the assessment of clinical evidence into three stages: 1) the quality of the individual studies; 2) the relevance of findings from individual studies to the Medicare population; and 3) overarching conclusions that can be drawn from the body of the evidence on the direction and magnitude of the intervention’s risks and benefits.
The issues presented here represent a broad discussion of the issues we consider when reviewing clinical evidence. However, it should be noted that each coverage determination has unique methodological aspects.
1. Assessing Individual Studies
Methodologists have developed criteria to determine weaknesses and strengths of clinical research. Strength of evidence generally refers to: 1) the scientific validity underlying study findings regarding causal relationships between health care interventions and health outcomes; and 2) the reduction of bias. In general, some of the methodological attributes associated with stronger evidence include those listed below:
- Use of randomization (allocation of patients to either intervention or control group) in order to minimize bias
- Use of contemporaneous control groups (rather than historical controls) in order to ensure comparability between the intervention and control groups.
- Prospective (rather than retrospective) studies to ensure a more thorough and systematical assessment of factors related to outcomes.
- Larger sample sizes in studies to help ensure adequate numbers of patients are enrolled to demonstrate both statistically significant as well as clinically significant outcomes that can be extrapolated to the Medicare population. Sample size should be large enough to make chance an unlikely explanation for what was found.
- Masking (blinding) to ensure patients and investigators do not know to which group patients were assigned (intervention or control). This is important especially in subjective outcomes, such as pain or quality of life, where enthusiasm and psychological factors may lead to an improved perceived outcome by either the patient or assessor.
Regardless of whether the design of a study is a randomized controlled trial, a non-randomized controlled trial, a cohort study or a case-control study, the primary criterion for methodological strength or quality is the extent to which differences between intervention and control groups can be attributed to the intervention studied. This is known as internal validity. Various types of bias can undermine internal validity. These include:
- Different characteristics between patients participating and those theoretically eligible for study but not participating (selection bias)
- Co-interventions or provision of care apart from the intervention under evaluation (confounding)
- Differential assessment of outcome (detection bias)
- Occurrence and reporting of patients who do not complete the study (attrition bias)
In principle, rankings of research design have been based on the ability of each study design category to minimize these biases. A randomized controlled trial minimizes systematic bias (in theory) by selecting a sample of participants from a particular population and allocating them randomly to the intervention and control groups. Thus, randomized controlled studies have been typically assigned the greatest strength, followed by non-randomized clinical trials and controlled observational studies. The following is a representative list of study designs (some of which have alternative names) ranked from most to least methodologically rigorous in their potential ability to minimize systematic bias:
- Randomized controlled trials
- Non-randomized controlled trials
- Prospective cohort studies
- Retrospective case control studies
- Cross-sectional studies
- Surveillance studies (e.g., using registries or surveys)
- Consecutive case series
- Single case reports
When there are merely associations but not causal relationships between a study’s variables and outcomes, it is important not to draw causal inferences. Confounding refers to independent variables that systematically vary with the causal variable. This distorts measurement of the outcome of interest because its effect size is mixed with the effects of other extraneous factors. For observational, and in some cases randomized controlled trials, the method in which confounding factors are handled (either through stratification or appropriate statistical modeling) are of particular concern. For example, in order to interpret and generalize conclusions to our population of Medicare patients, it may be necessary for studies to match or stratify their intervention and control groups by patient age or co-morbidities.
Methodological strength is, therefore, a multidimensional concept that relates to the design, implementation and analysis of a clinical study. In addition, thorough documentation of the conduct of the research, particularly study’s selection criteria, rate of attrition and process for data collection, is essential for CMS to adequately assess the evidence.
2. Generalizability of Clinical Evidence to the Medicare Population
The applicability of the results of a study to other populations, settings, treatment regimens and outcomes assessed is known as external validity. Even well-designed and well-conducted trials may not supply the evidence needed if the results of a study are not applicable to the Medicare population. Evidence that provides accurate information about a population or setting not well represented in the Medicare program would be considered but would suffer from limited generalizability.
The extent to which the results of a trial are applicable to other circumstances is often a matter of judgment that depends on specific study characteristics, primarily the patient population studied (age, sex, severity of disease and presence of co-morbidities) and the care setting (primary to tertiary level of care, as well as the experience and specialization of the care provider). Additional relevant variables are treatment regimens (dosage, timing and route of administration), co-interventions or concomitant therapies, and type of outcome and length of follow-up.
The level of care and the experience of the providers in the study are other crucial elements in assessing a study’s external validity. Trial participants in an academic medical center may receive more or different attention than is typically available in non-tertiary settings. For example, an investigator’s lengthy and detailed explanations of the potential benefits of the intervention and/or the use of new equipment provided to the academic center by the study sponsor may raise doubts about the applicability of study findings to community practice.
Given the evidence available in the research literature, some degree of generalization about an intervention’s potential benefits and harms is invariably required in making coverage decisions for the Medicare population. Conditions that assist us in making reasonable generalizations are biologic plausibility, similarities between the populations studied and Medicare patients (age, sex, ethnicity and clinical presentation) and similarities of the intervention studied to those that would be routinely available in community practice.
A study’s selected outcomes are an important consideration in generalizing available clinical evidence to Medicare coverage determinations. One of the goals of our determination process is to assess net health outcomes, and we are interested in the results of changed patient management not just altered management. These outcomes include resultant risks and benefits such as increased or decreased morbidity and mortality. In order to make this determination, it is often necessary to evaluate whether the strength of the evidence is adequate to draw conclusions about the direction and magnitude of each individual outcome relevant to the intervention under study. In addition, it is important that an intervention’s benefits are clinically significant and durable, rather than marginal or short-lived.
If key health outcomes have not been studied or the direction of clinical effect is inconclusive, we may also evaluate the strength and adequacy of indirect evidence linking intermediate or surrogate outcomes to our outcomes of interest.
3. Assessing the Relative Magnitude of Risks and Benefits
Generally, an intervention is not reasonable and necessary if its risks outweigh its benefits. Among other things, CMS evaluates whether reported benefits translate into improved net health outcomes. The direction, magnitude and consistency of the risks and benefits across studies are important considerations. Based on the analysis of the strength of the evidence, CMS assesses whether an intervention or technology's benefits to Medicare beneficiaries outweigh its harms.
Appendix C
Standards of Qualifying Clinical Trials
A. Required Elements of the Written Protocol 8
- The principal investigator must certify that he/she or the fiscal office of his/her institution will keep a copy of the final written protocol on file and, upon request, make it available to CMS.
- An abstract of the written protocol will be submitted as part of the registration process.
-
The written protocol must include the following information:
- Identifying information
- Scientific background
- Objectives and hypothesis
- Design
- Criteria for selection, exclusion, and withdrawal of subjects
- Interventions (where applicable) and other treatments for subjects under each arm of the study
- Outcome measures
- Statistical analysis plan
- Discussion of quality control, data management, and record keeping procedures, including plans to ensure compliance with prevailing privacy regulations
- Conflict of interest policies
- If the research is being conducted at an institution with a conflict of interest policy, this should be noted, with a statement that the policies are being followed;
- If there are no institutional conflict of interest policies, then the protocol should identify a set of policies that are being used; options include:
- U.S Public Health Service regulations: 42 CFR Part 50 Sec. 50.604; Institutional responsibility regarding conflicting interests of investigators: (http://www.access.gpo.gov/nara/cfr/waisidx_00/42cfr50_00.html).
- Association of American Medical Colleges Guidelines for Dealing with Faculty Conflicts of Commitment and Conflicts of Interest in Research: (http://www.aamc.org/research/dbr/coi.htm).
- American Medical Association Guidelines for Conflicts of Interest in Biomedical Research and Health Facility Ownership by a Physician: (http://www.ama-assn.org/ethic/ceja/report95.pdf) and (http://www.amaassn.org/ethic/ceja/06b.pdf), respectively.
- Other ethical issues, where applicable
- Publication policy:
- Protocol should describe the specific publication policies that are being followed.
- Principal investigator (P1) must certify that:
-Investigators have the right to publish findings from this trial without receiving approval from the trial's financial sponsors.
-Investigators agree to notify ClinicalTrials.Gov of initial publications based on data from this trial.
B. Institutional Review Board (IRB) review and approval
1. The principal investigator must certify that an IRB has reviewed and approved the trial. Evidence of this must be kept on file, and be made available to the Secretary for review on request.
2. Although the term IRB has been used to describe a range of committees, the use of the term here refers to a committee that is constituted and operates in a manner consistent with the definition and procedures specified in Department of Health and Human Services (DHHS) Regulations for the Protection of Human Subjects in the Code of Federal Regulations (45CFR Part 46).9
3. The Office for Human Research Protection (OHRP) is taking several steps that are designed to enhance the functioning of IRBs. These steps include developing a system of IRB registration and implementing a streamlined assurance program. In addition, IRB accreditation programs are being explored (and in the case of the VA, implemented). All of these steps are important to enhance the functioning of IRBs, and the panel believes that they should be required as part of the Medicare qualifying criteria as soon as appropriate systems are in place.
C. Scientific Review and Approval 10
1. Review of a trial protocol by two or more qualified individuals who are not part of the research team is important to ensure that the trial has scientific merit.
2. Critical elements of scientific review include the following:
- Importance and relevance of the research question(s)
- Soundness of the study's scientific rationale
- Previous research to support proceeding to clinical trials in human beings (if appropriate)
- Adequacy of the study design and procedures to evaluate the specific research question(s)
- Appropriateness of the study population (e.g., age, gender, health status)
- Appropriateness of statistical plan
- Feasibility of carrying out the study
- Qualifications of the investigators
- Evidence and assurance that risks to human subjects are minimized
3. Two or more individuals who have the appropriate range of expertise must conduct the scientific review (including clinical trial methodology and content area of the trial). The individuals who conduct the review should not have direct involvement with the research team, and should not have direct financial ties to or interests in the research. The review may be conducted by a standing scientific review committee or by two or more individuals identified by the principal investigator. The principal investigator must specify the names and contact information of the reviewers (or the standing committee and its chair) and the date of approval.
D. Certification that investigators have not been disqualified
The principal investigator must certify that none of the trial investigators have been barred from participating in human subjects research by the FDA, Office of Research Integrity (ORI), Office for Human Research Protections (OHRP), or any other Federal agency. The principal investigator must inform CMS if any investigator becomes disqualified over the course of the trial.
1 Decision Memo on Ultrasound Stimulation for Nonunion Fractures (HCFA, 2000) accessed at http://cms.hhs.gov/mcd/viewdecisionmemo.asp?id=76.
2 AAOS Fact Sheet, retrieved on November 2, 2004 from http://orthoinfo.aaos.org/fact/thr_report.cfm?Thread_ID=386&topcategory=General%20Information
3 Decision Memo on Ultrasound Stimulation for Nonunion Fractures (HCFA, 2000) accessed at http://cms.hhs.gov/mcd/viewdecisionmemo.asp?id=76.
4 Chang et al. Study of thermal effects of ultrasound stimulation on fracture healing. Bioelectromagnetics. 2002; 23:256-263.
5 NCD Coverage Issues Manual section 35-48 can be accessed at http://cms.hhs.gov/Regulations-and-Guidance/Guidance/Manuals/06_cim/ci35.asp#_1_53.
6 FDA status accessed at http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMA/PMA.cfm?ID=5779
7 Heppenstall et al. Non-invasive nonunion treatment by pulsed low-intensity ultrasound. AAOS Annual Meeting.1999
8 Mayr et al. Ultrasound - an alternative healing method for nonunions? Arch Orthop Trama Surg. 2000; 120 (1):1-8.
9 64 Adapted from the following: 1) International Conference on the Harmonization of Technical Requirements for Registration of Pharmaceutical for Human Use: Guideline forGoodClinicalPractice.May1996.(http://www.ich.org/MediaServer.jser?@_ID=482&@_MODE=GLB); 2)NIH scientific review group evaluations of clinical protocols: (http://grants.nih.gov/grants/guide/notice-files/not97-0l0.html); and 3)Elements of an NCI request for a proposal (http://rcb.nci.nih.gov/appl/rfp/85080/SOWMain.htm).
10 Code of Federal Regulations: Title 45 Public Welfare Department of Health and Human Services, Part 46:
11 Adapted from the following: 1) Hellen Gelband. A Report on the Sponsors of Cancer Treatment Clinical Trials and their Approval and Monitoring Mechanisms; prepared for the National Cancer Policy Board. February, 1999; 2) NIH scientific review group evaluations of clinical protocols: (http://grants.nih.gov/grants/guide/notice-files/not97-010.html); and 3) NHLBI guidelines for submission of investigator initiated clinical protocols: (http://www.nhlbi.nih.gov/funding/policies/clinical.htm).
