Proposed Guidance Document: Study Protocols That Use Real-world Data

Medtronic is the world's leading medical technology company, specializing in implantable and interventional therapies that alleviate pain, restore health, and extend life. We are committed to the continual research and development necessary to produce innovative therapies that improve health outcomes for all patients, including Medicare beneficiaries.

Medtronic has a strong history of working with CMS to generate meaningful evidence under the Coverage with Evidence Development (CED) framework, for technologies such as transcatheter aortic valve repair and leadless pacemakers. Across these programs, we have experienced different approaches to the design and execution of CED studies, including the novel use of real-world data such as Medicare claims. It is this history and experience that inform our comments on the Proposed Guidance Document: Studies That Use Real-World Data.

Medtronic commends CMS for publishing the guidance document “to encourage innovation and accelerate beneficiary access to new items and services,” and we are supportive of CMS’ proposed HARPER+ study protocol template for studies that rely on real-world data sources for NCDs with CED. Specifically, we appreciate CMS incrementally building on the HARPER template, which is well known and has been widely adopted by researchers conducting real-world evidence (RWE) studies. We view a standardized protocol template to be a beneficial starting point that will drive consistency in design, transparency, and review by CMS. Just as CMS modified the HARPER template to better align with CMS’ needs and expectations for CED studies, we expect there will be idiosyncrasies across studies that will necessitate adaptations, modifications, or amendments on a study-specific basis, and look forward to working collaboratively with CMS on all future CED studies.

Based on our past experiences designing and conducting RWE studies, including CED studies and using the HARPER template, we have two comments for CMS’ consideration regarding the proposed template structure. First, HARPER+ adds a “Statistical Analysis Plan” (SAP) as section 7.5.1. This addition is redundant given the study design elements described in the other sections. For example, the criteria for selecting data for analysis is captured in section 7.4, and section 7.5 captures all the details that would be in an analysis plan. Specifically, the SAP asks for details of the statistical methods and analysis procedures which are included in the original HARPER template in Table 10 for primary, secondary, and sensitivity analyses. Moreover, moving the original section 7.5.1 “Context and rationale” to section 7.2 will likely be confusing to reviewers as it puts the intuitive/conceptual design discussion after the SAP, which should detail how that design would be implemented technically. We encourage CMS to revert to the original HARPER format for section 7.5 to avoid unnecessary effort and confusion by researchers and reviewers.

Second, we are supportive of clear and consistent documentation of amendments and updates to the study protocol to ensure study transparency and rigor. We recommend that for clarity, CMS provide some examples of what would constitute a “major” versus a “minor” protocol update in Section 3, Amendments and Updates. For example, in Section 6, the explanation of Table 3 briefly describes primary and ancillary outcomes. Similar instructive text in Section 3 such as “major updates include changes in data source(s) or changes to the pre-defined primary or ancillary outcomes of interest; minor updates to include modification of operational definitions of variables” would be helpful to researchers and reviewers. We also recommend adding a column to Table 1 designating amendments as major or minor. Finally, for amendments requiring formal resubmission and review by CMS, we would further encourage CMS to clarify whether protocol amendments are permitted to become effective at the time of resubmission to CMS or at the time of affirmative response/approval from CMS. If affirmative approval is required, we request that CMS specify a timeframe (e.g., no later than 60 days from submission) for their review and response.

Conclusion: Medtronic strongly supports the use of CED as a mechanism to support coverage, and we appreciate CMS’ efforts in recent years through comment solicitations such as this to improve transparency and predictability under the CED framework. We look forward to continuing to work with the Agency to produce meaningful evidence evaluating the effectiveness of our devices in the Medicare population, as well as to support programs that appropriately balance expedited patient access with evidence generation.

Thank you for your consideration of our comments.

Less

Novitas Solutions, Inc. (Novitas) and First Coast Service Options, Inc. (First Coast) Endorsement of Proposed Guidance on Real-World Data (RWD) Study Protocols for Medicare Coverage Decisions

Novitas and First Coast appreciate the opportunity to provide comments and support the proposed guidance on study protocols that utilize real-world data (RWD). Integrating RWD into the broader body of evidence that informs policy is an important step to enhance clinical research contribution to evidence-based coverage decisions. RWD, data derived from actual patient care (such as electronic health records, claims, and registries), provides valuable insights into the safety and effectiveness of health interventions in routine practice, often at a significantly lower cost than traditional randomized trials. These real-world studies capture outcomes from varied patient populations over extended periods, offering evidence on long-term safety (including rare adverse events) and real-world effectiveness that complement the narrower scope of randomized clinical trials.¹ As Medicare serves a predominantly older and medically complex population, leveraging RWD is especially important to address evidence gaps about how new treatments perform in everyday use for our beneficiaries.² The proposed guidance on RWD study protocols is an opportunity to systematically harness this type of evidence for Medicare’s coverage determinations, and its adoption is both timely and strategic.

Enhancing Standardization and Transparency: This new guidance appears to be designed to standardize how RWD studies are planned and reported, thereby improving their quality and usability for coverage decisions. It introduces a uniform protocol template (the HARPER+ template) that sponsors would use when developing RWD-based studies under Coverage with Evidence Development. This standardized template ensures that all critical elements of a study, from patient population definitions to endpoints and analysis plans, are addressed in a consistent, rigorous manner.³ By reducing the wide variability in past RWD study proposals, the guidance will make study protocols more transparent, complete, and easier to evaluate. Novitas and First Coast are regularly asked, “…what evidence do you need to support coverage of our product.” In fact, CMS has previously emphasized that this effort is part of a broader modernization initiative to provide a “more transparent and predictable evidence-generation framework” for Medicare coverage. The clear structure and expectations set forth in the guidance will not only streamline internal review, but also signal to external stakeholders (e.g., healthcare providers, manufacturers, and patients) that Medicare-supported RWD studies adhere to high standards. Notably, we agree that using a standardized RWD protocol could “reinforce public confidence” that Medicare is applying rigorous, uniform criteria to RWD-based evidence. In sum, adopting the guidance will institutionalize transparency and consistency in how real-world evidence is generated and assessed for coverage decisions.

Improving Generalizability and Analytic Clarity: A core strength of RWD is its relevance to actual patient populations, and the proposed guidance smartly capitalizes on this by requiring protocols to address generalizability upfront. The template explicitly includes sections prompting investigators to discuss whether and how their study results are likely to generalize to Medicare beneficiaries.⁴ By doing so, Medicare will be assured that approved studies aren’t just internally valid, but also externally valid for the people it covers, an important consideration given that clinical trials often exclude elderly or medically complex patients. Alongside generalizability, the guidance markedly improves clarity for reviewers. It calls for detailed data analysis so that reviewers may ascertain whether it can answer the research question. Novitas and First Coast strongly support requirement of such details including specification of hypotheses, study design, statistical models, methods to control bias (e.g., confounding adjustment method, handling of missing data), and planned subgroup analyses.⁴ The section titled “Context and Rationale for Covariates” enhances analytical clarity by elucidating potential confounders and effect modifiers that are being addressed, as well as their possible impact on results. Novitas and First Coast support a more comprehensive approach in studies, emphasizing the importance of considering patient characteristics rather than merely looking at numbers. Additionally, the framework necessitates the inclusion of pre-defined effect size information, which will aid in removing the uncertainty currently associated with the evaluation of sample sizes in studies. Requiring this level of detail upfront means that both CMS reviewers and the broader research community can clearly understand how the study will be conducted and how data will be interpreted. This analytic transparency helps preempt ambiguities or methodological weaknesses, resulting in studies that are easier to appraise and reproduce. In effect, Medicare’s guidance sets the bar for scientific rigor, ensuring that real-world evidence is not “real-world” in name only, but truly meets the quality needed to inform national coverage decisions.

Anticipated Benefits of Adopting the RWD Guidance: Embracing the proposed RWD study protocol guidance will yield multiple benefits for Medicare and its stakeholders. In particular, we believe it will:

• Align Evidence with Clinical Practice: RWD is collected during routine care, reflecting how treatments perform in real clinical settings and across varied patient groups. Utilizing RWD means Medicare coverage decisions can be informed by evidence that mirrors everyday practice, improving relevance and patient-centeredness. Real-world evidence can fill in knowledge gaps left by tightly controlled trials, offering a more complete picture of treatment effects in the Medicare population.¹ This alignment with actual practice can lead to coverage policies that better anticipate real-world outcomes and variations in clinical practice and patient response.
• Improve Reproducibility and Quality through use of an established evidence-based framework: The HARPER+ template (built upon the HARmonized Protocol Template to Enhance Reproducibility) ensures that studies are designed and documented in a consistent format, which facilitates reproducibility of research findings. Adhering to standard definitions and protocol structures is “important for interoperability… [and] future reproducibility of the study.”⁴ In other words, other researchers and stakeholders can more easily replicate or validate studies conducted under this framework, strengthening the evidence base over time. Moreover, a standardized protocol format will improve the quality of submissions by ensuring no critical element is overlooked, thereby raising the overall standard of evidence that Medicare relies upon.
• Enable Long-Term and Post-Market Assessments: RWD studies are well-suited to evaluate long-term safety and effectiveness, as they can track outcomes over years and include large patient populations. By encouraging RWD protocols, Medicare can gain insights into how therapies perform long-term, well beyond the typical duration of pre-market trials. This is especially valuable for detecting rare or late-arising adverse events and for understanding sustained benefits (or diminishing returns) of treatments in the real world.¹ Adopting the guidance will facilitate studies that monitor long-term efficacy and safety in the Medicare population, informing whether initial coverage decisions remain appropriate as more data emerges. In the bigger picture, this contributes to continuous learning in our healthcare system, where coverage can adapt based on evolving evidence of real-world performance.
• Foster Efficient, Evidence-Based Decision-Making: With standardized RWD protocols, CMS’s review process will become more efficient and predictable. Sponsors who follow the guidance’s template are less likely to encounter delays due to missing information or protocol revisions, which means faster access to promising technologies for beneficiaries when coverage is granted. Novitas and First Coast agree that using the template once finalized could help avoid delays in Medicare coverage approvals. Over time, as more RWD studies adhere to these guidelines, Medicare will build a robust repository of high-quality real-world evidence, enabling more confident coverage decisions that keep pace with innovation while safeguarding patient outcomes. This proactive approach aligns Medicare with the scientific community’s emphasis on real-world evidence and with parallel efforts by regulators like the FDA to incorporate RWD in decision-making.

Risk of Bias Considerations in RWD Studies
Novitas and First Coasts’ main concern with observational RWD studies is the potential for bias. Medicare’s guidance wisely encourages protocol developers to confront these biases head-on. By mitigating selection bias, information bias, and confounders, RWD studies can achieve high internal validity (accuracy of results) and instill confidence that findings are reliable for decision-making. We recommend the expansion of mitigating strategies for selected types of biases below:

• Selection Bias: Selection bias occurs when systematic differences in who gets included in a study skew the results. For example, if certain patients (e.g., healthier or more engaged individuals) are more likely to be selected or captured in the dataset, the study sample may not represent the true population. This distortion can lead to conclusions that cannot be generalized to the overall population. To minimize selection bias, RWD protocols should define clear inclusion and exclusion criteria from the outset, ensuring the study population is appropriately scoped and comparable across groups. CMS’ draft guidance underscores that well-specified inclusion/exclusion criteria are “critical for ensuring that the study population is properly scoped so that the observed effects can be attributed to the exposure… (internal validity)” and for assessing “whether the results are likely to generalize to the target population” (external validity).⁴ In practice, researchers can mitigate selection biases by using propensity score matching to simulate randomization. Pairwise propensity score matching pairs individuals in different treatment groups based on similar baseline characteristics, balancing known factors across cohorts. This technique can effectively compensate for the lack of random assignment in RWD studies by creating a more equivalent comparison group.⁵ When appropriate, other statistical matching or weighting methods (e.g., inverse probability weighting) and multivariable regression modeling should be employed to adjust for differences in patient characteristics. By accounting for demographic and clinical covariates in a multivariable model, analysts can isolate the effect of the treatment or intervention from confounding factors. These approaches greatly reduce bias from non-random treatment assignment, strengthening the study’s internal validity. The bottom line is that selection biases are addressable with careful design and analysis, they need not entirely disqualify RWD findings if properly handled.
• Information Bias: Information bias (a form of measurement bias) arises from inaccurate or inconsistent data collection between groups. In real-world studies, data are often drawn from electronic health records (EHRs), claims, or registries not originally intended for research, which can lead to missing or misclassified information. Examples include a clinician documenting outcomes more diligently for one group of patients than another (observer bias) or patients failing to recall past treatments accurately (recall bias). Such errors can distort the apparent effectiveness or safety outcomes. Standardized data collection procedures are paramount to combat information bias. Researchers should use uniform definitions and instruments for key variables across all sites and participants. Schanbacher et al. provide guidance which states that, “continuous tracking of the patient and standardized, quantifiable recording procedures in real time can reduce this bias,” and implementing IT platforms with automated data integrity checks can improve data quality.⁶ In practice, this means establishing common data elements and timelines for outcome assessment, training personnel on consistent abstraction methods, and utilizing electronic data capture forms that minimize free-text variability. By ensuring that all study data are collected in a consistent manner, investigators prevent differential mismeasurement between comparison groups. High-quality, standardized data not only diminish information bias but also enhance the credibility of RWD evidence in the eyes of decision-makers.
• Confounding Bias: Confounding occurs when an uncontrolled third variable influences both the treatment exposure and the outcome, misdirecting the apparent effect of the treatment. In other words, the relationship between an intervention and outcome is “mixed up” with the effect of another factor. For example, suppose patients who smoke tend to receive a particular therapy, and smoking also independently worsens health outcomes; if smoking status is not accounted for, the study may underestimate or overestimate the treatment’s true effect.⁶ Confounding bias is a primary challenge in non-randomized studies, but it can be rigorously addressed through study design and analysis. First, RWD protocols should pre-specify known confounders (e.g., age, comorbidities, disease severity) based on clinical expertise and prior evidence, ensuring these factors are measured. Then, statistical techniques can be applied to adjust for these confounders. Multivariable regression modeling is a standard approach where all known confounding variables are included in the regression to statistically adjust their influence on outcomes. This helps isolate the independent effect of the treatment. Additionally, as noted above, propensity score methods are powerful tools against confounding bias, by matching or weighting patients on a propensity score (predicted probability of receiving the treatment given their covariates), the treatment and control groups become more comparable on observed confounders.⁶ It is also good practice to perform sensitivity analyses to gauge the impact of any unmeasured confounders. For instance, analysts can vary assumptions about an unknown confounder or use E-value calculations to estimate how strong an unmeasured confounder would have to be to negate the results. Such exercises demonstrate the robustness of findings. Experts emphasize that while we must “actively deal with” potential biases in RWD, these biases “can be addressed quantitatively,” meaning careful analysis can account for them, and they should not lead to the automatic dismissal of RWD studies.⁶ If a RWD study shows that its results hold even under potential hidden confounding biases, that is compelling evidence of validity.⁶ In fact, when an RWD study’s findings are robust to large hidden confounding biases, it can deliver valid insights not obtainable in randomized trials (for example, data on broader patient groups or longer-term outcomes).⁶

By incorporating the above bias mitigation strategies into RWD study protocols, clinical researchers can ensure that studies submitted under the new guidance meet a high bar for scientific rigor. Rather than viewing bias as an intractable flaw, the guidance treats it as a manageable risk, one that can be minimized through design and analysis safeguards. This proactive approach will yield RWD evidence with credibility approaching that of traditional randomized controlled trials. In sum, robust bias control strengthens the argument for relying on RWD in coverage decisions, as it signals that Medicare will only consider real-world studies that have been carefully designed to produce valid, trustworthy results.

Recruitment Methodology, Considerations for Varied Representation
In addition to controlling bias, the success of RWD studies’ contribution of actionable data depends on who is being studied. Ensuring that real-world evidence reflects the heterogeneity of Medicare beneficiaries is crucial. Medicare’s population spans different ages, races, ethnicities, sexes, socioeconomic backgrounds, and geographic regions. If a study’s participants do not adequately represent this variety, the findings may not generalize to all beneficiaries or could inadvertently ignore how certain subgroups respond to an intervention. CMS’s proposed guidance explicitly calls out this concern, requiring protocol authors to demonstrate that their study results can be expected to apply to the Medicare population at large.⁴ To meet this expectation, RWD study protocols should adopt recruitment and sampling strategies that maximize representativeness and minimize selection bias in assembling the study cohort. Novitas and First Coast recommend the following key best practices:

• Stratified Sampling for Representativeness: Rather than drawing the study sample haphazardly or solely from convenient sources, researchers should use stratified sampling techniques to ensure all relevant patient subgroups are represented. Stratified sampling involves dividing the target population into important categories (for example, stratifying by age group, sex, race, or region) and then sampling from each category so that the proportion in the study reflects the real-world population. This approach prevents under-representation of minority groups or over-representation of any single subgroup. By design, it guarantees heterogeneity in the study cohort. Regulatory guidance endorses such methods; the FDA, for instance, recommends using justified sampling strategies (including random sampling or stratified sampling) to achieve a study population that is adequately large and representative.⁷ In a Medicare context, a stratified approach might ensure that enough older seniors (80+), racial minorities, rural residents, etc., are included so that analyses can be done within these groups and results reflect their outcomes. Oversampling of smaller subpopulations can be employed as needed (with appropriate weighting in analysis) to acquire robust data on groups that are often overlooked. The goal is a study cohort that mirrors the demographic and clinical characteristics of the broader Medicare beneficiary pool, lending confidence that coverage decisions based on the study will benefit all eligible patients.
• Automated and Objective Enrollment Criteria: One powerful advantage of RWD (especially EHR-based studies) is the ability to leverage real-time clinical data to identify eligible patients without relying on subjective referrals. Best practice is to implement automated enrollment criteria using the electronic data source itself. For example, an EHR query can be programmed to find all patients with Condition X who received Treatment Y between dates A and B and meet inclusion criteria Z. This automation ensures that every patient meeting the pre-specified criteria is included or at least considered for inclusion, removing potential human biases in recruitment. By contrast, if enrollment were left purely to physician referral or patient volunteering, certain patients might be preferentially selected (or omitted) due to conscious or unconscious biases. An algorithmic approach treats all records equally and can systematically scan large databases to enroll a broad patient set. Efficiency and consistency are additional benefits – automated recruitment can rapidly assemble a cohort from multiple clinics or hospitals using the same logic, which standardizes the selection process. Of course, criteria must be carefully defined (and perhaps validated with a manual review sample) to ensure they correctly identify appropriate patients. When done well, this method yields a more inclusive and unbiased sample of real-world patients, as it draws from routine care data rather than a narrowly controlled experiment. It is an excellent way to maximize the yield of varied participants while adhering to the study’s eligibility definitions.
• Minimizing Selection Bias in EHR-Based Studies: Studies that use electronic health record data must contend with the fact that not all patient experiences are captured equally. An oft-cited issue is that a single health system’s EHR may not reflect patients who seek care elsewhere, and patients who frequently visit healthcare providers naturally have more recorded data than those who do not. Thus, RWD cohorts can inadvertently tilt toward patients who are more engaged with the healthcare system, leaving out those who might be less active or who receive care in underrepresented settings. This is a form of selection bias inherent to EHR data, “the population captured in the EHR rarely fully represents the population that is the source.”⁸ To mitigate this, RWD protocols should encourage broad data sourcing and inclusive cohort definitions. Practically, this means that wherever possible, studies should draw from multiple data sources or sites. For example, a study could combine data from a nationwide claims database with EHR data from varied health systems or use a clinical registry that includes both community clinics and academic medical centers. Including a mix of large urban hospitals and smaller rural hospitals can ensure regional variety. If using a single integrated dataset, researchers should examine and report the dataset’s demographics against the Medicare population – if certain groups (e.g., very elderly, minority populations, or low-income groups) are underrepresented, this should be acknowledged and, if feasible, remedied by targeted recruitment or supplementation with other data. Another strategy is to apply post-hoc weighting or statistical adjustments so that the analyzed results reflect the broader population distribution. Additionally, enrollment period and follow-up should be sufficiently long to capture patients with infrequent healthcare visits (to avoid excluding those who don’t show up in data in a short window). Stratified analyses can further check whether results are consistent across subgroups; if not, it may indicate a need for more targeted data collection for those subgroups in future studies. The overarching principle is to minimize any systematic differences between those in the study and those who are not (but should have been) represented. By actively addressing this, RWD studies can produce findings that truly reflect real-world patient outcomes across the spectrum. This broad representativeness is vital for Medicare – it ensures that a coverage policy informed by the study will likely hold true for the millions of heterogenous beneficiaries who rely on Medicare.

By following these recruitment best practices and establishing control group standards, RWD studies can enhance their external validity (generalizability). A study outcome that holds across a heterogenous, representative sample bolsters confidence that the intervention will work for real Medicare patients, not just an uncharacteristic subset. Emphasizing heterogeneity and representativeness in protocols also aligns with ethical and equity considerations. Medicare, as a public payer, has a responsibility to consider evidence that is relevant to all sub-populations it serves. Encouraging stratified sampling, automated broad-based enrollment, and avoidance of narrow inclusion of convenience samples shows due diligence in this regard. In the end, a RWD study that is both internally valid (through bias mitigation) and externally valid (through inclusive recruitment) provides an exceptionally strong foundation for coverage decisions. It means Medicare can confidently rely on the study’s results to determine whether an item or service is “reasonable and necessary” for its beneficiaries, knowing the evidence is scientifically sound and widely applicable.

Limitations of the Guidance
Novitas and First Coast wish to address two important concerns regarding the new guidance. Firstly, the guidance may impose challenges on smaller businesses and organizations that lack the resources to comply with the stringent requirements of the framework. Furthermore, we believe that the data management guidelines in Section 7.7 should be elaborated to include a more detailed explanation of data standardization and cleaning methods.

Conclusion
Novitas and First Coast believe this guidance is a forward-thinking initiative that marries scientific rigor with practical relevance. By explicitly addressing the methodological concerns of bias and representativeness, the guidance ensures that any real-world evidence used in coverage determinations will adhere to high standards. As detailed above, the adoption of bias mitigation strategies (like propensity score matching, standardized data collection, and rigorous multivariable adjustment) and robust recruitment methodologies (stratified sampling and comprehensive, unbiased enrollment of participants) will significantly enhance the quality of RWD studies. This means that Medicare can confidently utilize RWD to expand coverage for new treatments or technologies without compromising on scientific rigor. Importantly, this approach is fully in line with CMS’s broader goal of creating a “transparent and predictable evidence-generation framework” for coverage policy. Rather than relying solely on traditional trials that often exclude the very populations Medicare covers, RWD studies, when properly designed, offer a complementary source of evidence that is more generalizable to Medicare patients’ everyday clinical experience. In conclusion, Novitas and First Coast support the proposed RWD study protocol guidance.

Bibliography
1. Blonde L, Khunti K, Harris SB, Meizinger C, Skolnik NS. Interpretation and Impact of Real-World Clinical Data for the Practicing Clinician. Adv Ther. Nov 2018;35(11):1763-1774. doi:10.1007/s12325-018-0805-y
2. Tunis SR, Hatfield GM, Patel PB, Neumann PJ. Medicare Coverage of Medical Technologies In a New Era. Health Affairs Forefront. March 6, 2025. DOI: 10.1377/forefront.20250303.214272
3. Wang SV, Pottegård A, Crown W, Arlett P, Ashcroft DM, Benchimol EI, Berger ML, Crane G, Goettsch W, Hua W, Kabadi S, Kern DM, Kurz X, Langan S, Nonaka T, Orsini L, Perez-Gutthann S, Pinheiro S, Pratt N, Schneeweiss S, Toussi M, Williams RJ. HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force. Value Health. 2022 Oct;25(10):1663-1672. doi: 10.1016/j.jval.2022.09.001. PMID: 36241338.
4. Proposed Guidance Document: Study Protocols That Use Real-world Data. https://www.cms.gov/medicare-coverage-database/view/medicare-coverage-document.aspx?mcdid=39. Accessed March 9, 2025.
5. Rosenbaum PR, Rubin DB. The Central Role of the Propensity Score in Observational Studies for Causal Effects. In: Matched Sampling for Causal Effects. Cambridge University Press; 2006:170-184.
6. Schanbacher P, Paddock S, Braune S, van Hoevell P, Bergmann A, et al. (2024) Avoiding the Pitfalls of Confounding: A Practical Guide to Mitigating Bias in Real-World Data (RWD) Studies. J Clin Res Med Volume 7(3): 1–6. DOI: 10.31038/JCRM.2024731.
7. Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/real-world-data-assessing-electronic-health-records-and-medical-claims-data-support-regulatory. Accessed March 6, 2025.
8. Al-Sahab B, Leviton A, Loddenkemper T, Paneth N, Zhang B. Biases in Electronic Health Records Data for Generating Real-World Evidence: An Overview. J Healthc Inform Res. 2023 Nov 14;8(1):121-139. doi: 10.1007/s41666-023-00153-2. PMID: 38273982; PMCID: PMC10805748.

Less

Chiropractors are required to show subluxation in one of two ways. exam or x-ray. I feel if x-ray is listed as one of the way to show subluxations then it should be covered by Medicare. If Medicare says that an x-ray IS shown to show subluxation and is used in court to show subluxation, then I do not understand why it is not a covered service. (That is my personal feeling on this matter.) (Do you know how many times a Chiropractor not only finds subluxations and not only that they see broke bones and have to refer a patient out. sometimes they see other things that means they have to quickly refer the patient over to the ER, like objects, or masses. Those x-rays can saves lives, they also show Chiropractors what can not be seen on felt on an exam. Like Plaquing and Cancer. Can you imagine the poor chiropractor who could have done a simple x-ray to show subluxation as it is an acceptable way to show subluxation yet didn't only to adjust to discover the patient had cancer of the bones? (Again these are just my thoughts on the subject.)

Less

March 18, 2025

Submitted via Public Comment at https://www.cms.gov/medicare-coverage-database/view/medicare-coverage-document.aspx?mcdid=39

Tamara Syrek Jensen
Director, Coverage and Analysis Group
Centers for Medicare & Medicaid Services
Mailstop: S3-02-01
7500 Security Blvd
Baltimore, MD. 21244

Re: Johnson & Johnson Comments in Response to CMS Proposed Guidance Document: Study Protocols That Use Real-world Data

Dear Ms. Syrek Jensen,

Johnson & Johnson (“J&J”) is pleased to provide comments in response to the Centers for Medicare & Medicaid Services (CMS) Proposed Guidance Document: Study Protocols That Use Real-world Data. At J&J, we are driven by a passion to achieve the best version of health for everyone, everywhere, for as long as possible. In the next decade, we will see more transformation in health than in the past century – and we are ready to lead the way. Focusing exclusively on transformational healthcare innovation allows us to move with purpose and speed to tackle the world’s toughest health challenges. Innovating across the full spectrum of healthcare solutions puts us in a unique position today to deliver tomorrow's breakthroughs to our current and future patients, including Medicare, Medicaid, and Marketplace beneficiaries. Our strength in both biology and medical technology means we are accelerating advances in care – from cell therapy to AI-assisted robotic surgery. We are using our wide range of expertise to address healthcare challenges that can be tackled by medical technology and innovative medicine, such as cancer, cardiovascular disease, and eye health. Our reach and depth across a continuum of healthcare and technology solutions give J&J the ability to profoundly impact health for humanity.

At Johnson & Johnson, we are leaders in real-world evidence (RWE) innovation. We have experience in developing and using real-world data (RWD) strategies to evaluate and acquire new datasets to fill evidence gaps, particularly for regulatory purposes. Several examples come from our significant collaboration with the Food and Drug Administration’s (FDA) National Evaluation System for Health Technology’s Coordinating Center (NESTcc), where five out of eight funded NEST test cases came from J&J. Through this work, we incorporated RWE to expand a labeled indication, achieve more specific labeling, and for post-market surveillance. NEST and J&J continue to partner through “NEST Mark,” a process that structures the assessment and use of RWD for pre-market regulatory submissions for medical devices. , We have also published a large number of RWE studies for the payer community, as well as providers and patients, to further analyze the value proposition and outcomes associated with our devices (see endnotes for several examples of recent J&J real-world research publications).

We support CMS’ effort to create a framework for more predictable and transparent evidence development in the context of Medicare coverage. We are glad to note CMS’ interest in considering a broad range of study designs when a National Coverage Determination (NCD) requires Coverage with Evidence Development (CED), including study designs that rely on RWD. In general, J&J has supported the CED pathway for innovative medical devices when CMS believes that additional clinical evidence is needed and when the alternative is non-coverage. CED using RWD within a fit-for-purpose study paradigm will also be important for the Transitional Coverage for Emerging Technologies (TCET) pathway. Because a clear understanding of appropriate study designs for coverage is imperative for evidence development and eventual full coverage of medical devices, we further support the concept of a standardized template for RWD study protocols to improve efficiency for approval of CED studies. The HARPER+ framework is an excellent starting point.

Below we offer general comments on the proposed framework, as well as specific comments on the draft HARPER+ protocol template.

General Comments

Current definitions of real-world data typically include data from registries and surveys, as well as hospital databases and claims. It is important to note, however, that the approach for evidence generation can differ based on the data source and templates need to reflect these differences. HARPER+ provides an excellent basis for research but may be further tailored to address the data types utilized for evidence generation.

It is also important to keep in mind that medical device research differs from pharmaceutical research – device research can be more prone to confounding by indication, surgical skills and operator effects that may be associated with outcomes, and databases do not always capture the exact device types utilized for medical procedures. With this understanding in mind, while HARPER+ provides a comprehensive framework for protocol development, it would be helpful for researchers to have access to streamlined templates for each section of the protocol, specifying exactly how these would be completed in the context of medical device research. Examples of completed HARPER+-based protocols and streamlined templates of some of the fields related to this framework, would be helpful to all researchers.

As CMS moves to finalize this guidance document on studies that use RWD, we urge the Agency to consider providing examples of completed protocols for medical devices, as the current examples contain a number of “drug” fields. CMS should also consider making approved protocols available on the CMS website or another public website to help facilitate greater understanding of CMS’ expectations and assist companies with development of future study protocols. Finally, we encourage CMS to incorporate flexibility, and to follow an approach that encourages patient access to new technologies and reduces excessive burden on manufacturers and providers.

We encourage CMS to recognize the inherent difference between data collected as RWD and data developed through randomized control trials (RCT), and to incorporate the unique advantages of RWD into its evidence evaluation. We recommend that CMS partner with evidence generation experts such as J&J to develop a framework to determine fit-for-purpose data for various contexts. In particular, such a framework could build on the Data Quality Framework that J&J developed in partnership with NEST; expand the guidelines to better leverage the advantages of RWD (e.g., large-scale, representativeness, cost-effectiveness in design and execution); and incorporate guidelines on how to mitigate any data limitations in study design and results interpretation and applications.

Comments Specific to HARPER+ Template Subsections:

Section 7.4.2. Context and Rationale for Outcome(s) of Interest & 7.4.4 Context and Rationale for Covariates

Under this section, CMS provides detailed guidance on the operational definitions of outcomes. Specifically, the operational definitions include measurement characteristics/validation (e.g., PPV, sensitivity, specificity). We agree with CMS that outcome measurement validity is critical for RWD studies. Beyond this guideline, we recommend that CMS take further steps to enhance the validation of measuring outcomes in RWD.

First, we recommend that CMS provide more funding opportunities for outcome validation studies. Clinical outcomes related to medical devices are often first observed as radiographic results. As such, it is important to support validation studies to establish relationship between radiographic results and data elements (e.g., ICD diagnosis and procedure codes, and CPT procedure codes) measured in RWD.

Second, for the key RWD elements for outcome and covariate measurement, we recommend that CMS consider a process to evaluate and incorporate them into CMS programs (e.g., risk adjustment for reimbursement or quality reporting programs) to promote more complete documentation from healthcare providers and reporting into RWD. For example, nonunion and malunion after fractures is related to medical devices and procedures affecting many Medicare beneficiaries. Evaluating how ICD-CM diagnosis can be incorporated into CMS reimbursement or quality reporting programs will promote more accurate and complete reporting of these outcomes in RWD.

Section 7.6. Data Sources

In this section, CMS requests information about the data sources to understand the reasons for selection, as well as their strengths and limitations, and notes that this information will enable CMS to assess whether the data is fit-for-purpose to answer the research question being asked. Understanding the limitations of a given data set should be considered, but it is important to understand the intended purpose of the data and how it can be used to answer the clinical questions being addressed in the study. We encourage CMS to implement a process under which companies and CMS can have preliminary discussions, prior to the development of an entire protocol, in order to align on the data sources. Establishing this shared understanding of the data sources will be critical before pursuing further research.

We appreciate the opportunity to provide comments on the Proposed Guidance Document: Study Protocols That Use Real-world Data. J&J looks forward to continued partnership with CMS as the Agency develops guidance on evidence generation for coverage purposes. Please contact me or other members of the J&J Payment and Delivery Team if there is any further information we can provide or if the Agency has additional questions.

Sincerely,
Chandra N. Branham

Chandra N. Branham, JD
Senior Director, Payment & Delivery Policy, Health Policy
cbranham@its.jnj.com

Endnotes: Examples of Recent J&J RWE Research

1: Frankenberger EA, Resnic FS, Ssemaganda H, Robbins S, Dunbar MR, Coplan P, Zhang S, Bruno C, Maltenfort M, Benedetti JB, Matheny ME, Ghogawala Z. Evaluation of intervertebral body implant performance using active surveillance of electronic health records. BMJ Surg Interv Health Technol. 2022 Jul 8;4(1):e000125. doi: 10.1136/bmjsit-2021-000125. Erratum in: BMJ Surg Interv Health Technol. 2022 Aug 5;4(1):e000125corr1. doi:10.1136/bmjsit-2021-000125corr1. Erratum in: BMJ Surg Interv Health Technol.2022 Oct 19;4(1):e000125corr2. doi: 10.1136/bmjsit-2021-000125corr2. PMID:35909993; PMCID: PMC9272102.

2: Yu Y, Jiang G, Brandt E, Forsyth T, Dhruva SS, Zhang S, Chen J, Noseworthy PA, Doshi AA, Collison-Farr K, Kim D, Ross JS, Coplan PM, Drozda JP Jr. Integrating real-world data to assess cardiac ablation device outcomes in a multicenter study using the OMOP common data model for regulatory decisions: implementation and evaluation. JAMIA Open. 2023 Jan 10;6(1):ooac108. doi: 10.1093/jamiaopen/ooac108. PMID: 36632328; PMCID: PMC9831049.

3: Dhruva SS, Zhang S, Chen J, Noseworthy PA, Doshi AA, Agboola KM, Herrin J, Jiang G, Yu Y, Cafri G, Collison Farr K, Ervin KR, Ross JS, Coplan PM, Drozda JP Jr. Safety and Effectiveness of a Catheter With Contact Force and 6-Hole Irrigation for Ablation of Persistent Atrial Fibrillation in Routine Clinical Practice. JAMA Netw Open. 2022 Aug 1;5(8):e2227134. doi: 10.1001/jamanetworkopen.2022.27134. PMID: 35976649; PMCID: PMC9386540.

4: Dhruva SS, Zhang S, Chen J, Noseworthy PA, Doshi AA, Agboola KM, Herrin J, Jiang G, Yu Y, Cafri G, Farr KC, Mbwana MS, Ross JS, Coplan PM, Drozda JP Jr. Using real-world data from health systems to evaluate the safety and effectiveness of a catheter to treat ischemic ventricular tachycardia. J Interv Card Electrophysiol. 2023 Nov;66(8):1817-1825. doi: 10.1007/s10840-023-01496-x. Epub 2023 Feb 4. PMID: 36738387.

5: Peng M, Doshi A, Amos Y, Tsoref L, Amit M, Yungher D, Khanna R, Coplan PM. Does radiofrequency ablation procedural data improve the accuracy of identifying atrial fibrillation recurrence? PLoS One. 2024 Apr 5;19(4):e0300309. doi: 10.1371/journal.pone.0300309. PMID: 38578781; PMCID: PMC10997092.

6: Coplan P, Doshi A, Peng M, Amos Y, Amit M, Yungher D, Khanna R, Tsoref L. Predictive utility of the impedance drop on AF recurrence using digital intraprocedural data linked to electronic health record data. Heart Rhythm O2. 2024 Feb 2;5(3):174-181. doi: 10.1016/j.hroo.2024.01.006. PMID: 38560375; PMCID: PMC10980921.

7: Fortin SP, Spychaj K, Tomaszewski J, Grebeck H, Yalla R, Coplan PM, Zhang S. Clinical Outcomes Among Patients Undergoing Open Abdominal or Orthopedic Surgery with Wound Closure Incorporating Triclosan-Coated Barbed Sutures: A Multi- Institutional, Retrospective Database Study. Med Devices (Auckl). 2025 Mar 6;18:161-176. doi: 10.2147/MDER.S508169. PMID: 40071132; PMCID: PMC11894428.

8: Fortin SP, Petraiuolo W, Cafri G, Scapini G, Agarwal P, Chakke D, Johnston S, Johnson BH, Coplan PM, Zhang S. Comparison of Clinical Outcomes of Gripping Surface Technology Staple Reloads versus Standard Staple Reloads Used with Manual Linear Surgical Staplers. Med Devices (Auckl). 2022 Dec 15;15:385-399. doi: 10.2147/MDER.S393881. PMID: 36545363; PMCID: PMC9762253.

Less

March 18, 2025

Tamara Syrek Jenson
Director, Coverage and Analysis Group
Centers for Medicare & Medicaid Services
7500 Security Blvd
Baltimore, MD 21244

Re: Proposed Guidance Document: Study Protocols That Use Real-world Data

Dear Ms. Syrek Jenson,

On behalf of the MedTech Association (AdvaMed), we appreciate the opportunity to submit comments on the Proposed Guidance Document: Study Protocols That Use Real-world Data (RWD). AdvaMed and CMS share a common goal – the establishment of a clear and expeditious coverage process, based on scientifically sound clinical evidence with appropriate safeguards, for emerging technologies that will benefit Medicare-eligible patients, including the 54 percent of Medicare-eligible patients enrolled in Medicare Advantage organizations in 2023(1).

AdvaMed member companies produce the medical devices, diagnostic products, and health information systems that are transforming health care through earlier disease detection, less invasive procedures, and more effective treatments. AdvaMed members range from the largest to the smallest medical technology innovators and companies. We are committed to ensuring patient access to lifesaving and life-enhancing devices and other advanced medical technologies in the most appropriate settings.

Life-changing medical technologies allow millions of people to live longer, healthier, and more productive lives. From 1980-2019, five years were added to the U.S. life expectancy because of advancements in medical technology. Further, since 1980, there has been a 38% decrease in the number of patient-days spent in hospitals thanks to medical technologies. Innovative technologies improve quality of life, reduce time spent in hospitals, and help loved ones beat life-threatening diseases and conditions. Unfortunately, outdated Medicare laws and regulations often create unnecessary barriers to beneficiary access to many technological innovations.

Providing guidance on the use of RWD is one step towards modernizing CMS’ National Coverage Determination (NCD) process. The medical device industry supports evidence-based coverage policy, and we appreciate the challenges faced by CMS in trying to improve this process. We appreciate CMS’ ongoing efforts to improve the transparency and predictability of evidence generation, and are pleased to see additional guidance on RWD, which follow previous guidance documents on Coverage with Evidence Development (2) and CMS’ National Coverage Analysis Evidence Review (3).

General Comments

AdvaMed is supportive of CMS’ efforts to streamline and improve processes that allow Medicare beneficiaries access to medical technologies. When the available evidence is insufficient to demonstrate that the items and services are reasonable and necessary, coverage with evidence development (CED) has been used to support evidence development for certain items and services that are likely to show benefit for the Medicare population. AdvaMed supports CED when CMS determines additional clinical evidence is needed, including the use of RWD, as it enables evidence generation in the real-world, outside the context of a randomized clinical trial (RCT).

The proposed guidance document provides a standardized RWD study protocol template, called the HARPER+. The template is to be used by manufacturers or other sponsors when creating study protocols using RWD. CMS has done an excellent job integrating the existing HARPER framework into HARPER+. By building on solid foundation, this standardization will streamline the process of creating and submitting real-world evidence (RWE) study protocols to facilitate an efficient review process by CMS and ensuring that companies submit the required information uniformly. We agree with CMS’ objective that this guidance will lead to more efficient approval of proposed CED studies by reducing the need for multiple rounds of review.

RWD may be able to provide more information about subpopulations because of large sample sizes, and the fact that subpopulations are not excluded as part of inclusion or exclusion criteria typically required for RCTs. Furthermore, AdvaMed supports the use of a standardized template, which we believe has the ability to enable transparency, helping organize and align expectations between sponsors and CMS. It also allows for consistency across submissions, which helps sponsors understand what protocol elements are important and which enables a more efficient and streamlined review by the Agency. However, in some cases, submissions may need a level of flexibility based on the technology and the type of evidence generated. Therefore, we encourage CMS to work with submitters to adapt the template on a case-by-case basis. In that vein, it would also be helpful for CMS to provide guidance on the process to make amendments to study protocols.

As CMS finalizes and implements this guidance, we encourage CMS to offer workshops or webinars to demonstrate how sponsors can utilize the template effectively. CMS should also consider providing additional examples of completed protocols for medical devices, as the examples currently provided contain several “drug” fields. These efforts will help to improve understanding and adherence to the guidelines as well as enhance the usability and effectiveness of the guidance.

RWD Study Template Components

The introduction of the standardized RWD protocol template is a significant step forward as it has the potential to accommodate a wide range of study designs, which will satisfy the CED requirements and expedite the review and approval process. We offer the following recommendations to certain aspects of the template’s components, which we believe will help to improve the template’s effectiveness across all medical device manufacturers regardless of size or stage. While a standardized template can greatly improve the necessary level of detail and content submitted in proposed research protocols, we encourage CMS to maintain flexibility and communication with study sponsors as needs and experience will vary considerably.

7.5 Data Analysis
The section of the guidance document outlines extensive statistical analysis requirements, including detailed statistical analysis plans, multiple testing considerations, and complex interim analysis procedures. However, there is a concern that some early-stage companies may lack in-house analysts (e.g., biostatisticians, economists, data scientists, etc.) to handle such sophisticated analyses. To assist early-stage companies, we recommend that CMS consider providing a list of CMS-recommended statisticians for small businesses meeting Small Business Innovation Research (SBIR) criteria, and offering simplified templates for statistical analysis plans tailored to less complex studies.

7.5.1. Statistical Analysis Plan
This section of the guidance defines statistical analysis plan (SAP), which outlines the statistical methods and procedures for analyzing data in the study. As described, the SAP should include details on the statistical aspects of the trial design, the criteria for selecting data for analysis, and the procedures for analyzing various data items, as well as defines included populations and describes how the results will be presented. With regards to the criteria used for selecting data, this information is also captured in Section 7.6. The SAP focus should be on describing the statistical procedures chosen and providing details on how to use the chosen data in the analysis. Therefore, we recommend that information on data selection should only be included under Section 7.6 and removed from the SAP in Section 7.5.1.

Section 7.6. Data Sources
In this section, CMS requests information about the data sources to understand the reasons for selection and their strengths and limitations, which will enable CMS to assess whether the data is fit-for-purpose to answer the research question.

There is significant variability in RWD sources and manufacturers must weigh the potential advantages/disadvantages when determining which sources to use. To assist with this step, it would be helpful to have preliminary discussions with CMS, prior to the development of an entire protocol, to align on the data sources, as these are paramount to any further research.

We also recommend CMS consider single-arm studies in cases where a comparator group may not align with the specific attributes of a technology and its evidence gaps. We encourage CMS to consider the full range of fit-for-purpose study designs, including a range of data sources, methodological approaches and the requirement for a comparator group.

Additionally, this requires extensive documentation of data sources, including detailed data dictionaries, value sets, and data use agreements. Creating these documents from scratch can be time-consuming and technically challenging. To assist submitters, we recommend CMS consider the following approaches, such as: maintaining a repository of standardized data dictionaries and value sets for common RWD sources used in medical device studies (e.g., claims data, electronic health records (EHRs)); providing templates for data use agreements specific to small-scale RWD studies; and offering a streamlined data source documentation package for studies using a single, well-established RWD source (e.g., Medicare claims data), with pre-filled sections on data characteristics and limitations.

7.8 Quality Control
CMS requests that study sponsors outline steps to ensure data quality, including quality assurance and quality check procedures, double programming, source data verification, validation of endpoints, data transformation and linkages, and assessment of the data's reliability. The purpose is to assess whether the study sponsor has the appropriate quality control procedures to prevent errors in the study results.

The example quality control and validation activities outlined in the guidance such as double programming, source data verification, and comprehensive data quality assessments can be resource-intensive and technically demanding. AdvaMed recommends that CMS consider flexible approaches such as allowing tiered validation approaches or automated anomaly detection tools (e.g., open-source Python scripts) rather than full double-programming. Additionally, in cases when the data is retrospectively obtained from an owner or a vendor/aggregator, detailed quality control (QC) information might not always be available to the sponsor. In these cases, we recommend CMS clarify whether an attestation from the data provider is sufficient, and what it must contain.

Section 7.9. Study Size and Feasibility
This section provides projected study sizes to minimally detect a pre-specified effect with a pre-specified statistical precision. This section is used by CMS to determine if the study will be statistically powered to answer the research question. This section only requires sample size calculation whereas a feasibility study is more than just a power calculation. AdvaMed recommends CMS add more detailed information about the feasibility study, such as availability and completeness of key variables.

For retrospective RWD, various statistical techniques may be used. Such statistical techniques should take care to removal of or adjustment for confounders in the analysis. We suggest that CMS specify whether there are preferred methods for addressing potential sources of bias in the analysis of retrospective RWD, and whether there are specific requirements when different statistical methods are used. For instance, if propensity-score matching is used, we recommend that CMS specify a minimum match of <0.1(4,5,6), rather than 0.25 as is sometimes applied (7,8,9).

Furthermore, we encourage CMS to consider providing more specific guidance for the application of appropriate statistical methods in large data sets. Specifically, when large, retrospective data sets are used for development of RWE, we encourage CMS to rely on hypothesis testing and clarify any related requirements for establishing statistical significance. Where relevant, confidence intervals, effect sizes, standardized difference scores, or other methods should be considered rather than p-values, which tend to be size-dependent functions.

Thank you again for the opportunity to comment. We look forward to continuing our work with the Agency. If you have questions regarding these comments or if you require additional information, please contact me at tburke@advamed.org.

Sincerely,

Tara Burke, PhD
Vice President, Payment & Healthcare Delivery Policy
AdvaMed

References
1. “July 2024: A Data Book: Healthcare Spending and the Medicare Program.” Medicare Payment Advisory Committee (MedPAC), Chapter 9, 16 July 2024, www.medpac.gov/document-type/data-book/. Accessed 03 Feb 2025.
2. https://www.cms.gov/files/document/ced-guidance2024pdf.pdf
3. https://www.cms.gov/files/document/cms-evidence-review2024pdf.pdf
4. Austin PC. Using standardized difference to compare the prevalence of a binary variable between two groups in observational research. Communications in Statistics–Simulation and Computation 2009;38:1228–34.
5. Austin PC. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat Med 2009;28(25):3083–3107. pmid:19757444
6. Austin PC, Grootendorst P, Normand SL, Anderson GM. Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study. Stat Med. 2007;26(4):754-768. doi:10.1002/sim.2618
7. Rubin, D.B. Using propensity scores to help design observational studies: application to the tobacco litigation. Health Services & Outcomes Research Methodology. 2001; 2:169-188
8. Stuart, E.A. Matching methods for causal inference: a review and a look forward. Stat Sci. 2010; 25:1-21
9. Townsend SR, Phillips GS, Duseja R, et al. Effects of Compliance With the Early Management Bundle (SEP-1) on Mortality Changes Among Medicare Beneficiaries With Sepsis: A Propensity Score Matched Cohort Study. Chest. 2022;161(2):392-406. doi:10.1016/j.chest.2021.07.2167

Less

Dr. Dora Hughes CMO and Director, CCSQ Coverage and Analysis Group Centers for Medicare & Medicaid Services Mailstop S3-02-01 7500 Security Boulevard Baltimore, MD 21244-1850

Re: Proposed Guidance Document: Study Protocols that Use Real-world Data

Dr. Hughes:

IQVIA welcomes the opportunity to comment on the Medicare coverage document, Proposed Guidance Document: Study Protocols That Use Real-world Data.¹ IQVIA is a leading global provider of advanced analytics, technology solutions, and clinical research services to the life sciences industry. With approximately 87,000 employees, we conduct operations in more than one hundred countries. Over the past five years, IQVIA has planned and conducted more than 1,500 studies utilizing real-world data (RWD)and non-interventional study designs. Of these, more than 200 have pertained to the post-market evaluation of medical devices and in vitro diagnostics. Additionally, IQVIA staff have contributed to the design of CMS-approved studies for multiple therapies – including cochlear implantation, transcatheter aortic valve replacement (TAVR), and vagus nerve stimulation (VNS) for treatment resistant depression (TRD) – whose Medicare access is administered under the coverage with evidence development (CED) paradigm.

IQVIA appreciates CMS’s initiative to provide guidance on the elements of a study protocol that uses RWD, as it will help facilitate more efficient study planning and the interpretation of real-world evidence (RWE) submitted in the context of a National Coverage Determination for regulated medical products, especially for those subject to CED requirements. We commend your selection of the well-regarded HARmonized Protocol Template to Enhance Reproducibility (HARPER) – rigorously developed by the International Society for Pharmacoepidemiology (ISPE) in collaboration with The Professional Society for Health Economics and Outcomes Research (ISPOR) — as the foundation for the CMS template: “HARPER+.” IQVIA contributed to the development of HARPER, and today our study teams routinely use this robust framework in developing and/or evaluating study protocols for our clients.² Nonetheless, based on our experience in assessing the fitness-for-use of a wide range of real-world data sets for use in evaluating patient outcomes and other effectiveness measures, we have identified several areas where we believe the guidance can be strengthened. As listed below, these comments focus primarily on issues pertaining to the sourcing and evaluation of secondary data.

COMMENTS

1. Section 7.5 – Data Analysis. Table 12 in HARPER+ asks the researcher to outline any sensitivity analyses planned for the study. We recommend adding prompts focusing on standard concerns when analyzing secondary data, such as time-varying covariates and misclassification.



2. Section 7.6 Data Sources. It would be helpful to add a contextual statement to the guidance document describing how assessing data feasibility and setting other study design parameters are iterative processes when developing the protocol for a study analyzing secondary data. In order to assess whether a data source would be fit-for-purpose for the research question at hand, study sponsors must obtain feasibility counts and other details based on initial protocol assumptions regarding the study endpoints, population of interest, sample sizes, analytic approach, linkage strategy, and other factors. In some cases, these details may be found in published studies or in disclosures provided by the data vendor; however, especially for novel or new data sources, thoroughly evaluating a candidate data set often requires obtaining and analyzing a data extract. Once a study sponsor has thoroughly evaluated the data source in the context of the research question, they can move forward with adjusting the initial assumptions where necessary and finalizing the protocol, which should discuss the findings of the feasibility assessment in justifying the chosen study design.



3. Section 7.6.1 – Context and Rationale for Data Sources.
   • The guidance can be strengthened by providing more details about feasibility metrics and documentation needed to demonstrate if a data source is fit-for-purpose. The proposed guidance appropriately asks researchers to describe the strengths, limitations, and potential biases associated with different data sources, but should also ask for a discussion of these factors in relation to the research question of interest, (e.g., reasons why this data source is best positioned to capture the study outcomes and exposures completely), rather than listing them in general. The vague statement, “Understanding why researchers recommend using specific data sources is fundamental to the likelihood of success of the research proposal," could be bolstered by explaining that evaluating data sources requires both qualitative and quantitative assessments. This would be an appropriate place to list a few examples of possible metrics, such as those outlined in the Kahn framework, thereby demonstrating how users are to apply this valuable, prominently referenced resource.
   • Given the growing use of data linkage techniques to create new and enhanced data sources — while avoiding the costs associated with primary data collection — it is surprising that HARPER+ does not prompt template users to explain their proposed linkages in greater detail. We recommend that the guidance emphasize the importance of protecting patient privacy when linking data sets and instruct the user to document their linkage approach including tokenization, direct linkage, probabilistic matching, re-identification risk determination, or other approaches. This would build on the prompts provided in Table 13 (Metadata about Data Sources and Software), where users are asked to describe any intended data linkages.
   • The eighth row in Table 13, which prompts user to describe any “Conversion to common data model (CDM),” includes an unexplained asterisk. The final guidance should explain the asterisk via a footnote or omit it.
   • It is unclear why the completed HARPER+ template example provided with the draft guidance, which profiles the SAFE-PAD, study lists "N/A" under the section for “Data source provenance/curation.” Is this because no such information is available from CMS on the claims data sets used in the study? Is an explanation not needed because the AHA Annual Survey Database does not provide patient-level data? One of the more challenging issues regarding the use of RWD data for secondary purposes is describing data provenance. We recommend that CMS provide more complete discussions of data provenance and curation in the final guidance and future publications to help sponsors describe data provenance for secondary uses of RWD.



4. Section 7.7 – Data Management. For secondary data sources, considerations around data access and data privacy should be addressed. Typical data management plans will need to be adapted to account for unique aspects of secondary data (e.g., data often cannot be queried and corrected but data limitations can be acknowledged, and mitigation strategies can be implemented).



5. Appendix E. Value Sets. While this section of HARPER+ appropriately prompts users to list healthcare terminology codes for variables used in the proposed research study, we recommend also describing the importance of quality mapping between value sets. Vocabulary mapping is the process of linking terms and codes from different standardized vocabularies or ontologies to ensure consistency and interoperability across various data sources. A lack of provenance and versioning of mappings can have a significant impact on the quality of downstream data integration processes. When conducting studies using RWD, accurate and consistent data mapping is essential for ensuring reliability of study outcomes and ties back to the appropriate use of common data models.



6. Appendices. The formatting of the appendices is difficult to navigate. The draft guidance labels two separate documents as Appendix A and two others as Appendix B. We recommend that CMS clarify the formatting of the documents and/or sections currently labelled as appendices. For example, if the HARPER+ template and the completed example thereof were labelled as Annex 1 and Annex 2 of the guidance document, it would preserve the term “appendix” to apply to the five items currently listed as appendices to the HARPER+ template.

On behalf of IQVIA, thank you again for the opportunity to share our comments and recommendations. If you have any questions about the recommendations provided, please do not hesitate to contact Dan Campion at daniel.campion@IQVIA.com or by phone at 617-599-9409. We will be happy to connect you with any of our subject matter experts who work tirelessly on the application of real-world evidence. We look forward to participating further in this important conversation about coverage decision-making.

Regards,

Andrew T Barnhill

Andrew Barnhill, JD
Head of Policy
Global Legal

---

1 Centers for Medicare and Medicaid Services. Proposed Guidance Document: Study Protocols That Use Real-world Data (January 17, 2025), https://www.cms.gov/medicare-coverage-database/view/national-submit-public-comment.aspx?DocID=39&commentDocType=mcdoc&fromPage=doc&
2 Massoud Toussi, who served as IQVIA’s global category lead for evidence from secondary data at the time, was a member of the ISPE/ISPOR task force and a co-author of the 2022 publication, Wang SV, Pottegård A, Crown W, et al. HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force. Pharmacoepidemiology and Drug. 2023;32(1):44-55. doi:10.1002/pds.5507.

Less

Dear Acting Administrator Carlton,

Corewell Health appreciates the opportunity to provide comments on the proposed Real-World Data Study Protocol Guidance. Corewell Health is a Michigan-based nonprofit integrated health system with a team of 60,000+ dedicated professionals including more than 11,500 physicians and advanced practice providers and over 15,000 nurses delivering care in 21 hospitals and 300+ outpatient locations. In addition, as an integrated health system, Corewell Health includes Priority Health, a health plan that insures more than 1.2 million individuals. Corewell Health is not only Michigan’s largest health system but also the state’s largest private employer. Through experience and collaboration, we are reimagining a better, more equitable model of health and wellness. We submit these comments through the lens of an integrated health system.

Corewell Health would like to express our appreciation for the transparency and thoroughness demonstrated in the proposed Real-World Data Study Protocol Guidance. We applaud CMS’s goal of ensuring decision-making is clear for the public.

While we commend CMS for this effort, we would like to highlight a critical area for improvement that requires further review. It is imperative that CMS provides clear and consistent guidance that extends to state Medicaid agencies as part of this process. For example, our organization recently experienced a significant state Medicaid issue where a $1.6 million claim was held because our state Medicaid agency did not adhere to CMS guidelines.

We expect state Medicaid offices to follow the CMS guidelines for patients needing devices and therapies under “coverage with evidence development” (CED) requirements. The Michigan Medicaid office requires us to provide documentation to support that we are correctly billing a CED, which does not require an attestation per CMS guidelines. Accommodating the state office requirements delays payment to the organization and can create barriers for us to provide care in a timely manner and to as many patients as possible. We expect that, like Medicare, Medicaid would assess claims based on the unique identified present for the national clinical trial (NCT) and recognize those NCTs that indicate the patient is receiving care as part of a CED. We would also expect to see the claims process without the denial that would be typical of a qualifying clinical trial (QCT) without an attestation on file. In this case, we experienced a significant delay in processing the claim and ultimately, like previously stated, had a system loss of $1.6M, given that our state Medicaid would not follow CMS guidelines. This is a service that CMS deems appropriate and payable without attestation, and we hope that CMS will ensure that state Medicaid offices are complying with its guidance.

This incident underscores the need for more explicit instructions and support from CMS to ensure uniform compliance across all states.

We believe that by addressing this gap, CMS can further strengthen the implementation of the Real-World Data Study Protocol Guidance and ensure that all stakeholders, including state Medicaid agencies, are aligned and equipped to follow the established guidelines effectively.

Thank you for your consideration of our comments. Should you have any questions regarding these comments or if you would like any additional information, please contact Dave Walker. Public Policy Advisor, Corewell Health Government Relations & Public Policy, david.walkerii@corewellhealth.org.

Less

With the increased number of aged and people on Medicare services and the strain that it is placing on health care, especially SNFs and other inpatient facilities post acute care, I am recommending that the following items be considered for reimbursement or payment:

1. Bath/shower equipment
2. Compression stockings
3. Other safety equipment for home use, that are deemed necessary by a PT or OT, such has hospital beds and overlays.

This would allow patients to stay in their home, receiving home health services, instead of having to be admitted to a SNF or other high cost inpatient settings.

Less

Re: CMS Proposed Guidance Document: Study Protocols That Use Real-World Data, January 27, 2025

The Wound Care Collaborative Community (WCCC) would like to provide comments on the proposed guidance document concerning a standardized protocol framework for real-world evidence (RWE) studies used in national coverage determinations.

WCCC is a 501(c)(3) non-profit, FDA-recognized collaborative community consisting of over 150 wound care experts and various organizations committed to advancing best practice care for people with wounds. Our members include healthcare practitioners, clinical societies, associations, industry distributors, and manufacturers involved in medicine, geriatrics, dermatology, podiatry, surgery, physical therapy, nursing, research, and more. More information about our community can be found at http://www.woundcarecc.org.

The WCCC commends CMS for enhancing HARPER by creating HARPER+ to improve clarity and transparency in the development of RWE for use in Coverage with Evidence Development studies. Our organization reviewed the proposed guidance document and did not find any areas of concern. Standardizing real-world evidence protocols is important for all payers as a supplement to randomized clinical trials for effective coverage decision-making. Additionally, the alignment of this document with RWE guidance from FDA illustrates cross-agency collaboration and consideration of company sponsors navigating both agencies to bring their innovations to market.

However, we have two concerns outside of the framework itself:

• Potential for bias despite a standardized protocol framework
• Applicability beyond National Coverage Determinations

HARPER+ provides a standardized protocol framework to help ensure RWE studies are fit-for-purpose, but outputs are only as good as the inputs. There are three aspects of RW studies: 1) data collection and patient selection, 2) study protocols, and 3) data analysis. HARPER+ addresses study protocols, but not data collection nor analysis which we believe should also be standardized to reduce bias. Like randomized controlled studies (RCTs), patient selection bias and reporting bias can have a significant impact on RW study outcomes. Without knowing the method by which patients were selected for inclusion in a registry, for example, or whether all outcomes were reported, it is possible to prove that any intervention had a positive outcome by only reporting the outcomes of that intervention.

WCCC members have published papers on the standardization of RWE data collection and analysis^1,2. Although primarily focused on wound care research, we believe the recommendations could apply to research in other medical areas. In addition, the US Wound Registry³ published a white paper “Analysis of Bias Criteria Checklist for Wound Care Registries & EHRs” (USWR-White-Paper.pdf) that may also have applicability beyond wound care RW studies. We would be happy to discuss these papers with CMS should there be interest.

Our second area of concern is the primary intended use of HARPER+ for technologies that utilize a National Coverage Determination (NCD) pathway where Coverage with Evidence Development (CED) is available. While we believe this framework could be applied for real-world (RW) studies beyond those related to CED, we advocate for CMS to extend its application to products covered under Local Coverage Determinations (LCDs). It is noteworthy that no wound care innovations - despite numerous annual introductions - have undergone the CED process for an NCD to date. Moreover, only three wound care technologies are covered under NCDs:

• Hyperbaric Oxygen Therapy (HBOT) for Wound Healing (NCD: 20.29 - Hyperbaric Oxygen Therapy)
• Electrical Stimulation and Electromagnetic Therapy for Chronic Wounds (NCD - Electrical Stimulation (ES) and Electromagnetic Therapy for the Treatment of Wounds (270.1)
• Autologous Platelet-Rich Plasma (PRP) for Chronic Non-Healing Wounds (NCD - Blood-Derived Products for Chronic Non-Healing Wounds (270.3)

All other wound technologies are covered under LCDs issued by Medicare Administrative Contractors (MACs). The Medicare Program Integrity Manual (Chapter 13) details how MACs develop LCDs but does not provide a mechanism for MACs to implement CED requirements. The 21st Century Cures Act of 2016 introduced transparency requirements for LCDs but did not authorize MACs to employ CED. Although CMS has occasionally sought public input on expanding CED-like approaches, LCDs generally rely on randomized controlled trials (RCTs) and occasionally existing clinical evidence rather than conditioning coverage on future data collection. Our research indicates that wound RCTs can be narrowly focused on patient populations negatively impacting their generalization to the broader Medicare population, yet they remain the primary evidence mechanisms used by MACs for coverage decisions.

In 2009, a study using real world data in wound care demonstrated that more than 75% of patients would have been excluded from all prospective trials that had brought wound care treatments to market over the preceding decade⁴. In 2024, another analysis conducted by WCCC showed that the non-generalizability of clinical trials has worsened in the past 15 years. The analysis found that 92% of diabetic foot ulcer (DFU) patients and 90.7% of diabetic foot ulcers would be excluded from the typical prospective DFU trial. The results confirm the non-generalizability of most prospective DFU trials and explain, at least in part, the challenges to patient enrollment. The findings also have implications for Local Coverage Decisions relying solely or mostly on RCT-generated data. The 2024 study findings are planned for publication in late 2025.

In 2017, a novel project evaluated a new approach to clinical research by integrating clinical care, quality reporting and research⁵. A consortium of 6 hospital based wound care clinics engaged in 5 prospective RCTs evaluated the difference between the patients seen in the clinics and the subjects they were enrolling in clinical trials. Dramatic differences were noted both for DFUs and VLUs, with actual patients having many excluded conditions as well as larger, more numerous and more severe wounds than those enrolled in the RCTs. The study suggested that the integration of clinical care and research was possible through a purpose-built electronic health record. This capability has thus been in place, at least in the field of wound care, for many years, and can serve as a path forward to clinical research and coverage policy independent of the CED process.

“Big data” entities like IQVIA (IQVIA), Oracle (Oracle Health), Optum (Optum Business) and others used by CMS for CED studies extract data from huge EHR systems. Those projects cost millions of dollars and are affordable only by pharmaceutical and large medical device companies. Most device companies do not have the financial resources to access EMR nor have they shown much interest in conducting such evidence studies since CMS primarily utilizes RWE for National Coverage Determinations using CED. If, however, CMS were to consider real world data in Local Coverage Determinations (LCDs) with CED, we may see an uptick in the number of companies pursuing RW studies which we believe would be in the best interest of real-world patients.

The fact that many MAC-issued LCDs are identical across every coverage region means restricting CED to only NCDs is hard to understand if for no other reason as the MAC often collaborate in developing LCDs to create de facto national coverage decisions. Nonetheless, MACs today are prohibited from using the CED process, and they may consider real-world evidence (RWE) for coverage only in combination with RCT-level evidence. An example of this situation is the new LCDs on Skin Substitutes for diabetic and venous leg ulcers which go into effect on April 13th of this year. The policies across all eight A/B MAC regions are virtually identical. Below is a table with links to these LCDs.

Noridian Healthcare Solutions, LLC – JE	DL39764	A59628
Noridian Healthcare Solutions, LLC – JS	DL39760	A59626
First Coast Service Options, Inc.	DL36377	A57680
Novitas Solutions, Inc.	DL35041	A54117
Palmetto GBA	DL39806	A59691
National Government Services, Inc.	DL39828	A59712
Wisconsin Physicians Service Insurance Corporation	DL39865	A59740
CGS Administrators, LLC	DL39756	A59618

The WCCC believes that payers should make coverage decisions based on best available evidence generalizable to the real world to mimic the conditions in which payers pay. We are encouraged by the direction CMS is taking with HARPER+, but we would be further encouraged if CMS permitted its A/B MACs to evaluate the safety and effectiveness of new innovations using conditional coverage like CED when the MACs collaborate to develop identical LCDs that effectively serve as de facto NCDs.

---

It is the mission of the WCCC to eliminate barriers to innovation and improve access. Accordingly, we are fully aligned with CMS’ interest in utilizing a broader range of fit-for-purpose study designs for coverage decision making. We commend CMS' commitment to efficient coverage approvals and increasing public confidence in RW studies. WCCC is currently engaged in several targeted projects that support these commitments. Our work was recently featured in a full-day program conducted in collaboration with the FDA and the Symposium of Advanced Wound Care (SAWC) 2024 Spring. The “Driving Innovation in Wound Care Summit” was attended by over 200 experts in wound research and practice and addressed issues such as identifying and eliminating barriers to innovation, clinical study endpoints, generating and reporting evidence, real-world data in regulatory and payer decision making, and standards of care in clinical trials. During the Summit, we presented and discussed current projects aimed at removing barriers to innovation. Our efforts will be showcased again at the spring 2025 SAWC conference. We intend to continue our collaboration with both the FDA and CMS to integrate real-world evidence into product development and approval processes, and to utilize it for coverage policies as we strive to increase access to the most effective treatments for patients suffering from chronic wounds. The FDA (CDRH, CDER, CBER) has been an invaluable partner in our mission, and we appreciate their high level of support and collaboration.

The WCCC values the opportunity to provide these comments and welcomes further engagement with CMS to identify and remove obstacles to wound care innovation and access. We are open to meeting with CMS to discuss these issues and potential solutions in greater detail.

Sincerely,

Vickie R. Driver, DPM, MS, FACFAS
Chair, WCCC
chairperson@woundcarecc.org

Lucian G. Vlad, MD
Chair – WCCC Real World Evidence Group
lvlad@wakehealth.edu

1 Fife CE, Eckert KA. Harnessing electronic healthcare data for wound care research: standards for reporting observational registry data obtained directly from electronic health records. Wound Repair Regen. 2017;25:192-209.
2 Carter MJ. Harnessing electronic healthcare data for wound care research: Wound registry analytic guidelines for less-biased analyses. Wound Repair Regen. 2017;25:564-73
3 The US Wound Registry is an independent, 501(c)(3) non-profit organization that collects comprehensive data on all patients and wounds without industry sponsorship. US Wound & Podiatry Registries.
4 Carter MJ, Fife CE, Walker D, Thomson B. Estimating the Applicability of Wound-Care Randomized Controlled Trials to General Wound Care Populations by Estimating the Percentage of Individuals Excluded from a Typical Wound Care Population in Such Trials. Adv Skin Wound Care. 22: 316-24, 2009.
5 Serena TE, Fife CE, Eckert KA, Yaakov RA, Carter MJ. A New Approach to Clinical Research: Integrating Clinical Care, Quality Reporting, and Research Using a Wound Care Network-based Learning Healthcare System. Wound Repair Regen. 25(3): 354-365, 2017.

Less

March 18, 2025

Coverage and Analysis Group
Centers for Medicare & Medicaid Services
Mailstop: S3-02-01
7500 Security Boulevard
Baltimore, Maryland 21244-1850

Re: Proposed Guidance Document: Study Protocols That Use Real-world Data

To the Centers for Medicare & Medicaid Services (“CMS”):

The RWE Alliance appreciates the opportunity to comment on the Proposed Guidance Document: Study Protocols That Use Real-world Data (“Proposed Guidance”).[1] We are a coalition of real-world data (“RWD”) and analytics organizations with a common interest in harnessing the power of real-world evidence (“RWE”) to inform regulatory decision making to improve the lives of patients. Our members have deep knowledge and experience working with healthcare data across disease areas and patient populations, and we aim to bring these collective insights to bear in support of RWE policies.[2]

The RWE Alliance commends CMS for issuing the Proposed Guidance that describes critical elements for fit-for-purpose studies involving RWD and for providing a standardized template for sponsors to use when creating study protocols using RWD. CMS’s rationale for each section will help facilitate a researcher’s understanding of how the template is expected to be used. Moreover, we appreciate CMS aligning its Proposed Guidance with FDA’s guidance for use of RWE in regulatory decision making. We agree with CMS’s efforts to share clear expectations for the design, selection of data sources, and analysis methods for any studies using RWD intended for submission to CMS. We provide the following general comments for any approach that CMS considers for reviewing studies that incorporate RWD, including but not limited to potential updates to this Proposed Guidance.

• First, we agree with the general approach of using the HARmonized Protocol Template to Enhance Reproducibility (“HARPER”) framework as the basis for a modified framework (i.e., “HARPER+”). Given that many researchers use the HARPER framework for multiple submissions (e.g., regulators, health technology assessments), we suggest that the Proposed Guidance more completely document the modifications and additions introduced in HARPER+ to facilitate the identification and implementation of changes necessary for CMS studies.
  
  
• Second, we suggest that CMS move away from the terms “retrospective” and “prospective” when describing real-world studies. As FDA explains, these terms are “commonly but variably used”[3] and can have multiple meanings that add unnecessary confusion. To align with FDA, we suggest that CMS instead use the terms “primary data collection” and “secondary data collection” to distinguish the purpose for which the data were collected. Specifically, CMS should use “primary data collection” to refer to data that are collected de novo, i.e., directly from clinical sites, patients, or health care providers, for the purposes of the study at hand. CMS should use “secondary data collection” to refer to data that are obtained from an existing data source and were collected for a purpose other than the study. We recommend that CMS describe how the data collection type may influence the application of different parts of the Proposed Guidance, such as on the reporting of adverse events.
  
  
• Third, we ask that CMS clarify why the completed template example in Appendix B for the SAFE-PAD study states “N/A” under “Data source provenance/curation.” Specifically, it would be helpful to understand if “N/A” is listed because no such information is available from CMS on the claims datasets used in the study or because an explanation is not needed because the AHA Annual Survey Database does not provide patient-level data. Generally, we recommend that CMS provide more complete discussions of data provenance and curation in the Proposed Guidance and any future publications to help sponsors describe data provenance for secondary uses of RWD.
  
  
• Fourth, we find the formatting of the appendices to be difficult to navigate. The Proposed Guidance labels two separate documents as Appendix A and two other documents as Appendix B. We recommend that CMS clarify the formatting of the documents and/or sections currently labeled as appendices. For example, if the HARPER+ template and the completed example were labeled as Annex 1 and Annex 2, it would preserve the term “appendix” to apply to the five items currently listed as appendices to the HARPER+ template.

The comments in this letter focus on scientific methods related to developing study protocols and managing real-world data for clinical research. Our comments do not address the Transitional Coverage for Emerging Technologies pathway, Coverage with Evidence Development, or other CMS policies. The RWE Alliance appreciates the development of the Proposed Guidance and similar efforts to inform the design of studies involving RWD submitted to CMS. Thank you for considering these comments, and please let us know if you have any questions. We welcome the opportunity to discuss further.

Best regards,
The RWE Alliance

[1] CMS, Proposed Guidance Document: Study Protocols That Use Real-world Data (January 17, 2025), https://www.cms.gov/medicare-coverage-database/view/medicare-coverage-document.aspx?mcdid=39.
[2] For information about our members, please see our website, https://rwealliance.org/who-we-are/.
[3] FDA, Draft Guidance for Industry: Real-World Evidence: Considerations Regarding Non-Interventional Studies for Drug and Biological Products at 4, n.14 (Mar. 2024), https://www.fda.gov/media/177128/download.

Less

March 12, 2025

Coverage and Analysis Group
Centers for Medicare & Medicaid Services
7500 Security Blvd, S3-02-01
Baltimore, MD 21244

Re: “Proposed Guidance Document: Study Protocols That Use Real-world Data”

To the Centers for Medicare & Medicaid Services:

Thank you for publishing the draft guidance document entitled “Proposed Guidance Document: Study Protocols That Use Real-world Data” (the “Draft Guidance”). Aetion appreciates the Centers’ commitment to advancing the use of real-world data (RWD) and real-world evidence (RWE) in healthcare decision-making. Integrating RWE into healthcare decision-making will facilitate more efficient drug development and enhance understanding of medical products. The Centers for Medicare & Medicaid Services’ (CMS) work to provide robust, actionable guidance will help ensure that evidence submitted in support of national coverage determinations (NCDs) meet CMS’s appropriately high expectations.

Aetion welcomes and appreciates the opportunity to comment on the Draft Guidance. As a health technology company that specializes in the generation of RWE capable of supporting healthcare decision-making, we confront the matters addressed in the guidance on a daily basis. Our technology product, the Aetion Evidence Platform® (AEP), was originally created by two academia-based pharmacoepidemiologists to address a widely recognized need: scalable, transparent, and scientifically rigorous analyses of RWD to identify the safety and effectiveness of clinical interventions. A key feature of AEP and its suite of applications is data fluency, meaning that it enables users to run analyses against different data sources in their near-native formats, with minimal transformations required prior to analysis.

Aetion shares CMS’s commitment to fostering high-quality analyses of fit-for-purpose RWD that can provide RWE for healthcare decisions across the product life cycle, including to the extent that CMS reviews evidence in the context of NCDs. While we do not express a view on CMS’s use of Coverage with Evidence (CED) in general, or on specific programs such as the Transitional Coverage for Emerging Technologies (TCET) pathway, we believe that CMS’s review of RWD or RWE in any context should reflect current best practices and clear regulatory expectations. Greater public alignment on principles and best practices, and a commitment to clarity regarding CMS’s expectations, will enable sponsors and other researchers to implement RWD analyses in a transparent, auditable, reproducible, and scientifically valid manner.

Comments on the Draft Guidance

For the reasons described above, we very much appreciate the Draft Guidance’s clear and specific guidance to the public on the critical elements of a RWD-based study protocol for NCDs. Additionally, we commend CMS for explicitly acknowledging that “[s]tudies should be designed, and protocols should be written and submitted to CMS before study outcomes are available to those designing the study or writing the protocol” as a critical mechanism to ensure agency alignment and the generation of trustworthy evidence.

We strongly agree that pharmacoepidemiological studies should use reliable and relevant RWD, and that the study design should be sufficiently detailed so it is transparent and reproducible. We also agree with the Draft Guidance’s general approach of using the HARmonized Protocol Template to Enhance Reproducibility (HARPER) framework [1] as the basis for a modified framework (i.e., “HARPER+”) that is adapted for Medicare coverage criteria.

Nevertheless, we believe that the current operationalization of the HARPER+ framework could be made more practicable for researchers who will be submitting evidence not only to CMS, but also to regulators, health technology assessment bodies, and other key stakeholders. Given that many researchers will likely use the original HARPER framework for other submissions, it would be helpful if the modifications and additions introduced in HARPER+ were more explicitly documented, thereby better facilitating the identification and implementation of changes necessary for CMS submissions. Alternatively, rather than maintaining a standalone HARPER+ document, incorporating these additions and modifications into an “Appendix to HARPER” may provide a more streamlined approach for researchers to integrate the required elements into their preexisting HARPER-based protocols when submitting to CMS. Such refinements would directly respond to the increasing call for harmonization of RWE guidance across agencies and countries [2,3], thereby making it easier for researchers to meet various stakeholders’ standards efficiently.

To improve overall clarity, we also have a few additional suggestions for consideration:
- RWD should be defined early in the document, such as in the Background section, to ensure clarity and prevent potential misunderstandings.
- The guidance should move away from the terms “retrospective” and “prospective” studies [4], given that their multiple meanings can cause confusion (for example, see footnote 14 of the Food and Drug Administration's draft RWE guidance “Considerations Regarding Non-Interventional Studies for Drug and Biological Products” [5]). Instead, we suggest distinguishing between “secondary” and “primary” data collection, concepts which are already referenced indirectly throughout the text, and clarifying how aspects of the guidance (such as the need for the reporting of adverse events) may depend on the study type.

Below we include suggested section edits for CMS’s consideration.

Section Comments

1. Title Page

The Draft Guidance currently indicates that the protocol should be registered on clinicaltrials.gov. Although we fully agree that protocol registration is a key mechanism to ensure transparency, we recommend that the language here be expanded to allow for the registration of studies in other protocol repositories (e.g., the Real World Evidence Study Registry [6] and the HMA-EMA Catalogue of Real-world Data Studies [7]).

7.3.4. Generalizability of Study Population

We strongly agree that generalizability is a “crucial consideration in studies that may impact CMS coverage decisions,” especially in the context of the Medicare population. However, we recommend that the scope of this section be expanded to account for the submission of studies conducted outside of the United States. In such situations, the concepts of transferability/transportability are important considerations [8, 9]; we therefore recommend that CMS include the following sentence in this section: “When using data from outside of the United States, the transferability of the data (or the transportability of the results) to the United States setting should be clearly explained and/or rationalized.”

7.5.1. Statistical Analysis Plan

We believe that the use of the term “trial design” here might be confusing to the reader and thus recommend changing the framing to “study design” to be consistent with the phrasing of study design, study protocol, etc. used throughout the Draft Guidance.

7.6.1. Context and Rationale for Data Sources

The HARPER publication references various frameworks and tools for documenting and justifying study design and data source decisions, including “The Structured Process to Identify Fit-For-Purpose Data: A Data Feasibility Assessment Framework” (SPIFD) [10]. Using these tools to design RWD-based studies helps mitigate bias and ensure that RWD analyses will produce objective, interpretable, and clinically meaningful results for the assessment of the safety and effectiveness of medical products. We therefore recommend that the Draft Guidance explicitly reference these frameworks and tools as well as more recent additions in the field, such as “Staging and Clean Room: Constructs Designed to Facilitate Transparency and Reduce Bias in Comparative Analyses of Real-world Data” [11] and “A Structured Process to Identify Fit-for-Purpose Study Design and Data to Generate Valid and Transparent Real-World Evidence for Regulatory Uses” (SPIFD2) [12].

Aetion looks forward to collaborating with the Centers to support the successful use of RWD and RWE in healthcare decision-making. Please feel free to contact me at Nicolle.Gatto@Aetion.com with any questions regarding these comments or other issues related to RWE policy and development.

Sincerely,

Nicolle Gatto
Chief Science Officer, Aetion

References
[1] Wang SV, Pottegård A, Crown W, et al. HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force. Pharmacoepidemiol Drug Saf. 2023;32(1):44-55. doi:10.1002/pds.5507.
[2] Claire R, Cresswell K, Avsar TS, et al. D6.2 Report on Global Regulatory Best Practices Analysis: A Scoping Review of HTA and Regulatory RWD/RWE Policy Documents. 2024. IDERHA.
[3] Sarri G, Hernandez LG. The maze of real-world evidence frameworks: from a desert to a jungle! An environmental scan and comparison across regulatory and health technology assessment agencies. J Comp Eff Res. 2024;13(9):e240061. doi:10.57264/cer-2024-0061.
[4] Ankarfeldt MZ, von Osmanski BI, Blond K. Kill your darlings: Stop using the terms retrospective and prospective. Pharmacoepidemiol Drug Saf. 2023;32(4):506-507. doi:10.1002/pds.5598.
[5] US Food and Drug Administration. Considerations for the Design and Conduct of Externally Controlled Trials for Drug and Biological Products: Guidance for Industry. Published November 2023. Accessed March 3, 2025. https://www.fda.gov/media/177128/download.
[6] Real World Evidence Study Registry, https://osf.io/registries/rwe/discover.
[7] HMA-EMA Catalogue of Real-world Data Studies, https://catalogues.ema.europa.eu/catalogue-rwd-studies.
[8] Jaksa A, Arena PJ, Chan KKW, Ben-Joseph RH, Jónsson P, Campbell UB. Transferability of real-world data across borders for regulatory and health technology assessment decision-making. Front Med (Lausanne). 2022;9:1073678. Published 2022 Nov 16. doi:10.3389/fmed.2022.1073678.
[9] Levy NS, Arena PJ, Jemielita T, et al. Use of transportability methods for real-world evidence generation: a review of current applications. J Comp Eff Res. 2024;13(11):e240064. doi:10.57264/cer-2024-0064.
[10] Gatto NM, Campbell UB, Rubinstein E, et al. The Structured Process to Identify Fit-For-Purpose Data: A Data Feasibility Assessment Framework. Clin Pharmacol Ther. 2022;111(1):122-134. doi:10.1002/cpt.2466.
[11] Muntner P, Hernandez RK, Kent ST, et al. Staging and clean room: Constructs designed to facilitate transparency and reduce bias in comparative analyses of real-world data. Pharmacoepidemiol Drug Saf. 2024;33(3):e5770. doi:10.1002/pds.5770.
[12] Gatto NM, Vititoe SE, Rubinstein E, Reynolds RF, Campbell UB. A Structured Process to Identify Fit-for-Purpose Study Design and Data to Generate Valid and Transparent Real-World Evidence for Regulatory Uses. Clin Pharmacol Ther. 2023;113(6):1235-1239. doi:10.1002/cpt.2883.

Less

VIA ELECTRONIC SUBMISSION
Coverage and Analysis Group
Centers for Medicare & Medicaid Services
7500 Security Boulevard
Baltimore, MD 21244
March 18, 2025

RE: Proposed Guidance Document: Study Protocols That Use Real-world Data

Dear Coverage and Analysis Group,

The Robert J. Margolis, MD Institute for Health Policy at Duke University (“the Duke-Margolis Institute” or “the Institute”) appreciates the opportunity to comment on the “Proposed Guidance Document: Study Protocols That Use Real-world Data,” published on January 17, 2025.

The Duke-Margolis Institute generates and analyzes evidence across the spectrum of health policy and supports the triple aim of better care, better health, and lower cost. The Center focuses on transforming health care delivery and increasing the value of biomedical innovation through evidence-based solutions to the most pressing, relevant health care delivery and payment policy questions. Center experts are engaged in policy research and development efforts to improve both care delivery and the processes and infrastructure needed at the Centers for Medicare and Medicaid Services (CMS) to ensure efficient access to new and innovative technologies.

The comments below describe opportunities for strengthening the proposed guidance document to more closely align with CMS’s goals of improving the transparency, predictability, and timeliness of Medicare coverage processes. These comments are informed by the Institute’s independent analysis of the guidance document and engagement with the Institute’s Value for Medical Products Consortium, a diverse group of stakeholders, including manufacturers, real-world evidence experts, providers, researchers, and payers.

The Institute commends CMS for their continued efforts towards the goals of more transparent, predictable, and timely coverage processes. The template presented in this guidance document offers greater transparency into how CMS will evaluate studies that use RWD for Coverage with Evidence Development (CED). This guidance document is one important step to ensure clear communication regarding study protocol expectations. Noting that there are many potential data sources and collection strategies that could be used to collect RWD for studies that answer CMS-identified evidence questions, this template will be a helpful operational tool for manufacturers as they create and submit study protocols. The Institute provides below recommendations for language clarifications within the proposed guidance document to ensure that the template is as clear and effective as possible. Additionally, the Institute offers comments on further actions and steps that CMS can take beyond this template to support effective and efficient real-world evidence (RWE) generation in the post-market setting and further their goals of increasing the transparency, predictability, and timeliness of Medicare coverage processes.

The Institute recommends that CMS:

Clarify the preferred process for updating the study protocol;

Clarify how the new HARPER+ framework may be applied across studies that include primary and secondary data collection;

Publish guidance on fit-for-purpose (FFP) studies that can support Medicare coverage; and

Consider utilizing early, multi-stakeholder engagement to inform evidence thresholds and evidence generation strategies.

The comments below detail these recommendations.

Areas for Clarification
For this guidance document, CMS prepared a standard template for study sponsors to complete such that CMS can more efficiently and effectively review study designs for approval. This new template is based on the International Society for Pharmacoepidemiology (ISPE) and The Professional Society for Health Economics and Outcomes Research (ISPOR)’s HARmonized Protocol Template to Enhance Reproducibility (HARPER) framework, as well as the Food and Drug Administration (FDA) guidance on RWE use to support regulatory decision-making for medical devices.^1,2 The new template, the HARmonized Protocol Template to Enhance Reproducibility Plus (HARPER+), builds on these frameworks, with the following additions: a note that studies should seek to answer CMS-identified evidence questions, a section on generalizability to Medicare beneficiary populations, a section identifying threats to study completion, and language that aligns with CED criteria around protection of human subjects.

Protocol Updates
In the proposed template, there is a section on the title page that indicates the protocol version and the most recent update date. There is thus an opportunity for sponsors to update and edit the study protocol within this template. The types of changes that would warrant a manufacturer to update the study protocol, however, are unclear. There may be several appropriate junctures and study updates for which a manufacturer may want to submit an update to CMS. The threshold for considering what constitutes a “minor” change that may not require providing an update to CMS, as well as the appropriate timeframe for notifying CMS of a major change, is a point for clarification. Considering that recent process reforms have focused on helping manufacturers to navigate coverage processes, additional clarity around when and how to submit a study protocol update to CMS could provide further predictability. The Institute recommends that CMS clarify the preferred process for updating the study protocol.

Primary and Secondary Studies and HARPER+
The HARPER framework, as the foundation for HARPER+, is a standardized template designed to ensure reproducibility for RWE studies, specifically studies that make secondary use of RWD. Whereas primary data are collected directly from a population being studied for a specific study, secondary data are data collected for a non-research purpose that are then leveraged for a study (such as EHR data or administrative claims). CED studies may use either primary data collection, through prospective studies, or use secondary data to answer CMS-identified evidentiary questions. For example, some studies that support CED have been prospective and use primary data collection through registries, such as for amyloid PET imaging.³ Studies for Leadless Pacemakers and Autologous Platelet-rich Plasma have used existing Medicare claims data, which is a secondary use of RWD.^4,5 CMS may intend that HARPER+ be used for both studies that rely on primary or secondary data collection and analysis, though the distinction between studies that use primary and secondary data is not clear in the current guidance. Since the precursor to HARPER+ focused on secondary use of RWD, the Institute recommends that CMS clarify how the new HARPER+ framework may be applied across studies that include primary and secondary data collection.

Areas for Future Actions and Guidance
The recent guidance documents from CMS have provided insight into how CMS may evaluate what constitutes “reasonable and necessary," what is the strength and level of evidence needed to support coverage, and an example of how CMS will determine clinical endpoints of interest for a therapeutic area. These guidance documents were designed to add predictability and transparency into CMS processes. This guidance document on Study Protocols That Use Real-world Data adds an additional level of clarity into how CMS will review study protocols and the template that should be used. HARPER+ could further support timeliness of coverage processes by enabling CMS and manufacturers to communicate more effectively.

These reforms are intended to offer greater transparency around the strength and level of evidence needed in studies and the technical elements of data protocols submitted to CMS. However, there are still outstanding questions regarding the transparency and predictability around the types of data collection strategies and study designs that are most suitable for evidence generation to support Medicare coverage. Through this guidance document, CMS has the opportunity to inform studies that utilize RWD, even beyond studies for supporting Medicare coverage. This will ultimately increase CMS’ influence over the types of study designs, data collection methods, and RWE generation methods that are used in FFP studies more broadly.

One step that CMS could take to provide further clarity into their expectations for evidence generation strategies would be to propose a guidance document on FFP study designs. The updated CED guidance document outlined the key principles for CED studies, including expected level of evidence, and the recently published HARPER+ template details the study elements that CMS will review in a study protocol. CMS also provided an example of how they will determine clinical endpoints of interest in different therapeutic areas. A remaining area of uncertainty is how CMS will consider what is an appropriate evidence generation strategy, given the types of evidentiary questions CMS may have at the time of product approval.

CMS has expressed their interest in publishing a FFP guidance document, and the Institute supports this endeavor to clarify evidence expectations. CMS noted in their National Coverage Analysis Evidence Review Guidance Document that CMS “endorses the concept that studies should be fit-for-purpose” and lists the broad methodological principles that CMS uses to assess studies. CMS also expressed that one of the values of FFP post-market studies is that they can be more representative of the clinical experience, and in this recent guidance document on study protocols that use RWD, CMS reiterated their openness to FFP studies that support coverage. A guidance document for FFP study design could detail the types of data sources and evidence generation strategies that might be appropriate for supporting coverage based on the type of evidence questions that CMS identifies. For example, CMS could clarify how they might determine what types of causal inference study designs or data sources might be most suitable for answering questions regarding appropriate patient criteria for the product or service. This could help provide clarity into how CMS would evaluate studies that utilize novel RWE generation methods without being overly prescriptive, and would allow sponsors to think about potential data collection strategies earlier in the coverage process.

The Institute facilitated a series of expert stakeholder roundtables to determine evidence generation strategies for specific outcomes of interest or evidence questions for a particular product category, with the goal of making post-market evidence generation more streamlined and efficient. The Institute’s roundtable series focused on tricuspid valve interventions (TVIs). Participants included providers, researchers, RWE experts, commercial payers, FDA, and CMS. Participants identified potential evidentiary questions, the outcomes of interest for different stakeholders that may affect patient access and product uptake, the data sources that may be best suited to gathering data based on these identified outcomes, and data collection strategies that would support coverage without being overly burdensome or costly. The Institute created a preliminary framework for considering the data sources and evidence generation strategies that would best support coverage for TVIs given identified evidence questions.

A similar framework or guidance document from CMS could provide transparency into how they may evaluate the appropriateness of FFP studies for categorical evidence questions across different therapeutic areas. In conjunction with this Study Protocols that Use RWD guidance document, the evidence expectations for Medicare coverage would be more transparent and the coverage process more predictable. The Institute recommends that CMS publish guidance on FFP studies that can support Medicare coverage. CMS could also consider publishing an example of how they may evaluate FFP studies based on their evidentiary questions and expectations for one coverage determination, similar to the approach taken with the example Clinical Endpoints Guidance Document for Knee Osteoarthritis, or the example pre-populated HARPER+ template.

Early, multi-stakeholder engagement—in conjunction with FFP guidance—can support the development of efficient and novel RWD collection strategies that lower the burdens and cost of post-market evidence generation. For example, novel trial designs—such as decentralized trials that use remote patient monitoring devices or studies that utilize synthetic data—have the potential to produce rigorous evidence at a lower cost than traditional randomized control trials. As these take time to develop, any early guidance on FFP studies will allow manufacturers to streamline evidence generation.

CMS can facilitate early engagement with a variety of stakeholders to inform both the relevant outcomes of interest for a given disease or therapeutic area (which can be incorporated into clinical guidelines guidance documents) and the relevant considerations for FFP studies that support coverage. For instance:

Experts in RWD and RWE could offer feedback into the relevant considerations of implementing innovative evidence generation strategies, as well as ideas for how to ensure rigorous analysis for different sources of RWD, including patient-generated data.

Patients can elaborate on the outcomes that are most important to them as end users of a given product or service.

Providers can identify outcomes of importance that impact product uptake and how data collection methods may impact burden and clinical workflows.

For therapeutic areas for which there are a lot of novel products in development, CMS could consider implementing strategies to facilitate multi-stakeholder engagement to discuss evidence thresholds based on evidentiary questions and outcomes of interest to inform future guidance documents. Such efforts can inform the development of forthcoming Clinical Endpoints guidance documents and further clarify how manufacturers can use FFP guidance to inform their evidence planning.

This engagement could look similar to CMS’ Medicare Evidence Development and Coverage Advisory Committee (MEDCAC) meetings, which serve to provide expert guidance to CMS on clinical topics. These meetings, similar to MEDCAC meetings, could also be public to offer greater transparency into not only the processes, but the outcomes of these meetings. To support early and efficient data collection infrastructure development, the Institute recommends that CMS consider utilizing early, multi-stakeholder engagement to inform evidence thresholds and evidence generation strategies.

Conclusion
The Duke-Margolis Institute supports CMS’ continued efforts to make Medicare coverage processes more transparent, predictable, and timely. The proposed guidance document on Study Protocols that Use Real-world Data offers greater transparency into how CMS will evaluate elements of studies that support Medicare coverage. CMS can continue to pursue their goals through publication of a FFP study guidance document and through early engagement with a variety of stakeholders.

The Institute appreciates CMS’ consideration of our comments.

Sincerely,

Hannah Graunke – Senior Policy Analyst
Beena Bhuiyan Khan – Research Director of Payment and Coverage Policy

Disclosures
Beena Bhuiyan Khan is a former employee and shareholder of Abbott Laboratories.

References

1 Shirley V. Wang et al., “HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-world Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force,” Pharmacoepidemiology and Drug Safety 32, no. 1 (January 2023): 44–55, https://doi.org/10.1002/pds.5507.
2 Food and Drug Administration, “Use of Real-World Evidence To Support Regulatory Decision-Making for Medical Devices, Draft Guidance for Industry and Food and Drug Administration Staff; Availability,” Federal Register 88, no. 242 (December 19, 2023), https://www.federalregister.gov/documents/2023/12/19/2023-27852/use-of-real-world-evidence-to-support-regulatory-decision-making-for-medical-devices-draft-guidance.
3 Centers for Medicare & Medicaid Services, “Coverage with Evidence Development Amyloid PET,” CMS.gov, September 10, 2024, https://www.cms.gov/medicare/coverage/evidence/amyloid-pet.
4 Centers for Medicare & Medicaid Services, “Coverage with Evidence Development Leadless Pacemakers,” CMS.gov, September 10, 2024, https://www.cms.gov/medicare/coverage/evidence/leadless-pacemakers.
5 Centers for Medicare & Medicaid Services, “Coverage with Evidence Development Autologous Platelet-Rich Plasma,” CMS.gov, September 10, 2024, https://www.cms.gov/medicare/coverage/evidence/plasma.

Less

Dear Ms. Syrek Jensen,

Boston Scientific (BSC) is dedicated to transforming lives through innovative medical solutions that improve the health of patients worldwide. We develop and supply medical devices that serve the needs of many patients, including Medicare beneficiaries, in numerous clinical areas, including Cardiac Rhythm Management, Electrophysiology, Gastroenterology, Interventional Bronchoscopy, Interventional Cardiology, Interventional Radiology, Interventional Oncology, Interventional Peripheral Vascular, Pain Management, Urology, and Women’s Health. We appreciate the opportunity to provide comments regarding the Centers for Medicare & Medicaid Services’ (CMS) Proposed Guidance Document: Study Protocols That Use Real-World Data.

We commend and support CMS in developing proposed guidance for study protocols that use real-world data (RWD). The use of RWD in appropriate cases has the potential to provide valuable insights into the real-world effectiveness of treatments, including what treatments work best for specific patient populations across diverse settings of care, which may improve patient safety and outcomes. The use of RWD can be used to supplement the results and findings from randomized controlled trials, providing additional evidence to support FDA and CMS coverage decisions. Additionally, we support the use of standardized templates or tools to improve communication efficiency and consistency in study protocol development.

Recommendations
Overall, this guidance aligns with existing frameworks for RWD. However, three areas for additional clarification are recommended: 1) the definition of RWD; 2) study design methodology; and 3) standardization without making the RWD guidance too limited.

Definition of RWD and Integration of Study Findings into Coverage Decisions
We strongly recommend including definitions for RWD and fit-for-purpose study design with concrete criteria. While the guidance encourages adherence to a structured format (Section 5), it does not clearly define “fit-for-purpose” standards in a measurable way. To enhance clarity and consistency, we suggest specifying what criteria would be considered acceptable “objective success criteria” (Section 6) and providing examples. Additionally, we would like to see specific and unambiguous language regarding the use of health economics or finance data in CMS’s decision-making. Section 5 does not specify how findings from RWD studies will be integrated into CMS coverage decisions under Coverage with Evidence Development (CED). We recommend the guidance describes how CMS will use RWD from these studies to determine coverage and reimbursement. In addition, we highly recommend that CMS recognize the potential significant variations in RWD and include a section allowing reviewing on a case-by-case basis for those trial designs that are considered novel by applicants.

Study Design Methodology
When using RWD, unless using 100% Medicare claims data for the Fee-For-Service (FFS) Medicare population, we typically analyze data using a sample and use the results to make inferences about the target population. We recommend providing guidance on assessing generalizability beyond describing patient demographics. This could be achieved through the introduction of quantitative benchmarks, such as weighting techniques, inverse probability weighting, or sensitivity analysis, to ensure study populations accurately reflect real-world Medicare beneficiaries. Additionally, we suggest providing the specific name of the data source in section 7.6. For example, 100% Medicare fee-for-service claims Standard Analytical Files (SAFs), 5% SAFs, Medicare Research Identifiable Files (RIF), or Medicare Advantage data. Clarifying this information will assist in determining whether the findings of a study are generalizable to the Medicare population as mentioned in section 7.3.4.

The guidance acknowledges and provides a description of confounding adjustment and missing data handling (Section 7), which address common challenges in RWD studies. However, we recommend further clarification on the acceptance level by CMS for these potential limitations of RWD, which will help study sponsors better align their methodologies with CMS expectations.

Appendix A and B highlight the need for reporting data validation and quality control for studies using commercial vendor data. However, clarification is needed regarding whether CMS expects vendors to provide validation of the data for each study. While this guidance requests validation, it does not mandate third-party verification for data sources. We recommend incorporating independent data audits or source verification requirements, particularly for commercial datasets where data accuracy and transparency may not be consistent.

Standardization Without Making the Guidance Too Limiting
The current wording of the guidance suggests a potentially high burden for applicants. BSC recommends including data validation and quality control guidance that balances feasibility with the need for high-quality outputs. We recommend the use of standardized templates, such as those used in the National Coverage Analysis, to enhance consistency and streamline submissions. Additionally, ensuring the proposed statistical analysis requirements are rigorous yet flexible, allowing for methodological innovation without being limiting.

Conclusion
BSC supports the development of CMS’s proposed guidance for study protocols that use RWD. BSC recommends CMS focus on further defining RWD and study design methodology, while ensuring the guidance provides a structured framework without being too limiting. This balance will support rigorous, high-quality research while maintaining flexibility for diverse real-world study designs.

Thank you for considering our comments. Please contact me at liesl.hargens@bsci.com if you have any questions.

Sincerely,

Liesl Hargens
Vice President, Health Economics and Market Access
Boston Scientific Corporation

Less

To the Centers for Medicare & Medicaid Services:

Thank you for giving Novo Nordisk the opportunity to provide input to the draft guidance document entitled “Proposed Guidance Document: Study Protocols That Use Real-world Data” (the “Draft Guidance”). Please see our comments below.

Comments on the Draft Guidance:

Page 6: Comment regarding “contributions”. We suggest there should be a point about what are the new contributions by the study. so not only contribution, but also what is new.

Page 7: Comment regarding “Primary research question and objective”. We suggest to in-corporate the ICH E9(R1) Estimands framework including intercurrent events and strategies to handle these. Like we do for RCT. This would make research question even more clear. Intercurrent events including end of follow-up is not described until page 18. We see a benefit to describe the censoring criteria in the earlier section, especially when multiple entries is allowed for each patient.

Page 20: Comment to section on “Context and Rationale for Analysis Plan”. We strongly recommend to link this to the ICH Estimand framework.

Page 21-22: comment to section on “Primary Analysis” and “Secondary analysis”. We suggest to specify all five Estimand attributes here: Exposure, outcome, summary measure, target population, and strategies for handling intercurrent events. This information is needed to define a clear research question, and only by knowing what is to be estimated, it can be evaluated whether the suggested models, confounding methods etc are in fact estimating the right thing.

Page 21: comment to terminology “ Exposure contrast:”. We suggest to use terminology "summary measure" as this terminology is aligned with ICH Estimands framework.

Page 31: comment to section on “interim analysis and unblinding”. We do not think considerations regarding blinding/unblinding relevant to studies based on RWD from claims and EHR databases.

Page 32: Comment regarding “multiple testing”. We do not believe multiple testing is relevant for studies based on RWD from claims and EHR databases if sample sizes are extremely large and everything is significant. More important to look at the size of effect estimate, and whether it is a clinical relevant effect size.

Page 33: Comment regarding “randomization and blinding”. Above sections are more tailed observational study designs so we do not believe this section is applicable.

Page 35: Comment regarding terminology “value set”. Suggest to include short explanation or synonyms like codelist that explains what “value set” means.

Comments on Appendix B:

Page 8 : Comment to sentence “Objective success criteria of powered endpoints:”. Is this the same as “clinical relevant difference” used in the power calculation? Suggest to use the latter terminology.

Page 8: Comment to sentence “Median follow-up time of five years after exposure”. It is not clear from this when follow-up time begins and ends for the individual patient.

Page 9: comment to sentence including “retrospective”. Avoid terminology "retrospective". It is not clear whether you refer to analytical design, data collection or investigator perspective. STROBE guidance on observational studies recommend not to use this terminology. See also this publication https://pubmed.ncbi.nlm.nih.gov/36737851/

Page 10: Comment to “N/A”. Instead of N/A, we suggest to explain why you require one year of medicare claims data prior to index.

Page 11: Comment to the sentence “<1 year of Medicare claims data prior to their index procedure”. We suggest to either criteria is an inclusion or exclusion criteria. You are filtering on the same twice.

Page 13: Comment to terminology “prospective”. We suggest to avoid terminology retrospective and prospective. STROBE recommend not to use. see also this reference https://pubmed.ncbi.nlm.nih.gov/36737851/

Page 13: Comment to “exposure, comparator”. We suggest to call the populations as de-fined above, where they are named DCD and NDCD.

Page 13. Comment to outcome “Death observed within a median follow-up time of five years”. In our opinion this with the median follow-up time does not make sense as out-comes should be defined on subject level.

Page 14: Comment to the sentence “Will follow patients until study ends or death is observed”. This is not consistent with above where it says “max 7.5 years of follow-up”.

Page 15: Comment to “exposure, comparator”. We suggest to call the populations as de-fined above, where they are named DCD and NDCD.

Page 16: Comment o N/A. Suggest to replace “N/A” to “investigator”.

Page 19: Comment to the sentence “The Kaplan-Meier estimates of the cumulative incidence of survival and the log-rank test will then be computed.” Suggest to replace with “Cox model”.

Page 21: Comment on terminology “falsification endpoint ».We suggest to use terminology “negative control outcomes”.

Thank you for considering our comments. Feel free to reach out to me, if you have any questions or need additional clarification.

Less

Edwards Lifesciences (“Edwards”) appreciates the opportunity to comment on the proposed guidance document for “Study Protocols That Use Real-world Data” released by the Centers for Medicare & Medicaid Services (“CMS”) on January 17, 2025.

Edwards is the global leader of patient-focused innovations for structural heart disease. Our technologies address patient populations in which there are significant unmet clinical needs, such as structural heart disease, heart valve disease, and heart failure. Many of our technologies are utilized in the care of Medicare patients who often have complex needs with severe comorbidities and require specialized procedures and care. Because of this, we have a sincere interest in ensuring continuous improvement in the Medicare coverage landscape and that patients have more timely and predictable access to life-saving medical technologies and services.

Our comments focus on Edwards’ support of the following:

• Coverage pathways that allow for access to breakthrough devices; and
• The use of time-limited Coverage with Evidence Development (“CED”) in circumstances when CMS determines additional clinical evidence is needed; and
• The use of real-world data.

Additionally, Edwards recommends that CMS:

• Permit adequate flexibilities to accommodate fit-for-purpose study designs, including single-arm study designs; and
• Provide guidance on the process to make amendments to study protocols.

More detailed comments follow.

***

Access to Breakthrough Devices

Edwards supports coverage pathways that ensure patients have timely access to breakthrough medical therapies. Given the ongoing challenges and timeline for obtaining national coverage for emerging and innovative technologies, Edwards strongly supports policies that create processes and a framework for obtaining timely coverage for breakthrough designated medical technology. The process for obtaining coverage should be transparent and predictable. Edwards believes the proposed guidelines will help make the process more transparent and predictable for everyone involved, including CMS, study sponsors, and the public.

Coverage with Evidence Development

Edwards believes in and invests in creating evidence for our technologies. We support the use of National Coverage Determinations (NCD) with CED when CMS determines additional clinical evidence is needed to make a “Reasonably and Necessary” coverage determination. While we believe NCDs with CED requirements should be used carefully, we also believe that time- limited NCDs with CED can be a powerful tool for CMS to provide earlier and equitable access to promising technologies for Medicare patients. This also helps collect data to inform future clinical decision-making and coverage decisions.

Real-World Data

Because it is essential to better understand the outcomes of technologies outside the context of a randomized controlled trial (RCT), real world evidence (RWE) will play a critical role in continuously monitoring and analyzing the outcomes of interest during the maturation of the procedure and technology in the treatment of Medicare beneficiaries. Sources of RWE include electronic health records, registry data, claims data, and other documentation of patient health status outside the context of a trial.

Since real-world data (RWD) is typically collected in a standard care setting, RWE study results can be more generalizable to a broader population of beneficiaries. Furthermore, large sample sizes available in RWD allow for analyses that might not be feasible in RCTs (i.e., information about subpopulations). However, it is important to note that while RWE is a tool to supplement RCT data, RWE does not supplant the importance of robust clinical trial evidence.

Leveraging the strengths of both clinical trials and RWE study designs, the complete body of evidence that is generated under NCDs with CED should be used to support future reconsiderations and a durable coverage policy.

/ Recommendations /

1. Fit-for-Purpose Study Designs

Edwards agrees with CMS that the standardized format will facilitate efficient review of the protocol. Edwards believes a standardized template will enable transparency and help organize and align expectations between study sponsors and CMS. It allows for consistency across submissions, which helps sponsors understand what data elements are important and which enables a more efficient and streamlined review by the Agency. Edwards appreciates that, in developing the HARPER+, CMS solicited feedback from industry, government, and academia.

Edwards believes that the template, as proposed, is exhaustive and some elements may not be relevant to every study. We believe some level of flexibility is necessary when reviewing submissions of study protocols, given the range of study designs and unique attributes of data sources that may be utilized in these studies.

Furthermore, we encourage CMS to consider the full range of fit-for-purpose study designs, including a range of data sources, methodological approaches, and the requirement for a comparator group. Edwards recommends that fit-for-purpose evidence generation in the context of Medicare coverage determinations should:

• Be designed with patients and their needs in mind; and
• Be limited to the essential data needed to address unanswered questions and demonstrate to CMS that the innovation is reasonable and necessary; and
• Utilize, where appropriate, retrospective RWD, including payer claims and electronic clinical and patient data, as an alternative registry data collected prospectively; and
• Allow for the collection of data in the most efficient and least burdensome way for providers and the study sponsor.

Finally, we request that CMS consider single-arm studies in cases where a comparator group may not align with the specific attributes of a technology or a disease state and its evidence gaps. In some cases of breakthrough technologies, there may be valid reasons why including an active comparator group is not appropriate or is not possible. We ask CMS to consider such unique circumstances when evaluating whether a study is fit-for-purpose.

2. Process to Make Amendments

Edwards appreciates that CMS recognizes that “All protocols will likely undergo revisions based on discussions between the sponsor and CMS.” Edwards requests that CMS provide clarity on the types of revisions that would prompt a protocol amendment and the process to submit such revisions.

***

Edwards thanks CMS for its consideration of our comments and recommendations on the proposed guidance for study protocols that use RWD. We commend CMS for its efforts to improve transparency and efficiency in coverage pathways for innovative breakthrough technologies. We appreciate CMS’s continued willingness to collaborate with all healthcare stakeholders, including medical technology companies, to improve our nation’s healthcare for Medicare beneficiaries.

If you have any questions regarding the comments provided in this letter, please reach out to me.

Sincerely,
Leah Kegler
SVP, Government Affairs and Policy

Less

Abbott welcomes the opportunity to provide comments on the Centers for Medicare & Medicaid Services’ (CMS) proposed Medicare guidance document on Study Protocols That Use Real-world Data (Proposed Guidance).

Abbott is committed to helping people live their best possible life through the power of health. For more than 135 years, we’ve brought new products and technologies to the world — in nutrition, diagnostics, medical devices and branded generic pharmaceuticals — that create more possibilities for more people at all stages of life. Today, 114,000 Abbott employees are working to help people live not just longer, but better, in the more than 160 countries we serve.

As a leading medical technology and nutrition manufacturer, we seek to ensure that Medicare coverage policies promote beneficiary access to high-quality healthcare innovations that address unmet medical needs and improve health outcomes. To that end, we commend CMS for issuing its Proposed Guidance, which is intended “to provide a framework for more predictable and transparent evidence development.”

Abbott agrees that the Proposed Guidance and accompanying real-world data (RWD) protocol template would help to standardize the protocol development process. We offer specific comments below that focus on three main themes for improving the Proposed Guidance and template:

1. Providing clarifications to lessen ambiguity
2. Ensuring requirements are broadly applicable to real-world evidence studies
3. Accommodating information that cannot be shared publicly or is unavailable during protocol development

Our recommendations follow.

Background Section

CMS recommends that an Evidence Development Plan (EDP) be submitted with all coverage with evidence development (CED) study protocols to frame the overall evidence generation strategy, of which a RWD study may be one component. We suggest that CMS develop an EDP template as a separate guidance document, and we encourage CMS to provide a comment opportunity on the draft EDP template.

CMS strongly encourages prospective study sponsors to use the draft RWD protocol template, since “[d]epartures from the requested format and content may lead to delays in the study approval process.” We would appreciate additional clarity regarding the expected protocol review and approval timeline when the draft RWD study template is used.

We also recommend that CMS include information regarding the method that sponsors should use to submit the RWD protocol and other associated documentation.

Section 3. Amendments and Updates

CMS proposes that study sponsors document amendments and updates to the study protocol, but “minor changes to the protocol do not require a formal resubmission and review by CMS.” We believe additional clarity in this section would be helpful, including examples of minor changes and those that would require CMS review.

Section 5. Rationale and Background

CMS proposes that the protocol Rationale and Background section include “CMS-identified evidence deficiency and knowledge gaps.” We recommend that CMS revise this language to reference “CMS-identified evidence deficiency and knowledge gaps being addressed by CED study,” since some gaps in the evidence base may be addressed through separate clinical studies.

Section 7.5. Data Analysis

We request that CMS provide additional information regarding its criteria related to treatment effect for purposes of the RWD protocol. For instance, it would be useful to have clarification regarding whether CMS accepts either a superiority test or a non-inferiority test, and the extent to which these criteria are dependent on the therapeutic context.

Section 7.9. Study Size and Feasibility

CMS proposes that sponsors report projected study sizes to minimally detect a pre-specified effect with a pre-specified statistical precision. CMS states that it will review this section, including the assumptions used in the statistical power calculations, to determine whether the study will be statistically powered to answer the research question.

We suggest modifying the green helper text in this section to address only the feasibility of achieving the calculated sample size. We recommend that details regarding power and sample size calculation are more appropriate in the Statistical Analysis Plan section (Section 7.5.1).

Section 9.Top Threats to Successful Completion

CMS proposes that sponsors identify the top three threats to the successful completion of the study objective and associated mitigation strategies, which CMS intends to use to “rank the risks and prioritize the mitigation strategies.“

We note that a “top threats” section is not a standard element of research protocols. Concepts of risk mitigation are inherently embedded in different protocol sections, including the study design and statistical analysis plan sections.

If CMS seeks a narrative regarding risks to successful study completion, we suggest including this question as a component of the Limitation of the Methods section (Section 8). Furthermore, to the extent that CMS intends to review such information to rank risks and prioritize mitigation strategies, CMS should provide additional details regarding the criteria it would use to make such assessments and how the assessments impact the study approval process.

Section 12. Applicable Federal Regulations

This proposed section includes a list of “major tenets of compliance with the applicable federal regulations on privacy and patient protection.” CMS directs sponsors to confirm the statements or “write your own language for this section.”

We request that CMS either confirm that individual rows in the proposed text may be deleted if not applicable to the RWD study, or add an “N/A” column to indicate a provision that is not applicable to a specific RWD study (e.g., if a non-CMS data source is being used).

Appendix A. Additional Statistical Considerations

CMS references the terms “interim analysis” and “interim analyses” throughout this section. However, this terminology has formal statistical implications that are typically applicable to prospective clinical trials but not observational studies. We recommend that CMS use a more general term, such as interim data review or interim evaluation, to accommodate interim reviews of findings without implying formal statistical adjustments.

At 15.2.1, CMS notes that an independent data monitoring committee may be established to review “interim analyses for some complex studies.” We agree that an independent data monitoring committee may not be applicable for all studies, such as for studies involving retrospective observational data.

Appendix B. Data Validation Output

CMS proposes that “[t]o the extent possible,” sponsors should report the validation of the data used in the study in the protocol. CMS states that at a minimum, data validation and quality checks will be expected as part of the interim analysis.

We recommend that CMS amend the Data Validation Output section to account for situations in which it is not feasible to include the data validation in the study protocol. In such cases, the sponsor should be instructed to include a statement in Appendix B that the data validation output will be reported in the first interim report.

Appendix C. Data Use Agreement

CMS requests that sponsors provide a copy of the Data Use Agreement (DUA) as Appendix C “if applicable and permissible.” We observe that it is rarely permissible to share the data use agreement between the study sponsor and data vendor. In such cases, we recommend that in lieu of a copy of the Agreement, CMS accept a statement from the sponsor such as: "this protocol is being conducted on data accessed from XXX under DUA XXX."

Appendix D. Data Dictionary

CMS proposes that the study protocol include a data dictionary providing agreed-upon definitions, including code language, format, and categories for each variable used in the research data set. CMS adds that at a minimum, data validation and quality checks will be expected as part of the interim analysis.

We recommend that the data dictionary section be removed from the study protocol, since it is more appropriate to include the data dictionary in the interim and/or final report. As data collection begins, new variables may be identified or other clarifications may become appropriate, which would necessitate refinements to a data dictionary established in the protocol.

Confidentiality Considerations

CMS states in the preamble helper text that “CMS intends that information in this protocol will not be publicly shared; however, information contained in this protocol may be discoverable via the Freedom of Information Act.” However, in the August 12, 2024, Transitional Coverage for Emerging Technologies (TCET) Pathway final guidance, CMS states that “[e]lements of the CMS and AHRQ approved EDPs, specifically the non-proprietary information, will be made publicly available on the CMS website upon posting of the proposed TCET NCD.”

We request that CMS provide additional details regarding the confidentiality of the study protocol and related documents, including the EDP. Specifically, we would appreciate CMS identifying the specific data elements that may be publicly shared and the circumstances and timeline for any such disclosure.

* * * * *

We appreciate your consideration of our comments. Please feel free to contact me if you have any questions or if you need additional information.

Less

March 18, 2025

SUBMITTED ELECTRONICALLY VIA MEDICARE COVERAGE DATABASE AND CAGInquiries@cms.hhs.gov

Coverage and Analysis Group
Centers for Medicare & Medicaid Services
Department of Health and Human Services
7500 Security Blvd.
Baltimore, MD 21244-1850

RE: Comments on Proposed Guidance Document: Study Protocols That Use Real-world Data

Dear Centers for Medicare & Medicaid Services Coverage and Analysis Group:

Organogenesis, Inc. appreciates the opportunity to comment on the Centers for Medicare & Medicaid Services (CMS) Proposed Guidance Document: Study Protocols That Use Real-world Data (RWD). We are pleased that CMS is working on additional guidance to stakeholders on this important topic. Such guidance can help provide clarity for studies that involve real-world evidence (RWE) and RWD.

Organogenesis is a leading regenerative medicine company, with products that are critical to the treatment of chronic wounds, burns, and orthopedic injuries. We have been investing in innovation in this space for 40 years. We are also a leading, longstanding contributor to the development of evidence supporting use of skin substitutes in treating diabetic foot ulcers (DFU), venous leg ulcers (VLU), and other wounds. Our studies include randomized controlled trials (RCTs) and RWE studies that use RWD.

Studies that use RWD are important because they provide affirmative evidence demonstrating the effectiveness and safety of medical interventions in real-world settings and circumstances, leading to more informed clinical decision-making. They also allow for larger sample sizes and more robust inclusion of Medicare-eligible patients compared to RCTs, as recognized in CMS’s National Coverage Analysis Evidence Review guidance (https://www.cms.gov/medicare-coverage-database/view/medicare-coverage-document.aspx?mcdid=37). The 21st Century Cures Act likewise supports and reflects the importance of RWE and RWD.

At Organogenesis, our intentional evidence development program for a number of products has focused on RWD/RWE studies because of the benefits of RWD/RWE study design, such as:

• Larger numbers of patients and sites
• Longer-term follow-up demonstrating durable results
• Ability to include large numbers and percentages of patients generalizable to the Medicare population

Based on our extensive experience with both RCTs and RWD/RWE studies, we offer the following comments on CMS’s proposed template for study protocols that use RWD.

Overall, we believe the proposed template is workable and helpful as a framework for developing and designing study protocols that use RWD. In particular, it provides clarity and structure on the elements of a protocol to complete a RWD study.

Importantly, however, there are aspects of the proposed template that we believe are not appropriate for inclusion as required elements in a template for protocols with RWD/RWE study designs, because they are more consistent with RCT protocols than RWD protocols. For example:

• Proposed Template Sections 15.2.7 and 15.5 include elements related to randomization and blinding. Although the heading and first sentence of Section 15.5 helpfully and appropriately include qualifying “If Applicable” language after “Randomization and Blinding,” we strongly recommend revising the language in these sections of the template to be more direct and explicit in recognizing that randomization and blinding typically are not applicable to studies that use a true RWD/RWE design.

  While randomization and blinding are highly appropriate as required elements to address in RCT protocols, they are not appropriate to include as required elements in RWD/RWE study protocols. Indeed, for RWD/RWE studies, randomization and blinding often can undermine and be counter to the purpose of the real-world nature of the study, which is intended to reflect actual treatment practices at the relevant points of care and to show how patients respond to an intervention in those real-world settings with real-world circumstances. In other words, RWD/RWE studies often cannot be meaningfully blinded while still providing safety and effectiveness data generalizable to real-world practice settings.

  For example, in our experience with RWD/RWE studies in the wound care space, methods of blinding in wound care trials disturb the ability to study and assess the comparative effectiveness of an investigational therapy vs. a control therapy as used in the real-world practice setting. While RCTs show “whether a product can work in a controlled setting,” RWD comparative effectiveness analysis studies “show whether a product does work in widespread use”; as such, RWD “is likely to be more useful to clinicians in their everyday practice of medicine.” Sabolinski ML, Archambault T. Real-world data analysis of bilayered living cellular construct and fetal bovine collagen dressing treatment for pressure injuries: a comparative effectiveness study. J. Comp. E?. Res. (2024) e230109, https://becarispublishing.com/doi/10.57264/cer-2023-0109.

  As a result, randomization and blinding very often are simply not relevant to such studies. Any final guidance on this topic should clearly and directly acknowledge these realities of RWE studies that use RWD.

  Significantly, RWD/RWE study protocols can, and frequently do, involve other study design elements that help minimize the risk of bias. For example, although health care professionals in real-world settings are not blinded to the care they are providing, RWD/RWE studies can involve protocols that ensure the blinding of statisticians and individuals performing data analysis. These types of considerations appear to be reflected, for example, in the following question included in Section 15.2.7 of the proposed template: “Who will perform any final analyses and remain blinded?” While questions along those lines are appropriate to include in a template for RWD/RWE studies, we urge CMS to ensure that the template in any final guidance does not require or over-emphasize elements such as randomization and blinding, which are important for RCTs but often inapplicable to RWE/RWD studies and, indeed, can significantly undermine the key objective of studying “real-world” settings and circumstances.

  Additionally, as CMS has recognized in its National Coverage Analysis Evidence Review guidance, blinding is “especially important for subjective outcomes, such as pain or quality of life, where beliefs about an intervention may lead to a perceived outcome improvement by either the patient or investigator.” CMS Guidance Document, CMS National Coverage Analysis Evidence Review (Aug. 7, 2024), https://www.cms.gov/medicare-coverage-database/view/medicare-coverage-document.aspx?mcdid=37. Accordingly, risk of bias also is minimized in studies that focus on objective outcomes (e.g., measurements of wound size and closure) that do not involve subjective elements or assessments.

  RWD/RWE studies can further minimize bias by having designs ensuring that all patients who meet the study’s inclusion/exclusion criteria are included and evaluated in the analysis. In such studies, there is no loss to follow up, thus eliminating the risk and potential bias of missing data. This is a major advantage of many RWD/RWE studies as compared to RCTs.

  We therefore recommend revising Sections 15.2.7 and 15.5 of the proposed template for RWD/RWE study protocols to more directly and explicitly recognize that randomization and blinding typically are not applicable to studies that use a true RWD/RWE design.

• Proposed Template Section 15.2.3 addresses “Stopping Rules” and states that study sponsors should “[d]ocument any formal stopping rules for futility, efficacy, or lack of power.” This is another element of the proposed template that is not applicable to many RWD/RWE studies. For example, when conducting a retrospective study, the concept of “stopping” the study (or the relevant medical intervention) is not applicable. We therefore recommend directly and expressly recognizing in this section of the template that “Stopping Rules” will not be applicable for many RWD/RWE studies.

In summary, we urge CMS to keep this guidance document and template consistent with the nature and reality of RWD/RWE studies. We therefore recommend removing or revising any elements in the proposed template that pertain principally to RCTs but not RWD/RWE studies, such as those noted above.

We also note that, although the proposed guidance document directly applies only to RWD/RWE studies in the context of National Coverage Determinations (NCDs) using the Coverage with Evidence Development (CED) paradigm, the proposed guidance, if finalized, likely may be used to evaluate and assess RWD/RWE studies in additional contexts going forward, as it appears to be intended to identify core expectations regarding design, analysis, and implementation for rigorous RWD/RWE study protocols. We believe this further underscores the importance of ensuring that the final guidance, when issued, reflects appropriate changes to ensure that the template for study protocols that use RWD remains consistent with the scientific nature and realities of RWD/RWE studies and does not include elements that can undermine true RWD/RWE study designs.

Additional suggestions for modifications to the proposed guidance document are as follows:

• Registration of a protocol/study on clinicaltrials.gov should not be a universal requirement. Clinicaltrials.gov focuses primarily on registering and reporting progress, results, and termination of clinical trials. Not all types of RWD studies will necessarily align with information required by clinicaltrials.gov. Accordingly, registration on clinicaltrials.gov may not be relevant for many types of RWD studies (e.g., retrospective, observational studies, and non-interventional cohort studies).
  
  
• Tables can be useful in many circumstances but should not be required. We recommend that the guidance document be updated to recognize that either standardized tables or narratives that provide comparable, comprehensive information may be used. Depending on the specific study and its design, narratives may be more appropriate and useful than tables.

Finally, we recommend incorporating language in the guidance expressly recognizing the need for some flexibility in protocol elements and format based on specific study designs. We are concerned that certain language in the proposed guidance could be read as a requirement to adhere to every specific section and particular formatting aspect of the template in all cases. A lack of flexibility on these considerations can affect the purpose of specific RWD studies, which is to reflect what occurs in real-world practice settings and to create a report of patient data that is more representative of patient outcomes in real-world clinical practice, as compared to the highly controlled and more limited environments of RCTs. RWD studies provide equally important information as the data obtained in RCTs. If a guidance document focused on RWD were to be too prescriptive or were to push RWD study design too far in the direction of RCT requirements, then the most comprehensive review of all available medical and clinical evidence (i.e., RCT and true RWD study data) could not occur. Both prospective RCT data and retrospective RWD studies are extremely valuable, and both provide important data and evidence regarding safety and effectiveness.

As an example, the proposed guidance document states:

We strongly recommend that CMS adjust this language to note that while CMS “encourages prospective study sponsors to use the template,” CMS also recognizes that certain modifications of the template may be appropriate depending on the specifics of an individual study protocol, and that some study designs may not necessarily fit within every aspect of the template structure and format.

Thank you for your consideration of our comments on this important topic.

Sincerely,

Antonio Montecalvo
Vice President, Health Policy
Organogenesis

Less

March 17, 2025

Tamara Syrek Jenson
Director
Coverage and Analysis Group
Centers for Medicare & Medicaid Services
Mailstop: S3-02-01
7500 Security Blvd
Baltimore, MD. 21244

Submitted electronically to CAGInquiries@cms.hhs.gov

RE: Proposed Guidance Document: Study Protocols That Use Real-World Data

Dear Director Syrek Jenson,

On behalf of the members of the Alliance of Wound Care Stakeholders (“Alliance”) Chair Dr. Matthew G. Garoufalis, Vice Chair Kara Couch and I are pleased to submit our feedback on the CMS proposed Guidance Document: Study Protocols That Use Real-world Data. The Alliance is a nonprofit multidisciplinary trade association of physician medical specialty societies, clinical and patient associations, wound care provider groups, wound care clinics and wound care product manufacturers whose mission is to promote quality care and access to products and services for people with wounds through effective advocacy and educational outreach in the regulatory, legislative, and public arenas.

The Alliance supports coverage based on evidence. We also have a long history of being proponents of utilizing Real World Data (RWD) for use in Real World Evidence (RWE) – especially in wound care. We demonstrated this in our article “Consensus Principles for Wound Care Research Obtained Using a Delphi Process” where one of our principles are “National or formal wound registries should be developed with realworld data collection.”ⁱ

We appreciate the Agency’s commitment to utilize RWD for RWE as part of the national coverage determination (NCD) coverage with evidence development (CED) process and are supportive for the use of data from registries, claims, and electronic health records to support the evaluation and coverage of medical devices. In fact, in 2005 the US Wound Registry (USWR) was created, and was recognized by CMS as a Qualified Clinical Data Registry in 2014. USWR clinical registries are listed on ClinicalTrials.gov. Registries are ideal vehicles for providing comparative effectiveness research in wound care because they include real world patients often excluded from RCTs and provide insight into the actual effectiveness of treatments. A recent article reviewed the availability of real world data in wound care. Comprehensive Landscape Analysis for Usable Real-World Wound Care Data.ⁱⁱ

However, using RWE only for NCDs with CED is rather limiting as there are very few NCDs based on CED and we believe that if the Agency is permitting RWE to be used for coverage purposes the use should be extended to all coverage determinations and not just NCDs based on CED. The FDA, NIH and a significant body of medical literature recognize that RWE constitutes quality supporting evidence. Importantly, RWE addresses health disparities as it demonstrates efficacy. However, health disparities, patient outcomes and populations that are often underrepresented in randomized controlled trials (RCTs). RWE captures how care is administered to patients which is often missing from the highly controlled environment of an RCT. Accordingly, the Alliance submits that any coverage policy should consider RWE as an adjunct to Level 1/Level 2 evidence when determining product coverage since it aligns with the Congressional intent of the 21st Century Cures Act in 2016 encouraging the use and acceptance of RWE.ⁱⁱⁱ Therefore any guidance offered in terms of appropriate evidentiary standards including the use of RWE should apply not only to NCDs based on CED or NCDs in general but also to Local Coverage Determinations (LCDs).

Our specific comments related to the guidance document follow.

According to the guidance document, the evaluation of the evidence will be done via the HARPER+ protocol. As a general matter, HARPER is a very well thought out protocol for RWE studies and it is a sound method. The HARPER+ is a more expanded protocol that was the result of collaborative efforts between CMS and the FDA – which the Alliance supports. This standardization will streamline the process of creating and submitting RWE study protocols to facilitate an efficient review process by CMS - ensuring that companies submit the required information uniformly. This will lead to more efficient approval of proposed CED studies by reducing the need for multiple rounds of review.

We offer two comments specific to HARPER+:

1. HARPER/HARPER+ has nothing to do with evaluating the evidence for coverage purposes and therefore the Alliance questions how this will result in coverage decisions. What RWD/RWE will be evaluated in the process of making those determinations. This information is lacking from the guidance document and therefore more clarity is needed.
2. Additionally, since HARPER is already utilized by researchers for other submissions, the Alliance recommends that if HARPER+ is adopted following this guidance, any modifications and additions introduced in HARPER+ should be included in an "Appendix to HARPER." This approach would promote consistency and streamline the process for researchers, enabling them to seamlessly integrate required elements into their existing HARPER-based protocols when submitting to CMS.

The Alliance requests clarification in Section 7.5 regarding whether CMS will accept a superiority test or a non-inferiority test, and the extent to which these criteria are dependent on the therapeutic context. This information is lacking in this guidance document and therefore additional information is necessary for the RWD protocols.

In section 7.5.1. CMS discusses statistical analysis plans. Throughout the guidance document, CMS uses the terms “study design” and “study protocol”. However, in this section CMS uses the term “trial design”. In order to be consistent, the Alliance recommends using the term “study design” rather than “trial design”.

Furthermore, in Section 7.6, "Data Sources" the guidance document states, "Where appropriate, provide references to peer-reviewed literature that used the same RWD source for a similar target study population." This is a bit ambiguous and could mean different things to different people. As such, we recommend providing some clarifications as to what is meant by "same RWD source". For example, would this refer to:
a. A specific information system from a specific institution (e.g. the Epic EHR system at George Washington University Hospital in Washington DC)
b. A specific type of information system (e.g. EHR systems)
c. A registry which is used for multiple studies for the same class of product

The Alliance believes that this information needs to be clarified and examples provided in the guidance document.

In Section 7.9 Study Size and Feasability– there is no information as to what is considered a valid study size. In the wound care space study sizes tend to be small as a result of patients who have multiple co morbid conditions among other reasons. As such, we would recommend that wound care studies should not be judged the same as other sectors as it relates to study size. Any assessment of evidence should be conducted in the context of other studies in the same space utilizing acceptability criteria consistent with that for other treatments that have been deemed reasonable and necessary with the same exact indication.

Finally, the guidance document states, “minimize bias, it is important to specify at the beginning of the study who will perform the interim analysis, to what information they will have access, and what information, if any, might be released to the general public or other groups after the interim analysis and other study milestones” but there is no discussion on how bias is evaluated or how it will be treated as evidence is reviewed. Any clinical trial that is being reviewed comes with bias as it is always funded by the manufacturer. So, the Alliance requests that CMS provide further clarity on the issue of bias and how bias or perceived bias will be addressed when evaluating RWE for coverage purposes. It would be helpful to have some examples in which the Agency believes bias can be mitigated and in what way. We would direct you to the comprehensive paper from the USWR recommending standards for reporting observational registry data from electronic health records.^iv The USWR has also created a white paper summarizing these standards. ^v

The Alliance recommends that additional information be placed in this guidance document includes (but is not limited to) the following areas:

1. What type of registry data would be acceptable?
2. What are the trial designs that CMS will accept of a said data set to consider it as a surrogate marker for an actual trial?
3. When the protocol starts, what fields are needed since a retrospective look back would just be to compare for the control? More guidance on this issue would be welcome.
4. Does the database have to be registered?
5. What will be required in order to add an indication to a product that has already been approved/cleared?
6. Can data from outside the US be utilized?

The guidance document is silent or has limited information on these questions and it would be helpful to better understand more about what those controls are, what the process should be, as well as what is being accepted by CMS. As such, more information is necessary in this guidance document.

The Alliance believes that this guidance document is a good start in allowing for RWE to be used for NCDs with CED. The guidance will lead to more efficient approval of proposed CED studies by reducing the need for multiple rounds of review and the use of RWD for RWE will further allow data to be obtained from vulnerable populations (e.g., the elderly and children) that often are not able or willing to participate in trials and therefore will improve the underrepresentation of minorities. However, the guidance is rather limiting as NCDs with CED are the exception rather than the rule when assessing evidence for coverage purposes and the use of RWE should be extended to all coverage determinations. We would be happy to be a resource for CMS as it moves forward with evaluating the use of real world data for use as real world evidence for coverage of medical devices. If you require additional information or have any questions, please do not hesitate to contact us.

Sincerely,

Matthew G. Garoufalis, DPM, FASPS, FACFAOM, CWS
Alliance of Wound Care Stakeholders Chair

Kara Couch, MS, CRNP, CWCN-AP, FAAWC
Alliance of Wound Care Stakeholders Vice Chair

Marcia Nusgart R.Ph.
Chief Executive Officer

---

i Panel On Wound Care Evidence Based Research (POWERTM) “Consensus Principles for Wound Care Research Obtained Using a Delphi Process; Wound Rep Reg (2012) 20 284–293 © 2012 by the Wound Healing Society
ii Index Wounds. 2024;36(11):384-391. doi:10.25270/wnds/24076
iii 21st Century Cures Act, Pub. L. No. 114-255 (2016)
iv Carter MJ. Harnessing electronic healthcare data for wound care research: Wound registry analytic guidelines for less-biased analyses. Wound Repair Regen. 2017;25:564-73
v Fife CE, Eckert KA. Harnessing electronic healthcare data for wound care research: standards for reporting observational registry data obtained directly from electronic health records. Wound Repair Regen. 2017;25:192-209. Can also be accessed by clicking this link: USWRWhite-Paper.pdf

Less

My parents depend on Medicare. My Dad is a veteran who was in Vietnam. They depend on Medicare and social security to cover medical expenses in their golden years. If they had to pay out of pocket for [PHI Redacted], they could never afford it. The healthcare system is outrageously overpriced and this would be a death knell if they were cut off or reduced. The cost of all expenses these days is through the roof for basics, please do not touch Medicare and work on lowering health care and medication costs. Let’s not move in the wrong direction.

Less

First, the authors of the proposed guidance are to be commended for this thoughtfully written document, and their commitment to ensuring that use of real-world data conforms to sound scientific principles. The comments below are offered in the spirit of improving what is already a very strong start.

1. Re: Section 7, “Research Methods”
Tables 4-8 and 10 prompt for the source of any phenotyping algorithm(s) used to define time 0 criteria, inclusion criteria, exclusion criteria, exposures, outcomes, or covariates. The guidance states that if a novel algorithm is used, “investigators” should be stated as the source. It seems likely that in such cases, CMS would like the specifics of the novel algorithm to be provided, but this is not explicitly stated. We would recommend that such details be required, to allow CMS to determine the suitability of the algorithm(s) used and for purposes of reproducibility. Alternatively, if an algorithm is proprietary and/or cannot be explicitly described (e.g. because it was developed through machine learning), that details regarding its validation be provided.

2. Section 7.6, "Data Sources"
This section states: "Where appropriate, provide references to peer-reviewed literature that used the same RWD source for a similar target study population." We would like to suggest clarifying the meaning of "same RWD source", i.e. the level of specificity intended. For example, would this refer to:
a. A specific information system from a specific institution (e.g. the Epic EHR system at Acme Hospital in Philadelphia, PA)
b. A specific information system without reference other than a specific institutions (e.g. Epic EHR systems)
c. A specific type of information system (e.g. EHR systems)

3. Section 19, "Appendix E. Value sets"
We are encouraged to see this section included in the proposed guidance, since the details of value sets used in a real-world data study may have a significant impact on the study’s results, and yet their importance is often overlooked. We would recommend that the template go a bit further in the detail with which value sets are described, including the following:
1) Details of the semantic scope for the value set. This would consist of a human-readable description of precisely what was intended to be included and not included in the value set, including “gray areas” that might not be obvious from merely perusing the title of a value set. For instance, a value set for “diabetes mellitus” might or might not include gestational diabetes mellitus, based on the needs of the study; This proposed section would indicate that explicitly, removing ambiguity and ensuring clarity as to whether the inclusion or exclusion of a code was intentional or erroneous.
2) The version of the coding system on which the value set was based. All major health care coding systems are updated regularly, and these updates may include new codes. An explicit indication of the coding system version used could help clarify whether the omission of a code was intentional or due to the code appearing after the value set was created.
3) Methodology for identifying codes to review for inclusion. Most value sets are created by clinical experts who review codes to determine if they meet the semantic definition of the value set. However, they almost never review all codes in a coding system, since most coding systems are very large. Instead, they use various methodologies to identify subsets of the coding system to review for potential inclusion in the value set (including searching for codes whose description matches a particular text pattern, or selecting a section or “chapter” of a coding system for review, etc.). While a study would in theory be reproducible with the value set content only, since value sets are in constant development (as mentioned above), including this detail would facilitate true reproducibility over time.
4) Unambiguous referencing of coding systems (e.g. differentiating ICD-10 from ICD-10-CM, or SNOMED CT International from SNOMED CT US Edition). We would suggest this be handled by requiring coding systems to be identified using the unique URL identifiers for coding systems established by HL7 (see: https://terminology.hl7.org/external_terminologies.html).

Less

Proposed Guidance Document: Study Protocols That Use Real-World Data

Dear Centers for Medicare & Medicaid Services (CMS):

ISPOR – the professional society for health economics and outcomes research - is pleased to respond on behalf of its membership to your guidance document entitled “Proposed Guidance Document: Study Protocols That Use Real-world Data.”
ISPOR is a scientific and educational society with many of its members engaged in evaluating health technologies, including pharmaceuticals, medical devices, and other interventions. We have a large membership living and working in 110 countries globally, across a range of disciplines, including health economics, epidemiology, public health, pharmaceutical administration, psychology, statistics, medicine, and more, from a variety of stakeholder perspectives, such as the life sciences industry, academia, research organizations, payers, patient groups, government, and health technology assessment bodies. The research and educational offerings presented at our conferences and in our journals are relevant to many of the issues and questions raised in this request for information.
The response to this consultation was led by authors of the HARPER report. Comments were solicited from the ISPOR Real-World Evidence Steering Committee, and ISPOR Real-World Evidence Special Interest Group. The attached document provides a summary based on their comments. We hope they prove useful.
ISPOR would be happy to answer any questions about our response, to serve as a partner, or to participate in any follow-up consultations on the relevant program items mentioned within the report.

On behalf of the members of ISPOR – the professional society for health economics and outcomes research – we appreciate the Centers for Medicare & Medicaid Services’ efforts in preparing a guidance which provides clear and specific direction to the public on the critical elements of a study protocol that relies on real-world data (RWD). ISPOR supports RWD and real-world evidence (RWE) initiatives in many forms and stands firmly behind CMS’s efforts to require transparency in RWE studies. To that end, ISPOR is quite pleased to see that our good practices report, developed jointly in partnership with the International Society for Pharmacoepidemiology (ISPE), serves as the foundation for the guidance.

We noted that the guidance cites the Pharmacoepidemiology & Drug Safety article of the HARPER recommendations. Since its publication was subject to an agreement between ISPE and ISPOR that gives neither party first right of citation, we ask CMS to add the following Value in Health citation to the guidance:

Wang SV, Pottegård A, Crown W, et al. HARmonized Protocol Template to Enhance Reproducibility of Hypothesis-Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force. Value Health. 2022;25(10):1663-1672. doi:10.1016/j.jval.2022.07.012.

In terms of the contents of the guidance, we reached out to the ISPOR RWE Special Interest Group leadership to gather comments that would improve its clarity and impact. Given that the guidance largely focuses on the existing good practices report, our comments are limited.

1. Because the guidance is built from existing work, members noted that there were sections which would benefit from more thorough references to the source material (HARPER). One such instance is to clarify the intended audience for using the HARPER+ template. As noted in the guidance section titled “Development of the RWD Study Template”, HARPER+ is intended for use by manufacturers or other sponsors. It should emphasize that it is not limited to those stakeholders alone. Templates like HARPER and HARPER+ promote visibility and transparency in studies and study data, which benefits a variety of stakeholders beyond regulators and manufacturers. For example, a payer organization informing their decision-making process, or a data provider curating a fit-for-purpose dataset. Another instance where explicit reference to HARPER would strengthen the guidance would be to highlight examples from the original HARPER, particularly for the guidance section on “Research Question and Objectives”.

2. In the Preamble, it is stated that CMS proactively published this guidance to “provide a framework for more predictable and transparent evidence development”. To that end, we suggest that CMS require sponsors to post their protocols on a credible RWE-specific protocol registry and add this requirement to the guidance document. Protocol registries that are posted before initiation of a research study are crucial for ensuring the transparency of RWE designs. ISPOR collaborated with ISPE, the Duke Margolis Center, and NPC to offer such a registry, which is freely available on the Center for Open Science via this link: https://osf.io/registries/rwe/discover. Additional information about this registry, along with instructions for use, are here: https://www.ispor.org/strategic-initiatives/real-world-evidence/real-world-evidence-registry. We suggest that CMS include the registry and accompanying information as a resource in the final guidance. Note that users of this registry are asked to utilize HARPER as the template for their RWE protocols. A link to the protocol registry listing should also be included as part of the Title Page of protocols submitted to CMS, along with study contributors.

We acknowledge ISPOR members Shirley Wang and William Crown for their assistance in assembling these comments, as well as ISPOR staff Laura Pizzi, Kelly Lenahan, and Madeline Shipley.

Less

Philips is a leading health technology company focused on improving people's health and well-being through meaningful innovation. Philips’ patient- and people-centric innovation leverages advanced technology and deep clinical and consumer insights to deliver personal health solutions for consumers and professional health solutions for healthcare providers and their patients in the hospital and the home. Philips’ diagnostic and therapeutic solutions - diagnostic imaging, image-guided therapy, and cardiac monitoring – are provided to Medicare beneficiaries.

Philips is committed to supporting Medicare in the development of coverage policies that promote beneficiary access to innovations that address unmet medical needs and improve health outcomes. To that end, we commend CMS for issuing the Proposed Guidance Document: Study Protocols That Use Real-world Data (RWD), which is intended “to provide a framework for more predictable and transparent evidence development.” The Proposed Guidance and accompanying real-world data (RWD) protocol template can help standardize the trial protocol development process.

The integration of RWD into a broader body of evidence to inform policy is an important step to enhance evidence-based coverage decisions. Derived from actual patient care through electronic health records, claims, and registries, RWD offers valuable insights into the safety and effectiveness of medical interventions. Real-world studies capture outcomes from varied patient populations over extended periods, offering evidence on long-term safety, including rare adverse events and real-world effectiveness that complement the scope of randomized clinical trials. Because RWD is collected in routine clinical settings, study findings are often more generalizable to a wider beneficiary population. Additionally, RWD can generate meaningful evidence at a significantly lower cost than traditional randomized trials.

Philips appreciates the guidance and CMS’ recognition of the value of RWD. The use of a standardized template for a protocol will promote a thoughtful and consistent design of real-world evidence generation studies. While Philips agrees that the Proposed Guidance and accompanying RWD protocol template will help streamline protocol development, we believe additional guidance and flexibility are needed in certain areas.

CMS based the template on the HARmonized Protocol Template to Enhance Reproducibility (HARPER) and adapted it for medical devices and Medicare coverage criteria (HARPER+). While CMS adapted the template for medical devices, Philips urges CMS to also consider each subject’s applicability to the HARPER+ as the type of data generated and the technology may not fit within the confines of HARPER+. Careful consideration must be given to ensure the HARPER+ criteria are relevant and applicable to the specific medical device or technology being evaluated.

Not only does the HARPER+ template include provisions that may not be applicable to the technology but also includes provisions that may limit the submissions for smaller and less-resourced applicants. For example, CMS requires study sponsors to outline steps to ensure data quality, including quality assurance, quality check procedures, and double programming. The requirement for double programming presents challenges, as not all medical device companies conducting real-world data analyses may have the capacity to implement it. Many other processes outlined are resource-intensive and technically demanding for small teams. Therefore, we encourage CMS to provide flexibility in selecting quality control measures that are appropriate for each specific study.

Moreover, some aspects of the guidance are vague and ambiguous. For example, CMS requires study sponsors document amendments and updates to the study protocol but states that “minor changes to the protocol do not require a formal resubmission and review by CMS.” However, there is no clear distinction between minor and major changes. Greater clarity on this and other areas throughout the framework would be beneficial. CMS could address this by hosting open-door forums, providing additional examples, and issuing supplementary guidance documents.

We commend CMS for its thoughtful approach in developing this guidance and recognize the potential of the HARPER+ framework to standardize the protocol development process. However, submissions may require flexibility depending on the technology and the type of evidence generated. We encourage CMS to collaborate with submitters to adapt the template as needed on a case-by-case basis and implement educational outreach to interested parties.

Philips thanks CMS for their continued commitment to improving transparency and collaboration with stakeholders. We appreciate your consideration of our comments. If Philips can provide additional information, please contact me.

Less

Discriminatory algorhythmic use of pathways by CMS with use of NCD and pathways based on data is biased against populations of people with rare and or genetic diseases or multiple disease conditions as there is not enough robust data to be able to show statistical significance in treatment plan or care …one size does not fit all would deny access to medically necessary health care services and products to this population of patients increasing the risk of harm and progressing those disease processes to early death. CMS must include exceptions or the protections with full exclusions of patient populations with rare diseases, genetic diseases or multiple diagnoses from NCD and pathways as there is not enough robust data to input with statistical significance.

There are strict federal guidelines of randomized controlled trials that need to be applied with NCD pathways and algorithmic uses in healthcare due to the risk of harm to patients if only using data to create pathways as factors, variables, data can be manipulated by different use of testing methods, statistical size, or omitted data can affect the confidence interval and p levels as meting standards without COMIRB, proper oversight, accountability would be at zero without peer to peer professional healthcare review and multiple trials for monitoring must not be left tot he states to decide but rather controlled at the federal level with independent non contractor specialty physician non governmental entity impartial unbiased institutional review committee with rotating voting majority membership every 2 years and half of the membership to be current active or retired physician researcher members or speciality expert physicians in that field every 2 years with randmozied audits of existing pathways for patient outcomes to be done randomly on each pathway with current past 2 year data to confirm same statistical significance and benchmarking for another 2 year approval. Additionally, with current computer systems and advanced technology the use of artificial intelligence and machine learning should be fully excluded in healthcare NCD pathways due to the risk of harm and death to patients due to no current federal laws, guidelines, quality or risk assessments, oversight, and no requirement of ONC IT certification by DHHS at all per current published guidelines.

Furthermore, without added legal protections of this patient population it is a violation of human rights and discrimination against children and adults with complex medical needs as they are typically the patient populations that need increased access to medically necessary healthcare services, off the label black box medications and specialized products outside of the normal diagnosis of one disease process, illness or injury and additionally this population needs to fully be excluded from NCD pathways/ algorhythms as even evidenced based practice guidelines or treatments for short term illness and or injuries cannot be applied at times and must be individually reviewed by a physician for medical necessity criteria based on professional judgement, patient medical condition and physical assessment.

[PHI Redacted] is the only person in the entire world internationally ever to have a dual diagnosis of 2 very rare genetic disease processes one of which is life threatening and most evidence based practice guidelines, treatments and medications cannot be applied to his level of care even for a simple illness, injury, procedure or surgery as he has a team of doctors collaborating and making medical decisions together for best outcomes and if this was used in practice without added legal protections his risk of harm endangers his health, safety and life as a disabled beneficiary of the federal Medicaid medical assistance program.

Less

March 18, 2025

Coverage and Analysis Group
Centers for Medicare & Medicaid Services
7500 Security Boulevard
Baltimore, MD 21244

RE: Proposed Guidance Document: Study Protocols That Use Real-world Data

The American College of Cardiology (ACC) appreciates the opportunity to comment on the Proposed Guidance Document, Study Protocols That Use Real-world Data. The ACC is the global leader in transforming cardiovascular care and improving heart health for all. Since 1949, the ACC has been the preeminent source of professional medical education for the cardiovascular care team, credentialing professionals in over 140 countries and leading in the formation of health policy, standards, and guidelines. Through its world-renowned family of JACC Journals, NCDR registries, ACC Accreditation Services, global network of Member Sections, CardioSmart patient resources, and more, the College is committed to ensuring a world where science, knowledge, and innovation optimize patient care and outcomes.

The proposed guidance document references the benefits of professional society registries in the “RWD Study Template Components” section, specifically in sections 7.7.1, 7.8, 15.2.4, 16, 17, and 18. These sections highlight the advantages of using professional society registries for real-world data (RWD) studies. Notably, section 7.6.1 refers to one of several articles authored by representatives from multiple constituencies, demonstrating the power and cost-effectiveness of professional society registries. The College has included additional articles for your reference.

NCDR's Proven Track Record & Capabilities in Coverage with Evidence Development
RWD plays an increasing role in healthcare decisions and as part of the clinical evidence CMS uses to make coverage decisions. A single pivotal trial cannot provide all information on the safety and effectiveness of a medical device, and a device’s full capabilities and complications may not be identified until it is used and observed in the real world particularly when utilized in subgroups of patients not represented within the pivotal clinical trial. The National Cardiovascular Data Registry (NCDR) has a proven track record of providing high-quality, reliable data for cardiovascular research and clinical practice. For example, the STS/ACC TVT Registry™ has supported 23 regulatory decisions over the past decade, demonstrating the effectiveness of this model in generating real-world evidence.

Professional society registries, like NCDR, offer many advantages over other data sources like electronic health records (EHR) and claims data.
1. Data Quality and Accuracy
Professional society registries adhere to rigorous data quality processes. Physician members contribute to registry oversight through participation in NCDR committees. The registries utilize the clinical expertise of committee members to provide guidance such as ensuring all necessary data elements are collected, to develop sound data definitions and contribute to data quality initiatives. Good data management practices are employed in developing the data collection form. The Data Quality Program includes source verification, completeness checks, and validation. This ensures that the data collected are accurate, complete, and reliable. In addition, NCDR registries have a robust auditing program for further assessment and assurance as to the data quality. In contrast, EHR data often contains unstructured information, which can lead to missing or incorrect entries. Claims data, while comprehensive in capturing healthcare utilization, may include inaccuracies in diagnosis and procedure codes submitted for reimbursement.
2. Clinical Relevance and Detail
Professional society registries are designed to capture detailed clinical information relevant to specific medical conditions and treatments. This includes patient demographics, clinical outcomes, procedural details, and follow-up data. EHRs, on the other hand, may lack standardized data elements and often miss important clinical details due to variations in documentation practices. Claims data primarily focuses on billing information and lacks the clinical context necessary for in-depth research.
3. Procedural Data and Real-World Insights
Professional society registries provide detailed procedural data, including follow-up, which is essential for understanding the effectiveness and safety of treatments in real-world settings. Unlike claims data, which offers a broad view of healthcare utilization but lacks detailed clinical outcomes and patient experiences such as patient reported outcomes, these registries capture comprehensive procedural information and often, in addition, validated patient reported outcomes or PROMs. Additionally, registries have the ability to add additional data elements in a nimbler fashion, allowing the capture of additional endpoints that might be needed. EHR data, on the other hand, can be fragmented and may not consistently include long-term follow-up details. Therefore, the detailed procedural data from professional society registries is invaluable and often not possible to obtain from other data sources.
4. Comprehensive and Representative Data
Professional society registries often include data from a wide range of providers and institutions, ensuring a comprehensive and representative view of clinical practices and patient outcomes. This variety enhances the generalizability of research findings. In contrast, EHR data is often limited to specific healthcare systems and may not represent broader patient populations. EHR data does not conform to a standard format, terminology, or data elements & definitions. In contrast, claims data and registries follow standardized formats, which is highly beneficial for data analysis. This standardization allows large amounts of detailed data to be uniformly defined and structured, making it easier to analyze. The deployment of EHR systems can be unique to each installation, and this is compounded by the numerous EHRs in use. Large data aggregators may tout the benefit of the wide reach of their data across multiple EHRs and across multiple healthcare systems but normalizing that data can be challenging when each system uses its own data definitions. Claims data, while extensive, is restricted to the information required for reimbursement and may not capture all relevant clinical details.
5. Enhanced Data Validation and Auditing
Professional society registries implement robust data validation and auditing processes to ensure data integrity. For example, the NCDR conducts regular data audits and provides confidential Data Quality Reports (DQRs) to participating sites, allowing them to correct and resubmit data as needed.
This level of data validation is typically not present in EHR or claims data, which can result in lower data quality and reliability.
Clinical Event Adjudication has been done for some NCDR registries. The adjudication process provides external validation of adverse events, which promotes accuracy and consistency, thereby enhancing data. This process facilitates standardization, and reduces bias and errors, to support regulatory compliance. Claims data goes through an adjudication process, but the lag can be long from the time claims are submitted to final adjudication. This level of data validation provided by professional society registries is typically not present in EHR or claims data, which can result in lower data quality and reliability.
6. Specificity to Medical Conditions and Treatments
Professional society registries are tailored to specific medical conditions and treatments, capturing data elements that are directly relevant to the research questions being addressed. This specificity allows for more precise and meaningful analyses. EHRs and claims data, on the other hand, are designed for general healthcare documentation and billing purposes, respectively, and may not capture the nuanced details needed for specialized research.

High-quality professional society registries, like NCDR, offer several advantages over other registry models and methods of collecting RWD. While EHR and claims data each have their own strengths, professional society registry data stands out as a superior source of real-world evidence due to its high data quality, clinical relevance, longitudinal insights, comprehensive representation, robust validation, and specificity to medical conditions and treatments. These attributes make professional society registries invaluable for advancing healthcare research and improving patient outcomes.

NCDR's capabilities are particularly valuable in the context of Coverage with Evidence Development (CED). Our registry has been instrumental in supporting numerous CED initiatives, providing robust data that informs clinical and regulatory decisions. The use of RWD and real-world evidence (RWE) through the use of professional society registries is crucial for evaluating medical devices and ensuring their safety and effectiveness. The NCDR's rigorous data quality processes, including data audits and quality assurance measures, ensure that the data collected is accurate, complete, and reliable. Through our real-world data (RWD), NCDR has supported over 16 post-approval studies, leading to numerous label expansions and ensuring the continued safety and effectiveness of medical devices in diverse patient populations. Additionally, our data has been instrumental in numerous CED studies, directly influencing coverage and enhancing the regulatory landscape for medical devices.

Examples NCDR Supporting Regulatory Decisions
In 2004, CMS requested the formation of a National ICD Registry Working Group to help finalize the National Coverage Determination (NCD) for implantable defibrillators. This resulted in the creation of the ICD Registry (now the EP Device Implant Registry™ which in 2005, CMS announced would be a condition of coverage of ICDs for primary prevention. This national registry became the central component of the CMS NCD to support the collection of data to understand the quality of care and outcomes for patients receiving ICD therapy.

To better understand the utilization, safety, and effectiveness of left atrial appendage occlusion (LAAO) devices in “real world” clinical practice, the ACC and the Society for Coronary Angiography and Intervention collaborated with the FDA, the Centers for Medicare and Medicaid Services (CMS), and Boston Scientific to develop the NCDR LAAO Registry™. The registry was designed to function as the formal post-market surveillance vehicle and has been instrumental in supporting the NCD (1) and CED study for LAAO. The registry has allowed researchers and regulators to analyze outcomes beyond clinical trials, including post-procedural anticoagulation and antiplatelet treatment patterns for patients undergoing LAAO procedures. Data from the registry highlighted that patients receiving WATCHMAN left atrial appendage closure in the U.S. are often older and at higher thromboembolic and bleeding risk compared to those in clinical trials and early registries. These insights help regulators understand the patient population benefiting from LAAO procedures, guide decisions, and enhance the quality of patient care.

NCDR’s CathPCI Registry® provides real-world data on PCI procedures, patient outcomes, and device utilization. It collects information on adherence to ACC/AHA clinical practice guidelines, procedure performance standards, and appropriate use criteria for coronary revascularization. By evaluating real world practices against established guidelines, regulators can assess whether a PCI procedure aligns with evidence-based recommendations and assess the safety profile and potential risks associated with a new PCI device. The STS/ACC TVT Registry™ was created through the NCDR via a partnership of the Society of Thoracic Surgeons (STS) and the ACC in close collaboration with the FDA, CMS, and the analytical center at the Duke Clinical Research Institute. A study assessed the value of return on investment and time saved when conducting essential regulatory studies using coordinated profession society registry networks, specifically the TVT Coordinated Registry Network (CRN). The TVT CRN has gathered off-label use of devices that have demonstrated safety and effectiveness and led to timely FDA expansion of indications. The registry has also generated data on outcomes and adverse events when used in different populations. The TVT Registry was used to support four premarket decisions, including the expanded indications, and 19 post market surveillance studies in lieu of traditional preapproval and post approval studies and continues to be used successfully for FDA regulatory decision making. Additionally, there were significant cost savings along with substantial time saved in study enrollment and completion using the TVT CRN versus the traditional study design.
When TAVR was first approved under the NCD with CED in 2012(2), it was limited to high-risk patients that were too health-compromised to undergo open heart surgery for surgical aortic valve replacement (SAVR). Over time, intermediate and low risk populations have gained access to this once novel therapy based on the data collected through our registry. This method of evidence generation creates value for manufacturers and the broader device ecosystem with significant benefits to public health. The registry has also been pivotal in supporting the NCD for Transcatheter Edge-to-Edge Repair (TEER) for mitral valve regurgitation (3). Additionally, this model of data collection through the STS/ACC TVT Registry™ sets the standard for newer technologies like the recently FDA-approved devices for transcatheter tricuspid valve replacement and repair that are currently undergoing two national coverage analyses by CMS (4,5).

1) NCD - Percutaneous Left Atrial Appendage Closure (LAAC) (20.34).
2) NCD - Transcatheter Aortic Valve Replacement (TAVR) (20.32)
3) NCD - Transcatheter Edge-to-Edge Repair (TEER) for Mitral Valve Regurgitation (20.33)
4) NCA - Transcatheter Tricuspid Valve Replacement (TTVR) (CAG-00467N)
5) NCA - Transcatheter Edge-to-Edge Repair for Tricuspid Valve Regurgitation (T-TEER) (CAG-00468N)

Collaboration
Collaboration with industry is crucial to ensure that study protocols are comprehensive and meet all necessary standards. By leveraging NCDR's expertise in data collection and analysis, we can enhance the quality and reliability of real-world data studies. This collaborative approach will help ensure that protocols are well-designed and effectively implemented. ACC’s registry initiatives serve as a prime example of how collaboration can lead to successful data collection and analysis, ultimately benefiting both patients and healthcare providers.

Real-World Data
Through ACC's high-quality registries, clinicians offer valuable insights into the effectiveness of new cardiovascular procedures and the devices used in them, which are crucial for developing RWD. RWD collected during post-market surveillance is beneficial to patients, scientists, regulatory agencies, and device manufacturers. RWD is crucial to improve care for patients, particularly populations historically underrepresented in clinical trials. By confirming and validating data from clinical trials in a broader, more representative patient population, RWD enhances the generalizability of research findings. It also provides insights into off-label use, which can benefit patients, providers, and the industry by identifying new therapeutic applications and potentially leading to expanded indications. Recent NCDR research provides insights into sex-specific differences in patients undergoing ablation for atrial fibrillation (AFib). Other studies leveraging NCDR data have revealed predictors of outcomes associated with an ICD generator change, an increase in the use of prophylactic mechanical circulatory support in specific patients undergoing PCI, and an increase in the incidence of stroke during PCI.

Custom analysis of NCDR registry data, done in-house by the ACCF, offers a unique opportunity for stakeholders to obtain tailored analysis of clinical data from real-world populations to more deeply understand issues such as:

Adoption and use of medical devices and treatments

Quality of care

Geographic variations in care

The patient landscape

Trends and market conditions

Appropriate use criteria

Variation in healthcare delivery and outcomes

Real-world use of medical devices and drugs (including off-label use)

Data Quality and Reliability
The reliance on RWD from the NCDR necessitates that data are accurate, complete, and reliable. Registry data must have a high degree of accuracy to be viewed as a dependable source for research, regulatory post-market studies, and medical device surveillance. To certify the quality of data generated from the NCDR, a number of administrative, technical, and operational processes are in place for collecting, using, and sharing RWD.

Data are submitted directly and securely to the NCDR, which maintains a central database of all registry data. All data undergo rigorous quality checks to ensure data integrity. These checks ensure that data: 1) meet completeness thresholds; 2) fall within pre-specified ranges; and 3) are internally consistent.

A confidential Data Quality Report (DQR) is provided at the time of data submission. The DQR informs the site whether their data pass completeness thresholds to be included in risk-adjusted benchmark reports, site dashboards, and research. Sites can correct and resubmit data as often as needed until completeness criteria are met. Only data that have passed all checks and met pre-specified thresholds are used in national reports and for research.

The NCDR’s national Data Audit Program has been in place since 2003. Each year, a random sample of sites and patient records are selected to assess the accuracy of selected data elements. A contracted third-party quality improvement organization (QIO) conducts the data abstraction for selected facilities. Following the audit, each site receives a detailed report of the audit findings which identify opportunities for data collection and process improvements. Facilities are given an opportunity to review and appeal findings of the audit report before a final report is provided back to the site. An Executive Summary is produced that details the overall accuracy and completeness of data elements and the inter-rater reliability of the abstractors for that audit cycle. The Executive Summary is provided to the registry Steering Committee, Research and Publications Committee, Data Quality Subcommittee and the Clinical Science and Quality Committee. This allows for a comprehensive understanding of the quality of the real-world data being submitted and subsequently utilized for a variety of purposes.

Another component of the audit program is an evaluation of record compliance. This process evaluates the records submitted to the registry for a specific timeframe against billing records for that same timeframe to ensure that all eligible cases have been submitted to the NCDR. This completeness assessment assures that the registry has a comprehensive collection of cases.

The NCDR takes additional steps to identify data outliers. Data outlier reports are generated twice a year to identify statistical outliers among specified data elements. Sites receive a report if specific outlier metrics are flagged in their data.

Lastly, in large part because of the use of registry data to support FDA’s and CMS’ post-approval requirements, the NCDR introduced a clinical event adjudication process. This process, carried out by contracted data analytic centers, evaluates select clinical adverse events to ensure consistency in the interpretation of registry definitions. As stakeholders continue to utilize RWD, the ability to understand and evaluate the quality of the data is critical. Through its rigorous data quality efforts, the NCDR asserts the reliability of its data so that it continues to be fit for purpose for research, regulatory decision-making, and surveillance.

Community of Practice and Maturity Model
Real-world data is a robust and pragmatic solution for evaluating many priority medical devices and health technologies for regulatory and clinical decision-making. Well-developed and established registries, such as the NCDR, are increasingly recognized as a valid and reliable source of data to inform research and evaluation. Registry data not only complements evidence generated from clinical trials but also independently informs decision-making when it is fit for the purpose of timely evaluation of devices.

The increasing availability of registry data and use of appropriate analytic and statistical methodologies have the opportunity to improve quality and speed while decreasing the cost of generating evidence for medical device evaluations. Within various data sources for generating RWD, registries often form the core for the evaluation of specific procedures and related devices. With the increasing awareness and acceptance of RWD as a valid source of data, the concept of Coordinated Registry Networks (CRNs) has been advanced by the Medical Device Epidemiology Network (MDEpiNet). CRNs rely on a community of stakeholders such as clinicians, medical specialty societies, industry, patients, payers, academia, and regulators. RWD generated from a CRN such as the STS/ACC TVT Registry can help optimize clinical studies through a pragmatic registry-based approach using the existing infrastructure of the underlying registry and expanding the use of data routinely collected through already established operations at individual participating hospitals. Overall, CRNs have developed a very promising capacity for benchmarking and quality assurance for interventional procedures and contribute to specialty societies as well as to manufacturers to evaluate devices and meet regulatory requirements.

Professional societies have been instrumental in data collection for CED through registries. The NCDR is uniquely positioned to collaborate with stakeholders on RWD collection while minimizing the burden on hospitals. Consistent mechanisms provide clarity to providers and facilities who need to meet compliance requirements. With the new Transitional Coverage for Emerging Technologies (TCET) pathway and CED guidance, it is important to continue with what is already working well. Simplifying processes for providers and facilities ensures the best localized quality reporting and enhances future analytics. This approach benefits hospitals and clinicians by streamlining services. The ACC believes that registries are the best way forward and stands ready to work with CMS and industry to fulfill the requirements of real-world data collection for a CED study. Leveraging its extensive experience and resources, the NCDR can streamline the data collection process, providing valuable insights to assess the reasonableness and necessity of treatments for regulatory decision-making. We appreciate the opportunity to provide feedback on the proposed guidance. Please direct any questions or concerns to Amanda Stirling, Regulatory Affairs Associate, at 202-375-6553 or astirling@acc.org.

Sincerely,
Cathleen Biga, MSN, FACC
President, American College of Cardiology

References
Cavanaugh, K., Lottes, A., Raska, M., et al. Current Landscape for Global Regulatory Acceptance of Real-World Evidence for Medical Devices. Endovascular Today. 2023. https://evtoday.com/articles/2023-aug/current-landscape-for-global-regulatory-acceptance-of-real-world-evidence-for-medical-devices.
Cronenwett, J. L., Whatley, E., Malone, M., Svendsen, C., Kallok, M. J., Ouriel, K., & Eldrup-Jorgensen, J. 2024. The Value of Propensity-Matched Registry Data for New Device Evaluation. Endovascular Today. https://evtoday.com/articles/2024-apr/the-value-of-propensity-matched-registry-data-for-new-device-evaluation.
Malenka, D, Bhatt, D, Bradley, S. et al. The National Cardiovascular Data Registry Data Quality Program 2020: JACC State-of-the-Art Review. JACC. 2022 May, 79 (17) 1704–1712.https://doi.org/10.1016/j.jacc.2022.02.034
Pappas, G., Berlin, J., Avila-Tang, E., et al. Determining value of Coordinated Registry Networks (CRNs): a case of transcatheter valve therapies. BMJ Surgery, Interventions, & Health Technologies. 2019;.1: e000003. https://sit.bmj.com/content/1/1/e000003.
Whatley, E., Malone, M. Current Considerations on Real-World Evidence Use in FDA Regulatory Submissions. Endovascular Today. 2017. https://evtoday.com/articles/2017-oct/current-considerations-on-real-world-evidence-use-in-fda-regulatory-submissions.

Less

The Alzheimer's Association, the American College of Radiology, and the Department of Biostatistics and Center for Biostatistics and Health Data Science, Brown University School of Public Health appreciate the opportunity to comment on the Centers for Medicare & Medicaid Services's (CMS) Proposed Guidance Document: Study Protocols that Use Real-World Data (RWD). Overall, we support the proposed guidance, as we believe a template such as this is potentially beneficial, making the Coverage with Evidence Development (CED) protocol development and study design process more efficient by reducing unnecessary consultations with CMS and to provide the necessary framework for study development.

We note for CMS's consideration that all of the 19 elements within the template may not be applicable to every type of CED study so we encourage the agency to allow for that acknowledgement by a study team when drafting a protocol. Some limitations or threats to a study design may not be known until the study is underway which might have influenced the study design and eventual outcomes. For example, during the New IDEAS Study, the diagnostic radiopharmaceuticals came off pass-through funding status, resulting in bundled payments for hospitals and therefore limiting the number of hospitals participating in the study as well as patient access.

We also encourage CMS to consider that the data dictionary in the protocol (Exhibit D) could increase regulatory burden for sites, sponsors, and institutional review boards every time a data element is changed or updated throughout a study. As care is constantly changing in the real world and this guidance is intended for studies collecting real-world data, the data being collected needs to reflect that and keep pace. Therefore, a carefully curated selection of data elements would need to be identified for the protocol that is reasonable and feasible for collection in real world care settings. This balance would need to be considered by both the protocol team and CMS.

Furthermore, as the guidance primarily addresses the design, conduct, and analysis of specific, focused studies using real-world data, it is also important to clarify the relation of the guidance to registries, which are designed to collect and analyze data on specific diseases, conditions, or exposures to improve health outcomes. Registries are designed with a broader scope than a single study and are intended to provide data for use in a range of studies. As such, a registry needs to collect and report real-world data information to enable the individual studies to meet the requirements of the proposed guidance. It would be useful if the guidance addressed this matter and indicated the categories of required attributes a registry should have.

Finally, we would like to offer CMS several instances where additional clarification in an updated version of the guidance would be useful:
Section 15: The various elements listed in this section, entitled “Additional Statistical Considerations,” seem to recapitulate the general list of topics that might be addressed in the protocol for a clinical trial. It would be helpful to indicate if there is a specific real-world data angle for each element. If there is not, it might be best to leave the element out.
It would be useful to define the scope of sensitivity analyses. For example, such analyses would be indicated when, a-priori, it is thought that the conclusion of the study could be altered by the results of the sensitivity analysis.
The definition of covariates (Section 7.4.4) could be expanded to include potential provider or hospital level data. For example, this information would be needed to construct some propensity scores, as discussed in Section 7.5.1 of Appendix B, which states: “A propensity score model will first be fitted to connect group membership with patient and hospital characteristics”.
Table 3 in Appendix B has “Time (when follow up begins and ends): Median follow-up time of five years after exposure”. Is this definition made at the population level or the individual level? If the latter, this definition does not specify when follow-up begins or ends and many different distributions of individual level lengths of follow-up will give a median follow-up time of five years.
Appendix B, Table 3: has a hazard ratio as the measure of effect suggesting a time-to-event analysis. In such settings defining the outcome to be “time until death from all causes” might be more appropriate than what seems to be the binary outcome of all cause mortality.
Appendix B, Table 12: The instrumental variable Analysis in Table 12 in Appendix B indicates it is conducted “To examine the relationship between exposure and outcome without the influence of unmeasured confounders”. However, an instrumental variable analysis relies on a different set of assumptions compared to the IPW analysis (e.g., the instrument and the outcome have no common causes and the instrument affects the outcome only through the intervention). It would be useful to reconsider the statement “without influence of unmeasured confounders”. Indeed, both analyses rely on untestable assumptions.
Table 3 in Appendix B “Objective success criteria of powered endpoints: Less than a 5% difference in all-cause mortality between the DCD and NDCD groups” This is not entirely accurate as the non-inferiority margin is expressed in terms of hazard ratios and not differences in all-cause mortality.

We appreciate the opportunity to comment. Please do not hesitate to contact us as you work to implement this guidance if we can be of assistance.

Less

The Medical Device Manufacturers Association (MDMA) is a national trade association that provides educational and advocacy assistance to hundreds of innovative companies in the field of medical technology. Our members, the majority of which are small to mid-sized medical device companies, have a strong record of delivering breakthrough therapies to treat chronic diseases and life-threatening conditions while lowering the cost of care. MDMA’s mission is to ensure that patients have timely access to the latest advancements of safe and e?ective medical technologies that improve health outcomes.

We are pleased to submit these comments to the Centers for Medicare & Medicaid Services (CMS) in response to its draft guidance on critical elements for study protocols that use real world data (RWD) sources for the purpose of Medicare coverage with evidence development (CED) (the “Draft Guidance”). As noted in the Draft Guidance, providing additional clarity regarding RWD study protocols reflects CMS’ “interest in considering a broader range of fit-for-purpose study designs for CED.”

MDMA reiterates our strong support for the CED pathway. We believe that CED represents an appropriate use of the agency’s authority under section 1862(a)(1)(E) of the Social Security Act. The CED pathway is critical to providing timely access for Medicare beneficiaries and their physicians to cutting edge therapies and diagnostics, with appropriate beneficiary protections, and to generating additional clinical evidence sufficient to support future “reasonable and necessary” coverage determinations. In some cases, it provides an alternative to a finding of noncoverage, which would foreclose access and hamper ongoing development of clinical evidence within the Medicare population.

Clarity on acceptable fit-for-purpose (FFP) study designs is also critical to the success of the Transitional Coverage for Emerging Technologies (TCET) program, as it will provide the basis on which the manufacturer, CMS and AHRQ will reach agreement on an evidence development plan (EDP) prior to the start of the national coverage determination (NCD) process—a primary objective of the Program. It is also key to fulfilling one of the four principles that CMS has set for TCET—that “[i]f CMS determines that further evidence development is the best coverage pathway, the agency would explore how to reduce the burden on manufacturers, clinicians, and patients while maintaining rigorous evidence requirements.” (1)

MDMA believes that the goal of minimizing burden on manufacturers, clinicians, and patients has not been fully reflected in CMS’ approach to FFP studies, which the agency has defined under TCET as "where the study design, analysis plan, and study data are appropriate for the question the study aims to answer.” However, in a recent joint paper, leading experts on the evaluation of medical technologies from the University of Southern California, Tufts Medical Center and Dartmouth School of Medicine argued that “[a]lthough it is critical that approved CED registries and studies collect the data needed to address evidence gaps, it is equally important that data collection be ‘fit for purpose,’ meaning that the collection of any data not required to control for potential confounders and reliably address the research question(s) should be minimized.”(2) As CMS continues its efforts to delineate a framework for more efficient, predictable, and transparent evidence development, we urge the agency to extend the definition of “fit-for-purpose” to incorporate the consideration of burden on patients and practitioners. This revision is aligned with the data privacy and security principles of minimum data necessary, as well as the Food and Drug Administration’s approach to use of real world evidence (RWE) for medical device reviews.(3)

Another recent paper published in the Journal of the American Medical Association describes the growing health data infrastructure, developed through public and private partnerships, that can be used to collect real-world data efficiently and routinely.(4) CMS should encourage CED studies that leverage this growing infrastructure when feasible and practical.

With regard to the Draft Guidance specifically, MDMA appreciates the inclusion of a standardized study template, which can enable a more efficient and streamlined review of RWD protocols by CMS. We also appreciate the use of the HARPER template as a starting point, as it is well known and has been widely adopted for RWE studies. That said, we are concerned by the statement in the Draft Guidance that “[d]epartures from the requested format and content may lead to delays in the study approval process.”

MDMA recognizes the potential for CMS staff to expect manufacturers to include all template elements in RWD protocol submissions to the agency. We caution the agency against such an approach. In order to realize the promise of RWD, CMS should embrace flexible, least burdensome approaches that help to support the shared goal of facilitating prompt patient access and ongoing evidence development. CMS should clarify in the final guidance that different or a more limited set of elements may be appropriate for certain types of RWD. CMS also should explicitly state a commitment to consider the full range of RWD study designs, including a range of data sources, methodological approaches and the requirement for a comparator group.

MDMA looks forward to continuing our work with CMS to improve the efficiency of the NCD process, including the CED pathway and TCET program. If we can provide any additional information, please contact me at dwaldmann@medicaldevices.org or (202) 354-7171.

Respectfully,

Dan Waldmann
EVP, Health Policy & Reimbursement
MDMA

---

(1) Fleisher LA, Blum JD. A Vision of Medicare Coverage for New and Emerging Technologies: A Consistent Process to Foster Innovation and Promote Value. JAMA Intern Med. 2022;182(12):1241-1242. doi:10.1001/jamainternmed.2022.5085.
(2) Lakdawalla, Darius, et al. A Roadmap for Improving Medicare’s Application of Coverage with Evidence Development, Value in Health, Volume 27, Issue 9, 1191 – 1195. DOI: 10.1016/j.jval.2024.05.008.
(3) See U.S. Food and Drug Administration, Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices: Draft Guidance for Industry and Food and Drug Administration Staff, Dec. 19, 2023, https://www.fda.gov/regulatory-information/search-fda-guidance-documents/draft-use-real-world-evidence-support-regulatory-decision-making-medical-devices.
(4) Abbasi, Ali, et al. A Unified Approach to Health Data Exchange: A Report from the US DHHS, JAMA, Jan. 18, 2025. DOI: 10.1001/jama.2025.0068.

Less

The Food and Drug Administration’s (FDA’s) decision to approve Aduhelm for treatment of Alzheimer’s disease showed a stunning disregard for science and eviscerated the agency’s standards for approving new drugs. Because of this reckless action, the agency’s credibility has been irreparably damaged.

The approval of Aduhelm was based on seriously flawed post hoc analyses of two identical phase 3 trials that were stopped early because a preliminary review of the data found that the trials, if continued to completion, were unlikely to show the drug benefitted Alzheimer’s disease patients. Moreover, the integrity of the FDA’s review of the marketing application for Aduhelm was dangerously corrupted by the unprecedented and inappropriately close collaboration between Biogen and the FDA during the analyses of data from the key clinical trials of the drug after the termination of the phase 3 clinical trials because of futility.

CMS must not compound the FDA’s egregious error in approving Aduhelm on June 7, 2021. Given the lack of scientific evidence that Aduhelm provides any clinically meaningful benefit in terms of cognitive function outcomes in Alzheimer’s disease patients, the drug cannot possibly be deemed reasonable and necessary for treatment of such patients. I urge CMS to issue a national coverage determination that excludes Aduhelm from coverage under the Medicare program. It’s imperative to put patients and consumers before Big Pharma profits.

Less