The Clinical Trials Enterprise in the United States: A Call for Disruptive Innovation

Introduction

Over the past decade, the symbiotic relationship between the clinical trials enterprise (CTE) and the health care delivery system has been subject to increasing amounts of stress. During this period, the CTE has primarily focused on process improvement, seeking to create and maintain procedures and data systems that can satisfy a regulatory environment concerned with specific research procedures. These efforts have driven up the cost of research and left the CTE increasingly out of sync with the health care delivery system.

At the same time, there have been many significant changes in the health care delivery system, changes largely concerned with organization, quality improvement, operational efficiency, and error reduction and patient safety. Despite significant rhetoric (particularly in the United States) about “learning health systems” (Institute of Medicine [IOM], 2007), the CTE and the health care delivery system have continued to diverge. This situation is undesirable both because research done within such a partitioned, parallel system may not be broadly generalizable and because the cost of maintaining parallel systems limits our ability to address critical gaps in knowledge.

The end result of the widening separation between research and health care delivery will be a serious deficit in new knowledge about the benefits and risks of specific drugs and devices in medical practice at a time when biological knowledge is expanding exponentially (see Figure 1, p. 20) and warnings about the unsustainability of the system continue to escalate (IOM, 2008). This systemic divergence, in turn, deprives providers of reliable evidence upon which to base their practices, deprives patients of the expected return on research investments in science and medicine, and deprives policy makers of a rational basis for choosing one course of action over another.

 

 

The scope and pace of change in the overall health system will only increase as we enter the next decade, exacerbating the undesirable separation of research and practice. There is thus an urgent need to develop policies that will mitigate current stresses and increase CTE efficiency and effectiveness by expeditiously forming a deliberate plan to align the CTE and the emerging health care delivery system. Such an effort will require a new assertion of the societal values that underlie the CTE, a clarification of the objectives of such a plan, and accommodations on the parts of both the CTE and the health care delivery system to accomplish those objectives.

A decade ago, the National Institutes of Health (NIH) embarked upon a major effort to reshape the future of biomedical research. This framework for change, called the NIH Roadmap (Zerhouni, 2003), articulated a vision for clinical research in which each of more than 300 million Americans would have his or her own electronic health record (EHR). If individuals so chose (assuming appropriate privacy and confidentiality protections), the information in these EHRs could be used for research, and patients and their families would be joined together in networks that could answer critical questions about prevention and treatment. The fabric of constantly-accruing data would form the basis of a learning health system in which randomized trials could be performed by inserting randomization into the routine delivery of health care. This revolutionary approach would allow the rapid and definitive development of evidence at a relatively low cost.

Although this vision from a decade ago has not been achieved, much of it is within reach by 2020. Consolidations of service delivery and insurance organizations are changing the context of practice. Technological limitations are being overcome, disease-oriented networks and voluntary health organizations are evolving, and modern medical informatics is enabling the aggregation, classification, indexing, and analysis of information in volumes that were unimaginable a decade ago. Unfortunately, at the same time these tools are emerging, the CTE is increasingly being cordoned off from the rapidly-evolving, information-intensive world of medical practice. This paper provides a roadmap for integrating the clinical trials and health care delivery systems in ways that will improve the efficiency and effectiveness of both.

 

Current Status of the Clinical Trials Enterprise

The passage of the Food and Drug Administration (FDA) Amendment Act in 2007 (FDA, 2007) mandated the registration of most clinical trials involving medical products approved for use in the United States. The resulting increase in the number of trials registered and the quality of the information provided to entities such as ClinicalTrials.gov has afforded us a new opportunity to assess the state of the overall CTE.

When we examine these data, several significant patterns are manifest. First, the vast majority of clinical research comprises tiny trials performed on small numbers of patients (62 percent of studies plan to accrue fewer than 100 participants; 96 percent have a projected accrual of <1,000) (Califf et al. [in review]). Second, the majority of trials are sponsored by academic health centers (AHCs) and are focused on disease mechanisms, or are phase 1 and 2 studies sponsored by industry. The majority of research participants, however, are enrolled in industry-sponsored phase 2, 3, or 4 trials that are fewer but larger in size and increasingly globalized (Glickman et al., 2009). Third, relatively few trials are asking questions that directly address critical decision points in clinical practice.

A typical clinical trial today depends on the recruitment of investigators who, in turn, recruit trial participants and oversee the conduct of the research protocol. There is, however, no assurance that the recruited participants are sufficiently representative of broader populations to allow generalization of trial results. Further, these trials are often conducted at professional clinical trial sites that produce clean data, but at the cost of divorcing the trial from clinical care. At the same time, we now have multiple integrated health systems, including not only well-known health management organizations (HMOs) and the Department of Veterans Affairs (VA) system, but also numerous integrated health systems (IHSs) that possess data warehouses capable of supporting the selection of research participants in a much more systematic fashion.

These problems are compounded by a decline in the number of U.S. investigators, while at the same time the total number of clinical trials worldwide has nearly doubled in the last decade (Getz, 2005). This decline also coincides with a rapid expansion of the offshoring of clinical research activities, a particularly worrisome trend (Califf, 2011; Kim et al., 2011). In this context, “offshoring” of trials refers to the movement of research away from the United States purely because of the excess cost and difficulty of conducting research domestically, including long startup times, slow recruitment, and high rates of non-adherence and withdrawal of consent. Offshoring is not a synonym for globalization, which allows many countries and cultures to participate in research and which we regard as a positive development for all diseases. Globalization of clinical research can be particularly beneficial when the research addresses neglected diseases (Bollyky, 2011) or major chronic diseases that may have a different outcome or treatment effect with different genetic backgrounds or patterns of clinical care than those found in the United States.

Offshoring is a concern because it diminishes synergies between basic and clinical research that are vital to translational research, and because it weakens the interface with industry that speeds advances in drugs and devices made available to the American public. Furthermore, research performed in different populations or under different conditions may not be fully generalizable unless particular care is taken. And while the NIH Roadmap vision calls for a seamless and efficient integration of research and practice, in reality we see that the focus on efficiency in practice has led to the view that research is a source of additional expense and burdensome administrative and regulatory commitments (Califf, 2009). At the same time, when clinical research is not integrated with practice, its focus on efficiency in isolation from other factors distances it further from the environment crucial to producing generalizable results.

 

Part I: Organization of Health Services in 2020

This section describes our vision of the health systems of 2020. In health services, demography is destiny. At the largest scale, demographic patterns drive a substantial portion of the demand for services. Less obvious, however, is the manner in which American demography mandates the organization of the services that respond to that demand, how and by whom services are paid for, and who provides the services. The U.S. Census Bureau estimates that by 2020 there will be more than 341 million Americans, of whom nearly 55 million (16 percent) will be 65 years of age or older. Among these will be an estimated 135,000 centenarians (U.S. Census Bureau, 2008). The population of the country as a whole will be ethnically diverse; this will be especially true for those under the age of 20. Most patient and provider interactions will take place in the context of managing chronic conditions—chiefly depression, cancers, cardiovascular disease, diabetes, arthritis, asthma, and Alzheimer’s disease.

Throughout their lifetimes, most Americans will either be enrolled in or anticipating enrollment in an IHS that will provide all or nearly all of their health care. By 2020, there will be substantial but uneven progress toward establishing the IHS as the dominant form of health care organization, particularly in urban settings. Elements of IHS implementation will be in place in most communities.

The typical IHS is community hospital–centered and oriented toward providing primary care. Integration is driven by enrollment, bundled payment, EHRs, and standardized health care. It serves an enrolled (i.e., defined) population. It is further defined by the scope of and coordination among the comprehensive services that comprise its network, including directly owned or contracted multi-specialty medical group practices, ambulatory care centers (including imaging, walk-in “retail” clinics, employee wellness facilities, rehabilitation centers, etc.), home health agencies, nursing homes, extended care facilities, and hospital- and home-based hospice services within its primary service area. Most IHSs are affiliated with local public health departments and community health centers. All IHSs have collaborative agreements with AHCs for education and research support. The IHS and/or affiliated group practices serves as the locus of medical homes and accountable care organizations.

Integrated health care organizations and their affiliated service organizations either employ or contract with a substantial majority of health care providers, including physicians, nurses, advanced practice nurses, physician assistants, clinical social workers, and psychologists. Pharmacists are engaged in provider and patient counseling across the system. Although IHSs provide comprehensive data, patients/customers and their families switch from one IHS to another, with the result that interoperability is an indispensible characteristic of any viable data system.

Health care professionals have a strong sense of identity with the IHS, which is the focus of much of their professional activity. There is also a high degree of alignment between system objectives and professional expectations and roles that is achieved through recruitment, position and role description, promotion, remuneration, and incentives. The objective of implementing a culture of patient-centered care through the optimal deployment of appropriate competencies is reflected in the organization of teams that respond to the patient’s changing needs.

The EHR can be characterized as the clinical and administrative core of the IHS. It is designed to meet both administrative and clinical needs and employs nationally standardized interoperability and nomenclature. The scope of “meaningful use” of EHR technology has been expanded to embrace and incentivize research. Empowered by virtualization, the EHR is designed to enable preventive, clinical, and supportive services throughout an enrollee’s life. The development and maturation of EHR technologies promise to make practicable the application of genetic information to medical practice. The EHR is also designed to provide the continuous aggregate data that are essential for assessing the health of the enrolled population, continually improving the quality of care throughout the IHS, and supporting clinical research and professional education. Of paramount importance, however, is the fact that the EHR enables the system, the provider, and the researcher to follow up with the patient over time. Large population-based real-time evidence is the product of aggregated health system databases. Randomization is facilitated through super computer-based virtually-integrated information systems that merge administrative and clinical data from multiple sources. The virtually-integrated networks, spanning providers and payers, make it possible to enroll individuals in trials and to follow them going forward. The magnitude of the databases and the power of the tools make possible analyses that are responsive to the cultural diversity of the study populations.

The EHR is first initiated when the patient enrolls or enters any of the component organizations and services. Integrated with web-based mobile technology, the EHR “follows” the patient home, issuing reminders and monitoring compliance, as well as capturing incidents and wellness indicators (FasterCures, 2005). In 2020, there is wide public recognition that research is an integral component of community-based practice. The basic compact between the patient and the provider community is that every patient is a potential contributor to research. It is assumed that the patient record may be used for research in a de-identified data system. The patient-oriented part of the EHR is owned by the patient and is accessible only by the patient or by family members and providers to whom the patient grants access. Special permission is required to use such information in research projects that could potentially identify the individual.

The maintenance and improvement of professional competence is an objective of all IHSs and defines them, in the terms of an earlier IOM report, as “learning health systems” (IOM, 2011a). The formal relationships with providers across all participating settings as well as the aggregate clinical and administrative data generated by the EHR provide the foundation for feedback and for rigorous and sustained continuing education. Every health care delivery site is a learning site, providing continuing health education (CHE) that includes point-of-care reminders, links to clinical practice guidelines, and other online resources. This is in addition to traditional CME activities that are at once targeted to the objectives of the IHS and to the need for education where it is most effective—the point of patient engagement.

In 2020 the professional specialty societies, providing leadership for knowledge, are one of the most influential factors moving research participation into the definition of successful practice. Research participation is highlighted and promoted in the educational programs, publications, and recognitions of the societies. The societies support research directly by maintaining registries of clinical experience to which they have unique access and through fellowships that enable community-based practitioners to gain research experience. Board certification requirements recognize research and particularly translation skills.

Academic health centers are integrally related to IHSs. The missions and priorities of AHCs have been clarified to distinguish between those with a substantial investment in basic and translational research, and those that are primarily focused on education and service. The more comprehensive of the former have been designated as academic health science systems (AHSSs) (Dzau, et al. 2010). All AHSSs are organized to serve as research sites for enrollment in observational studies and interventional trials. All IHSs are affiliated with an AHC, which could be an AHSS, which serves them as a resource for education and clinical research through an affiliated IHS office or department. In most cases, the affiliated academic centers have access to the IHS databases for the purpose of maintaining registries and conducting collaborative projects.

A fundamental element undergirding mature electronic health and medical records is the application of an ontology that permits the same terms to be used for describing clinical phenomena and for billing; they also form the basis for quality measurement and for providing adjustments for severity of illness when efficiency measures are assessed. These same terms will also constitute the fundamental “vocabulary” for both observational research databases and randomized controlled trials (RCTs). When needed, additional data elements are added for more detailed investigation and randomization is applied to the record. While the nation has yet to reach this envisioned state of health system integration and effective use of the EHR, the implementation of national health care reform legislation and the efforts of organizations to display their accountability for health care outcomes and costs will likely bring us closer to this stated vision.

 

Part II: The Future of Clinical Research

Controlled clinical trials are the essential cornerstone of modern evidence-based medical care and health practice. Although multiple types of information are needed to fully inform practice, the interventional clinical trial (ICT) plays a particularly critical role. When we view scientific studies as a continuum, we see that they are translated into treatments through a series of steps, beginning with preclinical research that evolves into early-phase clinical trials designed to assess safety and demonstrate proof-of-concept for on-target and off-target effects. Successful interventions (drugs, devices, behavioral strategies) are next evaluated using controlled trials to determine whether the balance of risks and benefits merits the marketing of the technology or intervention. When such studies are robust, practice guidelines can be developed to guide clinical decision making; when they are definitive, guidelines can be distilled into performance measures to assess the quality of practice. At the heart of this paradigm is the measurement of process and outcomes, combined with the use of the measurement system to guide continuous education of practitioners and to assess therapeutic deficits that require new interventions to overcome.

When the effects of an intervention are modest (as is typically the case), randomization provides the most reliable method for determining the true effect of the intervention compared with an alternative. The first known description of an interventional trial is Lind’s 18th-century account of using fruit to combat scurvy (Lind, 1753). Following Fisher’s pioneering demonstration of randomization in a series of agricultural experiments in the 1920s (Fisher, 1926), the first randomized clinical trial was conducted by the British Medical Research Council to evaluate streptomycin for tuberculosis in 1946 (Medical Research Council, 1948). In the 1960s the NIH became the dominant force in developing clinical trials methodologies (Coronary Drug Project Research Group, 1973) and in 1962, following passage of the Kefauver-Harris Drug Amendments, the FDA took the position that efficacy must be established prior to the marketing of drugs (FDA, 2006). As a result of these developments, the RCT was adopted as the standard by which efficacy was determined. Soon, regulatory agencies in other countries joined in, and the majority of research participants were enrolled in trials sponsored by industry and intended to develop or evaluate drugs or devices. In the 1990s the CTE globalized, spurred by the simultaneous expansion of medical technology development together with the global need to understand the effects of health interventions (as well as any associated economic and scientific benefits) in all societies.

A major problem in the field of clinical research is the current absence of a standard ontology that adequately encompasses its activities, rendering it difficult to characterize the state of the enterprise. Common metrics must be based on common definitions so that valid comparisons can be made and policy decisions are based on solid evidence. This problem is illustrated by the multiple definitions of the term “clinical trial.” For the purposes of this report, we have adopted the definition used by the ClinicalTrials.gov registry: “biomedical or health-related research studies in human beings that follow a pre-defined protocol. Interventional studies are those in which the research subjects are assigned by the investigator to a treatment or other intervention, and their outcomes are measured” (National Institutes of Health, 2007). This broad definition thus includes nonrandomized interventions but excludes registries that simply measure practice, as well as research confined to databases. As the clinical research ontology develops, better classification will be key to accurately measuring the progress of the enterprise by comparing the same types of studies and methods over time.

Currently, more than 330 clinical trials are registered every week with the ClinicalTrials.gov registry, which now contains data on more than 110,000 studies. In reviewing data from ClinicalTrials.gov, the CTE is revealed as highly complex, with research studies fulfilling a wide variety of purposes (see Table 1, p. 18). As we noted earlier, the majority of clinical trials are small, conducted in AHSSs, and sponsored either by the federal government (typically through the NIH) or funded internally by the academic organization. However, the vast majority of patients are enrolled in industry-sponsored trials, due to academic trials’ small size and the fact that they are typically not intended to inform practice; rather, academic trials are oriented toward elucidating biological mechanisms or developing pilot data. However, industry resembles academic research in one respect: it conducts many more small, early-phase trials than it does large studies intended to inform practice. While trials are conducted in all disease areas, the distribution is dominated by cancer, cardiovascular medicine, and mental health. Both across disease areas and within specialty areas, it is clear that trial portfolios do not match public health or community medical practice needs in terms of either magnitude or urgency.

Historically, the United States has been a dominant presence in clinical research, but there is growing concern that the enterprise is in decline. There is ample evidence that U.S. trials are becoming more expensive (DeVol et al., 2011). Worse, 90 percent fail to meet enrollment goals, and additional evidence points to disillusionment among American investigators (Getz, 2005). The rate of attrition among U.S. investigators is increasing, even among experienced researchers with strong track records of productivity, while 45 percent of first-time investigators abandon the field after their first trial. The system has become so inefficient that even the NIH is offshoring clinical trials at a substantial rate (Califf, 2011; Kim et al., 2011), using taxpayer funding to conduct trials in countries with less expensive and more efficient CTEs, despite concerns about generalizability as noted above.

While the CTE is in decline in the United States, the need for trials, paradoxically, is increasingly well-recognized. Recent reports indicate that fewer than 15 percent of major recommendations in clinical practice guidelines in infectious disease (Lee and Vielemeyer, 2011) and cardiovascular disease (Tricoci et al., 2009) are based on solid evidence. Recent failures to perform proper trials have led to public health hazards after drugs were developed and marketed without proper supporting evidence. The wide-scale use of antiarrhythmic drugs provides a sentinel example: the CAST Trial (Pratt and Moye, 1990) demonstrated that a treatment that was being used to prevent death was actually causing excess mortality. Similarly, hormone replacement therapy (HRT) was thought to reduce cardiovascular disease until the HERS Trial (Hulley et al., 1998) and the Women’s Health Initiative (Rossouw et al., 2002) demonstrated an excess hazard for cardiovascular events with HRT. Most recently, high-dose erythropoietin appeared to provide substantial benefit in anemia related to renal failure and cancer, and clinical practice guidelines touted its use. But when controlled trials were finally done, high-dose erythropoiesis-stimulating agents were found to cause excess cardiovascular events (Bennet et al., 2008; FDA, 2011; Pfeffer et al., 2009; Singh et al., 2006). Never has the need for properly controlled interventional trials been so clear.

The decline of the U.S. CTE has been noticed by other countries, which are moving rapidly to fill in the gap. The United Kingdom has recently published a national strategy aimed at gaining a larger market share of clinical trials (The Academy of Medical Sciences, 2011) and has made research a primary mission of the National Health Service. Canada has developed a national plan (Canadian Institutes of Health Research, 2011) and China and India are focused on increasing their participation in clinical trials (Gupta and Padhy, 2011; Jia, 2005). At the same time, many countries are providing incentives for industry to locate clinical trials in their countries.

As we note above, we regard globalization of clinical trials as an important positive trend, but the motivating factor should be to provide population- and culture-specific medical evidence to inform local practice. If globalization takes place merely because the United States cannot enroll research participants or has priced itself out of the market, the net result may prove negative. Having countries assume a low-cost vendor status in circumstances where their own populations may not benefit from the research is also troubling from an ethical perspective.

Substantial evidence indicates that the fundamental problem is not rooted in the attitudes of the U.S. public. Although there appears to be variation among socioeconomic and ethnic groups, the public in general places high value on both research in general and clinical research in particular. The majority of Americans report that they would either certainly or most likely participate in research if asked (Getz, 2011). Additionally, an overwhelming majority of those who participate in clinical trials find them to be a positive experience and report that they would do it again. Rather, the problem is not with people but with the health care system—patients are not being asked to participate and many barriers exist (IOM, 2011b).

Because of these challenges, we believe that the CTE can and must be improved and integrated with the evolving health care delivery system as part of an essential evolution toward the learning health system envisioned by the IOM (2007). The CTE can be envisioned as four overlapping enterprises, which we refer to as “laboratories” to emphasize the vast needs that remain to be addressed by the continuing evolution of the entire system. These laboratories share a common fabric of methods, but are different enough that specialized training and education will be needed, intensive methodological work will be required, and investments must be made to produce results that can revolutionize our understanding of how to better prevent, manage, and treat disease.

 

The Innovator’s Lab

The majority of clinical trials will continue to enroll small numbers of research participants to address biological hypotheses, to develop initial evidence about the mechanisms of action of drugs, devices, and other interventions, or to develop preliminary evidence of their risk-benefit profile. We believe that the world of phase 1 units and academic studies must be combined into a much more effective approach to human systems biology. This new system should be highly networked, enabling researchers to leverage major advances in genetics, genomics, and biomarkers.

Because studies of biological mechanisms and early studies of new therapies require intensive measurements in human volunteers, these studies should be performed in an environment separate from the health care delivery system. Recent evaluation has implicated both a failure to fully engage the biological target of therapy as well as “off-target” effects in the high rate of attrition characteristic of early-phase trials. Also, when late-phase failures are valuated, off-target effects frequently emerge as the cause. Systems biological measurements, in which multiple biological outcomes can be monitored simultaneously, promise better characterization of on-target effects as well as better identification of off-target effects.

The environment for this type of research should be intensive in terms of highly qualified staff and sophisticated data-management capabilities. Because the studied populations will either be 1) normal volunteers or 2) patients with a disease of interest exposed to a new drug or device and/or undergoing intensive measurement, the most effective environment will likely be found within a hospital with advanced emergency care, imaging, and data capabilities.

Currently, these types of studies are either done in commercial “phase 1 units” or academic medical centers funded by the Clinical and Translational Science Awards (CTSA) program (CTSAs, 2011). However, phase 1 units lack sophisticated technologies for measuring biological mechanisms and are typically not collocated within a hospital. Academic centers are limited as well: they typically have inefficient units that are focused on specific investigator needs and not on throughput and sponsor needs.

 

The Traditional Lab

Even under the new system, there will still be a need for traditional trials designed to determine the efficacy of an intervention or to assess the balance of risks and benefits in carefully defined populations. Although these studies are labor-intensive and expensive, they are indispensible in determining whether a technology has any significant effect compared with placebo or standard treatment. In some cases, these studies should be done in the context of clinical care (e.g., when research participants are acutely ill). In other circumstances, they can be done in stand-alone research centers designed to effectively administer experimental treatment and collect extensive data on efficacy measures and adverse events.

There is overlap between the traditional lab and the health care delivery lab, especially in comparative-effectiveness research (CER). As the demand for comparative evidence grows, so too will the need for trials with pragmatic features that are carefully controlled, may be blinded, and are performed in the context of clinical practice (Eisenstein et al., 2008; Tunis et al., 2010; Yusuf et al., 2008).

One example of the first type of efficacy study can be seen in the setting of acute myocardial infarction (AMI). Patients presenting in the midst of a life-threatening emergency such as AMI must be enrolled in studies while at the same time receiving urgent medical care. For at least the first 1,500 patients enrolled on a trial of a new treatment, there must be intensive collection of adverse events to ensure that an unexpected toxicity (or benefit) is not occurring.

An example of the second type of efficacy study would be a trial of a new treatment for seasonal allergies, in which there is no reason to burden the health care delivery system with people who can elect to participate in a trial of a less serious condition. While this type of study could be done in primary care practices, it might also be done most efficiently in standalone research centers divorced from the pressures of the practice environment. As trials methodology advances, the efficiency afforded by a study site outside of clinical practice must be weighed against the generalizability of the findings.

 

The Health Care Delivery System as Laboratory

The central premise of this paper is that achieving the highest potential of clinical research depends upon the incorporation of clinical research into the broad scope of practice of health care delivery. Doctors, nurses, and other providers should regard it as their business to generate evidence as well as to translate it into practice. This research should include evaluation of the risks and benefits of technologies, the assessment of methods of delivering interventions at the point of care, as well as organizational aspects of health care.

The critical need for knowledge to guide diagnosis, assessment, and treatment of disease cannot be met by the current divided system, with its redundant domains for personnel and data collection. We have a significant window of opportunity arising from the development of EHRs and the drive to collect well-defined data for quality improvement and population health endeavors. If data from integrated health systems can be used for clinical trials as well as quality improvement/population health, the incremental cost of clinical trials could be reduced significantly.

In addition to the development of standardized data collection and ontologies, this transformation requires the adoption of and financial support for knowledge generation as a fundamental value of the health care delivery system. Providers, as well as patients and their families, must be made aware of the cost in lives and disability that inevitably accompanies failure to define benefits and risks in the context of health care. If we successfully cultivate this awareness, obtaining consent and inserting randomization into the treatment paradigm will not be regarded as an impediment to efficiency; instead, they will be viewed as essential elements of ethical and appropriate health care delivery. The immediate results of such a fundamental change to the system will be a dramatic increase in the amount and quality of evidence available to inform practice, combined with a fine-tuning of the system to achieve best practices as a matter of course.

As mentioned above, there is overlap between the traditional lab and the clinical practice lab in the area of CER. These regions of overlap will evolve further as data warehouses are used to conduct postmarketing studies of drug and device safety (Behrman et al., 2011).

 

The Community Engagement Lab

Many interventions that have a potentially great impact on health either do not involve the health care delivery system or do so only peripherally. Daunting questions about disease prevention, managing a healthful lifestyle, and living with chronic illness require the application of a variety of methods, but clinical trials should be an element of the repertoire.

One important type of study will formally assess approaches to prevention or behavioral intervention in chronic disease. While the health care delivery system may be involved, the majority of a given intervention may well take place in homes, schools, or neighborhoods. This research will require adoption of principles of community engagement, so that the needs of individuals and their communities are understood, allowing them to participate with full knowledge of the methods and rationales for specific clinical trials.

An essential method in this major area of research will be cluster randomization, an approach that randomly allocates “units” to different interventional strategies. These units may include clinics, hospitals, neighborhoods, or communities. Cluster randomization may provide the most efficient method for determining optimal public health policies.

A new and rapidly evolving approach to clinical trials is being developed by voluntary health organizations and other entities that have pioneered using the Internet for this purpose, without significant health delivery system involvement or control. In fact, the use of patient-reported outcomes (PROs), both through voluntary health organizations and through IHS records, provides a basis for incorporating subjective elements into the research paradigm, so that the impact of interventions on quality of life can be assessed in a systematic, generalizable manner.

 

Part III: Specific Steps to Achieve the Vision: The Innovator’s and Traditional Labs

Revival of these arenas in the United States will depend upon successful reinvigoration of the NIH’s clinical research efforts and the development of effective public-private partnerships to drive research. The newly formed National Center for Advancing Translational Sciences (NCATS) (Collins, 2011) has the potential to catalyze the many elements of the system that must be combined to achieve successful transformation. In particular, the Innovator’s Lab would be stimulated by networking of the former general clinical research centers (GCRCs) and the development of a common informatics infrastructure, with core technologies available for assessing systems biology and linking phenotypic information with genomic and physiological data.

The Traditional Lab should be rejuvenated by linking more efficient traditional research-site functions with the evolving health care delivery lab. Given the higher cost of labor in the United States, the only way to remain competitive in this arena is through innovation in data collection and study procedures. Of note, recent findings demonstrating heterogeneity in treatment effects as a function of the country of enrollment (Mahaffey et al., 2011; O’Connor et al., 2011; Simes et al., 2010) have led the FDA to require more U.S. enrollment for studies of products intended for domestic marketing, so success in this arena is a critical element of the overall plan. Simply offshoring this research is not a viable approach to informing the U.S. population of the best choices with regard to treatment and diagnostic strategies.

 

How the Vision Can Be Achieved Through the Integrated Health System as Health Care Delivery Laboratory

It is both possible and essential that we expand participation in controlled clinical trials in a cost-effective manner, ensuring that such research reflects the general population, their providers, and the range of settings in the health care delivery system of 2020. However, success will require a new approach to the partnership between the CTE and the health care delivery system, one that builds on the centrality of the emerging IHSs and their access to community-based practitioners and hospitals. Expanding the base of clinical research to include many if not most of the non-academic IHSs will result in important contributions to the vitality of service-delivery systems and to the richness of professional practice in all of the participating entities.

In addition to nurturing clinician research, these Health Care Delivery Laboratories will facilitate the translation of research from the bedside to community practice. And they will also stimulate essential feedback from community practice to academic research, with community-based clinicians being encouraged to initiate and participate in knowledge development. In addition, they will enhance the professional development of clinicians across integrated systems and increase community engagement and support. These learning organizations will provide health professions students with the opportunity to observe the professional and practical benefits of participating in clinical research across the full spectrum of practice settings.

 

A Practical Roadmap to Implementation

Direction and commitment on the part of health system leadership will be essential to forging this new partnership. Both IHS governance and management leadership must understand and communicate the value added by explicitly building research into the mission, objectives, and strategies of the organization; further, they must be charged with shaping a supportive culture. Their message is that participation in clinical and system-improvement research is an essential dimension of the social compact among the health care delivery system, health care providers, the public, and the scientific enterprises that serve them. We recognize that this definition of scope of practice to include research is disruptive. That is the point. We cannot continue a practice and education pattern that is not optimally responsive to, or aligned with, the needs of science and society.

Research in this context is broadly defined to include clinical trials, but also embraces “a much broader range of investigative methods. These methods include epidemiological observations, clinical observations, quasi-experimental evaluations of natural experiments, time series experiments, case studies of apparently successful projects or organizations, rapid-cycle learning and qualitative studies, either alone or as mixed studies in combinations with quantitative studies” (Kottke et al., 2008). It includes health services and operations research to improve the qualitative and economic performance of the system.

A typical health system mission is “to provide quality of care to the community and contribute to medical knowledge.” An example of a specific priority objective of many systems might be to build specialty-service lines such as cardiology and oncology with the aim of becoming the leading center in the region. Participating in research, including clinical trials, is a realistic strategy toward accomplishing that objective, and may in fact be a prerequisite of success.

Incorporating research as a core element of a health system’s mission, objectives, and strategies will enhance the system’s stature and attract political, community, and financial support. It will also expedite the recruitment of practitioners and other employees who aspire to be associated with an organization at the leading edge of health care development.

A supportive infrastructure, aligned with a culture that values research and education, depends upon the commitment of clinical leadership to implement a research business plan. An important objective of the plan is to provide potential research sponsors with a single portal to the system—one that is responsive to their needs and facilitates system-wide access to practitioner investigators.

A well-developed business plan will address specific goals, resources and their allocation, and the timeline for implementation. It will address how the system will communicate with patients to establish the expectation that they may be invited to participate in clinical research. It will address how the organization can identify, reduce, or remove impediments to practitioner participation and overcome the understandable reluctance of busy providers to engage in the research process.

Research business plans may be implemented at the level of the integrated delivery system, affiliated physician groups, and other community-based entities, or all three, if those entities are well-coordinated. There are medical groups, not affiliated with an academic medical center, that are implementing successful research business plans that focus on clinical trials. These community-based sites, or networks, can enter into agreements with research institutes and contract research organizations (CROs) to recruit patients and implement clinical trial protocols. These groups are profitable and have high levels of practitioner participation and patient enrollment and indicate that active clinical practices outside of an academic medical center can successfully add clinical trials to their business plans. We can advance the vision of this paper’s Practical Roadmap by looking to these groups as learning laboratories.

Small systems and medical groups may establish leadership by broadening the Chief Medical Officer role to include responsibility for research development and possibly for education. Others may create a new position of Research Director, while more complex systems may establish a Research Institute or Department. The designated research leader can encourage practitioners to become involved in research by conducting “tentative ideas” seminars, offering methods coaching, and arranging for mentoring.

The research objective will be further enhanced over time by building research promotion and support into the position descriptions of department heads, practice directors, and chairs, so that individuals are recruited into the organization with that expectation. The incentive system in a community hospital-based system can be designed to recognize research participation, including providing supplementary income and the opportunity for protected time.

Customer-friendly administrative support is a major factor in encouraging investigator-initiated clinical research, as well as expanding practitioner participation in government- and industry-supported clinical trials. Such support includes assistance in proposal writing, ethical review, privacy questions, grant administration, IRB relationships, incident reporting, data management and reporting, and preparation of publications.

Some start-up investments are of course necessary. Assuming that the sponsor’s business plan includes appropriate reimbursement, it is likely that a robust clinical research program will be self-supporting, if not profitable. Modest strategic investments will attract and stimulate practitioner interest. These may include small seed grants for proposal development, research assistants, and attendance at meetings. A small fund would also support participation in networks and collaboratives.

The IHS will further its research objectives by celebrating the research participation and accomplishments of practitioner-researchers. The national goal of expanding Health Care Delivery Labs will be enhanced if the research initiatives of community-based practitioners in integrated delivery systems lead to those practitioners being recognized as “clinical research associates,” thereby positively differentiating them from their peers. Participation in clinical trials and other research can be a focus of presentations within the system and provide a basis for communications to the community, enhancing the message that their health system is a learning organization.

 

Part IV: Implementing the Research Agenda

This vision of disruptive innovation, one that is necessary to transform the challenged 2011 CTE into the robust CTE of 2020, generates researchable questions to be addressed with alacrity by the health services community. These questions reside largely in the domain of health services research (HSR), which has been defined as “a multidisciplinary field of inquiry, both basic and applied, that examines the use, costs, quality, accessibility, delivery, organization, financing and outcomes of health care services” (IOM, 1995).

As implementation of the integrated community-based health system model proceeds, the following questions are examples of the research agenda that is called for:

• What are successful cases of securing health system cultural change and system-wide buy-in?
• What are the specific impediments to change in organizational culture?
• How have each of the impediments been managed?
• What are the financing and costing alternatives and their implications?
• How have the systems that have adopted a research agenda organized to implement it?
• What are the characteristics of community medical groups and practitioners that participate actively in clinical research?
• What approaches to enlisting practitioner participation are most successful?
• What approaches to securing trial enrollment and participant retention are most successful in the community practice setting?
• How have community-based research programs succeeded in engaging culturally and linguistically diverse populations?
• How can participation in clinical trials be successfully extended into chronic disease care settings?
• How can clinical trials be organized to maximize practitioner participation?
• What are successful methods of achieving public engagement?
• How have systems, hospitals, and medical groups that are engaged in research developed the necessary workforce competencies?

 

Summary

The United States faces a pressing need for a revitalized clinical trials infrastructure. But rather than attempting to optimize clinical trials conduct independently of health care delivery, we recommend that clinical trials and clinical care be integrated into the learning health system. The rapid evolution of integrated health systems provides a natural vehicle for accomplishing this goal, using electronic health records aggregated at the health system level and beyond. The one major exception to this approach should be for intensive biological research that we envision as taking place within a system of Innovator’s Labs connected by sophisticated informatics and strategic technology investments. The continuation of standardized control trials in the Traditional Labs and the development of Health Care Delivery Labs embedded within integrated health systems will provide a major improvement in the quantity, quality, and generalizability of clinical trial results (see Figure 2, p. 21). Community Engagement Labs will need to be expanded to conduct critical trials outside of traditional health care delivery systems. Intense efforts will be needed in order to understand the requirements for an effective national system capable of interdigitating with developing global systems, including efforts focused on workforce development, economic analysis, and concrete operational plans.

 

 

The vision described in this paper for the future of clinical research will require improved alignment among patients, payers, and providers with respect to the implications of research and the choices of each stakeholder regarding receiving, funding, or providing specific services. Assuming that such alignment among stakeholders can be attained, the financial incentives to improve care, avoid waste, redundancy, and unnecessary services could result in better health outcomes, moderation of cost increases, and the opportunity for a jointly financed research and development engine to provide for continuous measurement, innovation, and improvement across the health care continuum. Given the fragmented nature of the American payer and provider landscape, a national “clinical research tax” applied to health care premiums could be a potential source of the funds needed to fuel the research and development engine envisioned in the integrated health care and clinical trials enterprise of the future. Some of the savings enabled by continuous quality improvement in health care could also lower the costs of care and premiums paid by patients, or at the least moderate their continued increase.

The authors developed this paper in the context of the IOM Forum on Drug Discovery, Development, and Translation’s 2-year effort to assess the current status of the clinical trials enterprise and to put forward a plan to assure a robust and optimal future for clinical research. The consequences of the problems in the extant system are clear and the need to address them is urgent. There is a compelling opportunity to effectively address the gap between the research enterprise and the delivery system in the near future. We suggest attention to the development of a national agent that will bring the stakeholders together and catalyze the essential realignment of organizational mission and system-wide incentives.

 

 

---

References

1. Behrman, R. E., J. S. Benner, J. S. Brown, M. McClellan, J. Woodcock, and R. Platt. 2011. Developing the Sentinel System—a national resource for evidence development. New England Journal of Medicine 10;364(6):498-499. https://doi.org/10.1056/NEJMp1014427
2. Bennett, C. L., S. M. Silver, B. Djulbegovic, A. T. Samaras, C. A. Blau, K. J. Gleason, S. E. Barnato, K. M. Elverman, D. M. Courtney, J. M. McKoy, B. J. Edwards, C. C. Tigue, D. W. Raisch, P. R. Yarnold, D. A. Dorr, T. M. Kuzel, M. S. Tallman, S. M. Trifilio, D. P. West, S. Y. Lai, and M. Henke. 2008. Venous thromboembolism and mortality associated with recombinant erythropoietin and darbepoetin administration for the treatment of cancer-associated anemia. JAMA 299:914-924. https://doi.org/10.1001/jama.299.8.914
3. Bollyky T. J. 2011. Safer, Faster, Cheaper: Improving Clinical Trials and Regulatory Pathways to Fight Neglected Diseases: Report of the Center for Global Development Working Group on Clinical Trials and Regulatory Pathways. Washington, DC: Center for Global Development. http://www.cgdev.org/files/1425588_file_Bollyky_Clinical_Trials_FINAL.pdf (accessed April 11, 2012).
4. Califf, R. M. 2009. Clinical research sites—the underappreciated component of the clinical research system. JAMA 302(18):2025-2027. https://doi.org/10.1001/jama.2009.1655
5. Califf, R. M. and R. A. Harrington. 2011. American industry and the U.S. cardiovascular clinical research enterprise: An appropriate analogy? Journal of the American College of Cardiology 58(7):677-680. https://doi.org/10.1016/j.jacc.2011.03.048
6. Califf, R. M., D. A. Zarin, J. M. Kramer, R. E. Sherman, L. H. Aberle, and A. Tasneem. Characteristics of Clinical Trials Registered in ClinicalTrials.gov, 2007-2010. JAMA 307(17): 1838-1847. https://doi.org/10.1001/jama.2012.3424
7. Canadian Institutes of Health Research. 2011. Annual Report 2010-2011. http://www.cihrirsc.gc.ca/e/44144.html (accessed October 18, 2011).
8. Clinical and Translational Science Awards (CTSAs). 2011. http://www.ctsaweb.org/index.cfm?fuseaction=home.aboutHome (accessed September 27, 2011).
9. Collins, F. S. 2011. The NIH National Center for Advancing Translational Sciences: How Will It Work? https://www.dtmi.duke.edu/website-administration/files/Collins%20NCATS%20slides.pdf (accessed September 27, 2011).
10. Coronary Drug Project Research Group. 1973. The Coronary Drug Project: Design, methods, and baseline results. Circulation 47(Suppl 1):11-179. https://doi.org/10.1161/01.cir.47.3s1.i-1
11. DeVol, R. C., A. Bedroussian, and B. Yeo. 2011. The global biomedical industry: Preserving U.S. leadership. The Milken Institute. http://www.milkeninstitute.org/publications/publications.taf?function=detail&ID=38801285&cat=resrep (accessed October 18, 2011).
12. Dzau, V. J., D. C. Ackerly, P. Sutton-Wallace, M. H. Merson, R. S. Williams, K. R. Krishnan, R. C. Taber, and R. M. Califf. 2010. The role of academic health science systems in the transformation of medicine. Lancet 375(9718):949-953. https://doi.org/10.1016/S0140-6736(09)61082-5
13. Eisenstein, E. L., R. Collins, B. S. Cracknell, O. Podesta, E. D. Reid, P. Sandercock, Y. Shakhov, M. L. Terrin, M. A. Sellers, R. M. Califf, C. B. Granger, and R. Diaz. 2008. Sensible approaches for reducing clinical trial costs. Clinical Trials 5(1):75-84. https://doi.org/10.1177/1740774507087551
14. FDA (Food and Drug Administration). 2006. Promoting Safe and Effective Drugs for 100 Years. http://www.fda.gov/AboutFDA/WhatWeDo/History/CentennialofFDA/CentennialEditionofFDAConsumer/ucm093787.htm (accessed September 28, 2011).
15. FDA. 2007. Public Law 110-85. The Food and Drug Administration Amendments Act of 2007. http://www.fda.gov/RegulatoryInformation/Legislation/FederalFoodDrugandCosmeticActFDCAct/SignificantAmendmentstotheFDCAct/FoodandDrugAdministrationAmendmentsActof2007/FullTextofFDAAALaw/default.htm (accessed May 31, 2011).
16. FDA. 2011. FDA Drug Safety Communication: Modified Dosing Recommendations to Improve the Safe Use of Erythropoiesis-Stimulating Agents (ESAs) in Chronic Kidney Disease. http://www.fda.gov/Drugs/DrugSafety/ucm259639.htm (accessed October 19, 2011).
17. FasterCures. 2005. Think Research: Using Electronic Medical Records to Bridge Patient Care and Research. White Paper. http://www.fastercures.org/objects/pdfs/white_papers/emr_whitepaper.pdf (accessed October 17, 2011).
18. Fisher R. A. 1926. The arrangement of field experiments. Journal of the Ministry of Agriculture of Great Britan 33:503-513. Available at: https://digital.library.adelaide.edu.au/dspace/handle/2440/15191.
19. Getz, K. A. 2005. Number of active investigators in FDA-regulated clinical trials drop. Tufts Center for the Study of Drug Development CSDD Impact Report 7:1-4.
20. Getz, K. A. 2011. Public Confidence and Trust Today: A Review of Public Opinion Polls. http://www.ciscrp.org/downloads/articles/Getz_publicopinion.pdf (accessed October 18, 2011).
21. Glickman, S. W., J. G. McHutchison, E. D. Peterson, C. B. Cairns, R. A. Harrington, R. M. Califf, and K. A. Schulman. 2009. Ethical and scientific implications of the globalization of clinical research. New England Journal of Medicine 360(8):816-823. https://doi.org/10.1056/NEJMsb0803929
22. Gupta, Y. K. and B. M. Padhy. 2011. India’s growing participation in global clinical trials. Trends in Pharmacological Sciences 32(6):327-329. https://doi.org/10.1016/j.tips.2011.02.017
23. Hulley, S., D. Grady, T. Bush, C. Furberg, D. Herrington, B. Riggs, and E. Vittinghoff. 1998. Randomized trial of estrogen plus progestin for secondary prevention of coronary heart disease in postmenopausal women: Heart and Estrogen/progestin Replacement Study (HERS) Research Group. JAMA 280:605-613. https://doi.org/10.1001/jama.280.7.605
24. Institute of Medicine (IOM). 1995. Health Services Research: Work Force and Educational Issues. Washington, DC: The National Academies Press. https://doi.org/10.17226/5020.
25. IOM. 2007. The Learning Healthcare System: Workshop Summary. Washington, DC: The National Academies Press. https://doi.org/10.17226/11903.
26. IOM. 2008. Evidence-Based Medicine and the Changing Nature of Health Care: 2007 IOM Annual Meeting Summary. Washington, DC: The National Academies Press. https://doi.org/10.17226/12041.
27. IOM. 2011a. Engineering a Learning Healthcare System: A Look at the Future: Workshop Summary. Washington, DC: The National Academies Press. https://doi.org/10.17226/12213.
28. IOM. 2011b. Public Engagement and Clinical Trials: New Models and Disruptive Technologies: Workshop Summary. Washington, DC: The National Academies Press. https://doi.org/10.17226/13237.
29. Jia, H. 2005. China beckons to clinical trial sponsors. Nature Biotechnology 23(7):768. https://doi.org/10.1038/nbt0705-768
30. Kim, E. S. H., T. P. Carrigan, and V. Menon. 2011. International participation in cardiovascular randomized controlled trials sponsored by the NHLBI. Journal of the American College of Cardiology 58(7):671-676. https://doi.org/10.1016/j.jacc.2011.01.066
31. Kottke, T. E., L. I. Solberg, A. F. Nelson, D. W. Belcher, W. Caplan, L. W. Green, E. Lydick, D. J. Magid, S. J. Rolnick, and S. H. Woolf. 2008. Optimizing practice through research: A new perspective to solve an old problem. Annals of  Family Medicine 6(5):459-462. https://doi.org/10.1370/afm.862
32. Lee, D. H., and O. Vielemeyer. 2011. Analysis of overall level of evidence behind Infectious Diseases Society of America practice guidelines. Archives of Internal Medicine 171:18-22.
33. Lind, J. 1753. A Treatise on the Scurvy in Three Parts, Containing an Inquiry into the Nature, Causes, and Cure, of That Disease. http://www.jameslindlibrary.org/illustrating/records/atreatise-of-the-scurvy-in-three-parts-containing-an-inquiry/title_pages (accessed September 27, 2011).
34. Mahaffey, K. W., D. M. Wojdyla, K. Carroll, R. C. Becker, R. F. Storey, D. J. Angiolillo, C. Held, C. P. Cannon, S. James, K. S. Pieper, J. Horrow, R. A. Harrington, L. Wallentin, and the PLATO Investigators. 2011. Ticagrelor compared with clopidogrel by geographic region in the Platelet Inhibition and Patient Outcomes (PLATO) trial. Circulation 124(5):544-554. https://doi.org/10.1161/CIRCULATIONAHA.111.047498
35. Medical Research Council. 1948. Streptomycin treatment of pulmonary tuberculosis: A Medical Research Council investigation. British Medical Journal 1948(ii):7692.
36. National Institutes of Health (NIH). 2007. Understanding Clinical Trials. http://ClinicalTrials.gov/ct2/info/understand#Q01 (accessed October 19, 2011).
37. O’Connor, C. M., M. Fiuzat, K. Swedberg, M. Caron, B. Koch, P. E. Carson, W. Gattis-Stough, G. W. Davis, and M. R. Bristow. 2011. Influence of global region on outcomes in heart failure β-blocker trials. Journal of the American College of Cardiology 58(9):915-922. https://doi.org/10.1016/j.jacc.2011.03.057
38. Pfeffer, M. A., E. A. Burdmann, C. Y. Chen, M. E. Cooper, D. de Zeeuw, K. U. Eckardt, J. M. Feyzi, P. Ivanovich, R. Kewalramani, A. S. Levey, E. F. Lewis, J. B. McGill, J. J. V. McMurray, P. Parfrey, H. H. Parving, G. Remuzzi, A. K. Singh, S. D. Solomon, and R. Toto for the TREAT Investigators. 2009. A trial of darbepoetin alfa in type 2 diabetes and chronic kidney disease. New England Journal of Medicine 361:2019-2032. https://doi.org/10.1056/NEJMoa0907845
39. Pratt, C. M. and L. Moye. 1990. The Cardiac Arrhythmia Suppression Trial: Implications for anti-arrhythmic drug development. Journal of Clinical Pharmacology 30:967-974. https://doi.org/10.1002/j.1552-4604.1990.tb03580.x
40. Rossouw, J. E., G. L. Anderson, R. L. Prentice, A. Z. LaCroix, C. Kooperberg, M. L. Stefanick, R. D. Jackson, S. A. A. Beresford, B. V. Howard, K. C. Johnson, J. Morley Kotchen, J. Ockene, and the Writing Group for the Women’s Health Initiative Investigators. 2002. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: Principal results from the Women’s Health Initiative randomized controlled trial. JAMA 288(3):321-333. https://doi.org/10.1001/jama.288.3.321
41. Simes, R. J., R. L. O’Connell, P. E. Aylward, S. Varshavsky, R. Diaz, R. G. Wilcox, P. W. Armstrong, C. B. Granger, J. K. French, F. Van der Werf, I. A.c Marschner, R. Califf, and H. D. White. 2010. Unexplained international differences in clinical outcomes after acute myocardial infarction and fibrinolytic therapy: Lessons from the Hirulog and Early Reperfusion or Occlusion (HERO)-2 trial. American Heart Journal 159(6):988-997. https://doi.org/10.1016/j.ahj.2009.12.044
42. Singh, A. K., L. Szczech, K. L. Tang, H. Barnhart, S. Sapp, M. Wolfson, D. Reddan, and the CHOIR Investigators. 2006. Correction of anemia with epoetin alfa in chronic kidney disease. New England Journal of Medicine 355:2085-2098. https://doi.org/10.1056/NEJMoa065485
43. The Academy of Medical Sciences. 2011. A new pathway for the regulation and governance of health research. http://www.acmedsci.ac.uk/p47prid88.html (accessed September 28, 2011).
44. Tricoci, P., J. M. Allen, J. M. Kramer, R. M. Califf, and S. C. Smith, Jr. 2009. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA 301:831-841. https://doi.org/10.1001/jama.2009.205
45. Tunis, S. R., J. Benner, and M. McClellan. 2010. Comparative effectiveness research: Policy context, methods development and research infrastructure. Statistics in Medicine 30;29(19):1963-1976. https://doi.org/10.1002/sim.3818
46. U.S. Census Bureau. 2008. Projections of the population by selected age groups and sex for the United States: 2010 to 2050 (2008 projections). http://www.census.gov/population/www/projections/summarytables.html (accessed September 27, 2011).
47. Yusuf, S., J. Bosch, P. J. Devereaux, R. Collins, C. Baigent, C. Granger, R. Califf, and R. Temple. 2008. Sensible guidelines for the conduct of large randomized trials. Clinical Trials 5(1):38-39. https://doi.org/10.1177/1740774507088099
48. Zerhouni, E. 2003. Medicine. The NIH Roadmap. Science 302(5642):63-72. https://doi.org/10.1126/science.1091867.

Tweets by @theNAMedicine