Drug development has always been a high-risk enterprise. It requires sustained optimism in the face of repeated failures. Define a successful drug as a drug that has been approved for marketing by regulators. Define success rate as the rate of successful drugs among all drugs that have been entered into the clinical testing phase of development. This rate has decreased in recent years. In 2004, the Food and Drug Administration (FDA) in the United States (US) issued a Critical Path Initiative Document (http://www.fda.gov/ScienceResearch/SpecialTopics/CriticalPathInitiative/CriticalPathOpportunitiesReports/ucm077262.htm#fig3). In the document, FDA quoted a “current” success rate around 8% and a historical success rate of 14%.
Success rates differ among drugs for different therapeutic areas. Systemic anti-infectives tend to have a higher success rate, partly due to the existence of highly predictive animal models for most infections. On the other end, the success rate of drugs for treating the central nervous system disorders is generally lower. There has not been a new molecular entity approved for the Alzheimer’s Disease (AD) in the US since 2003 when memantine was approved. Up until now, only five AD drugs have been approved, all for treating the symptoms of AD. Among them, one (tacrine, approved in 1993) is no longer available due to safety issues. Many experimental AD drugs, both small molecules and immunotherapies, have either failed to show an advantage when added to a background therapy or have been associated with unacceptable toxicity (Cummings et al., 2014). The good news is that there are currently some promising drugs for AD in the late-stage development. I am hopeful that one of them will be successful, capable of improving not only AD symptoms but also modifying the disease process.
Along with a decrease in the overall success rate, the expense in developing a successful drug has steadily increased. A study released by the Tuft’s Center for the Study of Drug Development in 2014 suggests that the average pre-tax industry cost in bringing a new medicine to market is now around $2.56 billion USD (DiMasi, 2014). The study included 106 investigational new drugs from 10 mid- to large-size pharmaceutical companies and the drugs were first tested in humans during 1995-2007. Cost included clinical development up to 2013. By comparison, in 2003, the cost was estimated to be about $1.04 billion in the 2013 dollars. While some critics questioned the accuracy of these figures, the figures nevertheless suggest a doubling in cost for successfully turning an innovative drug into a commercial product.
Having worked in the pharmaceutical industry for 30 years, I am acutely aware of the cost associated with certain trials. For example, since 2008, FDA has required sponsors of new molecular entities for type 2 diabetes to conduct a large cardiovascular (CV) outcome trial to assess the cardiovascular risk of these drugs (FDA, 2008). The CV outcome trial is in addition to the 7 or 8 phase II and phase III trials typically conducted to support the marketing application for a new type 2 diabetes medicine. These CV outcome trials enroll thousands, if not tens of thousands, of patients and treat them for 2 years. An industry colleague told me that their CV outcome trial had an initial cost estimate of $400M USD. As is often the case, the initial cost estimate could under-estimate the ultimate cost.
A trend to combat high risk and high development cost is for companies to adopt a co-development business model to share cost and risk. Of course, alliance companies will share financial returns if the drug turns out to be commercially successful. The concept of co-development is not new, but we have seen more examples of it in recent years.
Because of the need to contain development cost and manage investment risk, many companies are looking for data-driven quantitative approaches to make better decisions so that if a development program is to fail, it can be terminated earlier and more efficiently. Making efficient decisions require designing trials that can offer the needed evidence efficiently. While looking for more efficient study designs has always been an area of intense research interest for many scientists, the need to look for new design options has accelerated since 2004. A class of designs beyond the traditional group sequential design has emerged from these efforts. A common feature of these designs is to use interim data of a trial to modify certain aspects of the trial so that the trial can better address the questions it is designed to answer. These designs are categorically called adaptive designs. Group sequential designs (Jennison and Turnbull, 2003) belong to this class of designs.
The “new” adaptive designs include features such as sample size re-estimation, modifying randomization ratios to different treatment groups, dropping a dose, adding a dose, and limiting future enrollment to a subgroup of patients etc. Bauer et al. (2015) gave a historical overview of the history of confirmatory adaptive designs over the 25 years since 1989. They describe the early days of adaptive design research, review the key methodological concepts and summarize regulatory and industry perspectives on adaptive designs. The article also includes a critical review of how expectations from the beginning of adaptive design journey were fulfilled, and discusses potential reasons why the expectations were not fulfilled in some cases. Another good reference for adaptive designs is a recent book edited by He, Pinheiro and Kuznetsova (2014).
Against the backdrop of a lower success rate and higher cost for large development programs is a trend for the pharmaceutical industry to invest more heavily in precision medicine or medicines for rare diseases. In 2014, FDA approved 41 new molecular entities. According to Gulfo (2015) of Breakthrough Medical Innovations, 40% of the new molecular entities approved are for rare diseases, underscoring industry’s shift to niche products. There are several reasons for this shift. First, a large number of products that are highly effective in treating common disorders (e.g. high cholesterol, high blood pressure, CNS disorders) have become generic in the past decade. It is hard to demonstrate extra value beyond what the now generic products could offer. On the other hand, value proposition is easier for rare diseases that have traditionally received less attention and nevertheless continue to represent unmet medical needs. The smaller patient populations afflicted with rare diseases require out-of-box thinking in study design and nimble applications of innovative approaches to clinical studies. Adaptive designs are attractive options in these cases.
On July 9 2012, the US Congress signed the FDA Safety and Innovation Act. The Act allows FDA to designate a drug as breakthrough therapy if (1) the drug, used alone or in combination with other drugs, is intended to treat a serious or life-threatening disease or condition; and (2) preliminary clinical evidence indicates that the drug may demonstrate substantial improvement over existing therapies on at least one clinically significant endpoint. A manufacturer can submit the breakthrough designation request to the FDA for their drug and the agency has 60 days to grant or deny the request. Once a drug is designated as breakthrough therapy, FDA will expedite the development and review of such drug. Between October 1 2014 and September 4 2015, the Center for Drug Evaluation and Research (CDER) at the FDA received 87 requests for breakthrough designation. As of today (September 14 2015), FDA granted the breakthrough status to 25 requests and denied it in 37 instances. The remaining 25 requests are still awaiting decisions. (http://www.fda.gov/drugs/developmentapprovalprocess/howdrugsaredevelopedandapproved/drugandbiologicapprovalreports/indactivityreports/ucm373559.htm).
Having a drug designated as a breakthrough therapy is highly desirable. It increases the prestige of the drug and offers the possibility of a faster time to market. A requirement for a breakthrough therapy is preliminary clinical evidence of substantial improvement over existing therapies on at least one clinically meaningful endpoint. The preliminary clinical evidence could come from an early trial in a small number of subjects. Pereira et al. (2012) reported findings from an empirical investigation on how often very large treatment effects were replicated in subsequent trials of the same comparison, disease and outcome. They concluded that most large treatment effects observed in small studies became much smaller when additional trials were performed.
In a recent article, a colleague and I commented on similar experiences in drug development. Promising results observed in early proof-of-concept (POC) trials either decreased in magnitude or failed to replicate completely (Chuang-Stein and Kirby, 2014) in subsequent trials. Despite our training, we statisticians often forget about the regression-to-mean phenomenon when using the very positive results from a completed trial to predict results of future trials. The more rare a successful therapy is in a therapeutic area, the more we want to believe that we have successfully unlocked the mystery of a disease and have found the holy grail in treating the disease upon observing an overwhelming effect in an early trial. We could be so wrong in our optimistic belief at times!
One way to understand the need to be cautious about early promising results is to treat clinical trials at the various stages of drug development as a series of diagnostic tests for the effectiveness of the drug (Lee and Zelen, 2000). Under the diagnostic test analogy, sensitivity of the diagnostic test is given by the power at an assumed drug effect and specificity is 1 minus the false positive rate. The POC trial is the first test applied to the drug. Let’s assume that the POC trial has a sensitivity of 80% and a specificity of 90%. Let’s further assume that the average probability of success is 10%. Using the Bayes rule, we can calculate the positive predictive value (PPV) of our POC trial, which is the probability that the drug indeed has the assumed effect if the trial returns a positive result.
In the above example, the PPV is 47%, meaning that more than half of the time, the drug that delivers a positive POC readout does not have the assumed effect. The probability drops to 30% if the prior average success rate is only 5%. We have found this to be an easy way to explain to non-statisticians why replication is needed to verify positive early results before making huge development investment, especially for diseases where prior successes are rare.
Developing life-saving drugs is an exciting and rewarding endeavor. It requires optimism and a lot of hard work. Needless to say, it is very helpful to have luck on our side too.
By Christy Chuang-Stein
Independent Consultant
Formerly, Vice President and Head of the Statistical Research and Consulting Center, Pfizer
References
Bauer P, Bretz F, Dragalin V, König F, Wassmer G. (2015). 25 years of confirmatory adaptive designs: opportunities and pitfalls. Statistics in Medicine. Available at http://onlinelibrary.wiley.com/doi/10.1002/sim.6472/pdf.
Chuang-Stein C, Kirby S. (2014). The Shrinking or disappearing observed treatment effect. Pharmaceutical Statistics; 13:277-280.
Cummings JL, Morstorf T, and Zhong K. (2014). Alzheimer’s disease drug-development pipeline: few candidates, frequent failures. Alzheimer’s Research & Therapy; 6:37. Available at http://alzres.com/content/6/4/37 (accessed January 20 2015).
DiMasi J. (2014). Cost to Develop a Drug More Than Doubles to $2.56 Billion. Available at http://www.bloomberg.com/news/2014-11-18/cost-to-develop-a-drug-more-than-doubles-to-2-56-billion.html (accessed May 4 2015)
FDA Guidance to Industry. Diabetes Mellitus — Evaluating Cardiovascular Risk in New Antidiabetic Therapies to Treat Type 2 Diabetes (2008). Available at: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm071627.pdf (accessed January 21, 2015).
Gulfo JV. (2015). FDA 2014 approvals – the message behind the numbers. The Hill, January 8. Available at http://thehill.com/blogs/congress-blog/healthcare/228803-fda-2014-approvals-the-message-behind-the-numbers (accessed January 20 2015).
He W, Pinheiro J, Kuznetsova OM. (2014). Practical Considerations for Adaptive Trial Design and Implementation. Springer.
Jennison C, Turnbull BW. (2000). Group Sequential Methods with Applications to Clinical Trials. Chapman and Hall/CRC, Boca Raton, FL.
Lee SJ, Zelen M. (2000). Clinical trials and sample size consideration: another perspective. Statistical Science; 15(2):95–110.
Pereira TV, Horwitz RI, Ioannidis JPA. (2012). Empirical evaluation of very large treatment effects ofmedical intervention. Journal of the American Medical Association; 308(16):1676–1684.