An analysis of 211 biotech companies compared the success rates of NCEs (new chemical entities) and biologicals over six years and concluded that the “results confirm the notion that biologicals have higher success rates from entry into man to approval than NCEs”. With overall success rates of 8.9% and 16.8% for NCEs and biologicals, there is certainly a difference, but is it significant? Could it all be down to chance?

The data (from Avance’s website, with correct rounding and eliminating a couple of typing errors) are as follows.

Table 1: Success rates by phase for NCEs in biotech companies.

	Failures	Successes	Total	Rate
Phase I	46	236	282	83.7%
Phase II	80	89	169	52.7%
Phase III	34	19	53	35.8%
Approval	7	9	16	56.3%
Overall				8.9%

Table 2: Success rates by phase for biologicals in biotech companies.

	Failures	Successes	Total	Rate
Phase I	53	236	289	81.7%
Phase II	58	76	134	56.7%
Phase III	19	14	33	42.4%
Approval	1	6	7	85.7%
Overall				16.8%

The success rate for each phase is the ratio of successes to the total sample of projects assessed. The “overall” success rate is the product of the three clinical phases and approval success rates. (This last calculation makes many heroic assumptions about independence and all that, but we will live with it.)

There appears to be a big difference in the overall success rates; biologicals have almost twice the chance of succeeding from first-in-man clinical trials through to approval by the regulatory authorities. But some of those sample sizes are awfully small. Could this result just be down to luck?

The easy way to test this is to build a simple Monte Carlo simulation. Assume for a moment that there is no difference, and pool the data sets together. Assume also that the pooled success rates are the “true” rates. Then pick, for NCE Phase I, 282 random samples with the “true” rate of 82.7%, and 289 samples for Biologicals Phase I. Repeat for the other phases and recalculate the overall success rates for this one example. Is the difference in the overall success rates larger than that observed in the tables above? Using the computer’s random number generator, repeat this several thousand times.

It turns out that a result at least as dramatic as that observed in the tables above occurs in 6.4% of the simulated cases. (Statisticians call this the “P-value”.) This is more than the conventional 5.0% used by most people, and so we reject the idea (statisticians would say “hypothesis”) that there is any difference between the two underlying probability distributions for NCEs and biologicals.

Sorry, you guys at Avance. With more data maybe (probably!) things will change. Until then, the jury is out.

See also:

• Chapter 20 of Sam Savage’s excellent book, The Flaw of Averages, introduced me to the idea of using Monte Carlo simulation to answer the question “did it happen by chance?” It’s a much more powerful approach than the classical statistics and hypothesis testing I learned in my maths degree.

Avance is a consulting company with a focus on the valuation of biotech and pharmaceutical opportunities.