Your CDMO Quality Dashboard Is Green. What's It Hiding?
By Jeffrey S. Buguliskis, PhD, Deputy Chief Editor, Outsourced Pharma
Consider a hypothetical investigation due on day 30. Its deadline moves to day 60, then day 90, and the report is signed on day 88. When measured against the latest deadline, a dashboard records an on-time closure. The investigation still took 58 days longer than planned.
In governance meetings between a drug sponsor and its CDMO, that arithmetic hides behind emerald squares. Illustrative figures look reassuring: deviation closures at 98 percent, corrective and preventive actions (CAPAs) are on track, and batch defects are near zero. To a lean team racing toward a milestone, those green squares suggest quiet control. Yet, as Goodhart’s law warns, once a measure becomes a target, it stops being a good measure.
That vulnerability took center stage during a recent Outsourced Pharma Live discussion titled “Your CDMO Quality Oversight Is Not Working.” Panelists Andrea Goddard, Chief Technical Officer at CatalYm, and Greg Whitehead, Executive Vice President and Chief Technical and Quality Officer at Editas Medicine, unpacked how routine metrics obscure unresolved risks. In a follow-up response, Andrew Mackintosh, Chief Operating Officer at Douglas Pharmaceuticals (which operates event sponsor Douglas CDMO), agreed, stating that “a closure rate on its own tells you very little.”
Keep the Original Clock
Quality agreements often set a target window for closing a deviation, such as 30 days, though timelines do vary. And during the broadcast, Goddard used a hypothetical reporting example to show how a headline figure can create false confidence when measured against revised deadlines.
“What is your deviation on-time closure rate? Well, you can say it’s 100 percent, but I’ve extended 90 percent of my deviations, and so I’m closing on time to the extended date.”
Extending a deadline is reasonable when a complex excursion needs extra testing and immediate risk is controlled. Routine postponements, however, can signal an under-resourced quality unit. Whitehead noted that habitual deadline resets and months-long batch disposition reviews point to deeper strain.
“The metrics are all green. The goal is not to be green. The goal is to drive these to closure.”
Follow the Problem After Closure
A fast closure rate also deceives if an investigation stops at the surface. Both panelists singled out repetitive “human error” findings. “Show me your fishbone, your 6M that you’ve done. How did you eliminate those root causes?” Goddard asked. “If in their root cause analysis, they are often coming up with human error, then I’m going to really question whether their investigation and deviation process is robust.” In one active investigation where her team lacked confidence in a CDMO’s inquiry, Goddard’s team prepared its own analysis: “Hey, we did a 6M, here’s our 6M, please look at all these potential root causes.”
Prescribing generic retraining when an operator misreads a batch record is, in Whitehead’s words, “not truly a corrective action or preventive action. It really is a quick fix.” It closes a ticket without asking whether a confusing protocol or layout invited the mistake. When the deviation resurfaces, the CAPA merely fed a dashboard.
The FDA’s ICH Q10 guidance on pharmaceutical quality systems calls for evaluating CAPA effectiveness and using investigations to deepen process understanding. “Closing an action is not the same as reducing the risk,” Mackintosh notes, pairing investigation health with recurrence tracking because repeated events indicate that teams are treating symptoms. Even then, an investigation often closes before enough subsequent runs have occurred to prove the fix worked. For a therapy made only a few times a year, several quiet months offer little opportunity for recurrence, so sponsors must weigh how many runs or tests have occurred since the CAPA.
Leading indicators and process variability offer a second early-warning beat while targets stay green. Whitehead watches for delayed environmental monitoring reports. Goddard tracks post-release deviations: “If you’ve already released a batch and you’re routinely getting post-release deviations, to me, that’s a really clear sign that something’s going on there.” Mackintosh monitors rising rework, shifting yields, and atypical lab results. As the FDA’s process validation guidance notes, ongoing statistical review detects unintended variability without overreacting to isolated fluctuations.
Look Beyond Your Batches
A sponsor’s batch history captures only a slice of the facility supporting its medicine. In an illustrative scenario where three clinical runs a year proceed smoothly, a product dashboard stays green even as a shared lyophilizer, water loop, or lab accumulates excursions on other lines. Under 21 CFR 211.192, any unexplained discrepancy or batch failure must be thoroughly investigated, and the investigation “shall extend to other batches of the same drug product and other drug products that may have been associated with the specific failure or discrepancy.” Yet extracting that broader view is difficult. “They may give you deviation rate for your product,” Goddard notes. “But what you really want to understand is what is their overall deviation rate for their facility?”
Whitehead agreed, calling facility-wide rates “one of the hardest metrics to get.” Confidentiality is the usual barrier, yet Mackintosh notes that sponsors do not need another client’s records to evaluate the building. By agreeing on transparency upfront, a CDMO can share anonymized trends covering shared equipment, utilities, lab events, CAPAs, and inspections.
Context cuts both ways: a site-wide average can blur unrelated operations, while a problem elsewhere in the plant does not threaten every client. The CDMO must explain which systems the sponsor’s product touches. “Good oversight therefore requires a conversation, not simply the exchange of a dashboard,” Mackintosh says.
Ask How the Numbers Were Produced
Every dashboard depends on people reporting problems. Mackintosh cautions that a site reporting more deviations can have a healthier culture than one reporting very few. Sponsors should cross-check a low or falling count against rejects, rework, lab investigations, complaints, and audits, since a decline during a production slowdown means something very different from a decline during steady output.
Site visits and governance meetings reveal what spreadsheets omit. Speaking directly with operators and technical staff on the shop floor rather than relying on project managers shows “what’s actually happening,” Goddard said. She watches how staff respond to anomalies and urges sponsors to bring curiosity to difficult updates: “If you’re overly reactive, then that’s not going to create transparency.” When technical teams disagree over product risk, Whitehead advises moving the issue promptly to a joint steering committee: “Escalation is not a bad word.”
At your next review, pick one recently closed investigation that needed an extension. Follow it from its original due date through root-cause analysis and CAPA, then examine the manufacturing or lab data collected afterward to see what those runs prove and when enough evidence will exist to judge the fix.
“Transparency is often a better indicator of maturity than a dashboard that is always green,” Mackintosh says. “By the time a KPI turns red, the underlying trend has often been visible for some time.”