From The Editor | October 6, 2015

The Paradox In Measuring Your CMO


By Louis Garguilo, Chief Editor, Outsourced Pharma

The Paradox In Measuring Your CMO

Something Denise McDade of Capricor Therapeutics said during a session at Outsourced Pharma West in San Diego stuck in my mind: “Metrics are fundamentally important in measuring and managing any relationship, but there’s a paradox when it comes to outsourcing.”

That paradox stems from an awareness that while there is a rising criticality to establishing a tangible set of metrics for your outsourcing partners, those increasingly important metrics still don’t offer as complete a picture as many would believe. Even with more sophisticated performance measurements for items like cycle time, throughput, deviations per batch, time to resolve deviations, on-time delivery, surpassing quality or quantity goals … these results may lead a sponsor astray at times.

According to McDade, VP of Quality, and whose career has spanned various positions at Genentech, Novartis, Amgen and other biopharma before joining Capricor, there are equally critical elements – such as trust and the spirit of the supplier relationship – that defy even today’s finer metrics. Here’s a closer look at what McDade – and others – are saying.

Do You Know Your CMO Even If You Measure?

McDade made her initial comments as a panelist at on OPW session titled, “How do you know you're on target if you don't measure your CRO/CMO?” Other panelists included Michele Johnson, Director, Supplier Relationship Management, Sandoz; Sylvie Sakata, Sr. Director, External Research Solutions West, Pfizer; and Nicholas Virca, President & CEO, HedgePath Pharmaceuticals, Inc. John Newsam, CEO, Tioga Research, was the moderator.

No panelist or member of the audience was advocating less measurement of CMOs. The pendulum has certainly swung to the side of “the more measurement the merrier,” away from a substantially less vigorous approach of even a few years ago. A separate panel at OPW dealt specifically with new FDA statements exploring the need to measure “culture” as a function of reliably deriving a quality product. This panel also included discussion on the FDA’s plan to receive quality data directly from CMOs (see here for more, and even more to come in follow-up articles at Outsourced Pharma).

But back to McDade, she has arrived at her convictions during a career that spans more than 25-year. About half that time she’s focused on the outsourced world, and the other half on the manufacturing owner-operator side.

“I’ve worked with CMOs where we had a robust and healthy management process, amble performance feedback, and a lot of measurement,” she explains. “However, there was still little trust in the supplier relationships. Ultimately, those types of relationships were not successful, or healthy in an overall business sense. It’s difficult to measure exactly what the issue was with trust, but it was crucially important for the CMOs to try and maintain – or salvage – it as a part of their relationship management efforts with us. Therefore, I’ve seen that both sides – the measurable and the more intangible components – are equally important. In my experience, you can’t be successful unless you have both.”

The 5am Call Builds Trust

At the conference, and in a follow up discussion, McDade walked me through an example that draws out her feelings.

At one point in her career, she was outsourcing the same product to a CMO in the U.K., and one in the U.S. There were similar quality agreements in place stipulating that the CMOs inform McDade, as head of quality, within 24 or 48 hours for different levels of deviation that might occur at the facilities during the manufacture of the products. In McDade’s words:

“The U.K.-based CMO had a facility contamination issue in an area of their facility that ultimately didn’t impact our product. However, I got a call at 5am my time from their QP [Qualified Person, as referred to in Europe] so I could be informed as soon as they found out. There were other times as well where the QP would pick up the phone to let me know immediately of anything that he thought I should know.

“There was also an issue at the U.S.-based CMO, this time a critical contamination in their WIFI [purified water] system. This also was not in our part of the facility, so by a strict reading of the quality agreement they were not obligated to inform us. We would like to have known, though, and only found out about it after they were inspected by the FDA and received a warning letter.

“By strict measurement, both CMOs were within the language of the contract, but only one was within the spirit of the relationship. Which one did I trust and continue to work with? You could look at the numbers on the metrics reports and they might be similar, but the quality of the relationships were totally different.”

Can You Metricize Trust?

The OPW panel agreed that the trust-factor would not diminish in importance because more and better metrics are put in place at the sponsor to measure provider performance. In fact, the same reason for an increased need for CMO metrics also bolsters the need to have the intangibles – like personal relationships – contributing positively to outsourcing outcomes. As one panelist said: “You can’t have a relationship with an organization; you have a relationship with individuals.”  While systems and processes are set up so that projects, processes and relationships are not “completely person dependent,” sponsors will always be “somewhat dependent upon the individuals on the other end.”

It’s important to have an open structure where the letter and spirit of the relationship can be transitioned as personnel changes – at either the sponsor or provider. One panelist talked about working with a supplier that had to replace project leaders a few times, but “how the transition was accomplished had a huge impact on how successful and seamless we continued to move forward.” Trust as an independent variable gets high marks for how well productivity, quality and reliability – and the relationship itself – is sustained over time and as people change. This then becomes the yardstick for the entire organization as a fully reliable partner.

McDade has had outsourced supply chains find ways to document intangibles to head off any paradox cropping up in the more standard metrics. These broader measurements became an important part of an overall supplier scorecard.

“We measured items that were subjective by applying a numeric rating to them, and then presenting these as part of our quarterly review meetings with the supplier. We actually spent a fair amount of time talking about these items with our CMOs, so they understood the importance to us,” says McDade. She adds: “We measure and measure. On paper, performance can look similar, but from a relationship standpoint performance can turn out to be completely different. For me there is always another equally important ingredient in that relationship, and that is trust.”