Stuart R. Gallant, MD, PhD

Sooner or later, something goes wrong in our work lives.  In pharmaceutical manufacturing, errors or “deviations” are common events, and there is a lot at stake.  When I started working at Bayer AG, just after my graduation, there was an informal group called the “million-dollar club.”  You joined the club if you made a mistake that caused the product from a manufacturing batch to be compromised and unfit for sale.

Underlying the moniker “million-dollar club member” is a kind of macabre humor which springs from the emotional question, “Who’s to blame?”

Another thing that was remarkable to me as I started working in biotechnology is how young the candidates for the million-dollar club are—in pharmaceutical manufacturing, decisions are often made by young people only a few years out of school.  For them, the stress of the unexpected is higher because they have a small personal database of failure—it is harder for them to tell themselves, “It’s going to be okay,” and there may be a tendency to catastrophize as a result.

Today’s post is about what to do when something goes wrong—the focus is pharmaceuticals, but the methods are widely applicable in work and in life.

Recognize

Even before you get started there needs to be a plan.  When I am preparing for pharmaceutical manufacturing of a new active ingredient or drug product, I write a process description which includes everything I know about the manufacturing process. This becomes the source document for reports, regulatory filings, and manufacturing procedures.  The process description is in plain English, chronologically organized, and populated with as many photos and diagrams as possible.  The headings for a drug product process description include:

• Product Description:  English description of product, formulation with amounts of all chemical components, as well as source manufacturer, and drug product container.
• Compounding Table:  Amounts of chemicals required for a batch.
• Process Description:  English language description of the step-by-step manufacturing process.

Preparing clear, well-organized documents allows the team to anticipate problems, manage risk, and recognize when things are not going correctly.

Respond to Deviation

In pharmaceutical manufacturing, any occurrence which is outside of the anticipated range of input or response is a deviation.  Response to deviations may need to be real-time.  For instance, take the example of the most basic manufacturing operation:  dissolution of a chemical in a solvent; what does the team do if the chemical is not dissolved in the prescribed amount of time?  Options include:  stop the manufacturing operation and possibly lose the batch, extend the mixing time, increase mixing speed, increase temperature.

All of the above are valid possible responses—the point is that the manufacturing leader must make a decision as to how to respond at the moment a deviation is recognized.  That decision may be enhanced by advice from outside experts, may be handicapped by limited information, and may be pressured by constraints on available time.

Psychologically, humans under stress can have a range of reactions.  Sometimes these are adaptive (for example, the response of the crew of US Airways Flight 1549 to engineer failure—in that case, the plane was landed successfully in the Hudson River [1]).  Sometimes these are non-adaptive (for example, the actions of the operators of the nuclear plant at Chernobyl—in that case, the operators made the situation progressively worse [2]).  Factors affecting the quality of response during deviation include:

• Quality of Training:  How well do the personnel understand the operation they are performing and have they engaged in sufficient training to know what to look for and how to respond?
• Quality of Documentation:  Has a proper risk analysis been performed in advance to anticipate where failures could occur and define responses?  Is the team operating from a playbook or are they on their own?
• Stress Response:  Under stress, humans exhibit cognitive biases too numerous to list here [3].  Two worth are worth mentioning.  First, ”limited database”—many folks who work in pharmaceuticals are comparatively young and have not seen things go sideways many times—so, they can find themselves in situations which they have not experience previously, leading to poor deviation response.  Second, “how does this affect me”—sometimes people under stress focus away from the situation at hand and fixate on how the likely outcome will affect them personally—the idea that they might be blamed can cripple good decision making.

In summary, good training and documentation act to counter poor decision making due to stress.

Investigate Deviation

After a manufacturing operation is completed, deviations are “closed”—documentation is put in place to assess the magnitude of the error and prevent its recurrence.  The level of effort is proportional to the magnitude of the deviation.  For example, if someone forgot to write their initials on a particular document, a simple note is appended to the document explaining why the omission occurred and verifying that the check symbolized by the initials did (or did not) actually occur.  On the other hand, a more major deviation, for example failure of equipment to reach the correct temperature, requires a more extensive effort at closure which is documented in a closure memo.  The elements of a closure memo include:

• Description of Deviation:  What actually happened as best as can be determined?
• Action Taken:  How did the team respond in real time to the deviation?
• Product Impact Assessment:  The magnitude of the deviation on product attributes may range from mere procedural without impact on the product (a “minor deviation”) to significant without compromising the safety of the medication (a “major deviation”) to deviations which could compromise the safety of the medication (a “critical deviation”).  In general, the bias is toward the most conservative interpretation of the available data—the team needs to prove that there is “no impact.”  Nothing is assumed.
• Root Cause:  Root cause analysis starts with the proximate cause—the equipment did not reach the correct temperature—then, it looks deeper to find the real causes.  The preference is to look at how the system is engineered and run, rather than simply blaming an operator for forgetting or misjudging.  Ultimately, the cause may be assignable, probable, or undetermined.
• Corrective Action:  What will be done to prevent a repetition of the deviation?  Again, the preference is for systems which absolutely prevent repetition, rather than retraining the operators and hoping that the problem does not recur.

Today, a variety of computer systems are used to track deviations.  The visual output of these systems can be unappealing and difficult to read.  One solution is to draft clearly written memo which can be attached to the computer record.

Root Cause Analysis

Root cause analysis is widespread across industries (for example, aircraft and spacecraft crash investigation, military after action report, and medical error investigation).  An excellent dramatization of the process is 2013 film The Challenger Disaster.  In the film, William Hurt plays the physicist Richard Feynman.  He displays two very useful tools of root cause analysis:

• Interviewing the Key People:  Today, many of us work remotely.  Even, if you are at the same site, you maybe be separated physically and divided by badge access authorizations.  Nevertheless, it is tremendously useful to go to the place the error occurred and talk with the people who were at the event.  Reading their emails or hearing second hand is not as probative.
• Trending the Data:  The space shuttle Challenger crashed on a cold day—plotting temperature versus space shuttle mission makes that obvious.  Humans are very good at spotting deviation if the information is displayed in an effective manner [4].

Attaching figures and graphs may be helpful in making the investigation report clear.  And, listing relevant SOPs, plans, batch records, and references clarifies the process knowledge at the time of the event, as well as providing a tabulation of documents that may need to be revised in light of the investigation.

There are numerous tools for root cause analysis.  Two to consider are:

• Failure Mode and Effects Analysis (FMEA):  FMEA is very useful before an event occurs to anticipate and rank risks.  FMEA is an organized way of categorizing risk into the types of risk that need to be remediated through better engineering, monitoring, and training versus risk that the operation will “live with.”  Taking a look at the FMEA during the investigation can help in understanding whether the event was anticipated or not.  The strategy for ranking risk in FMEA is shown below:

• Ishikawa Diagram:  So useful in performing root cause analysis, the Ishikawa diagram (“fishbone” diagram) is often used interchangeably with the term “root cause analysis.”  It allows the many possible causes to be enumerated without bias.

The Role of Emotions

When something goes wrong, many emotions can come to the surface:

• Embarrassment:  Manufacturing organizations track failures.  The management may need to report this event in the future as a failure—for example, a sterility failure—that will make auditors view the organization as unreliable.
• Fear:  Individual personnel may fear harm to their reputation, loss of financial bonus, or even in the extreme job loss.
• Regret:  Some personnel may feel that they “saw it coming,” but failed to take action in time.  They may blame themselves—whether the blame is merited or not.

Leaders need to provide clear messages when deviations occur and when investigations are conducted:

• Lead from a Position of Emotional Integrity:  Do not use words like “blame” and “fault.”  Make it clear that the investigation is about objectively reviewing the events in a scientific way and taking corrective actions to prevent the problem from happening again.  As a senior person, your words carry extra weight.  Think about who is listening when you speak.
• Consider Finding a Sounding Board:  Do you have someone you can talk to confidentially?  Preferably, this is a person at a similar level of experience to you, or perhaps a higher level—someone not involved in the investigation.  Talking through possible scenarios can help you release your emotions and perform better on the investigation team.
• Trust the Process:  Often there is a lot of money on the table.  If the batch is compromised, then resources and time will be required to replace the batch.  It may be tempting to push the investigation toward a conclusion of “no impact.”  It is important to remember that an inaccurate conclusion of the investigation has two risks:  first, the problem is more likely to recur and second, there may be follow on effects.  For example, if defective product is released and later determined to be defective, then an expensive product recall, as well as legal liability, are now involved.
• Check In:  Frequently, investigations of complex deviations involve short periods of activity in the investigation team, followed by long periods waiting for additional testing to be completed.  When the team is together, agree on what the plan is, what next steps will be based on how any testing goes, and what the check-in interval will be—so that the investigation does not stall.

Conclusions

Deviation investigation can be disheartening because the game is over—often there is nothing that can be done to repair the damage.  But, deviation investigation is also an opportunity to learn.  Frequently, participation in investigations signals a point at which a leader or future leader transitions from the role of line operator or individual producer toward the role of leader.  Leaders think systematically about problems and work actively to prevent problems from occurring, so incident investigation is a type of leadership training.

[1] See the 2016 film Sully: Miracle on the Hudson.

[2] See the 2019 miniseries Chernobyl.

[3] Baron, J.  Thinking and Deciding, Cambridge University Press (2008).

[4] Tufte, E.R.  Visual Explanations, Graphics Press (1997).

[5] CMC Biotech Working Group.  A-Mab:  a Case Study in Bioprocess Development (2009).

Disclaimer:  PharmaTopo^TM provides commentary on topics related to drugs.  The content on this website does not constitute technical, medical, legal, or financial advice.  Consult an appropriately skilled professional, such as an engineer, doctor, lawyer, or investment counselor, prior to undertaking any action related to the topics discussed on PharmaTopo.com.