Efficacy vs Effectiveness

I know I’ve written about efficacy, student success per dollar, and related topics before, but my mind continues to be drawn back to the issue. I can’t adequately express how important I believe this conversation to be.

Consider the following fictitious scenario:

A pharmaceutical company is developing a drug to cure a rare form cancer. During the clinical trials the drug eliminates cancer in 35% of the patients in the treatment group. The drug is hailed as a breakthrough, promising a second chance to the million people worldwide with the cancer. The company takes the drug to market as a pill to be taken once daily for six months, with a six-month supply costing $100,000. Individuals and governments accuse the company of price gouging and insurance companies refuse to cover the exorbitant cost of the drug, but the company holds firm on the price. Of the million people who need the drug, 1700 are able to purchase it during the first year of availability. 35% of the 1700 purchasers – 595 people – beat their cancer.

The question we are interested in is this – how effective is this new drug? In “the lab” the drug cured cancer in 35% of patients. In “the real world” the drug cured less than one tenth of one percent of patients. So how effective is the drug?

The confusion is partly due to a lack of specificity in our vocabulary. The FDA uses more specific language:

Efficacy refers to whether a drug demonstrates a health benefit over a placebo or other intervention when tested in an ideal situation, such as a tightly controlled clinical trial. Effectiveness describes how the drug works in a real-world situation. Effectiveness is often lower than efficacy because of interactions with other medications or health conditions of the patient, sufficient dose or duration of use not prescribed by the physician or followed by the patient, or use for an off-label condition that had not been tested. (How FDA Approves Drugs and Regulates Their Safety and Effectiveness, Congressional Research Service, p. 4. h/t wikipedia)

Our typical conversations about the efficacy of educational materials completely miss this critical distinction. Why does the distinction matter? Just as there are many sick people experiencing an “insufficient dose or duration of use” because they can’t afford their medicine, there are many students who experience an “insufficient dose or duration of use” of educational materials because they can’t afford them. When students who can’t afford their textbooks have to borrow them from friends or check them out from the library, they’re likely receiving an insufficient dose or duration of use. When students without friends in class or time to get to the library try to get by without using textbooks at all, they’re receiving no dose whatsoever.

When major publishers talk about the efficacy of their educational products, I expect they mean efficacy in the sense described in the Congressional Research Service report above. Something along the lines of “What do student success metrics look like in a tightly controlled, ideal circumstance where every student in a class is using our product?” The problem with talking about efficacy – in the FDA sense – is that it completely misses the fact that so many students go without access to textbooks, online homework systems, and other educational materials because of their cost:

More than half (64%) [of students] reported not having purchased the required textbook because of the high cost, and almost one-fourth reported doing without frequently (23%). Academically, 45% reported not registering, 49% took fewer courses, 27% dropped a course, and 21% withdrew from a course [because of the high cost of textbooks]. (Florida Student Textbook Survey, p. 4.)

If we care about improving grades, enrollment intensity, graduation rates, and other student success metrics, we don’t need educational materials whose efficacy has been demonstrated in the lab, we need educational materials whose effectiveness has been demonstrated in the world. Both the medicine you can’t afford and the textbook you can’t afford are perfectly ineffective for you.

Returning to our fictitious cancer cure above, it becomes clear that two kinds of progress need to be made in future versions of the drug. First, chemical and other innovations are necessary to increase the drug’s efficacy in the lab. Second, manufacturing and other innovations are necessary to improve the drug’s affordability, and thereby its effectiveness in the world (ignoring price elasticity considerations). Pause here for a moment to ponder this question: which would have the largest practical significance in terms of curing cancer – improving the drug’s efficacy in the lab by 50% or decreasing the cost of the drug to consumers by 50%?

In an ideal world we would pull both these levers – improving both efficacy and affordability – at once, because the delta in real world effectiveness of pulling both levers at once is multiplicative.

This is exactly what we’re trying to do when we aggregate, align, and use OER in accordance with principles of effective instructional design – increase both the efficacy and affordability of educational materials at the same time. In the case of educational materials, we might even say:

effectiveness = efficacy x affordability

With apologies for my own lack of terminological precision in conversations past, when we talk about educational resources we desperately need to focus on their effectiveness. All of our discussions with vendors, peers, and others that deal with educational resources should come back to this point: the single most important characteristic of an educational resource is its effectiveness, and effectiveness is a function of both efficacy and affordability.

Comments are closed.