Efficacy vs Effectiveness

I know I’ve written about efficacy, student success per dollar, and related topics before, but my mind continues to be drawn back to the issue. I can’t adequately express how important I believe this conversation to be.

Consider the following fictitious scenario:

A pharmaceutical company is developing a drug to cure a rare form cancer. During the clinical trials the drug eliminates cancer in 35% of the patients in the treatment group. The drug is hailed as a breakthrough, promising a second chance to the million people worldwide with the cancer. The company takes the drug to market as a pill to be taken once daily for six months, with a six-month supply costing $100,000. Individuals and governments accuse the company of price gouging and insurance companies refuse to cover the exorbitant cost of the drug, but the company holds firm on the price. Of the million people who need the drug, 1700 are able to purchase it during the first year of availability. 35% of the 1700 purchasers – 595 people – beat their cancer.

The question we are interested in is this – how effective is this new drug? In “the lab” the drug cured cancer in 35% of patients. In “the real world” the drug cured less than one tenth of one percent of patients. So how effective is the drug?

The confusion is partly due to a lack of specificity in our vocabulary. The FDA uses more specific language:

Efficacy refers to whether a drug demonstrates a health benefit over a placebo or other intervention when tested in an ideal situation, such as a tightly controlled clinical trial. Effectiveness describes how the drug works in a real-world situation. Effectiveness is often lower than efficacy because of interactions with other medications or health conditions of the patient, sufficient dose or duration of use not prescribed by the physician or followed by the patient, or use for an off-label condition that had not been tested. (How FDA Approves Drugs and Regulates Their Safety and Effectiveness, Congressional Research Service, p. 4. h/t wikipedia)

Our typical conversations about the efficacy of educational materials completely miss this critical distinction. Why does the distinction matter? Just as there are many sick people experiencing an “insufficient dose or duration of use” because they can’t afford their medicine, there are many students who experience an “insufficient dose or duration of use” of educational materials because they can’t afford them. When students who can’t afford their textbooks have to borrow them from friends or check them out from the library, they’re likely receiving an insufficient dose or duration of use. When students without friends in class or time to get to the library try to get by without using textbooks at all, they’re receiving no dose whatsoever.

When major publishers talk about the efficacy of their educational products, I expect they mean efficacy in the sense described in the Congressional Research Service report above. Something along the lines of “What do student success metrics look like in a tightly controlled, ideal circumstance where every student in a class is using our product?” The problem with talking about efficacy – in the FDA sense – is that it completely misses the fact that so many students go without access to textbooks, online homework systems, and other educational materials because of their cost:

More than half (64%) [of students] reported not having purchased the required textbook because of the high cost, and almost one-fourth reported doing without frequently (23%). Academically, 45% reported not registering, 49% took fewer courses, 27% dropped a course, and 21% withdrew from a course [because of the high cost of textbooks]. (Florida Student Textbook Survey, p. 4.)

If we care about improving grades, enrollment intensity, graduation rates, and other student success metrics, we don’t need educational materials whose efficacy has been demonstrated in the lab, we need educational materials whose effectiveness has been demonstrated in the world. Both the medicine you can’t afford and the textbook you can’t afford are perfectly ineffective for you.

Returning to our fictitious cancer cure above, it becomes clear that two kinds of progress need to be made in future versions of the drug. First, chemical and other innovations are necessary to increase the drug’s efficacy in the lab. Second, manufacturing and other innovations are necessary to improve the drug’s affordability, and thereby its effectiveness in the world (ignoring price elasticity considerations). Pause here for a moment to ponder this question: which would have the largest practical significance in terms of curing cancer – improving the drug’s efficacy in the lab by 50% or decreasing the cost of the drug to consumers by 50%?

In an ideal world we would pull both these levers – improving both efficacy and affordability – at once, because the delta in real world effectiveness of pulling both levers at once is multiplicative.

This is exactly what we’re trying to do when we aggregate, align, and use OER in accordance with principles of effective instructional design – increase both the efficacy and affordability of educational materials at the same time. In the case of educational materials, we might even say:

effectiveness = efficacy x affordability

With apologies for my own lack of terminological precision in conversations past, when we talk about educational resources we desperately need to focus on their effectiveness. All of our discussions with vendors, peers, and others that deal with educational resources should come back to this point: the single most important characteristic of an educational resource is its effectiveness, and effectiveness is a function of both efficacy and affordability.


For almost three years Lumen Learning has been helping faculty, departments, and entire degree programs adopt OER in place of expensive commercial textbooks. In addition to saving students enormous amounts of money we’ve helped improve the effectiveness of courses we’ve supported, as we’re demonstrating in publications in peer-reviewed journals co-authored both with faculty from our partner schools and other researchers. We’re making great friendships along the way. It’s been absolutely amazing.

Last year we received one of seven grants from a Bill and Melinda Gates Foundation competition to create next generation personalized courseware. We’ve spent the last year working with something like 80 faculty from a dozen colleges across the country co-designing and co-creating three new sets of “courseware” – cohesive, coherent collections of tools and OER (including some great new simulations, whose creation was led by Clark Aldrich, and newly CC licensed video from the BBC) that can completely replace traditional textbooks and other commercial digital products.

As part of this work we’ve been pushing very hard on what “personalized” means, and working with faculty and students to find the most humane, ethical, productive, and effective way to implement “personalization.” A typical high-level approach to personalization might include:

  • building up an internal model of what a student knows and can do,
  • algorithmically interrogating that model, and
  • providing the learner with a unique set of learning experiences based on the system’s analysis of the student model

Our thinking about personalization started here. But as we spoke to faculty and students, and pondered what we heard from them and what we have read in the literature, we began to see several problems with this approach. One in particular stood out:

There is no active role for the learner in this “personalized” experience. These systems reduce all the richness and complexity of deciding what a learner should be doing to – sometimes literally – a “Next” button. As these systems painstakingly work to learn how each student learns, the individual students lose out on the opportunity to learn this for themselves. Continued use of a system like this seems likely to create dependency in learners, as they stop stretching their metacognitive muscles and defer all decisions about what, when, and how long to study to The Machine. This might be good for creating vendor lock-in, but is probably terrible for facilitating lifelong learning. We felt like there had to be a better way. For the last year we’ve been working closely with faculty and students to develop an approach that – if you’ll pardon the play on words – puts the person back in personalization. Or, more correctly, the people.

It’s About People

Our approach still involves building up a model of what the student knows, but rather than presenting that model to a system to make decisions on the learner’s behalf, we present a view of the model directly to students and ask them to reflect on where they are and make decisions for themselves using that information. As part of our assessment strategy, which includes a good mix of human graded and machine-graded assessments, students are asked to rate their level of confidence in each of their answers on machine-graded formative and summative assessments.


This confidence information is aggregated and provided to the learner as an explicit, externalized view of their own model of their learning. The system’s model is updated with a combination of confidence level,  right / wrong, and time-to-answer information. Allowing students to compare the system model of where they are to their own internal model of where they are creates a powerful opportunity for reflection and introspection. 

We believe very strongly in this “machine provides recommendations, people make decisions” paradigm. Chances are you do, too. Have you ever used the “I’m Feeling Lucky” button on the Google homepage?



If you haven’t, here’s how it works. You type in your search query, push the I’m Feeling Lucky button, and – instead of showing you any search results – Google sends you directly to the page it thinks best fulfills your search. Super efficient, right? It cuts down on all the extra time of digging through search results, it compensates for your lack of digital literacy and skill at web searching, etc. I mean, this is Google’s search algorithm we’re talking about, created by an army of PhDs. Of course you’ll trust it to know what you’re looking for better than you trust yourself to find it.

Except you don’t. Very few people do –  fewer than 1% of Google searches use the button. And that’s terrific. We want people developing the range of digital literacies needed to search the web critically and intelligently. We suspect – and will be validating this soon – that the decisions learners make early on based on their inspection of these model data will be “suboptimal.” However, with the right support and coaching they will get better and better at monitoring and directing their own learning, until the person to whom it matters most can effectively personalize things for themselves.  

Speaking of support and coaching, we also provide a view of the student model to faculty and provide them with custom tools (and even a range of editable message templates written from varying personalities) for reaching out to students in order to engage them in good old-fashioned conversations about why they’re struggling with the course. We’ve taken this approach specifically because we believe that the future of education should have much more instructor – student interaction than the typical education experience today does, not far less. Students and faculty should be engaged in more relationships of care, encouragement, and inspiration in the future, and not relegated to taking direction from a passionless algorithm.

A Milestone

This week marks a significant milestone for Lumen Learning, as the first groups of students began using the pilot versions of this courseware on Monday. Thousands more will use it for fall semester as classes start around the country. This term we’ll learn more about what’s working and not working by talking to students, talking to faculty, and digging into the data. We’ll have an even more humane, ethical, productive, and effective version of the courseware when we come out of the pilot in Spring term. And an even better version for next Fall. (We’re really big on continuous improvement.)

This stuff is so fun. There’s nothing quite like working with and learning from dozens of smart people with a wide variety of on the ground, in the trenches experience on the teaching and learning side, and being able to bring the results of educational research and the capabilities of technology into that partnership. You never end up making exactly what you planned, but you always end up making something better.


MHA Convocation Address, June 2015

Back in June I had the great privilege of speaking at the Mountain Heights Academy Convocation Ceremony. This is largely the talk I to gave, except a few paragraphs at the end I skipped over as I ran out of time. I’ve been speaking about this theme more and more recently, starting with my 2013 AECTx talk You Have Superpowers. I can’t talk about it often enough.


MHA Convocation Address
Abravanel Hall
June 3, 2015

I’m extraordinarily humbled and grateful both to be affiliated with Mountain Heights Academy and to have the opportunity to speak with you today on the occasion of your graduation from high school. Today represents the culmination of over a decade of effort on your part, and it’s quite a momentous event. Thank you for allowing me to be part of it.

I want to begin with a question. What do the following movies have in common?

  • Harry Potter movies
  • The Matrix movies
  • Star Wars movies
  • The Lord of the Rings movies
  • The Lego Movie

Take a moment to discuss with your neighbor. (pause)

These movies have many things in common. For example, they each have amazing special effects. They each have great music. They each enjoyed incredible financial success at the box office. But none of that is what I want to talk about today.

Each of these stories – and in many cases these are arguably the stories that shaped an entire generation of people – centers around a character who is a bit of a loser when we first meet them. Neither Harry, Neo, Luke, Emmett, nor Frodo are what we would call successful or popular at the beginning of their stories. They’re not even normal. Each of them is a misfit.

Until they’re not. Early in each of these stories the main character learns two life-changing pieces of information. First, they have superpowers they didn’t know about. Harry’s a wizard. Luke has the force. Frodo is inexplicably immune to the corruption of the Ring. And second, the world or universe as we know it is in imminent danger, and armed with their newfound superpowers each of these characters is in a unique position to save humanity (and other races). Neo is The One. Emmett is The Special. Each of these characters is called from their humdrum life to pursue a higher purpose, and each is armed with higher powers that will enable them – just barely – to accomplish those purposes.

There are thousands of stories that follow this pattern. Why is that? What is it about these stories that draws us each to them? Take a moment and discuss with your neighbor. (pause)

I believe we’re drawn to these stories because each and every one of us desperately wants our own Hagrid Day. The day when some one kicks down the door and says to you, “Yer a wizard, Harry, and a thumpin’ good one I’d wager, once yer trained up a bit, o’ course.” Only later does Harry learn that he’s not just a wizard; he’s the wizard. The wizard that will be responsible for ridding the world of its greatest source of evil. As Uncle Ben told another one-time slacker named Peter Parker, “With great power comes great responsibility.”

The reason we’re each drawn to the stories where the bumbling underdog is endowed with great power and called to save the world is that, deep down, we all know that this is the true storyline of our life. We can run away from that potential – from that opportunity – from that responsibility. But that doesn’t make it any less real. If you try hard enough you actually can outrun your destiny to help save the world, like Tom Bombadil in the Lord of the Rings books. Or you can avoid the responsibility for a time, but realize your mistake and take up the sword that was broken, as Aragorn did.

You may be thinking, “David this whole talk is completely pointless because I don’t have a wand, a lightsaber, or the Piece of Resistance.” I can’t stop bullets in mid-air by raising my hand, or fly. And I’ll admit that’s true. But blockbuster movie superpowers aren’t the only superpowers.

Another example of superpowers you may not be familiar with are in the Alcatraz books by Brandon Sanderson. Alcatraz Smedry has an unlikely superpower – he breaks things. His “Smedry talent,” as it’s called, sounds like a curse, but it’s not. Alcatraz can break locks, for example. He can break weapons and other machines that threaten him and his friends. Grandpa Smedry’s superpower is arriving late to things. His power has saved his life many times, allowing him to arrive late to bullets, which always miss him. And his blood can arrive late to wounds so that he doesn’t bleed to death. In his own words,”I’ve been arriving late to my own death for years now.” Alcatraz’s cousin Sing’s talent is the ability to trip and fall to the ground. Quentin, another cousin, has the talent to speak gibberish.

Now we’re finally getting somewhere. Migrating into the real world, look at the examples of Gandhi and Martin Luther King, Jr. What was their superpower? Nonviolence. Their gift was the ability to not hit things. The ability to not shoot things. The ability to no blow things up. It almost sounds like a Smedry talent, doesn’t it? And yet look at what they did – how profoundly they changed the world.

Today is your Hagrid Day. I’m here to tell you that you do have superpowers. What is your superpower? What is your Smedry talent? Take a silent moment and think about this question. (pause)

I bet you know what your superpower is, but you’re not willing to admit it to yourself. As Marianne Williams wrote, “Our deepest fear is not that we are inadequate. Our deepest fear is that we are powerful beyond measure.”

Now why would you be afraid of that? Perhaps because of Uncle Ben’s Rule, “With great power comes great responsibility.” As long as you pretend you don’t have superpowers, you can pretend you have no responsibility to save the world. Well, if that’s the case I’ve got bad news for you – you do have superpowers, and it is your responsibility to save the world.

You will be correct to observe that neither Lord Voldemort, nor the Matrix, nor Darth Vader, nor the Ring, nor even the Kragle are actual threats to the real world. What is it, then, that the world needs saving from? A brief list includes poverty, ignorance, hunger, disease, and violence.

These issues sound huge and unsolvable. When we’re feeling optimistic we sometimes call them “Grand Challenges.” When we’ve recently lost a battle we sometimes call them “Wicked Problems.” You can Google either of those terms and find that lots has been written about them.

But I have two important pieces of good news for you, too. The first is that you don’t have to solve all the world’s problems. And this is critically important to understand. You only have to save the world from the problem that matches your superpower. This matching of your superpower with a critical need in the world is what Jeff Thompson and Stuart Bunderson refer to as a “Calling.” In their book by that same title, they write:

Let’s imagine that a young girl says to herself, “I’m an animal person. I’ve been given these special abilities to interact with animals and to love them in ways that other people don’t. I’m hardwired for this kind of work.” As a child, she was perhaps satisfied to play with her pets just to have fun.

But as she began to pursue her passion, to read more about animals or perhaps to associate with others who share her devotion, she made another key realization. “There are animals in need! There are species that are dying out because their habitats are threatened. Some endangered species can only survive in human captivity. That’s got to be very hard on the animals. What can we humans do to help them survive and thrive?”

So now this young girl has discovered that there is some crying need in the world that is related to her particular gifts. The moment that she actually discovers her calling as a zookeeper might sound like this: “Wait a minute, if I’m hardwired to help animals, and there are animals in need, then it is up to me to be the one to help them! Who else but someone like me is going to be able to meet this need? Caring for animals isn’t just about having fun anymore. I have an obligation to help these captive animals because of who I am!” And thus, a calling as a zookeeper is born.

So the first piece of good news is that you don’t have to do it all – you are only called to do the part that you are uniquely capable of doing.

The second piece of good news is that you don’t have to do it alone – in fact, you can’t do it alone. Harry finds Ron and Hermione and others. Neo finds Morpheus and Trinity and others. Luke finds Han Solo and Princess Leia and others. Emmett finds Wyldstyle and Vitruvius and others. Frodo finds an entire Fellowship. In every case, each character brings his and her unique superpowers into a team that works together to save the world. You will have the same experience, and let me tell you – it’s awesome.

Mountain Heights Academy follows this very pattern. MHA is a huge Fellowship with a wide range of people with incredible gifts in caring, in supporting, in sharing, in loving, in administering effectively, in balancing budgets, and in believing that every student can succeed. These and other Smedry talents combine in our school family to save the world, in very concrete ways, every year.

Your superpowers will be only barely enough to help you accomplish the great work that lies before you (the movie of your life wouldn’t be exciting otherwise!). But take a few inspiring thoughts into account as you consider the work ahead of you. The first is attributed to Gandhi:

First, they ignore you.
Then, they laugh at you.
Then, they fight you.
Then, you win.

Changing the world is slow going. For a long time it feels like you’re getting nowhere. Then, when someone does notice, it’s often to ridicule you and tell you how hopeless your cause is. Then, others notice and begin actively working against the change you’re trying to create in the world. Without the proper perspective, it can feel like you’re just trading one obstacle for another. But you aren’t you’re progressing steadily toward the path to winning. As you pass these signposts you’ll know you’re almost there. As Winston Churchill said, “Success consists of going from failure to failure without loss of enthusiasm.”

And next a word from Willy Wonka. In the original (and best) version of the film, Gene Wilder sings:

If you want to view paradise,
simply look around and view it.
Anything you want to – do it.
Want to change the world?
There’s nothing to it.

Wonka is perhaps our greatest literary example of the childlike belief that you can change the world. Never lose that hope – never give up on that faith. No matter what life throws at you, no matter how difficult it is to walk the path that Gandhi described, never give up.

Graduates, family and friends, Mountain Heights Academy faculty and staff, you absolutely do have superpowers. And you absolutely do have a solemn responsibility to save the world. There will be no revolutionary breakthroughs in renewable energy if you don’t get out there and create them. There will be no end of corruption in Congress if you don’t go out there and end it. There will be children and others struggling with poverty, ignorance, hunger, disease, and violence until you personally go out there and end them. We need you. Desperately.