Stop Saying “High Quality”

The Open Business Models conversation at the Hewlett Foundation grantees meeting (#oer2015) was a lot of fun. The biggest surprises to me were the number of times the phrase “high quality” came up, and what a strong, negative reaction I had each time I heard the word.

After some reflection I think the reason the phrase gets my goat is that “high quality” sounds like it’s dealing with a core issue, while actually dodging the core issue. The phrase is sneaky and deceptive. (Now I don’t mean that the people who were using it were trying to be deceptive; they weren’t. But the phrase itself tends to blind people.) And by “core issue” I mean this – the core issue in determining the quality of any educational resource is the degree to which it supports learning. But confusingly, that’s not what people mean when they say that a textbook or other educational resource is “high quality.”

It’s very easy to demonstrate that “the degree to which it supports learning” is the only characteristic of an educational resource that matters. If an educational resource is written by experts, copyedited by professionals, reviewed by peers, laid out by graphic designers, contains beautiful imagery, and is provided in multiple formats, but fails to support learning, is it appropriate for us to call it “high quality”? No. No, no, no. A thousand times no. Despite this fact, which is intuitively obvious, when people say “high quality” they actually mean all these things (author credentials, review by faculty, copyediting, etc.) except effectiveness. In the world of textbooks and other educational materials, “high quality” describes the authoring and editorial process, and is literally unrelated to whether or not the educational resource supports learning.

In this way, saying “high quality” obscures the issue we should care about most. Instead of letting people and companies off the hook by checking boxes during the pre-publication process, we should care about whether or not a particular resource supports learning for each of our particular students. Seen this way, the true desideratum of educational materials is “effective.” I really don’t care what the pre-publication processes was like as long as my students are learning (unless the process was unethical in some way).

So please – let’s stop saying “high quality.” We don’t want “high quality” educational materials – we want “effective” educational materials. In the future, when you catch yourself saying “high quality,” stop and correct yourself. When you hear others say “high quality,” take that teachable moment to help them understand that the phrase is a ruse. If we can change this one element of the education conversation, we’ll have done something powerful.

(And don’t forget – when materials are so expensive that students can’t afford them, they are perfectly ineffective.)


The Remix Hypothesis

For several years my colleagues and I have been conducting and reviewing empirical research on the impact on student outcomes when OER are adopted in place of commercial materials. Suffice it to say the research results are highly variable. Some studies of OER adoption show essentially no change in student outcomes. Many of these studies report small positive and negative changes in outcomes that, aggregated across several courses, fail to achieve statistical significance. Some studies including larger numbers of students find small changes in students outcomes that achieve statistical significance while failing to achieve practical significance. Based on these studies, we can say that sometimes OER save students significant amounts of money while obeying the “do no harm” rule in terms of student outcomes. Achieving the same outcomes for free, or for 95% less than students were previously paying, is a solid “win” for OER.

However, there are other studies of OER adoption that show large positive changes in student outcomes. (For sake of completeness, we are unaware of any studies showing large negative impacts of OER adoption on student outcomes.) Of these studies we can say that OER save students significant amounts of money while actually improving their learning outcomes. Clearly, supporting more learning at lower cost is also a solid “win” for OER.

Seeing two solid wins for OER, and no clear losses, reassures those of us pursuing this line of work that we are on the right track. However, as this irregularity in research results persists – some showing essentially no change in student outcomes while others show large improvements in students outcomes – you have to begin asking yourself, “Self, what is going on here? Why is there such a large difference in the results of many of these studies?” For the past several months I have been pondering this question and speaking to some of the faculty who taught the courses reported in the studies. While I don’t have any research to report on the issue today, I have developed an initial working understanding of the discrepancy. Until I think of something more descriptive I’ll call it “The Remix Hypothesis.”

In it’s simplest form, The Remix Hypothesis states that changes in students outcomes occurring in conjunction with OER adoption correlate positively with faculty remixing activities. Specifically, I hypothesize relationships between (at least) three levels of remix activity by faculty who adopt OER and changes in student outcomes, based on what I’ve seen in the research to date.

Level 0 – Replace
At this level faculty engage in no remixing whatsoever. They simply adopt OER (most often an open textbook) in place of a commercial textbook and preserve other aspects of the course as they taught it previously. I hypothesize no changes in student outcomes when faculty Replace – except possibly in one special case. In the case of students who are particularly financially disadvantaged, where faculty were previously assigning very expensive textbooks, there may be a small positive effect attributable to the increased percentage of students who can access the core instructional materials of the course.

Level 1 – Realign
At this level faculty remix their open course materials. In my work to date, this has most often involved faculty stripping a course’s content down to its bare learning outcomes, and then selecting the OER from multiple sources that they feel will best support student learning of specific course learning outcomes. I hypothesize small to modest positive changes in student outcomes when faculty Realign.

Level 2 – Rethink
At this level faculty remix both course materials and pedagogy. In conjunction with the Realign activities described above, faculty create or select new learning activities and assessments – possibly inviting students to co-create and openly share them – often leveraging the unique pedagogical possibilities provided by the 5R permissions of OER. (This is what I refer to as open pedagogy.) I hypothesize modest to large positive changes in student outcomes when faculty Rethink.

Remixing and revising at the Rethink level will be significantly more effective if faculty make those decisions after gathering, analyzing, and reflecting on empirical data about their courses. Helping them ground these decisions in their own data is extremely important. I’ve personally seen several cases where faculty’s own memories about what is and isn’t working in their own classes are contradicted by data in their own gradebooks. It’s an amazing process to watch faculty struggle to understand the conflict between their intuitions and their data. We need to support faculty in this reflection process every time they teach the course – supporting ongoing, continuous improvement. Rethink is ideally an iterative process by which pedagogy and supporting materials come into increasing harmony, supporting deeper and deeper student learning.

Very roughly, we might say in terms of the 5Rs that at Level 0 faculty take advantage of their permissions to Retain, Reuse, and Redistribute educational materials. At Level 1 they add Remix. At Level 2 they add Revise, and expand from open materials into open pedagogy. Strictly speaking this characterization isn’t completely accurate, but I think it provides a good approximation to get our thinking going.

The three levels build on one another. First, a faculty member decides to Replace her textbook with OER. A logical next step a semester or three later is to Realign, making more sophisticated choices about which specific OER to use to support specific student learning outcomes. Finally, as faculty become more familiar with the benefits the 5Rs provide them as teachers, they may begin to grasp the potential benefits the 5Rs can provide to learners. This will lead them to Rethink their assignments and assessments so that they maximize the learning-related benefits of openness to their students.

As I said above, I don’t have empirical data from a specifically designed study to corroborate The Remix Hypothesis yet, but I hope to either validate or disprove it empirically in the next few years in collaboration with my awesome partners in the Open Education Group.


At this year’s SXSWEdu I gave a Future15 talk, a 15 minute talk given in a “TED style” format. These are the notes I wrote while preparing for the talk. The talk itself differed from these notes somewhat, including some improvisations, but this will give you a broad sense of the argument.

What is a bubble?

King, Smith, Williams and Van Boening (1993) define a bubble as “trade in high volumes at prices that are considerably at variance with intrinsic values.” Krugman describes a bubble as a situation in which prices appear to be based on implausible or inconsistent views about the future (2013). (wikipedia)

Is the textbook market a bubble?

Before the Internet, educational materials were printed, making them scarce resources. Scarcity was a defining attribute of educational resources. Appropriately, commercial publishers tailored their business models to this core characteristic of their product and the environment.

Since the Internet, educational materials have become digital, making them abundant resources. But rather than adapting their business models to the new truth of their product, publishers have invested significantly in technologies that attempt to make educational materials artificially scarce. DRM technologies frequently used by publishers make it difficult (and illegal, thank you DMCA anti-circumvision provisions) to copy and share digital educational resources. These technologies cripple the Internet’s inherent ability to support the instantaneous and free copying and redistribution of content. (See The Importance of Getting in the Air.) Publisher’s refusal to sell copies of educational materials, now only allowing students to access educational materials as long as they are subscribed, wages a war on private ownership of property. (They’ve also brought this war into the physical realm with lawsuits to deprive students of their first sale rights in printed books). Invisible ink. Contrived “permissions.” (See Disappearing Ink, Textbook Affordability, and Ownership.)

Motivated students are terrific problem solvers. Consequently, some students are finding ways to succeed without buying textbooks. While this is good for students, it is bad for publishers. The major publishers are publicly traded companies with growth and earnings expectations they have to meet. And when the number of units you sell is dropping, the only way to meet revenue goals is to raise prices on the few units you do sell.

Rapidly rising prices provide stronger incentives for students to avoid buying textbooks, so they invent additional clever ways to succeed without buying textbooks, driving sales further down. This, in turn, drives prices even higher. This vicious cycle is a swirling vortex of academic doom.

What should, in the 21st century, be a completely frictionless and painless activity – owning a copy of your required educational materials – has instead become an arms race between billion-dollar multinational corporations and smartphone-wielding teenagers. The time and effort spent by students trying to find ways to get the learning resources they need to succeed is a huge waste. It’s time they’re spending on school, but time they’re not spending learning.

Importantly, only students with a significant degree of digital literacy or those who are part of a sophisticated social network can typically makes this kind of accommodation. First generation students, financially at-risk students, and other traditionally underserved students – those least able to afford to purchase textbooks – are left with very limited choices: go to class unprepared, take fewer classes, indenture themselves by taking on student loans to pay full-ticket for textbooks, or try to make their way in the world without post-secondary education.

Not only are commercial textbooks impossibly expensive, there is almost no evidence in the peer-reviewed, scholarly literature about their effectiveness or impact on learning. A major publisher recently announced a new initiative designed to study the efficacy of the materials they sell to students. I hope this initiative takes the scholarly path and submits its results to the rigors of the peer-review process for publication in academic journals. However, should it turn out that some of this company’s products are effective (or more effective than their competitors’ offerings), what do you suppose will happen to pricing? I doubt it will come down; it will likely only rise higher. (See Efficacy, the Golden Ratio, and the OER Impact Factor.)

Efficacy efforts are meaningless if they don’t address issues of affordability. Textbooks are becoming like a $10,000 per dose cancer treatment, “proven effective” in the lab but completely out of the reach of the majority of people who would benefit from them.

Is openness the pin that will pop the textbook bubble?

Which open are we talking about? We’re not talking about things that are free. The entire internet is free to access and read – CNN, BBC, National Geographic, HuffPo, etc. Free is, by itself, completely undifferentiating. To be called “open,” something must be free AND provide you with a broad set of copyright permissions. We call these the 5R permissions – Retain, Reuse, Revise, Remix, Redistribute. (See The Access Compromise and the 5th R.)

Anything which is free to use but traditionally copyrighted is absolutely not “open.” Materials or software that claim to be “open” but are traditionally copyrighted “All Rights Reserved” are more appropriately called “faux-pen” (fake open).

The inferiority of free to open is not simply a matter of philosophy. As we conduct original research and review the literature on empirical impacts on student outcomes when faculty make a displacing adoption, we’re beginning to see an interesting trend. Adopting an open textbook in place of your commercial textbook, and using it the same way you used your commercial textbook, will get you the same student success outcomes at a radically lower cost for students. And that’s terrific. But when a faculty member leverages the possibilities provided by OER to improve their pedagogy in ways that are only possible in the context of the 5R permissions, drastic improvements in student success are possible. (Again, see The Importance of Getting in the Air.)

Just as the core internet infrastructure runs on open source software – apache, sendmail, postfix, mysql, postgres, redis, mongo, hadoop, php, python, etc. – the core content infrastructure supporting education is shifting rapidly toward OER.

It started slowly with a faculty member adopting OER in place of a commercial textbook, what we call a “displacing adoption.” One happened here, another happened there, scattered and uncoordinated. Then it began happening in a coordinated way at scale, at schools like Tidewater Community College. Their “Z Degree” replaces commercial textbooks across their entire Business Administration degree program with OER. (Lumen Learning was privileged to work together with Tidewater on that project.) After Tidewater’s success, Z Degree-like projects are in planning stages all across the country.

Just as high enrollment general education courses subsidize much of what happens in colleges and universities, high sales volume general education titles are primary revenue streams for traditional publishers, subsidizing other activities. As the highest enrolling general education courses and highest enrolling degree programs replace commercial textbooks with OER, sales will drop to historic lows and prices will rise to historic highs.

As a bit of context, in less than five years we think Lumen Learning can be helping faculty replace commercial textbooks with OER for 10 million students a year. This will pull around $1B US out of the textbook market each year and give those dollars back to students. Other organizations that focus on OER adoption will have additional impacts. Every time publishers raise prices of textbooks and digital products, they make it easier for us. But publishers can’t help it. They will keep raising prices, faster and faster.

By 2020, the textbook market bubble will pop, completely resetting faculty and student expectations about (1) the prices of educational materials and (2) what we should be permitted to do with educational materials (5R activities). “Open” is the pin.

{ 1 comment }