instructional-technology open content research

Fall in Love with the Problem, Not the Solution

Curt Bonk and I recently published a Preface for a special issue of ETR&D on Systematic Reviews of Research on Learning Environments and Technologies. It is largely a collection of personal stories and reflections about the arc of learning technologies over the last 30 years. However, we close with some advice which I believe to be profoundly important for everyone working in and around the learning technologies field, include open advocates.

Perhaps the most frustrating thing about the field of learning technologies is the way it obsesses over technologies while devaluing or even ignoring problems faced by learners around the world. For decades, learning technologies like those discussed in this special volume have been elevated to objects of study in and of themselves. All too frequently, those working in our field respond to questions about their research agenda with answers like “I study iPads,” “I study augmented reality,” or “I study open educational resources.” We question whether this fetishization of learning technologies will help us make sustained, meaningful improvements to the world in the future. As long as we are focused on the tools themselves, the ongoing march of learning technologies will resemble an endless series of waves eternally breaking on the shore only to draw out and come crashing in again without making a visible difference in the surrounding landscape.

We encourage learning technologists to follow the old advice, ‘fall in love with the problem, not the solution.’ The world is full of so very many problems that desperately need solving—racism, poverty, crime, climate change, war, Internet access, educating refugees… the list goes on and on and range from the local to the global. At the very least, we encourage the reader to consider adding a problem to their answer to the question above. For example, “I study how to help young women maintain their interest in science and math into their high school years. iPads show real promise for mitigating this problem.” Or “I study how to make higher education more effective and affordable to students who are most at-risk. Open educational resources have an important role to play in making that happen.”

Fall in love with a problem—let it be your “standing wave.” Then as the inevitable extended, connected, and repeated waves of learning technologies roll past over the years, you will have a steady foundation from which to evaluate and use them instrumentally to make the world a better place.

You can read the full article online.

open content research

From Static to Interactive and From Open to Free: Consequences Both Intended and Unintended

The most recent issue of IRRODL included an article titled Effectiveness of OER Use in First-Year Higher Education Students’ Mathematical Course Performance: A Case Study, by Juan I. Venegas-Muggli and Werner Westermann. Quoting from the article:

The main aims of this research were to examine the effect of OER use among higher education students and to analyze teacher and student views on OER use in order to better understand how these resources are used and valued. This was justified by the fact that there is a lack of empirical evidence to support expanding the use of OER. Moreover, recent societal demands to improve education quality in Chile have made this a relevant case study environment in which to examine the potentials of OER.

In relation to the first aim, the most important result is that students in face-to-face arithmetic/statistics courses using Khan Academy resources achieved significantly better exam grades than students who did not use any extra resources (p < 0.05) or those who used open textbooks as an extra resource (p < 0.01). The fact that the final exam was the same for everyone makes this a valid comparative measure of students’ performance.

As I read the article, I couldn’t help but reflect on a recent article in PLOS One by Phil Grimaldi and his colleagues at OpenStax, which challenged us to be more specific in our theorizing about why OER adoption might improve student learning. Despite popular rhetoric along the lines of “you can’t learn from materials you can’t afford,” their rather ingenious experiment showed that increasing access to learning materials by adopting OER instead of traditionally copyrighted resources (TCM) will almost never measurably improve learning. Learning will only improve measurably if you ALSO do something else.

So why did the Chilean math students who used Khan Academy do better than those who used either an open textbook or a traditionally copyrighted textbook? The underlying explanatory mechanism is extraordinarily simple. Students who have more opportunities to practice, receive feedback, reflect on that feedback, and practice again learn more than students who don’t have as many of those opportunities.  Whether you’re talking about conducting a literature review, playing a musical instrument, solving math problems, or literally anything else, practice with feedback is critical to learning.

Why, then, did the Chilean math students who used Khan Academy do better than those who used an open textbook? The students who used Khan Academy – specifically, students who used Khan Academy’s online homework system – had significantly more opportunities to practice, receive feedback, reflect, and practice again than students who used an open textbook. Unsurprisingly, the students with more opportunities to engage in online interactive practice did significantly better on the final exam. You have to read a bit between the lines of the article to see that students using Khan Academy weren’t just watching OER videos, but were also using the online interactive practice. I reached out to Werner and specifically verified this. He said teachers were also tracking student progress using Khan Academy’s analytics dashboards.

When Grimaldi et al. call on us to make the explanatory mechanisms underlying our research clearer (rather than just saying “OER was better”), this is the kind of clarification they’re calling for. If you didn’t know how Khan Academy worked before reading this article, you could easily have missed the role the online interactive practice system played in improving student learning and just thought “OER was better!”

This is not the first example, nor will it be the last example, of peer-reviewed research demonstrating that OER plus a system that provides opportunities to engage in online interactive practice results in better student learning than OER alone. And it shouldn’t surprise anyone that it’s true.

Research at CMU

A series of research papers from Carnegie Mellon going back almost 15 years has established this over and over again. In a 2016 article, Koedinger and colleagues wrote:

The “doer effect” is an association between the number of online interactive practice activities students do and their learning outcomes, that is not only statistically reliable but has much higher positive effects than other learning resources, such as watching videos or reading text… [We] provide generalizable evidence across four different courses involving over 12,500 students that the learning effect of doing [online interactive practice activities] is about six times greater than that of reading.

If you’re not familiar with educational research, 6x is a massive effect.

Other research from CMU explicitly connects the dots between (1) the significant impact of doing online interactive practice activities and (2) the somewhat common student practice of printing out their online learning materials.  Unsurprisingly, students who print out their online learning materials don’t do as many of the online interactive practice activities (“Volqs” in the path diagram below). Consequently, students who print do worse on both quizzes and final exams than their peers who work online.

Scheines, Leinhardt, Smith, Cho, (2005)

For more research along these lines from CMU, read just about anything by Ken Koedinger.

If you’ve heard that research says print is superior to online in terms of supporting student learning, take a closer look at that research. It generally compares reading print to reading online. But the CMU research, the Chilean research, and related research is comparing doing online interactive practice to reading online (in the CMU case) and reading offline (in the Chilean case). Regardless of the format of the materials students are reading, actively engaging in online interactive practice is more effective than simply reading. Again, this should not be a surprise to anyone.

Homework Systems

Unfortunately, there’s something akin to a conspiracy theory floating around the open education community about the systems that provide these opportunities to students. The statement of the conspiracy goes something like this. “Since publishers can’t really sell static content for exorbitant prices anymore, they’ve started selling homework systems. Far from adding value to student learning, these systems simply put a paywall in front of activities required to pass the course – like submitting homework and doing exams – making it impossible for students to look for cheaper alternatives. These systems are really just attempts by publishers to increase revenues by forcing students to buy access codes that can’t be resold.”

As with every good conspiracy theory, there’s some truth in that statement. But there are some huge falsehoods, as well.

First, homework systems aren’t a recent development. They’re not a response to the way OER caused the bottom to fall out of the textbook market. They’ve been around for a very long time, and they emerged in order to serve market demands that textbooks did not meet (and do not meet, and cannot meet). First, they provide students the opportunity to engage in online interactive practice and therefore have the potential to greatly increase student learning as described above. Second, they automate a large portion of the grading in a course, giving faculty back a large amount of time to do higher value activities, like meeting with students during office hours (or evaluating OER to adopt in place of their current traditionally copyrighted textbook).

Second, publishers can’t force students to do anything. Ever. Publishers don’t get to make adoption decisions, only faculty get to make adoption decisions. And for every homework system out there, there are dozens of static textbooks available that faculty are completely free to choose. However, faculty choose to require homework systems for their courses because they see benefits for both their students and themselves. So, if you’re looking to criticize someone for “making students pay to submit their homework,” 100% of your criticism should be directed at faculty – they’re the only ones who can require students to submit their homework in a particular way.

Now, do these systems need to be priced unconscionably high, as publishers have done historically? Absolutely not. While there absolutely are real, ongoing costs associated with hosting, maintaining, and securing online systems that hold student data, these costs are something like an order of magnitude lower than what publishers have traditionally charged. So if you want to criticize homework systems, but you don’t want to criticize faculty for choosing to adopt learning materials that actually improve student learning, criticize publishers for the way they price these systems. That’s the part that publishers control and, as we have seen with textbooks, it’s something publishers can and will change in response to market pressure.

And we have to talk a little more about costs.

From Open to Free

You may not have discerned it yet, but the open education movement is pulling apart. There’s a growing group of people and institutions whose primary commitment has shifted from open to free. You can see this primarily in the emergence of the “zero textbook costs / Z degree / Zed cred” movement, which generally has a different primary goal than the OER movement. Rather than focusing primarily on supporting the adoption of OER in order to both lower costs and facilitate OER-enabled pedagogy and other open educational practices, these programs focus on eliminating learning materials costs for students without regard for whether the materials used by faculty are open or merely free.

Sister causes, to be sure. Siblings, certainly. But not the same movements.

The interaction of this shift from open to free with what we know (and are continuing to confirm) about systems that enable students to participate in online interactive practice has a terrible unintended consequence. As an example, see this question and answer from a FAQ for California’s ZTC grant program:

12. Do fees for access to online materials disqualify a program? Do materials fees for lab classes and would it matter if they are actual materials fees or items purchased in a bookstore?

Fees for access to online instructional materials like MyMathLab are not allowed because components of these type programs [sic] include e-textbooks. Fees for materials in certain labs are allowed such as materials kits in cosmetology courses/labs.

Given what the research says about the significant learning impact online interactive practice systems can have, ZTC and similar policies can cross a line that OER advocates said we would never cross – sacrificing student learning for cost savings. Traditionally, one of the cornerstones of OER advocacy has been that faculty should never adopt OER in order to save students money if they thought that doing so would harm student learning. If they’re not careful, ZTC and related policies can abandon that foundational commitment, replacing it with something closer to “free at any cost.” And I don’t believe any of us want to be painted with this “free at any cost” (to student learning) brush.

The way ZTC / Z degree / Zed cred program policies can make it difficult for students and faculty to use and benefit from these systems seems to be a genuinely unintended consequence. Fortunately, like all policies, these policies aren’t laws of nature and can be changed after unintended negative consequences like this are identified.

A Concluding Thought

I know it makes people unhappy when I say it, but we’ve got to stop talking about “textbooks” if we want to improve student learning. Textbooks are universally understood by both faculty and students to be static compilations of words, tables, and images. Yes, the first conversation with a faculty member (or policymaker) can be easier when you say “textbook” rather than “learning materials” or “learning resources” or “courseware” or something else. But the price of the lubricant you use to grease the skids of that first conversation is a restriction of imagination that precludes the possibility of something interactive. Something that will improve student learning. The language of “textbooks” prevents faculty, administrators, policymakers, students, and others – including OER advocates – from seeing the adjacent possible. Maybe we needed to use the language of “textbooks” 20 years ago when we first started advocating for faculty to adopt open content. But, for the sake of student learning, it’s time to re-evaluate that decision.

open content research textbooks

OER Cost Savings and Adoption Rates: New Methodologies, New Data, and New Results

At the OpenEd Conference in 2013, Nicole Allen and I challenged the OER community to save students one billion dollars. Five years later, SPARC have collected a significant amount of data in order to answer the question of whether or not we have achieved that goal. You can read more about the data collection methodology and their ongoing work on this question here. SPARC have made the data available under the CC0 dedication and you can download them here.

As a brief summary, part of the data collection involved sampling the prices for required course materials across a range of material formats, including new print, used print, print rental, digital rental, and loose-leaf. SPARC started with a stratified sample of 120 US post-secondary institutions. Then, working from a pool of 20 courses for which adoptable OER exist, they randomly selected five courses to examine at each institution and collected pricing information for all available formats from each campus’ online bookstore. Quite the task!

Below I present the results of some exploratory data analysis intended to answer basic questions. In this post I’ll “stick to the facts,” providing color commentary in a later post. (The source code used to perform the EDA below is also available for download here and released under CC0.)

The tl;dr

  • The overall average price for “Traditionally Copyrighted Materials” (TCM), defined as the price halfway between the highest cost option (e.g., new print) and the lowest cost option (e.g., a digital rental), is $134.26.
  • The overall average price for “OER Only” (Open Educational Resources in all formats, including new print, used print, rental, loose-leaf, and digital) is $10.69.
  • The overall average price for “OER Hybrid” (Open Educational Resources in all formats, paired with a homework system, personalized learning platform, or other value-added service) is $34.71.
  • The overall average price for all OER (both OER Only and OER Hybrid) is $17.32.
  • Average student savings from OER, calculated as the overall average price for TCM minus the overall average price for all OER, is $116.94.
  • The adoption rate of OER in these 20 courses for which good OER exist is 6.3%.

Exploratory Data Analysis

What types of institutions are represented in the data, and how many of each type are there?


Across all available course materials formats, how do the average prices of the most expensive options and least expensive options vary across type of institution?


Across all available course materials formats, how do the average prices of the most expensive options and least expensive options vary across courses?


How do the average prices of the most expensive options and least expensive options vary across TCM, OER Only, and OER Hybrid?


If we take the midpoint between the most expensive option and least expensive option as the “overall average price”, what is the overall average price for each type of required course material?


How much money do students save when their faculty adopt OER? We can calculate student savings with OER by starting with the overall average price of TCM and subtracting each of the overall average prices of OER in turn.


What is the rate of OER adoption in the data set?

Type of Materials Adoption Rate
TCM 93.68%
OER Only 4.58%
OER Hybrid 1.74%