I’m a huge proponent of OER. In fact, I’m the original OER advocate - since 1998, when I created the first open license for educational materials and began working to persuade people to openly license them. Creating and sharing OER is a powerful lever for increasing educational opportunity. (Generative AI is now an even more powerful way of accomplishing this goal, but that’s a subject for another post.)
Because I’m an OER advocate, I’m frustrated by the unfounded claims I hear people make about OER, especially the claim that ‘adopting OER improves student outcomes!’ That’s simply not true. Adopting OER does NOT reliably improve student success. In some studies adopting OER is associated with better outcomes, in some studies there’s no change at all, and it some studies adopting OER is actually associated with decreases in student success. This variability in study results is partly due to problems with the ways studies are designed.
Three years ago I wrote a primer for critically reading OER research. The primer encourages readers to ask three questions as they read OER research:
- Did the study control for differences in instructors?
- Did the study control for differences in instructor support?
- Did the study control for differences in the instructional design of the learning materials?
When research that purports to be about OER doesn’t answer these questions, it doesn’t actually answer questions about the effectiveness of OER. Instead:
When a study fails to control for differences in instructors, what that study is really measuring is, “How do outcomes differ when students take courses from the kind of instructors who choose to adopt innovations like OER compared to the students who take classes from the kind of instructors who choose not to?”
When a study fails to control for differences in instructor support, what that study is really measuring is, “How do the students of well-supported instructors perform compared to the students of instructors who are not as well supported?”
When a study fails to control for differences in instructional design between the OER and the TCM, the study is really asking, “How do students taught using pedagogical approach X perform compared to students taught using pedagogical approach Y?”
A New Report from AAC&U
I was excited yesterday to see that the American Association of Colleges and Universities (AAC&U) had released a new research report titled “Open Educational Resources: A New High-Impact Practice.” This is a very large-scale research report whose lead author, Eddie Watson, is a terrific scholar (I’m sure the other authors are as well; Eddie is the only one I know).
The title caused me some concern right off the bat, as AAC&U defines high impact practices as “well established in the research literature as powerful levers for student success in higher education.” But there’s actually plenty of nuance in the report, which I really appreciated.
I don’t know if the authors of the report had read my primer, but they did specifically call out each of the primer’s three issues (which previous OER research rarely - if ever - did):
The authors control for instructor differences by comparing the success of instructors’ students before they adopted OER with the success of those same instructors’ students after they adopted OER. This is genuinely awesome and I was excited to see this.
The authors dig into the kinds of supports instructors receive, including financial support, release time support, support from specialized personnel like librarians and instructional designers, and professional development.
The authors also capture coarse-grained differences in instructional design using Hofer’s notion of “high” and “low” OER implemention - operationalized in this study as whether the instructor adopted an OER textbook as-is (low) or revised, remixed, or created OER materials (high).
While the authors did collect information about instructor support and instructional design, I will admit to being a little disappointed that they never used instructor support as a covariate in any of the student-outcome analyses, and used high/low implementation as a covariate in only a single analysis - the withdrawal-rate model. The grade and time-to-completion analyses use only OER yes/no, institution type, and learner complexity as predictors. That prevents us from getting a clean quantitative read on the degree to which improvements in student success are due to differences in instructor support or instructional design, as opposed to differences in instructional materials (OER or TCM).
But, as we’ll see below, the authors do explicitly recognize the critical role instructor support and pedagogy play in student success! This is a positive for the report, and another rarity in the OER efficacy literature.
Hooray for Nuance!
There is some great nuance in this paper, which means we have to pay close attention so we don’t miss it!
The report predominantly uses the language of “OER implementation” as opposed to “OER adoption.” This is a critically important distinction that will be easy for readers to miss.
On the second-to-last page, the report summarizes its findings by explaining that OER is only a high-impact practice when its implementation is “done well.”
In other words, you can’t just swap Traditionally Copyrighted Materials (TCM) for Open Educational Resources (OER) and expect to see meaningful changes in outcomes. OER adoption is only one part of a broader intervention that needs to include both instructor support and changes in pedagogy if you want to reliably improve student success.
(If you want to talk about the impacts of just adopting OER without the other components of a high impact implementation, the truth is that adopting OER will save your students money and get them roughly the same outcomes. And getting the same outcomes for less money is a great deal! But if you’re looking for something ‘well established in the research literature as a powerful lever for student success in higher education,’ you need the entire, “well done” implementation.)
The one place the report’s attention to nuance seems to lapse is in the description of OER’s impact on final grades. The report states that A grades rose in every context - nearly five percentage points overall - and then attributes that change to OER. But the grade analysis does not include the effects of instructor support or implementation level. Consequently, that bump in A grades can’t be attributed confidently to OER - the change in grades could have been due to OER, or it could have been due to the support instructors received, or it could have been due to changes instructors made to their pedagogy, or some combination of the three. The report can say with confidence that grades rose, but it can’t confidently say why they improved. This thorny tangle of effects is the same confound the authors acknowledge in their earlier work (Colvard et al., 2018), and which I have acknowledged in some of my earlier work as well.
Final Thoughts
This is an excellent report that will make a great contribution to the OER research literature. I just hope that advocates share what it actually says: that if you want to go beyond saving money and actually improve student success, you have to implement OER effectively - meaning you adopt OER, and are well supported with funding, time, access to specialists, and PD, and that the support leads you to adopt more effective teaching practices. That’s when student success improves meaningfully.
Let’s not wave this report around and say “See?!? Adopting OER improves outcomes!” Let’s say, “When combined with meaningful instructor supports and changes in instructional design, OER adoption contributes to improved outcomes!”