As I travel the country (and the world) telling people about open educational resources, open textbooks, etc., I frequently receive questions about the quality of openly licensed instructional materials. I’ve answered this question enough that I thought it might be time to actually write something on the topic.
A Tiny Thought Experiment
Imagine you had a favorite textbook (hey – it’s a thought experiment). Now imagine receiving a letter informing you that the author has passed away and left you all the copyrights to the book. You immediately walk across the room and pull your copy off the shelf and open to the copyright page. You carefully cross out the words “All Rights Reserved” and replace them with the words “Some Rights Reserved – this book is licensed CC BY.” Have you changed the quality of the book in any way? No. Simply changing the text on the copyright page does not change the rest of the book in any way.
Consequently, we learn that quality is not necessarily a function of copyright status. We are forced to admit that it is possible for openly licensed materials to be “high quality.” We are also forced to admit that taking poor quality instructional materials and putting an open license on them does not improve their quality, either.
No Monopoly on Quality
Because quality is not necessarily a function of copyright status, neither traditionally copyrighted educational materials nor openly licensed educational materials can exclusively claim to be “high quality.” There are terrific commercial textbooks and there are terrific OER. There are also terrible commercial textbooks and terrible OER. Local experts must vet the quality of whatever resources they choose to adopt, and cannot abdicate this responsibility to publishing houses or anyone else.
Accuracy and OER
Some people are unable to believe that any process other than traditional peer review, licensing, and publication can result in content that is highly accurate. If you were to create a kind of content wild west, where anyone could publish anything and anyone could edit anything published by anyone else, this would obviously result in horrifyingly inaccurate content when compared to content produced via the traditional process.
Except that it doesn’t.
In 2005 Nature conducted an experiment in which they directly compared the accuracy of Wikipedia articles with the accuracy of traditionally reviewed, licensed, and published articles in Encylopedia Britannica.
They explain,
We chose fifty entries from the websites of Wikipedia and Encyclopaedia Britannica on subjects that represented a broad range of scientific disciplines. Only entries that were approximately the same length in both encyclopaedias were selected. In a small number of cases some material, such as reference lists, was removed to bring the length of the entries closer together.
Each pair of entries was sent to a relevant expert for peer review. The reviewers, who were not told which article came from which encyclopaedia, were asked to look for three types of inaccuracy: factual errors, critical omissions and misleading statements. 42 useable reviews were returned. The reviews were then examined by Nature’s news team and the total number of errors estimated for each article.
In doing so, we sometimes disregarded items that our reviewers had identified as errors or critical omissions. In particular, as we were interested in testing the entries from the point of view of ‘typical encyclopaedia users’, we felt that experts in the field might sometimes cite omissions as critical when in fact they probably weren’t – at least for a general understanding of the topic. Likewise, the ‘errors’ identified sometimes strayed into merely being badly phrased – so we ignored these unless they significantly hindered understanding.
The results?
Only eight serious errors, such as misinterpretations of important concepts, were detected in the pairs of articles reviewed, four from each encyclopaedia. But reviewers also found many factual errors, omissions or misleading statements: 162 and 123 in Wikipedia and Britannica, respectively.
With 42 usable reviews returned to Nature, this means the average article in both encyclopaedias contained 4 / 42 = 0.09 seroius errors, and 162 / 42 = 3.8 smaller errors per article for Wikipedia and 123 / 42 = 2.9 smaller errors per article for Britannica.
In other words, alternative authoring and review processes used to create openly licensed resources like Wikipedia can result in content that is just as accurate as the traditional peer review, publication, and licensing processes used to create works like Encyclopedia Britannica.
Distracting People from the Issue at the Core of Quality
Beyond issues of accuracy, when publishers, their press releases, and the media who reprint them say “quality” with regard to textbooks and OER, they actually mean “presentation and graphic design” – is the layout beautiful, are the images high resolution, are the headings used and formatted consistently, is the book printed in full color?
But this is not what we should mean when we talk about quality. There can be one and only one measure of the quality of educational resources, no matter how they are licensed:
- How much do students learn when using the materials?
There are two ways of thinking about this definition of quality.
- One is to realize that no matter how beautiful and internally consistent their presentation may be, educational materials are low quality if students who are assigned to use them learn little or nothing.
- The other way to think about it is this: no matter how ugly or inconsistent they appear to be, educational materials are high quality if students who are assigned to use them learn what the instructor intended them learn.
Really. For educational materials, the degree to which they support learning is the only meaning of quality we should care about.
Publishers put forth the beauty = quality argument because they have the capacity to invest incredible amounts of money in graphic design and artwork that visually differentiate their textbooks from OER. But when learning outcomes are the measure we care about, we see over and over again that many OER are equal in quality to commercial textbooks. (That is, over and over again we see OER resulting in at least the same amount of learning as commercial textbooks.)
We should never give into the temptation to focus on vanity metrics like number of pages or full color photos simply because they’re easy to measure. We have to maintain a relentless focus on the one metric that matters most – learning.
