On the (im)Possibility of OER Research

In Lorna’s review of the OER presentations at CETIS 2008, I read this bit about the new OLNet program:

The OU and Carnegie Mellon University have now received additional funding from the Hewlett Foundation for OLNet – a network to support sharing methodologies and evidence on the effectiveness of OERs. This next wave is about impact, evidence and effectiveness.

I realize that when blogging summaries of conference presentations you seldom quote the presenters with complete accuracy. So I mean neither disrespect for Lorna or Patrick, but something about this characterization of the “next wave” work rubbed me the wrong way. About once a year I have a student burst into my office and announce they have found their dissertation topic – comparing the effectiveness of OERs with traditionally copyrighted learning materials. I now have a well rehearsed shtick about how such a study would be the most pointless dissertation ever conducted (and if you read many dissertations, that’s really saying something!). Please join me in the following thought experiment:

Dr. Wiley takes a textbook from his shelf. He rips the (c) statement page out of the book, and inserts a new page with a Creative Commons license. Now riddle me this: is the textbook more or less effective instructionally than before?

The answer is that there is no difference in the educational effectiveness of the textbook. The same would be true for a simulation, an intelligent tutor, or any other form of educational material that changed from (c) to openly licensed. Let’s be perfectly clear – licensing plays no role in the effectiveness of a learning resource. Licensing does not appear in the long list of attributes of a resource that contribute to its educational effectiveness, including all the topics we study in instructional design programs like rehearsal scheduling, types of feedback, managing cognitive load, the use of worked examples, etc. There is, in fact, exactly zero reason for us to expect a resource to be more effective just because it is open, all other design considerations being equal.

The design of effective educational materials, software programs, and other experiences is already its own “applied” field of study – instructional design (drawing on “basic” fields of study like cognitive science). Those of us who work in OER need to be (at a minimum) keenly aware of this kind of research, but let’s not confuse instructional design research with OER research. (OER researchers can, of course, choose to go down the ID research path, ignoring existing research in instructional design, cognitive science, and other areas. They would actually be following a time-honored tradition in educational research if they did.) If open licenses do not directly impact the effectiveness of educational materials, then research questions about the effectiveness of educational materials are not OER research questions. We must ask ourselves, then: what kinds of research questions actually fall within in the field of open education?

The OLNet Proposal describes its core research questions as follows:

Research Questions

The driving research question behind OLnet pinpoints what we see as the next evolutionary step in the OER movement, namely:

• How can we build a robust evidence base to support the design, evaluation and use of OERs?

This high level question is refined into three sub-issues:

• How to improve the process of OER reuse/design, delivery, evaluation and data analysis?
• How to make the associated design processes and products more easily shareable and debateable?
• How to build a socio-technical infrastructure to serve as collective, evolving intelligence?

Since questions about the effectiveness of OERs are ID questions and presumably out of scope, what kinds of evidence would an OER study look to gather? In other words, what is it that makes OER special or different from other digital educational media? The answer must include their “4R” potential for reuse, redistribution, revision, and remixing. If we take the 4Rs of buckets, some questions that present themselves might include:

• Reuse – Does the process of making effective use of an OER in the classroom, online, or in another setting differ from the process of making effective use of (c) digital media in those same settings? If so, how? When people use OERs, do they use them effectively? Do those uses differ from the effectiveness with which they use other media / materials? What does noncommercial mean? Do we need a common definition? If not, and it doesn’t matter what noncommercial means, why do so many people / institutions use it?
• Redistribution – Can people find OERs? Are people accessing OERs? Are people making and sharing / trading copies of OERs? Who? How often? When? Why?
• Revision – How do you know what to leave in an OER and what to change? How can you insure that your design changes make OERs more effective and don’t accidentally botch their careful pedagogic design? Can an OER iterate over time toward something “better”? If some local revisions are improvements for the local learners but detrimental to learners in other locales, is forking actually desirable in OER? Should we be encouraging forking? How can student performance data inform the revision process?
• Remixing – While we spend huge resources clearing and openly licensing materials, do remixers care about licensing incompatibility issues? If they care neither in word nor in practice, what is the point of open licensing?

There are dozens more (including questions about sustainability models), but Lorna’s post just got me thinking, so I shared. This list is not, of course, comprehensive. Finding research questions that properly belong to the field of open education (as distinct from instructional design or cognitive science) is both a fun and important exercise. Fun, just because it is. Important, because if we can’t find questions unique to this field of study then it isn’t really a field.