On the (im)Possibility of OER Research

In Lorna’s review of the OER presentations at CETIS 2008, I read this bit about the new OLNet program:

The OU and Carnegie Mellon University have now received additional funding from the Hewlett Foundation for OLNet – a network to support sharing methodologies and evidence on the effectiveness of OERs. This next wave is about impact, evidence and effectiveness.

I realize that when blogging summaries of conference presentations you seldom quote the presenters with complete accuracy. So I mean neither disrespect for Lorna or Patrick, but something about this characterization of the “next wave” work rubbed me the wrong way. About once a year I have a student burst into my office and announce they have found their dissertation topic – comparing the effectiveness of OERs with traditionally copyrighted learning materials. I now have a well rehearsed shtick about how such a study would be the most pointless dissertation ever conducted (and if you read many dissertations, that’s really saying something!). Please join me in the following thought experiment:

Dr. Wiley takes a textbook from his shelf. He rips the (c) statement page out of the book, and inserts a new page with a Creative Commons license. Now riddle me this: is the textbook more or less effective instructionally than before?

The answer is that there is no difference in the educational effectiveness of the textbook. The same would be true for a simulation, an intelligent tutor, or any other form of educational material that changed from (c) to openly licensed. Let’s be perfectly clear – licensing plays no role in the effectiveness of a learning resource. Licensing does not appear in the long list of attributes of a resource that contribute to its educational effectiveness, including all the topics we study in instructional design programs like rehearsal scheduling, types of feedback, managing cognitive load, the use of worked examples, etc. There is, in fact, exactly zero reason for us to expect a resource to be more effective just because it is open, all other design considerations being equal.

The design of effective educational materials, software programs, and other experiences is already its own “applied” field of study – instructional design (drawing on “basic” fields of study like cognitive science). Those of us who work in OER need to be (at a minimum) keenly aware of this kind of research, but let’s not confuse instructional design research with OER research. (OER researchers can, of course, choose to go down the ID research path, ignoring existing research in instructional design, cognitive science, and other areas. They would actually be following a time-honored tradition in educational research if they did.) If open licenses do not directly impact the effectiveness of educational materials, then research questions about the effectiveness of educational materials are not OER research questions. We must ask ourselves, then: what kinds of research questions actually fall within in the field of open education?

The OLNet Proposal describes its core research questions as follows:

Research Questions

The driving research question behind OLnet pinpoints what we see as the next evolutionary step in the OER movement, namely:

  • How can we build a robust evidence base to support the design, evaluation and use of OERs?

This high level question is refined into three sub-issues:

  • How to improve the process of OER reuse/design, delivery, evaluation and data analysis?
  • How to make the associated design processes and products more easily shareable and debateable?
  • How to build a socio-technical infrastructure to serve as collective, evolving intelligence?

Since questions about the effectiveness of OERs are ID questions and presumably out of scope, what kinds of evidence would an OER study look to gather? In other words, what is it that makes OER special or different from other digital educational media? The answer must include their “4R” potential for reuse, redistribution, revision, and remixing. If we take the 4Rs of buckets, some questions that present themselves might include:

  • Reuse – Does the process of making effective use of an OER in the classroom, online, or in another setting differ from the process of making effective use of (c) digital media in those same settings? If so, how? When people use OERs, do they use them effectively? Do those uses differ from the effectiveness with which they use other media / materials? What does noncommercial mean? Do we need a common definition? If not, and it doesn’t matter what noncommercial means, why do so many people / institutions use it?
  • Redistribution – Can people find OERs? Are people accessing OERs? Are people making and sharing / trading copies of OERs? Who? How often? When? Why?
  • Revision – How do you know what to leave in an OER and what to change? How can you insure that your design changes make OERs more effective and don’t accidentally botch their careful pedagogic design? Can an OER iterate over time toward something “better”? If some local revisions are improvements for the local learners but detrimental to learners in other locales, is forking actually desirable in OER? Should we be encouraging forking? How can student performance data inform the revision process?
  • Remixing – While we spend huge resources clearing and openly licensing materials, do remixers care about licensing incompatibility issues? If they care neither in word nor in practice, what is the point of open licensing?

There are dozens more (including questions about sustainability models), but Lorna’s post just got me thinking, so I shared. This list is not, of course, comprehensive. Finding research questions that properly belong to the field of open education (as distinct from instructional design or cognitive science) is both a fun and important exercise. Fun, just because it is. Important, because if we can’t find questions unique to this field of study then it isn’t really a field.

3 thoughts on “On the (im)Possibility of OER Research”

  1. I took a different take on the quote entirely. When I read “This next wave is about impact, evidence and effectiveness”, I don’t think of effectiveness in the sense of “more effective learning materials”, but instead in the sense of “how effectively do OER materials accomplish the things that we say they will accomplish?” If you phrase is that way, then all of the research topics that you described do indeed apply to OLNet.

  2. Dave, your analogy about the textbook is missing something. OER isn’t just about replacing the copyright page with a new CC page, is it?. There’s also the aspect that has to do with freeing the textbook to be made widely available in many different formal and informal learning environments. Doesn’t that create whole new situation that is ripe for research in terms of the book’s impact and effectiveness. For example wouldn’t it be interesting to know about the educational impact of the textbook in informal learning situations, with online learners studying in a peer-to-peer community, in classrooms with teachers of differing skills, or in completely different cultures?

  3. Right on Dave! Even though I wasn’t there, reading between the lines of the quote, I have to agree with Andre.. “impact, evidence and effectiveness” sounds more like a first call to the publishing industry to get on board and make OER investments .. sustainable.. that word will probably be their 3rd wave. None of this has ever been about improving the effectiveness of teaching and learning has it?

    So another research questions might be:
    *How the publishing industry can profit from OER

    And, actually, I think this is a good move. If we can get the publishing industry to take these mostly DIY resources and add graphical and formatting quality to it, then the average teacher will start using the resources in class, like they do any other resource that’s good enough. They will continue to photocopy, burn and handout – only this time it will be more legal. It’ll be a long time (if ever) before a teacher or a student comes to appreciate the geekiness of copyright and the legalities of their effectiveness.

Comments are closed.