Toward Renewable Assessments

For some time now I’ve been critical of “disposable assessments.” An assessment can be characterized as “disposable” if everyone understands that its ultimate destiny is the garbage can. Take an all-too-typical example:

  • Faculty member assigns student to write a two page compare and contrast essay
  • Student writes the paper and submits it to faculty
  • Faculty grades the paper and returns it to student
  • Student checks what grade they received, briefly peruses any written comments, and then throws the paper away

(This example assumes physical paper, but the principles are exactly the same in the context of assessments submitted, graded, and returned electronically.)

A “renewable assessment” differs in that the student’s work won’t be discarded at the end of the process, but will instead add value to the world in some way. Take, for example, the Murder, Madness, and Mayhem assessments from 2008:

The University of British Columbia’s class SPAN312 (“Murder, Madness, and Mayhem: Latin American Literature in Translation”) contributed to Wikipedia during Spring 2008. Our collective goals were to bring a selection of articles on Latin American literature to featured article status (or as near as possible). By project’s end, we had contributed three featured articles and eight good articles. None of these articles was a good article at the outset; two did not even exist.

Rather than writing essays to submit to their instructor and then throw away, these students contributed good quality research and writing to Wikipedia, where others will be able to benefit from their work for years to come. That’s the core idea between renewable assessments like Murder, Madness, and Mayhem, or Project Management for Instructional Designers, or Blogs vs Wikis, or the DS106 Assignment Bank, or The Open Anthology of Earlier American Literature, and many of the other examples listed by the community here.

In many ways, I think the most powerful part of renewable assignments is the idea that everyone wants their work to matter. No one wants to struggle for hours or days on something they know will be thrown away almost as soon as it is finished. Given the opportunity, people want to contribute something, to give something back, to pay it forward, to make the world a better place, to make a difference. Few right thinking person will invest their heart and soul in work that is academic in the way that non-faculty use the term – “not of practical relevance; of only theoretical interest. The debate has been largely academic.”

It’s no wonder people hate homework so much. They don’t hate learning – they hate wasting time and energy and effort. Try to imagine dedicating large swaths of your day to work you knew would never be seen, would never matter, and would literally end up in the garbage can. Maybe you don’t have to imagine – maybe some part of your work day is actually like that. If so, you may know the despair of looking forward and seeing only piles of work that don’t matter. And that’s how students frequently feel. Your results may vary, but I estimate that the 20 million postsecondary students in the US spend over 150M hours per year on disposable assessments. Every year. Year after year. When time is being used so poorly at such scale, I can’t believe it doesn’t negatively impact society.

Replacing disposable assessments with renewable assessments goes a long way toward re-humanizing education, giving students a reason to care about and truly invest in their work. Without this broader motivating context, students are just waxing cars, sanding decks, and painting fences.

“You promise learn. I say, you do. No question. That your part.”

Research on Renewable Assessments

A change of this magnitude – and really, any change in assessment strategy – deserves to be well understood. So how do we conceptualize research about renewable assessments (and perhaps other forms of open pedagogy)? What kinds of questions are appropriate and useful to ask in this context?

My colleagues in the Open Education Group and I like to say that when you’re considering the outcomes of research on OER adoption, there are “two ways to win.” First, think about three possible outcomes of OER adoption in terms of change in cost and change in learning:

  • Students save money and learn less
  • Students save money and learn the same amount
  • Students save money and learn more

When OER is adopted in place of commercial resources, students save a substantial amount of money. But what happens to learning? Two of the three possible outcomes are “wins” for OER – the same amount of learning for less money is a win, and more learning for less money is a win. Hence our “two ways to win” mantra.

Are there parallels to this set of questions in the assessment context? I believe so. Instead of cost and learning, I think we should begin by examining the value students recognize in their work and the amount of learning these new assessments support. I realize this requires some additional explanation.

[We now interrupt this essay with a brief, unscheduled rant. The overwhelming majority of assessments used by faculty to assign grades to students and, in a very real sense, determine some of their future life prospects, are created by faculty with no training in psychometrics. These assessments are never evaluated in terms of the reliability and validity of their results. (The test item banks and other assessments that commercial publishers provide with textbooks are also almost never subjected to this level of rigor in their design.) To say that the current state of play among faculty is a widespread ignoring of issues of reliability and validity in assessment is to give faculty too much credit. I would wager that over 90% of faculty don’t know these are technical terms in the assessment context, that over 99% of faculty couldn’t properly define the terms in this context, and that over 99.99% of faculty couldn’t describe a reasonable process for establishing the reliability and validity of an assessment’s results. So the first person who objects to the idea of renewable assessments on the grounds that they “might not be as good” as the assessments they’ve traditionally used has some serious explaining to do.]

In the early days of OER adoption, we found that there are ways of adopting OER that actually cost more than using commercial materials. (See Wiley, Hilton, Ellington, and Hall (2012) for an example of how a poorly planned print-on-demand strategy can make OER more expensive than publisher textbooks.) In similar fashion, I think it’s reasonable to anticipate that in the early days of renewable assessment design we’ll see assessments that students find no more motivating than their disposable counterparts. Just as we spent time in the early years of OER adoption research specifically investigating the whether-or-nots and hows of cost savings, we’ll need to spend time in the early years of renewable assessment design specifically investigating the value students find in doing this work, how motivating or engaging they find it, etc. Just writing that sentence I can see there’s still some construct clarification to do here.

As we work to establish common patters for designing renewable assessments that students find significantly more valuable to do than their disposable counterparts (just as we found OER adoption patterns that consistently save students money), we can also ask questions about the assessments are functioning. At a minimum we can begin by asking questions about outcome alignment. For example, should a rubric for grading a renewable assessment differ from the rubric used to grade the disposable assessment it is replacing? If so, how? We’ll have to guard carefully against “construct irrelevance creep” in rubrics for renewable assessments. For example, it might be tempting to award points for a renewable assessment published on YouTube based on how many views or likes it gets. Unless the course context is marketing with social media, this is likely completely irrelevant to the learning outcomes we ought to be assessing in introductory sociology or biology. If I replace a two page compare and contrast essay with a renewable assessment, should it not assess the same (or very highly overlapping) set of learning outcomes? Establishing some degree of comparability in what is assessed and the rigor with which it is assessed will be key to persuading faculty to abandon disposable assessments for renewable assessment strategies.

If you take the (inexplicably radical) position that assessments can be a productive part of learning and not just an autopsy of the learning process, we might also hypothesize better learning outcomes for students whose faculty use renewable assessments strategies. (Establishing the comparability described in the previous paragraph will also be helpful here.) Given a disposable assessment and a renewable assessment that both assess the same learning outcomes, might we hypothesize that students who find the renewable assessment work valuable, and consequently invest more time and effort in it, will display higher levels of mastery on the outcomes we care about? While only a hypothesis, it appears reasonable on its surface. And I have a few years of anecdotal evidence that give me confidence that it’s a hypothesis worth testing.

Looking for a topic for that dissertation or for your next journal article? You might think about attacking questions like:

  • Do students assigned renewable assessments find them more valuable, interesting, motivating, or rewarding than traditional assessments? Why or why not?
  • Do students assigned renewable assessments demonstrate greater mastery of learning outcomes than students assigned traditional assessments? Why or why not?

And what do you suppose will be the result of research into these and similar questions? Again, going back to the OEG mantra, there are two ways to win:

  • Assessments that students find significantly more rewarding to do that result in lower levels of mastery,
  • Assessments that students find significantly more rewarding to do that result in the same level of mastery, and
  • Assessments that students find significantly more rewarding to do that result in higher levels of mastery.

Renewable Assessments and Open

Open licenses allow faculty and students to revise and remix materials (both content and assessments) in a broad range of ways. As you look through the examples of renewable assessments above, you will see that many of them involve revising and remixing – demonstrating that renewable assessments are enabled by the 5R permissions granted by open licenses. (It’s true that a student could do a renewable assessment completely “from scratch,” but that doesn’t appear to be the way they’ve worked to date.) In other words, “open” makes possible renewable assessments that would otherwise be illegal. This is why I think renewable assessments are the best examples of open pedagogy we have now. You might argue that a student could use a range of copyrighted materials in a homework assignment and claim it was a Fair Use. However, I suspect many people would hesitate to share this kind of material broadly given the ambiguities of Fair Use, which kind of undermines the “give something back” philosophy underlying renewable assessments. And without providing the permissions for others to revise, remix, build on, and improve the work, it’s difficult to really call it “renewable.”

Students are the authors and, thanks to Berne, the copyright holders of the homework and other artifacts they create as part of their education. There is no morally or ethically appropriate scenario in which faculty require students to openly license their homework or other creations as part of an assignment. However, faculty can espouse the benefits of openness and advocate for students to license their works under a Creative Commons license. This advocacy will be significantly more effective (and less hypocritical) if the faculty member is using OER in the class and can point to OER they have created and shared.

If some portion of the over 150M hours higher ed students currently spend on disposable assessments can be spent on renewable assessments instead, and if some portion of those students choose to openly license their work, questions about the sustainability and maintainability of the OER ecosystem can be answered. Over time, we could see a transition to a place where the majority of content and assessments a learner encounters were created “by students, for students” with editorial support from faculty (what we used to call “grading”). What an incredible, inspiring, sustainable world that would be…

Let’s create and share more renewable assessments as OER – open renewable assessments – that others can adopt, improve, and share broadly. And let’s get this research going.


OER-based Degrees: Momentum

Fifteen years ago MIT announced its OpenCourseWare project. Above all else, this groundbreaking project demonstrated that an institution can openly share it’s core instructional resources without materially harming itself. Inspired by MIT’s example, hundreds of other institutions around the world began openly publishing the resources they created in support of their courses. Of critical importance is the fact that neither MIT nor any of the hundreds of other schools that launched OCW initiatives has ever reported suffering a decrease in enrollments because of its program of open sharing. Creating and sharing OER did not harm their ability to succeed in accomplishing their core missions – the education of their students. And there are many reasons to believe that their efforts in creating and sharing OER actually advanced their core missions. However, by the end of the decade growth of new programs had slowed and I haven’t heard of any new OCW initiatives launching in the last several years.

Over a decade after MIT broke ground with OCW, Tidewater Community College broke new ground with its Z Degree. Rather than committing to create and share its own OER as MIT and others had done over a decade earlier, Tidewater committed to adopt, adapt, and reuse OER instead of commercial educational resources across all the elective and required courses necessary to earn one of it’s most popular degrees. This initiative instantly cut the cost to graduate by 25% and greatly increased faculty’s pedagogical freedom. Like MIT OCW before it, the Tidewater Z Degree proved inspirational:

  • Almost immediately Northern Virginia Community College offered two OER-based degrees.
  • Shortly thereafter, the Virginia Community College System created its Zx23 program, providing support for every community college in the Commonwealth to create an OER-based degree.
  • A few weeks ago Achieving the Dream announced a major initiative in which 38 community colleges across the US will completely replace the commercial instructional materials in their courses with open educational resources (OER).
  • Yesterday, California Governor Jerry Brown signed into law the state’s budget for 2016-2017, including a $5M allocation (see p. 18) for the creation of OER-based degrees (see SEC. 16. Article 4) at community colleges across California. This funding will facilitate the creation of another 25 OER-based degrees at CA community colleges.

Just as we’ve seen rates of adoption increase over successive generations of technological innovations, the same seems to be happening in open education. OER-based degrees are spreading across institutions much more rapidly than OCWs did. This is truly extraordinary. It’s quite easy to imagine half of all US community colleges offering an OER-based degree by 2020, and not hard to imagine even more colleges doing so (or the same number of colleges each offering more than one OER-based degree).


{ 1 comment }

Of OER and Free Riders

This began as a comment on Heather’s post, but grew unwieldy and so ended up here. Heather’s post is reacting to this quote from an article she read recently: “There is one additional requirement for widespread OER adoption. Incentives need to discourage ‘free-riders’.”

This statement is demonstrably false. Of the 50 colleges in the US today with widespread OER adoption initiatives underway (by “widespread” here I mean that so many faculty across the institutions are adopting OER that it is – or will soon be – possible for students to earn complete degrees using only OER), literally none of them have discouraging incentives like those Annand describes. I could have ended this post here, but there’s more to say.

If you believe that open educational resources are public goods, which they appear to be since they are both non-excludable and non-rivalrous, then it can be hard to avoid bringing the rest of that conceptual framework (including the idea of the free rider problem) to your thinking about OER. If you don’t want to start from scratch as you think about ensuring the long-term sustainability of OER, the empirical and theoretical work already done on the problem of underprovision of public goods can be quite helpful.

Rajiv is right that the term “free rider” can, unfortunately, sound offensive and off-putting – especially if it is used in ways that sound like a criticism of a specific individual rather than a description of macro-level, society-wide behaviors. As we are thoughtful and careful, I think we can reap the benefits of the research already done on this problem without making people feel like we are singling them out. The open education family, as I think of us, has a deep moral and ethical responsibility to be accepting and welcoming of everyone regardless of their specific relationship to OER (e.g., whether they are contributors to OER or users of OER).

I spend quite a bit of time thinking / worrying about these problems. As I vocally and energetically advocate for universally replacing traditional materials with OER, I am acutely aware that there are essentially no OER available for 300 and 400 level courses or graduate courses. Importantly, the free rider problem does not describe a situation in which an individual uses open educational resources without contributing to their creation. It describes a situation in which so few people contribute to their creation that the OER needed by students and faculty never get created – and that accurately describes the current state of upper-level and graduate-level courses today.

If our only model for creating the OER necessary to replace traditional textbooks is to spend $250k of government or philanthropic funding for each and every course offered at each and every university, there is literally no path from here to there. We need to enable and facilitate alternative development models if our vision of universal OER adoption is to become a reality. (It’s no secret that I believe that these future models must be significantly more distributed and stigmergic than current models.)

We don’t need all users of OER to be contributors to OER for there to be a vibrant, healthy ecosystem of open content, assessments, simulations, and other resources for all courses at all levels. But no such ecosystem can ever emerge if no one (or as it stands today, an insufficient number of someones) contributes to the creation and continuous improvement of OER. Regardless of how we label this problem, we have to solve it to create the kind of educational future we want.

If only 2% – 5% of all faculty and their students (who are doing renewable assignments) were active creators and improvers of OER, that would likely be sufficient. If we could then persuade the other 95% – 98% of faculty to universally adopt OER in place of traditional resources, even without contributing any original or improved OER, I would be ecstatic. And I certainly wouldn’t be inclined to call them names.