Of Sunlight, OER, and Lumen

We recently installed solar panels on our home. The benefits of adding them were immediate and obvious – the very first month they were on the roof our electric bill dropped to $9 (the fee required to stay connected to the grid) and we generated more power than we used, pushing the excess back out to the grid. Because I can’t stop thinking about open, I’ve been pondering the relationship between solar power and OER.

At the same time, I’ve been thinking about how to answer several questions I’m often asked. People who don’t work directly with Lumen sometimes have a hard time understanding what we do, and this leads to a range of confused questions like, “What does Lumen do, anyway?”, “How can you sell OER if they’re free?”, and “If OER are free, why would anyone pay you?”

As I’ve continued to think about these two topics, I realize they’re actually closely related. In fact, I believe the simplest way to answer to many questions about Lumen is by analogy. Let me explain…

Sunlight is perhaps the ultimate example of a public good. Both nonexcludable and nonrivalrous, sunlight is available to anyone and everyone for free. Sunlight is highly versatile and can do everything from making your garden grow to melting the snow off your driveway. For years now I’ve been hoping to use solar to power my home – to harness the sunlight so that it consistently and reliably does what I want (i.e., provides electricity to cool my house, run my lights, keep the wifi on, etc.). But in order to get the sunlight to do what I wanted it to, I needed to partner with someone else who had the right combination of expertise and technology.

There were dozens of questions to answer… Things like, what type of panels should I use? What’s the right number of panels to install given the amount of power I need versus what I’m likely to generate? How and where will I mount them? Should I use a system with microinverters on each panel, or one large inverter? How do I integrate the power the panels produce into my home without burning the house down? How do I tie into the grid so you can push excess power there? Does a battery system make sense in my circumstances? If so, how many should I use and what kind? Do I want to be able to live monitor my energy production, or is checking the power bill at the end of the month sufficient? Etc.

You get the idea. Of course, I could have stopped everything else in my life for a few months and learned most of what I needed to know to answer these questions myself. I could probably even have done a passable job with the installation and tying into the house and the grid. But the end product of working with a company who already had the expertise, experience, and knew the relevant technology inside out was far beyond what I could have done myself.

And yes, I paid them – even though sunlight is free.

OER are a lot like sunlight. They’re another prime example of a public good. They are freely available to anyone and everyone. They’re extremely adaptable. And like sunlight, in order to use OER to reliably and predictably meet our goals (improving student outcomes and saving students money), we need to apply a combination of expertise and technology.

Again, there are dozens of questions to answer. Are you trying to replace commercial materials with OER across an entire degree program or in just a single course? Is the primary goal to improve student learning, increase graduation rates, or save students money? How, specifically, do you optimize for each of these outcomes? Are you willing to take a fresh look at your pedagogy? How would renewable assessments work in your discipline? Which OER should you use, and where do you find them? What tools will you use to revise and remix the OER you select? How are students going to access these OER – online or in print? If online, how will you integrate them into your LMS in a sustainable way? If print, how are you going to manage that process? Is this a math or other quantitative course that requires algorithmically generated and graded practice problems? What role would you like automated systems to play in personalizing the learning experience for your students? How can learning analytics help you strengthen your relationship with your students? Based on last semester’s learning results, where and how should you engage in continuous improvement of your OER?

You get the idea. Of course, many faculty could stop everything else they’re doing for a few months and learn much of what they need to know about copyright law and open licensing, instructional design and learning science, open source platforms for revising and remixing, relevant technical standards, statistics and data science, etc., to answer these questions themselves. They could probably even do a reasonable job of pulling it all together. But when faculty collaborate with Lumen, the end products are far better than what they can typically do by themselves.

And yes, institutions pay us – even though OER are free.

I think this analogy between sunlight and OER, and by extension between Lumen and a renewable energy company, works pretty well. Returning the the questions above:

Q. What does Lumen do, anyway?
A. In the same way a solar power company helps people harness sunlight to power their homes, Lumen helps faculty harness OER to power student learning.

Q. How can you sell OER if they’re free?
A. Solar power companies don’t sell sunlight and Lumen doesn’t sell OER. Sunlight and OER are free.

Q. “If OER are free, why would anyone pay you?”
A. We provide expertise and technology that help people make effective use of OER, just like solar power companies provide expertise and technology that help people create electricity from sunlight.

PS. It occurs to me that there may be more to do with this analogy. If we can fruitfully compare OER to sunlight and Lumen to a renewable energy company, should we compare commercial textbooks to oil and traditional publishers to extractive energy companies? Should we compare the broad move away from commercial materials and toward OER-based degrees with the move away from fossil fuels and toward renewable energy? These are questions for another time.


Although this was written as a critique of physics, truer words were never spoken about educational research:

Science is corrupted when it abandons the discipline of empirical validation or dis-confirmation. It is also weakened when it mistakes its assumptions for facts and its ready-made philosophy for the way things are. (Smolin and Unger)

Oh, how I wish more people would embrace this way of thinking about educational research…

{ 1 comment }

Toward Renewable Assessments

For some time now I’ve been critical of “disposable assessments.” An assessment can be characterized as “disposable” if everyone understands that its ultimate destiny is the garbage can. Take an all-too-typical example:

  • Faculty member assigns student to write a two page compare and contrast essay
  • Student writes the paper and submits it to faculty
  • Faculty grades the paper and returns it to student
  • Student checks what grade they received, briefly peruses any written comments, and then throws the paper away

(This example assumes physical paper, but the principles are exactly the same in the context of assessments submitted, graded, and returned electronically.)

A “renewable assessment” differs in that the student’s work won’t be discarded at the end of the process, but will instead add value to the world in some way. Take, for example, the Murder, Madness, and Mayhem assessments from 2008:

The University of British Columbia’s class SPAN312 (“Murder, Madness, and Mayhem: Latin American Literature in Translation”) contributed to Wikipedia during Spring 2008. Our collective goals were to bring a selection of articles on Latin American literature to featured article status (or as near as possible). By project’s end, we had contributed three featured articles and eight good articles. None of these articles was a good article at the outset; two did not even exist.

Rather than writing essays to submit to their instructor and then throw away, these students contributed good quality research and writing to Wikipedia, where others will be able to benefit from their work for years to come. That’s the core idea between renewable assessments like Murder, Madness, and Mayhem, or Project Management for Instructional Designers, or Blogs vs Wikis, or the DS106 Assignment Bank, or The Open Anthology of Earlier American Literature, and many of the other examples listed by the community here.

In many ways, I think the most powerful part of renewable assignments is the idea that everyone wants their work to matter. No one wants to struggle for hours or days on something they know will be thrown away almost as soon as it is finished. Given the opportunity, people want to contribute something, to give something back, to pay it forward, to make the world a better place, to make a difference. Few right thinking person will invest their heart and soul in work that is academic in the way that non-faculty use the term – “not of practical relevance; of only theoretical interest. The debate has been largely academic.”

It’s no wonder people hate homework so much. They don’t hate learning – they hate wasting time and energy and effort. Try to imagine dedicating large swaths of your day to work you knew would never be seen, would never matter, and would literally end up in the garbage can. Maybe you don’t have to imagine – maybe some part of your work day is actually like that. If so, you may know the despair of looking forward and seeing only piles of work that don’t matter. And that’s how students frequently feel. Your results may vary, but I estimate that the 20 million postsecondary students in the US spend over 150M hours per year on disposable assessments. Every year. Year after year. When time is being used so poorly at such scale, I can’t believe it doesn’t negatively impact society.

Replacing disposable assessments with renewable assessments goes a long way toward re-humanizing education, giving students a reason to care about and truly invest in their work. Without this broader motivating context, students are just waxing cars, sanding decks, and painting fences.

“You promise learn. I say, you do. No question. That your part.”

Research on Renewable Assessments

A change of this magnitude – and really, any change in assessment strategy – deserves to be well understood. So how do we conceptualize research about renewable assessments (and perhaps other forms of open pedagogy)? What kinds of questions are appropriate and useful to ask in this context?

My colleagues in the Open Education Group and I like to say that when you’re considering the outcomes of research on OER adoption, there are “two ways to win.” First, think about three possible outcomes of OER adoption in terms of change in cost and change in learning:

  • Students save money and learn less
  • Students save money and learn the same amount
  • Students save money and learn more

When OER is adopted in place of commercial resources, students save a substantial amount of money. But what happens to learning? Two of the three possible outcomes are “wins” for OER – the same amount of learning for less money is a win, and more learning for less money is a win. Hence our “two ways to win” mantra.

Are there parallels to this set of questions in the assessment context? I believe so. Instead of cost and learning, I think we should begin by examining the value students recognize in their work and the amount of learning these new assessments support. I realize this requires some additional explanation.

[We now interrupt this essay with a brief, unscheduled rant. The overwhelming majority of assessments used by faculty to assign grades to students and, in a very real sense, determine some of their future life prospects, are created by faculty with no training in psychometrics. These assessments are never evaluated in terms of the reliability and validity of their results. (The test item banks and other assessments that commercial publishers provide with textbooks are also almost never subjected to this level of rigor in their design.) To say that the current state of play among faculty is a widespread ignoring of issues of reliability and validity in assessment is to give faculty too much credit. I would wager that over 90% of faculty don’t know these are technical terms in the assessment context, that over 99% of faculty couldn’t properly define the terms in this context, and that over 99.99% of faculty couldn’t describe a reasonable process for establishing the reliability and validity of an assessment’s results. So the first person who objects to the idea of renewable assessments on the grounds that they “might not be as good” as the assessments they’ve traditionally used has some serious explaining to do.]

In the early days of OER adoption, we found that there are ways of adopting OER that actually cost more than using commercial materials. (See Wiley, Hilton, Ellington, and Hall (2012) for an example of how a poorly planned print-on-demand strategy can make OER more expensive than publisher textbooks.) In similar fashion, I think it’s reasonable to anticipate that in the early days of renewable assessment design we’ll see assessments that students find no more motivating than their disposable counterparts. Just as we spent time in the early years of OER adoption research specifically investigating the whether-or-nots and hows of cost savings, we’ll need to spend time in the early years of renewable assessment design specifically investigating the value students find in doing this work, how motivating or engaging they find it, etc. Just writing that sentence I can see there’s still some construct clarification to do here.

As we work to establish common patters for designing renewable assessments that students find significantly more valuable to do than their disposable counterparts (just as we found OER adoption patterns that consistently save students money), we can also ask questions about the assessments are functioning. At a minimum we can begin by asking questions about outcome alignment. For example, should a rubric for grading a renewable assessment differ from the rubric used to grade the disposable assessment it is replacing? If so, how? We’ll have to guard carefully against “construct irrelevance creep” in rubrics for renewable assessments. For example, it might be tempting to award points for a renewable assessment published on YouTube based on how many views or likes it gets. Unless the course context is marketing with social media, this is likely completely irrelevant to the learning outcomes we ought to be assessing in introductory sociology or biology. If I replace a two page compare and contrast essay with a renewable assessment, should it not assess the same (or very highly overlapping) set of learning outcomes? Establishing some degree of comparability in what is assessed and the rigor with which it is assessed will be key to persuading faculty to abandon disposable assessments for renewable assessment strategies.

If you take the (inexplicably radical) position that assessments can be a productive part of learning and not just an autopsy of the learning process, we might also hypothesize better learning outcomes for students whose faculty use renewable assessments strategies. (Establishing the comparability described in the previous paragraph will also be helpful here.) Given a disposable assessment and a renewable assessment that both assess the same learning outcomes, might we hypothesize that students who find the renewable assessment work valuable, and consequently invest more time and effort in it, will display higher levels of mastery on the outcomes we care about? While only a hypothesis, it appears reasonable on its surface. And I have a few years of anecdotal evidence that give me confidence that it’s a hypothesis worth testing.

Looking for a topic for that dissertation or for your next journal article? You might think about attacking questions like:

  • Do students assigned renewable assessments find them more valuable, interesting, motivating, or rewarding than traditional assessments? Why or why not?
  • Do students assigned renewable assessments demonstrate greater mastery of learning outcomes than students assigned traditional assessments? Why or why not?

And what do you suppose will be the result of research into these and similar questions? Again, going back to the OEG mantra, there are two ways to win:

  • Assessments that students find significantly more rewarding to do that result in lower levels of mastery,
  • Assessments that students find significantly more rewarding to do that result in the same level of mastery, and
  • Assessments that students find significantly more rewarding to do that result in higher levels of mastery.

Renewable Assessments and Open

Open licenses allow faculty and students to revise and remix materials (both content and assessments) in a broad range of ways. As you look through the examples of renewable assessments above, you will see that many of them involve revising and remixing – demonstrating that renewable assessments are enabled by the 5R permissions granted by open licenses. (It’s true that a student could do a renewable assessment completely “from scratch,” but that doesn’t appear to be the way they’ve worked to date.) In other words, “open” makes possible renewable assessments that would otherwise be illegal. This is why I think renewable assessments are the best examples of open pedagogy we have now. You might argue that a student could use a range of copyrighted materials in a homework assignment and claim it was a Fair Use. However, I suspect many people would hesitate to share this kind of material broadly given the ambiguities of Fair Use, which kind of undermines the “give something back” philosophy underlying renewable assessments. And without providing the permissions for others to revise, remix, build on, and improve the work, it’s difficult to really call it “renewable.”

Students are the authors and, thanks to Berne, the copyright holders of the homework and other artifacts they create as part of their education. There is no morally or ethically appropriate scenario in which faculty require students to openly license their homework or other creations as part of an assignment. However, faculty can espouse the benefits of openness and advocate for students to license their works under a Creative Commons license. This advocacy will be significantly more effective (and less hypocritical) if the faculty member is using OER in the class and can point to OER they have created and shared.

If some portion of the over 150M hours higher ed students currently spend on disposable assessments can be spent on renewable assessments instead, and if some portion of those students choose to openly license their work, questions about the sustainability and maintainability of the OER ecosystem can be answered. Over time, we could see a transition to a place where the majority of content and assessments a learner encounters were created “by students, for students” with editorial support from faculty (what we used to call “grading”). What an incredible, inspiring, sustainable world that would be…

Let’s create and share more renewable assessments as OER – open renewable assessments – that others can adopt, improve, and share broadly. And let’s get this research going.