Reducing Friction and Expanding Participation in the Continuous Improvement of OER

I’m going to write a post or three about some of the friction that exists around using OER. There are some things about working with OER that are just harder or more painful than they need to be, and getting more people actively involved in using OER will require us to reduce or eliminate those points of friction.

I’ve been writing about continuous improvement in the context of OER for a few years now. To date, I’ve written about and worked on reducing the friction involved in a relatively centralized model for continuous improvement of OER – a “top down” approach, if you will:

Today I want to write about the other side of continuous improvement – a complementary, “bottom up” approach to facilitating broad participation in the continuous improvement process based on individual’s experiences as opposed to data analyses.

It’s a well established principle in open source software, open content, and other volunteer settings that when it’s hard to contribute, not many people contribute. When it’s easy to contribute, more people contribute. In fact, one of the most important keys to unlocking participation by a community is removing any friction they might experience in the process of participating.

Stop and think for a minute about how the process of suggesting an improvement to an educational resource typically works. It generally happens in one of two high-friction ways. In the first model, you send an email to an errata@publisher.com or errata@oerpublisher.org email address. In that email you have to try to describe exactly where the improvement should be made, probably by including a URL or page number, followed by a description of where on the page the change is supposed to go (“in the second sentence of the seventh paragraph…”). Then you can finally describe the specific change you think needs to be made. Or perhaps the provider wants you to use their ticketing system, and you end up in a piece of software like Zendesk. Once you figure out how to create a ticket, then you can write one up that includes all the information described above (you might even be asked to do some tagging of your ticket). Either way, if they choose to make the improvement you suggest it will likely be months or years before students in class benefit from your suggestion since it will be rolled into the next edition.

In other words, there’s a ton of friction in this process. And because it’s so painful, countless suggestions are never made that would have been made if the process were easier.

This semester at Lumen we’ve launched a continuous improvement pilot in which I believe we’ve removed just about all the friction that’s possible to remove from this process. Here’s how it works:

  1. There’s a new button at the bottom of every page of content. It says “Improve this page.”
  2. When a student or teacher or other user from the public web clicks the button, they’re linked directly to a Google Doc which includes all the content from the page. The Google doc is shared publicly and has Track Changes turned on. So you can just begin typing or commenting immediately, and your suggestions are highlighted and tracked.
  3. You’re done making your suggestion!

How easy is it to suggest an improvement to OER now? Faster than 30 seconds easy!

The pilot is active in three courses and we’re already receiving great feedback. Much to our excitement many of the suggestions appear to be coming from students. And they’re sending everything from spelling errors they catch to suggestions about how to make course content more inclusive.

The awesome folks on Lumen’s continuous improvement team have developed some tools and workflow that allow us to track the amount of time it takes us to vet these suggestions and get them implemented in the canonical copy of the courseware.  We’re averaging well under 24 hours from the time a suggestion is made until it’s vetted and implemented, and we think we can continue to be that responsive even as the number of suggestions grows after the pilot.

And here is where Lumen’s model particularly shines – since our courseware is embedded in the LMS via LTI (rather than copied and pasted into the LMS), all these improvements are immediately available to everyone using the courseware the instant we make them. The OER is literally getting a little better every single day – benefiting both the teachers and students who have formally adopted in their LMS as well as the informal learners who access our OER on our website. That’s the beauty of transclusion – the OER embedded in the courseware and the OER published on our public-facing website are the same copy of the OER. Update once, improve everywhere.

So far the suggestions people are making aren’t something you could openly license and attribute – they’re either high-level ideas or lower-level issues like fixes to spelling or grammar (i.e., not copyrightable works that can be licensed and attributed). So to make sure people are recognized for their efforts to make things better, we’ve created a new Acknowledgments section on the About This Course page in the pilot courses, and we’re adding the names of everyone (faculty, student, and otherwise – it doesn’t matter who you are) who uses their name when making suggestions that we integrate into the courseware.

 

Thoughts on Continuous Improvement and OER

Recently I’ve been doing both more thinking and more roll-up-your-sleeves working on continuous improvement of OER. Below I’m cross-posting two short pieces on this topic I recently published on Lumen’s site (here and here).


Improvement in post secondary education will require converting teaching from a solo sport to a community-based research activity. (Herbert A. Simon, 1986)

Photo by Perry Grone on UnsplashThe faculty Lumen work with carry an enormous workload. Some have research, grant writing, and publication responsibilities in addition to teaching their courses. Some teach five or six courses per semester. Some have committee assignments and additional service responsibilities. Some drive across town several times per day as they try to string adjunct appointments at three institutions together into a career that pays the rent. All of our faculty have expertise in their discipline. Few have formal training in teaching or learning.

Herbert Simon, quoted above, was an “above average” faculty member. He won both the Turing Award for his work in computer science and the Nobel Prize for his work in economics. But even he realized that we can’t expect individual faculty to stay at the cutting edges of their discipline, teaching and learning practice, educational research, and the ever-changing technologies that can be used in the service of learning. This is why Simon called for us to come together as a community – there are countless ways in which education needs to be improved, and no one person, institution, or organization has the time or expertise to do it all alone. We need each other.

The role Lumen is choosing to play in the community working to improve education is to enable and empower learners and faculty with highly effective learning materials that become more effective every semester. And this process of making OER more effective every semester – also known as “continuous improvement” – is where we see some of the most exciting opportunities to collaborate with faculty.

Continuous improvement is an iterative cycle. In the case of OER, the continuous improvement cycle involves:

  • Creating or selecting OER for use in your course,
  • Instrumenting the OER for measurement,
  • Measuring the effectiveness of OER in supporting student learning,
  • Identifying areas where student learning was not effectively supported,
  • Making changes to the learning design of the OER in those underperforming areas, and
  • Beginning the cycle again.

Developed with funding from the Bill and Melinda Gates Foundation, Lumen’s Waymaker courses are designed specifically to support this continuous improvement process, and we have been refining our process for several years in collaboration with a small group of faculty. You can see an example of the difference in OER before and after we applied this internal continuous improvement process here:

While we’re still refining the tools we’ve created to support this work, we are now eager to open our continuous improvement process to all faculty members, with the goal of making it a genuinely community-based research activity. Here’s what we’re doing this fall:

  • We have analyzed data from Spring 2018 to empirically determine which learning outcomes students struggled with the most in five Waymaker courses. (Learn more about this process in this accompanying blog post.)
  • For each course, we have published a collection of “Learning Challenges  Leaderboards” listing the learning outcomes students struggled with the most, together with links to the OER that didn’t adequately support student learning.

The RISE and Shine Initiative

We invite you to engage with us in a community-based continuous improvement process. We’re calling this initiative RISE & Shine. RISE is the analysis that identifies which content needs work (you can read more about RISE here). Once we’ve identified that content, we invite faculty to Shine by contributing their expertise to the improvement of OER.

You can participate by taking one or more of these steps:

  1. Raise your hand. Complete this form to let us know you’d like to be part of conversations about improving learning with OER. We’ll share Learning Challenges updates and include you in what’s happening in your discipline.
  2. Reflect. Look at the Learning Challenges Leaderboard in your discipline. Think about what you do to make learning better for your students as you’re tackling these challenging topics, and compare that with the approach taken in the aligned OER. How would you do things differently?
  3. Share ideas. Have ideas about how we should make the OER supporting these difficult topics more effective? Share them here.
  4. Share improvements. Do you have a short video, an interactive activity, an edited version of the existing OER, or any other improved content you’ve developed to improve your students’ understanding? If so, submit them using this form. Whenever your contributions are included in Lumen course materials, your work is attributed. And you’ll be able to see the effect your contributions have on student learning in the next semester’s Learning Challenges Leaderboard update.

At Lumen we’re serious about making improving education a community-based research activity. That’s why we collaborate with faculty throughout the course improvement process, openly license the improvements we make to content, publish our continuous improvement frameworks in open access journals, and open source many of the tools we create to support our continuous improvement efforts.

However, we’re just one company. Truly transforming education will require more people and organizations to adopt a continuous improvement mindset. Given the amount of effort and the range of expertise required to engage in continuous improvement, Simon’s admonition to do this work collaboratively resonates with us as being deeply true.

We hope you’ll become part of this community-based effort with us.


The potential of the “data revolution” in teaching and learning, just as in other sectors, is to create much more timely feedback loops for tracking the effectiveness of a complex system. In a field where feedback is already well established as a vital process for both students and educators, the question is how this potential can be realized through effective human-computer systems (Buckingham Shum and McKay, 2018).

Open educational resources (OER) are educational materials whose copyright licensing grants everyone free permission to engage in the 5R activities, including making changes to the materials and sharing those updated materials with others. Consequently, everyone who wants to continuously improve OER has permission to do so. (Not so with traditionally copyrighted materials, whose licensing allows only the rightsholder to alter and improve the content.)  Permission to make changes is a necessary – but not sufficient – condition for continuous improvement.

In addition to permission to make changes, improvement requires a capacity for measurement. We can say we’ve changed OER without measuring the impact of those changes, but we can only say we’ve improved OER when we have measured student outcomes and confirmed that they have actually changed for the better.

Continuous improvement of OER, then, is the iterative process of:

  • Instrumenting OER for measurement,
  • Measuring their effectiveness in supporting student mastery of learning outcomes,
  • Identifying areas where student mastery of those learning outcomes was not effectively supported,
  • Making changes to the learning design of the underperforming OER aligned to those learning outcomes, and then
  • Beginning the cycle again so we can:
    • Measure the impact of those changes and determine whether or not they were actually improvements (not just changes), and
    • Identify additional areas that need strengthening.

Engaging in the continuous improvement of OER in this manner allows us to make OER support learning more effectively each semester.

Learning Design and Continuous Improvement

Lumen instruments OER for measurement at the individual learning outcome level. Outcome alignment is at the very core of both our learning design process and our continuous improvement process. The outcome alignment process has three parts.

A visualization of the relationships between the more than 250 learning outcomes in Waymaker Microeconomics

A visualization of the relationships between the more than
250 learning outcomes in Waymaker Microeconomics

First, we collaborate with faculty to identify each of the individual skills we want to support students in mastering. These detailed outcomes are, like all the content Lumen creates, licensed CC BY. Second, we align each individual page of content with the one or more outcomes whose mastery it supports. Finally, we align each assessment item with the outcome it is designed to assess. In the case of Waymaker Microeconomics, for example, that means aligning over 2,350 individual assessment items appearing in pre-tests, interactive practice opportunities, self-checks, and end of module quizzes with the appropriate learning outcome.

If that sounds like an incredible amount of work, that’s because it is!

But it’s worth it. In addition to providing benefits in the learning design process that we don’t discuss here, outcome alignment is fundamental to the continuous improvement process. With assessment items aligned to individual outcomes in pre-tests, practices, self-checks, and end-of-module quizzes, we can model learning over time, from the beginning of the module (the pre-test occurs before students see any OER) to the second attempt on the end of module quiz (after students have used and reused the OER). Similarly, because all course content is outcome-aligned, we can examine how patterns of OER usage correlate with performance on aligned assessments.

Analyzing the Effectiveness of OER

This process begins with a RISE analysis. I published the RISE framework last year with Bob Bodily and Rob Nyland, two amazing PhD students at BYU. Earlier this year I also published an open source implementation of RISE in the Journal of Open Source Software. RISE analysis divides performance on assessments into two categories, higher and lower, and usage of OER into the same two categories, higher and lower. These are matrixed to create four ways of diagnosing how OER are working in support of student learning.

Higher Grades High student prior knowledge, inherently easy learning outcome, highly effective content, poorly written assessment Effective resources, effective assessment, strong outcome alignment
Lower Grades Low motivation or high life distraction, too much material, technical or other difficulties accessing resources Poorly designed resources, poorly written assessments, poor outcome alignment, difficult learning outcome
Lower Use of OER Higher Use of OER

Each outcome in the course is placed in one of these four categories, as in the visualization below. We focus first on those outcomes in the lower right corner, where usage of OER is high but performance on aligned assessments is low. These are places where effort invested in improving OER is most likely to improve student learning. Below we have drawn a blue diamond three standard deviations out from the origin (mean OER usage on the x-axis and mean assessment performance on the y-axis) to make it easier to visually identify outliers in need of immediate attention.

RISE analysis visualization of Introduction to Business

Making Targeted Improvements to OER

In the past, once the OER most in need of improvement were identified, we reached out to individual faculty to invite them to participate in the process of analyzing and improving course materials in collaboration with Lumen’s learning engineers and course designers. Moving forward, we will use the Learning Challenges Leaderboards to make this information public and invite the community to participate in the process of revising, remixing, finding, or creating new OER to better support student learning.

(In addition to continuously improving the OER based on outcomes data, we also make a wide range of other updates to our courses. For example, we update OER based on faculty feedback, current events, and the availability of new OER. We make improvements to assessments based on the results of item analysis, make improvements to features of the Waymaker platform (like faculty and student nudges) based on ways they correlate with student performance, and make improvements to supplementary materials based on faculty feedback.)

The Role of Learning Materials in Education

It would be easy to look at the effort Lumen invests in improving OER and other courseware components and come to the conclusion that we think learning materials are the most important part of education. That would be a mistake. We believe deeply that the contributions made by the learner and the faculty both significantly outweigh the importance of learning materials. However, we also believe that highly effective learning materials can dramatically amplify the efforts of learners and faculty. For example, we know that highly effective learning materials can help learners reach the same levels of mastery in half the time compared to materials that follow a traditional textbook design (Lovett et al., 2008).

There are myriad ways in which education needs to be improved. The role Lumen is choosing to play in the community working to improve education (which extends far beyond problems relating to learning materials) is to enable and empower learners and faculty with highly effective learning materials that become more effective every semester.

We’re working to engage a broad community of educators and institutions in the work of improving education by continuously improving OER course materials. We’re trying to make this complex task more transparent, measurable, and participatory. Given the creativity and commitment of the community we serve, we have every hope of success.

Of Progress, Problems, and Partnerships

In 2012 Kim Thanos and I founded Lumen Learning because, through our Gates-funded work on the Kaleidoscope Project, we had seen first-hand how hard it was for faculty to replace publisher materials with OER. The 2000s were an inspiring decade as institutions and individuals created and published a huge amount of openly licensed educational materials (e.g., MIT OCW, Wikipedia, Khan Academy), but in 2010 it was difficult to find a faculty member who had made the switch. It seemed like lots of people wanted to publish and share their own OER, but no one wanted to use anyone else’s.

Actually, plenty of faculty wanted to use OER – they just didn’t want to use them badly enough to fight through all the obstacles that involved.

The difficulties faculty had – and still have – adopting OER is a huge problem for several reasons. First, when faculty don’t adopt OER there’s no opportunity for OER to save students money. Second, when faculty don’t adopt OER there’s no opportunity for OER to facilitate new forms of pedagogy that invigorate both teaching and learning. And third, when faculty don’t adopt OER there’s no opportunity for OER to support significant improvements in student learning.

These are the problems Lumen is still chasing today. How can we help as many faculty as possible adopt OER, leading directly to significant student savings? How can we help those faculty who have made the choice to adopt OER wrap their heads around the pedagogical benefits enabled by OER? How can we help faculty and students use OER in ways that will result in improvements in measures like final grade, completion rate, persistence rate, and graduation rate?

It’s been a humbling and amazing experience. We’ve learned some things about working effectively with faculty through the course redesign and OER selection process. About how to carry forward the best work of previous faculty to a new group of faculty so they can stand on the shoulders of those who came before and go much faster. About hosting, managing, and integrating OER into LMSs where faculty and students already are. About license vetting and CC attribution management. About using data to support the continuous improvement of both content and assessments (watch for more on this topic in weeks to come). About the intersection of instructional design, personalization, learner agency, metacognition, and behavioral economics (nudging). About building and running entire degree programs on nothing but OER.

As we’ve learned more we’ve been able to help more. We more than tripled the number of students we supported from 2014 (~10,000 students) to 2015 (~35,000 students), and more than tripled again from 2015 to 2016 (~110,000 students). Across those three years these students saved almost $15 million. Peer-reviewed research has shown, for a range of outcomes, that when we support students using OER they perform the same or better than their peers using publisher materials. (And the most exciting research, looking in much more depth at our Gates-funded work on personalization and OER, is still to come.) And faculty are slowly coming to appreciate the range of novel things (like renewable assignments) that OER adoption enables in their pedagogy. There’s a mountain of work left to do, but it feels like we’re off to a solid start.

But it’s really just a start. While Kim is magically staying the same age, I’m getting older – and neither of us is really interested in running a cool little project that improves things for a small fraction of postsecondary students and faculty. Neither are the other amazing people who’ve found their way to Lumen over the years. (And did I mention we currently have five positions open?) To only lightly edit the language from my original application for the Shuttleworth Fellowship back in 2012:

We want to push the field over the tipping point and create a world where OER are used pervasively throughout schools, colleges, and universities. In our vision of the world, OER supplant traditional textbooks for all courses at all levels. Organizations, faculty, and students at all levels collaborate to create and improve an openly licensed content infrastructure that dramatically reduces the cost of education, increases student success, and supports rapid experimentation and innovation.

I’m keenly aware that this kind of system-level change can’t be facilitated by a single person or a single organization. This scale of change requires collaborative efforts on the part of many, many people and organizations. So what kind of collaboration could move the field meaningfully closer to this vision of a world where OER are the default?

First, what do we know about why OER aren’t already the default? According to the most recent Babson survey of faculty about OER, the biggest obstacles to OER adoption in higher ed are:

  • the majority of faculty don’t know that OER exist, and
  • for the minority who do know about them, OER are too hard to find.

So how do you address that problem at a scale large enough to make a difference?

Today Lumen announced a major new partnership with Follett. Follett operates more than 1,200 local campus bookstores, and they’ve made significant investments in tools, processes, and people to make it easy for faculty to review and adopt course materials. The new partnership will integrate Lumen’s open courseware offerings into Follett’s systems, making them super easy for faculty at about 1/3 of all US higher education institutions to find, review, and adopt. The partnership also adds, for the first time, the option for students to pay Lumen’s course support fee rather than the institution. (Previously our model only allowed institutions to pay these fees, and that has made it difficult for some schools to work with us.) This new option makes it possible for individual faculty to choose to work with Lumen, which hasn’t been possible due to the institutional focus of our model. Now there’s an easy way for everyone to work with Lumen – from an entire institution to a single faculty member. The press release has all the details.

I’m incredibly excited about this partnership because I think it will go a long way toward overcoming the biggest obstacles to OER adoption Babson identified. It should result in millions more students and faculty using OER and enjoying the wide range of benefits they provide. And it will provide a terrifically interesting case study of a new OER adoption model that we haven’t seen or studied before, hopefully inspiring more creative thinking in this space.

“We’ve only just begun,” as the song says. We won’t realize our entire vision by taking this one step, but this one step certainly moves us forward.