Anna asks about the “Key Challenges for the OER Movement.” While the field earlier faced different challenges, here in late 2011 I believe there is one primary challenge the movement faces in the coming decade, and it is almost never discussed.
1. The Complete and Utter Lack of Assessment in the OER Space. Humans are famously terrible at judging whether they’re “getting it” or not during learning. One of the primary reasons the CMU OLI courses are (and have been shown to be) so incredibly effective in supporting learning is because they include frequent formative assessments that help learners check their own understanding. These assessments provide immediate feedback, allowing informal learners to determine with greater confidence whether or not they’re “getting it.”
The vast majority of OER in the world do not include any assessments. This means that you can use the OER in a variety of ways, but you can never really be sure (beyond your own biased estimates) if you’re grokking it or not. This is kind of like having access to a wide range of weaponry (guns, bows, crossbows, canons, catapults, etc.) and an unlimited amount of ammunition, but never being able to see whether (or not) you’ve hit the target.
A tiny minority of OER provide non-interactive assessments – these are things like practice problems delivered as text in a pdf. You can work the problems, but you don’t know whether or not your work is accurate because there is no feedback.
The CMU OLI courses, an even tinier minority of the world’s OER, provide interactive assessment, but do so using a technology which is neither reusable, revisable, remixable, or redistributable. Awesome for them, #fail for the rest of us.
Now that the OER snowball is rolling down the hill and growing in size every day, at least some people in the field need to turn their attention to the creation of Open Assessment Resources (OAR). And unless we believe that a single practice opportunity per OER is sufficient, we need multiple OAR per OER. If there are 500M OER in the world, we need something on the order of 1.5B OAR. And somebody needs to align these things (a fabulous task for a tool like delicious, btw) so that, after I’ve studied some OER, I can find an associated OAR.
The work on the Open Badge Infrastructure is only slightly encouraging with regard to this challenge. Much like the LOM work of the last decade it seems to be driven primarily by technologists and not educators. Given that there is a very rigorous academic field dedicated to assessment (psychometrics), one might expect that a few big name psychometricians would be part of the core development team on such a high profile project. That doesn’t appear to be the case. Please don’t get me wrong – I think it’s awesome that someone is dealing with issues relating to the technological requirements of alternative credentialing. However, we can’t credential (responsibly) without assessments, and in the midst of all the excitement about badges I’ve yet to hear of any novel (or even uninteresting) work on the underlying assessments themselves. (And I would love to learn that I’m wrong if someone could just point me to work out there I’ve just missed.)
Who is going to create new and appropriate assessment models? Once we have appropriate models, who will create the assessments themselves? This is the key challenge for OER in the coming decade, and a challenge standing in the way of alternative credentialing as well.
Given the OAR acronym, users of OER are literally up a creek without a paddle. And will be until we start making some progress on the OAR issue.
Khan Academy has developed a system of YouTube video presentations and related on-line exercises using the “10 exercises correct in a row = understanding” model. Stanford’s AI Class uses sequences of YouTube video clips, some of which end with questions; the student is expected to work out an answer before starting the next clip in the sequence. As you point out, these video-based resources are very problematic to reuse. Moreover, they provide less assessment of the effectiveness of the pedagogy than we would like (remembering that assessment cuts both ways, reflecting on the performance of both the learner and the educational system).
My PhD research into open source software development has led to the creation of a text-to-speech-based system which could address the problem as you frame it in several ways:
1. Reuse. We know from Wikipedia that we do not need many modified copies of a modular bit of knowledge, an encyclopaedia article. We need one article which can be modified (reused) in place, multiple times, and translated into other languages. In fact, the smaller the allowable changes (the granularity of the module), the more likely collaborators will be to contribute and improve it.
Wikipedia is in a sweet spot for this type of collaborative reuse. It has reasonably small, well bounded modules (articles) and the edits to each article’s wiki page can be as trivial as fixing a typo. In contrast, the boundary for a traditional education module is a course taught by a single teacher — usually too large and idiosyncratic to be easily reworked, even if the changes happened to be small ones.
Reusable on-line learning content should not require a separate OAR for each OER if a system is built to have an integrated, searchable combination of the two. In such a system, each piece of content is wrapped in questions (written as text, experienced as speech) to help the learner follow a chain of ideas. Before the content, the question is (paraphrased here): “Answer this question. Do you understand the prerequisites? (if not, see this other module).” After the content, the question is: “Try this exercise. If you have learned X, consider learning Y or Z next (pick one to follow the link provided). If not, try learning X again in this different way.”
The syntax for creating this kind of question / answer / response structure on a wiki page requires very little markup (only a semicolon and a link):
This is the question text.
This is an answer followed by a semicolon ; This is the verbal response
A second answer ;[link] Verbal response to second answer
…
For more examples, see http://code.google.com/p/open-allure-ds/wiki/SeparateContentFileSyntax
2. Process Improvement. What we really want to know as educators from assessment is not just “has the student learned†but “what has the student not learned and how can we remedy that.†Ultimately, this leads to “how effectively has the student learned†and an optimization of the learning sequences which achieve the desired objective in the shortest time.
Khan Academy is able to track the progress of students over time through their completed exercises. A system with integrated content and assessment can track every step of the process — and time each student’s progress along the way. Taken in aggregate, those numbers can reveal when and where learning slows (becomes difficult).
For a short video of progress tracking, see http://youtu.be/cxx7YxiW_V8
3. Credentialing. Currently a degree represents an industrial age measure of learning: time spent at the degree-granting institution. Both value assumptions, that 1) spending more time and 2) spending the time in school lead to optimal learning, are challenged by research which shows that learning can take place faster (more effectively) on-line. If possible, we would like a digital age credential to reflect acquired knowledge, rather than a time-invested proxy for it Bright students should be able to demonstrate their learning capabilities by 1) completing their studies faster than others and/or 2) studying more material, neither of which is well represented by a traditional degree.
The difficulty here again relates to the industrial age size of a learning module: a course. On-line, we can and should be tracking smaller modules. In fact, an on-line system can track every click. A student should be able to monitor their own progress continually. The implications for this approach for stimulating learning are explained by Tom Chatfield in this TED Talk: http://www.ted.com/talks/tom_chatfield_7_ways_games_reward_the_brain.html
As someone who has been working on interactive auto-grading open assessment resources for over 5 years, one of the challenges I face is the cost of delivering interactive assessment content. Especially if the content is to be remixable, it is hard to make interactive assessment standalone, so web applications like OLI or a LMS are the natural choice. Unlike OER, which can typically be redistributed for negligible cost, hosting dynamic web applications is not as cheap. This is especially true if the application is hosted for a wide audience, which would be necessary to make use practical for independent learners or faculty without web hosting experience or IT departments.
If we view assessment as being about feedback rather than merely accreditation then this perhaps offers us a different view. If oers are framed and presented within community networks or include opportunities for peer and tutor dialogue (possibly through discussion forums or other social networking tools) then that feedback can be available.
Perhaps oers (and broader open practice) provide the opportunity for us all to rethink our attitude to assessment and value feedback from ourselves through reflection, from peers and from those who are acting in capacity as teacher/tutor.
Hi David,
Thank you very much for picking up on this strand and raising an important question. So just to stick my OAR in – Assessment seems to be playing a couple of possible roles with OER:
1) You need some metrics and feedback to know you are on track
2) We need to have some proof that you are up to scratch if this is to count for you.
In the first category I think we need to be careful to not go down an test and assess approach. As Lou says the feedback element can be as important as the metrics. As my prime example I might use David – his blog provides plenty of evidence of capability and feedback from the community validates much of what appears there as valuable. The MOOC approach also appeals to this approach with most activities avoiding formal assessment.
For the second case the work of OER University and the Badges movement both indicate potential ways forward. I can also see holes in those – accreditation of prior learning or exam led assessment are both quite difficult tools to apply, the badges are a very nice idea well thought through but depend on trust that will need to be earned.
In saying this I do want to also add that working on assessment tools and question banks may well be a very fruitful contribution to OER – or rather OAR :-).
[The following is a copy of an earlier comment with all links removed.]
Khan Academy has developed a system of YouTube video presentations and related on-line exercises using the “10 exercises correct in a row = understanding†model. Stanford’s AI Class uses sequences of YouTube video clips, some of which end with questions; the student is expected to work out an answer before starting the next clip in the sequence. As you point out, these video-based resources are very problematic to reuse. Moreover, they provide less assessment of the effectiveness of the pedagogy than we would like (remembering that assessment cuts both ways, reflecting on the performance of both the learner and the educational system).
My PhD research into open source software development has led to the creation of a text-to-speech-based system which could address the problem as you frame it in several ways:
1. Reuse. We know from Wikipedia that we do not need many modified copies of a modular bit of knowledge, an encyclopaedia article. We need one article which can be modified (reused) in place, multiple times, and translated into other languages. In fact, the smaller the allowable changes (the granularity of the module), the more likely collaborators will be to contribute and improve it.
Wikipedia is in a sweet spot for this type of collaborative reuse. It has reasonably small, well bounded modules (articles) and the edits to each article’s wiki page can be as trivial as fixing a typo. In contrast, the boundary for a traditional education module is a course taught by a single teacher — usually too large and idiosyncratic to be easily reworked, even if the changes happened to be small ones.
Reusable on-line learning content should not require a separate OAR for each OER if a system is built to have an integrated, searchable combination of the two. In such a system, each piece of content is wrapped in questions (written as text, experienced as speech) to help the learner follow a chain of ideas. Before the content, the question is (paraphrased here): “Answer this question. Do you understand the prerequisites? (if not, see this other module).†After the content, the question is: “Try this exercise. If you have learned X, consider learning Y or Z next (pick one to follow the link provided). If not, try learning X again in this different way.â€
The syntax for creating this kind of question / answer / response structure on a wiki page requires very little markup (only a semicolon and a link):
This is the question text.
This is an answer followed by a semicolon ; This is the verbal response
A second answer ;[link] Verbal response to second answer
…
For more examples, see the project website.
2. Process Improvement. What we really want to know as educators from assessment is not just “has the student learned†but “what has the student not learned and how can we remedy that.†Ultimately, this leads to “how effectively has the student learned†and an optimization of the learning sequences which achieve the desired objective in the shortest time.
Khan Academy is able to track the progress of students over time through their completed exercises. A system with integrated content and assessment can track every step of the process — and time each student’s progress along the way. Taken in aggregate, those numbers can reveal when and where learning slows (becomes difficult).
For a short video of progress tracking, see the project YouTube channel.
3. Credentialing. Currently a degree represents an industrial age measure of learning: time spent at the degree-granting institution. Both value assumptions, that 1) spending more time and 2) spending the time in school lead to optimal learning, are challenged by research which shows that learning can take place faster (more effectively) on-line. If possible, we would like a digital age credential to reflect acquired knowledge, rather than a time-invested proxy for it Bright students should be able to demonstrate their learning capabilities by 1) completing their studies faster than others and/or 2) studying more material, neither of which is well represented by a traditional degree.
The difficulty here again relates to the industrial age size of a learning module: a course. On-line, we can and should be tracking smaller modules. In fact, an on-line system can track every click. A student should be able to monitor their own progress continually. The implications for this approach for stimulating learning are explained by Tom Chatfield in his TED Talk.
Interested to hear the idea “Analytics can help me find the people who need my help today” in the recording of the Change MOOC session. While I appreciate the perspective that human intervention (a teacher) is necessary to further learning, I wonder about all the cases where that intervention is unavailable. Sir John Davies of Commonwealth of Learning gave a figure of 400 million students worldwide entering the 12-to-17 age group who lack learning opportunities in schools. As Sugata Mitra has said, the place where educational technology is most useful is in the poorest, rural communities.