I had the good fortune of being invited to speak at the ETS Future of Assessment internal conference today. The slides are available at slideshare, but here are the three main points from my talk today.
“Badges are not assessments.” OER provide a huge content infrastructure on which educational innovations can be built more quickly and less expensively than before OER existed. The Open Badge Infrastructure (OBI) provides a standard, interoperable system for issuing, managing, and displaying credentials on which educational innovations can be built more quickly and less expensively than before OBI existed. However, no one is paying sufficient attention to the gap between learning anything anywhere (OER) and receiving a recognition (OBI) – this gap is called “assessment.” A badge is not an assessment anymore than a blue ribbon is a foot race. Someone has to pay attention to designing the assessments, experiences, and challenges people will complete in order to EARN badges. There is a huge opportunity for “open assessment infrastructure” in this chasm between OER and OBI.
“Assessment as status update.” People already invest significant effort updating Facebook statuses, tweeting, writing book and product reviews, blogging, uploading videos, etc. Given the opportunity, people will complete simple in-place assessments in order to let the world know what they’re learning from what they’re reading / watching. In addition to the existing “status update” motivations already driving people’s behavior, lots of organizations have a vested interest in seeing this body of data come into existence. Assessment will be ubiquitous in the very near future.
“Browser history as high stakes exam.” If an entity like ETS can establish predictive validity around different performance / behavior patterns and college completion or success, one can easily imagine submitting their usernames for Google Web History, Facebook, Twitter, Delicious, Blogs, Google Reader, YouTube, etc. IN PLACE OF taking a four hour high stakes exam like the ACT or GRE. Why make a high stakes decision based on a few hundred data points generated in one morning (when you could be sick, distracted, etc.) when you could get 1,000,000 data points generated over three years? Organizations that can figure out how to leverage big, messy data will win. While some will run the other direction screaming “privacy!,” many people will opt to take this non-test path into college. The precedent exists in our willingness to give all our financial data to companies like LifeLock or Mint to monitor against identity theft or recommend better products to us. When sufficient value is available, we are typically willing to pay with personal data.
I’m curious if you’ve heard any discussion of open-licensed item banks for high stakes assessments. There was some talk about this awhile back but I haven’t heard anything lately.
David,
Awesome post! I completely agree, there is still a big a gap between OER and credentials. Badges are a step toward closing this gap but they are not assessment. I’d love to get some feedback on this Open Assessment wiki page I have been writing. Do you think this could be steps toward closing this gap? https://wiki.mozilla.org/User:Prawsthorne/Assessment
Thank-you, Peter
This is a must read for clarifications that are being requested in the above comments.
(Trying again) Peter’s link above is a must read for concerns and questions raised by Jeremy and Chris. This link provides clarity on the scope of open assessment. And…it’s nothing new.
If you want a software engineering job, the boss is not interested in the high stakes test you took to prove your competence in the field. In fact, there isn’t one and for good reason.
He/ she wants to see your portfolio. What was the last project you designed and developed? How well did it do? With whom did you collaborate? Are they respected and well-known in the field? Portfolios, peer testimonials, rubrics, Number of peer reviews, etc.
@font-face {
font-family: “Cambria”;
}p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: “Times New Roman”; }div.Section1 { page: Section1; }
It’s good to see someone talking about assessment-OER, but wouldn’t have been nice for it to go the other way? I would have preferred OER to be talking about it in more than a cursory way 10+ years ago. Water under the bridge. Now that we’re here…
Western Governors has been working the middle ground between OER (and other resources) and accreditation for years. From a blog post I wrote in response to one of David’s:
“If you look at the truly innovative institutions (e.g. WGU), they have shifted the gate-keeping mechanism from credit-for-participation-in-information-distribution (the role of lecture and the lecture-hall walls) to credit-for-evidence-of-proficiency (assessments). This model affords greater efficiency in instruction without sacrificing the gate-keeping (read: profit-generating) responsibilities of the institution.” http://brownelearning.org/blog/2010/03/why-do-lectures-prevail/—On “browsing history as assessment”:Even if “predictive validity” (a term depreciated in the “latest” APA/AERA/NCMA standards) is established between browsing history and college preparedness, I would be *very* hesitant to try to replace formal testing with it. The SAT already inherits a lot of bias form its criterion; I’d bet browsing history would inherit even more.
Let’s try this again (apparently, my pasting caused some formatting errors…):
It’s good to see someone talking about assessment-OER, but wouldn’t it have been nice for it to go the other way? I would have preferred OER people approaching assessment people about it years ago. Water under the bridge. Now that we’re here…
Western Governors has been working the middle ground between OER (and other resources) and accreditation for years. From a blog post I wrote in response to one of David’s:
“If you look at the truly innovative institutions (e.g. WGU), they have shifted the gate-keeping mechanism from credit-for-participation-in-information-distribution (the role of lecture and the lecture-hall walls) to credit-for-evidence-of-proficiency (assessments). This model affords greater efficiency in instruction without sacrificing the gate-keeping (read: profit-generating) responsibilities of the institution.” http://brownelearning.org/blog/2010/03/why-do-lectures-prevail/
—
On “browsing history as assessment”: Even if “predictive validity” (a term depreciated in the latest APA/AERA/NCMA standards) is established between browsing history and college preparedness, I would be *very* hesitant to try to replace formal testing with it. The SAT already inherits a lot of bias form its criterion; I’d bet browsing history would inherit even more.
Isn’t there a difference between what you have read, browsed, viewed and what you know, understand or can do?
David, You are one step ahead…as always… 😉
Assessment will give sustainability to the badge system.
David’s slides (linked above) are also a must view. They are quick to click through and read (2-3 minutes).
My favorite slide #55: “We can’t condemn next-gen assessment to selected response.”
Fully agreed.
I fear to question how many brilliant minds have been halted or deterred from academic and intellectual progression due to an inaccurate representation (ACT and SAT) of their ability to succeed in higher education. I know I’ve met at least 50 of such individuals.
I took your advice and looked through the slides. I would have been more direct in my original comments had I known these were David’s ideas. I went easy because I thought he was summarizing other presenters’ ideas.
People who fear constrict-response items need to take a lesson from Donald Campbell who had similar misgivings about self-reported data. He struggled for a decade to find a better way before concluding that there wasn’t one. Rather, it was better to acknowledge the format’s shortcomings and mitigate them. We have more theory and empirical evidence to help us interpret constricted-response items than we do any other assessment format. By all means, we should explore alternatives, but until those alternatives have sufficient evidence, they should not replace the most rigorously developed means of assessment.
I’m sorry for those whose development may be impeded by the format of an assessment. They should have an avenue to appeal, etc. But replacing one assessment method with another *without changing the policies of how the results are used* will not change anything. This is clear from a century of experience in the medical licensure community, as told by Swanson, Norman, & Linn (1995). As those authors state, “All high-stakes assessments, regardless of the method used, have an impact on teaching and learning.”
Jeremy, due to the emotion of your response I’m having some difficulty deciphering your intent and am somewhat hesitant to respond.
I take it you are saying, “Research-based, high stakes tests (such as the ACT/SAT) should not be replaced with other forms of assessment that are not research based.”
Do you agree with this statement?
I think you are trading one problem for another. I am not sure how the whole badge system makes anything better. I am not a fan of the current system, but this does not seem like a viable option right now.
I would point out that assessment has some alternative meanings, particularly when modified as formative assessment. I agree that data trails can generate excellent evidence of accessing educational resources and even perhaps of some engagement with them. But assessment as I and many other educators understand the term implies deep learning engagement in which the assessment is the key mechanism for the achievement of learning outcomes. OER that is built around tasks, inquiry, and which actively produces collaborative learning will produce the kind of outcomes which we seek. Without it, I suspect the future is a little less rosy.
We can’t use tests designed to demonstrate knowledge learning to determine skill acquisition. High stakes exams like ACT or GRE are part of our accepted college entrance qualification system because they purport to demonstrate how well a student has learned knowledge, ia. a given set of curricula. The ability to authentically demonstrate an individual student’s increased knowledge in a given field may be imprecise in even the best such assessments. We have accepted this for the most part and yet continue to depend on these assessments. End of year exams that states administer are similar. Credentialing exams are different. They seek to authentically demonstrate an individual’s increased competence in a given skill, taking for granted any knowledge that must have been gained to develop that skill. Showing a list of sites one has clicked on may contribute to the perception that one has taken time to access knowledge but doesn’t seem to me to be in and of itself a viable way of determining actual competency in a skill. Personally, I’d like to see a version of the old tradesman system of apprenticeship, journeyman etc. where one’s competence was attested to by an acknowledged master. When one’s Jedi master has presented the young Padwan learner to be subjected to questions from the Jedi Council, then one may be determined ready.