With apologies to psychometricians who may read, let me set some vernacular context for additional thoughts (prompted originally by Dan Hickey‘s, and then Alex Halavais’, writing) regarding my own thinking on badges and assessment.
It is beyond argument that we cannot crack open a learner’s head, insert a magnifying glass, and make direct, error-free observations of what the learner “knows.” Since we can’t actually take a “direct” measure of what someone knows, we collect evidence that allows us to increase or decrease our beliefs about the likelihood that they know, or are able to do, something.
For example, I can’t fMRI your brain in order to see if you are able to multiply two three digit numbers. However, if you successfully multiply two three digit numbers I will start to believe that you know how to do it. If you do it three times in a row without making a mistake I will believe it more. If you do it 100 times in a row without error I will have a very strong belief that you “know how” to multiply these kinds of numbers.
The same, high-level “direct measurement is impossible so we settle for gathering evidence” argument applies to all sorts of knowledge and skills, from multiplying, to naming state capitals, to troubleshooting TCP/IP networks, to arranging orchestral scores, to interpreting and critiquing a new philosophical work.
Assessment, then, is about having people engage in activities that provide this kind of evidence.
This evidence can be used in a number of ways. It can lead others to believe that you are qualified for employment, or it can lead others to believe that you will succeed in graduate school. It can lead you to believe that you don’t need to study for the final exam anymore, or it can lead you to believe that you’re ready to sign up for that Udacity class. How evidence is used and who it is used by is a related – but separate – issue from the extremely thorny process of helping learners create the most valid, reliable body of evidence possible.
To me, a badge – which strictly speaking is a few lines of JSON and a PNG image – is a form of evidence. However, these two files stored on a server are clearly NOT an activity (like writing a 1000 word compare and contrast essay) that results in evidence.
If you think about it, not only is the badge not an activity, it is also not the evidence (e.g., artifact) directly created by engaging in the activity. The activity of comparing and contrasting the North and South’s motivations for engaging in the civil war does not result directly in a badge. This process results directly in an essay (for example). After a learner has engaged in the activity and created the evidence, someone judges the essay and then represents their beliefs about what the person knows – based on the evidence – by awarding or not awarding a badge.
Those of you who have poked around in the JSON know that the word “evidence” is used in exactly this way inside the badge file. The common way of thinking about this in the badge world is “Ms. Third Party, if you don’t believe the person really deserved this badge you can click through and look at the evidence yourself!” But -importantly – this is the same evidence that led a different third party to believe that the learner deserved the badge in the first place.
So that’s a lot of explanation to say that we design (1) an activity, which results in (2) evidence, which is (3) judged, and if judged sufficient is awarded a (4) badge.
You see that a badge is a proxy for evidence, which evidence itself is a proxy for what a person “actually knows or can do.” We provide second-order proxies like badges, GPAs, and ACT scores so that every future person who is interested in your ability doesn’t have to grade your essays, review your portfolios, and view the video of your
performance assessment themselves. While these second-order proxies provide lossy compression (they contain less detail and less information), they greatly increase the efficiency of decision making processes later on. Imagine trying to narrow a 300 person applicant pool without these second-order proxies (with only access to their original evidence / artifacts).
All this rambling to say that I hope that as a community we will commit to being agnostic with regard to (1) the activity, (2) the evidence, and (3) the judgment. Regardless of whether these three steps are radically modern or terribly traditional, there is no a priori reason that any arbitrary configuration of these could not result in a (4) badge. In fact, the technical approach Mozilla has taken to badges assures that this agnosticism is possible. Only social pressure could close this door Mozilla has architected open.
This is what I was trying to get at the other day when I said that badges are credentials and not assessments. To me, an assessment is the (1) the activity, (2) the evidence, and (3) the judgment. Whether the “thing” awarded out the back end of that process is a grade, a certificate, a pat on the back, or a badge, these second-order proxies are credentials and not the actual assessments. Perhaps this is just a difference in terminology. I hope to find out…
I’m a bit sceptical about the concept of Badges. I think one major factor IN that, is benchmarking. If you have 20 badges, I might not have time to click through to view your evidence. So how do I know the badge you have is at the same level as the badge I have?
I think there will be three main “solutions” that emerge:
1) The knowledge gained while earning badges will be a differentiator. If you’re interviewing me, you might not care about or understand my badges, but you will notice and care that I know more.2) Having -any- badges will become a differentiater, without needing to assess the badges themselves. If you have two job candidates, and one has many badges that seem to be related to the job while the other has no badges and no apparent self development, then you’ll be inclined to learn toward the candidate that seems to be continuing their learning.
3) More reputable/renowned badges will emerge. Existing assessment firms will eventually move into the badge space. They’ll start by offering badges alongside the certifications they already provide, and eventually may move to offer more variety – badges for individual courses, etc.
If you restrict it to a narrow situation, then yes, it will always be difficult to assess unknown badges. Assessing unknown honors (degrees from unknown institutions, awards from unknown organizations) has always been a challenge, but in reality employers are able to draw on additional context or research to get past the unknown…badges will be no different.
Peter,
The same could be said for GPA or degree. How do I know your bachelors degree in business is equivalent to my bachelors degree in business?
Well in the UK we have (some) benchmarking for degrees through the Higher Education Qualification Framework. This would provide level descriptors to which all degree schemes (and each year within) should meet, and University quality procedures would further ensure this. These validation panels normally include external academics from other Universities to help ensure validity.
In relation to school and college level qualifiations; the assessments are normally set from an examining body, conducted locally and papers sent to external examiners.
So this all adds some assurances related to quality of qualification. Surely if our higher education system didn’t have some type of benchmarking for qualifications they would be absolutely useless!
David–
This is really helpful post and the exchange has been quite enlightening. Yes indeed much of this is terminology, or perhaps referents. You are correct that a badge is ultimately a few lines of code. My references to “badges AS” aims to get at the broader social practices that are enacted around badges.
I think that this document from Educause will likely serve as the final word for now. I understand that this wording was negotiated by a bunch of folks who are involved in initiative. There definition is quite consistent with yours:
Badges are digital tokens that appear as icons or logos on aweb page or other online venue. Awarded by institutions, organizations,groups, or individuals, badges signify accomplishmentssuch as completion of a project, mastery of a skill, or marks ofexperience. Proponents suggest that these credentials herald afundamental change in the way society recognizes learning andachievement.
More at http://net.educause.edu/ir/library/pdf/ELI7085.pdf
Dan
Good post and useful clarifications made. In general, Badges are SIGNS to indicate to the world that the badge-holder has met certain specific objectives. The process of meeting those objectives involves learning actions and assessments, and after one meets the objectives, one gets a Badge to conveniently convey this achievement to anyone in the future.
A badge is a credential. A credential should not abstract or vague. It should be able to completely convey WHAT objectives were met, HOW WELL they were met and IN WHAT MANNER the assessment was made. (e.g. in the case of Coursera MOOCs, multiple-choice quizzes can be quite tough if only 1 attempt is allowed, and that too within 1 timed session, as compared to another quiz in which 3 attempts are allowed, with same questions asked every time, info about correct/incorrect result revealed after every attempt, and all this being un-timed). So, the badge must reveal not only the quality of the badge-holder, but also the quality of the underlying assessment (or in other words, its own quality). We could call this the “meta-assessment”. This will make it simpler to compare different badges issued for similar-sounding objectives, and provide a context for making good decisions.
More ramblings about creating awesome open universal learning credentials and making OpenBadges revelant can be found here: https://medium.com/@alphabeta_sociologist/how-to-make-openbadges-relevant-and-useful-a-brief-rant-fd96791576fa
Thanks and happy hacking education! 🙂
@alphabetasocio:disqus