More on Badges and Assessment

With apologies to psychometricians who may read, let me set some vernacular context for additional thoughts (prompted originally by Dan Hickey‘s, and then Alex Halavais’, writing) regarding my own thinking on badges and assessment.

It is beyond argument that we cannot crack open a learner’s head, insert a magnifying glass, and make direct, error-free observations of what the learner “knows.” Since we can’t actually take a “direct” measure of what someone knows, we collect evidence that allows us to increase or decrease our beliefs about the likelihood that they know, or are able to do, something.

For example, I can’t fMRI your brain in order to see if you are able to multiply two three digit numbers. However, if you successfully multiply two three digit numbers I will start to believe that you know how to do it. If you do it three times in a row without making a mistake I will believe it more. If you do it 100 times in a row without error I will have a very strong belief that you “know how” to multiply these kinds of numbers.

The same, high-level “direct measurement is impossible so we settle for gathering evidence” argument applies to all sorts of knowledge and skills, from multiplying, to naming state capitals, to troubleshooting TCP/IP networks, to arranging orchestral scores, to interpreting and critiquing a new philosophical work.

Assessment, then, is about having people engage in activities that provide this kind of evidence.

This evidence can be used in a number of ways. It can lead others to believe that you are qualified for employment, or it can lead others to believe that you will succeed in graduate school. It can lead you to believe that you don’t need to study for the final exam anymore, or it can lead you to believe that you’re ready to sign up for that Udacity class. How evidence is used and who it is used by is a related – but separate – issue from the extremely thorny process of helping learners create the most valid, reliable body of evidence possible.

To me, a badge – which strictly speaking is a few lines of JSON and a PNG image – is a form of evidence. However, these two files stored on a server are clearly NOT an activity (like writing a 1000 word compare and contrast essay) that results in evidence.

If you think about it, not only is the badge not an activity, it is also not the evidence (e.g., artifact) directly created by engaging in the activity. The activity of comparing and contrasting the North and South’s motivations for engaging in the civil war does not result directly in a badge. This process results directly in an essay (for example). After a learner has engaged in the activity and created the evidence, someone judges the essay and then represents their beliefs about what the person knows – based on the evidence – by awarding or not awarding a badge.

Those of you who have poked around in the JSON know that the word “evidence” is used in exactly this way inside the badge file. The common way of thinking about this in the badge world is “Ms. Third Party, if you don’t believe the person really deserved this badge you can click through and look at the evidence yourself!” But -importantly – this is the same evidence that led a different third party to believe that the learner deserved the badge in the first place.

So that’s a lot of explanation to say that we design (1) an activity, which results in (2) evidence, which is (3) judged, and if judged sufficient is awarded a (4) badge.

You see that a badge is a proxy for evidence, which evidence itself is a proxy for what a person “actually knows or can do.” We provide second-order proxies like badges, GPAs, and ACT scores so that every future person who is interested in your ability doesn’t have to grade your essays, review your portfolios, and view the video of your
performance assessment themselves. While these second-order proxies provide lossy compression (they contain less detail and less information), they greatly increase the efficiency of decision making processes later on. Imagine trying to narrow a 300 person applicant pool without these second-order proxies (with only access to their original evidence / artifacts).

All this rambling to say that I hope that as a community we will commit to being agnostic with regard to (1) the activity, (2) the evidence, and (3) the judgment. Regardless of whether these three steps are radically modern or terribly traditional, there is no a priori reason that any arbitrary configuration of these could not result in a (4) badge. In fact, the technical approach Mozilla has taken to badges assures that this agnosticism is possible. Only social pressure could close this door Mozilla has architected open.

This is what I was trying to get at the other day when I said that badges are credentials and not assessments. To me, an assessment is the (1) the activity, (2) the evidence, and (3) the judgment. Whether the “thing” awarded out the back end of that process is a grade, a certificate, a pat on the back, or a badge, these second-order proxies are credentials and not the actual assessments. Perhaps this is just a difference in terminology. I hope to find out…