More on Fuchs & Woessmann

Am I biased in my perspectives? Absolutely. Is there anyone who is not? No. So, now that I’ve owned up wholly and completely to the accusation that I am biased in my reading and interpretation of the study in question…

Commenting on my previous post about the Fuchs & Woessmann study, Mihaly Nyary writes: “You have an agenda, and not the authors,” and provides the abstract to the study, which is in no way whatsoever related to the point I made yesterday. Let me try once more, and make a handful of other points along the way.

According to the article abstract, which Mihaly quote in his comment, the study authors’ method is to “estimate the relationship between students’ educational achievement and the availability and use of computers at home and at school in the international student-level PISA database.” This just seems inappropriate, as I started to describe in my last bit of writing. Why? I’ll try to be a bit more detailed here.

First of all, what possible, reasonable hypothesis could there be in which the mere availability of anything impacted learning? Would the availability of human experts (not interaction with them, now) improve learning? Would the availability of books? Would the availability of anything? If simple proximity to anything were going to correlate with learning outcomes, I would go back to sleeping with books under my pillow. This hypothesis was ill-begotten to start and muddies whatever actual message the authors want to make. Its inclusion in the report suggests to me that there was a pre-existing bias toward finding no or negative results.

Second, when the study does look at “use” of computers in finding relationships with learning, it looks at “use” so generically as that we find no good reason to expect this activity to contribute to learning outcomes. We have almost no idea how the students are using the computers.

The first construct, “internet access,” is examined from two perspectives:

(On the positive side) internet access can help students exploit enormous information possibilities for schooling purposes and increase learning through communication… (On the negative side) internet access could offer distraction by chat rooms or online games, reducing the time spent on doing homework or learning.

Internet access at home is correlated positively with outcomes in math and reading. Internet access at school is negatively correlated. SO WHAT? What are they doing when they access the internet? Its so vague as to be equivalent to “teaching is positively correlated with learning at home but negatively at school”. What does use mean???

A related construct, “use of email and webpages,” is reported and evaluated with respect to learning outcomes. The study compares a group who “never or hardly ever” use email and webpages with a group that uses them “several times a week.” Does use of webpages mean finding and viewing pornography? Using BitTorrent tracker websites to begin peer-to-peer downloads of movies, tv shows, software, and music? As these two web-based activities account for the majority of traffic on the Internet, it would make sense to assume that they account for at least some of what this group of teenagers is doing online. (Not to mention the slew of web-based Flash games to which so many are completely addicted.) When we know nothing beyond the fact that students launch their browsers or open their email clients, why would we expect to find that learning occurs?

Finally, I have to make a few quick points about the “educational software” category of computer “use”. First, what instructional strategies do these titles supposedly employ that would lead us to believe they should facilitate learning? Again, lumping “educational software” all into one category is like lumping “teaching” all into one category. What specifically is the teacher doing? But we get none of this absolutely crucial detail. My final point is… self-report data from 15 year olds on whether or not they have educational software at home (yes, the article indicates that this is how it was gathered)?

I could go on. As my pal Andy Van Schaack is fond of reminding me, Jay Gould has commented that confusing correlation for causation is just about the worst mistake you can make when dong research. And yet this study seems to frequently flirt with this issue:

Thus, the mere availability of computers at home seems to distract students from learning, presumably mainly serving as devices for playing computer games.

Coming back to the point I made in yesterday’s post, I believe there is a perfectly plausiable alternative explanation for the negative correlation the researchers find between their constructs of computer use and academic achievement, in which teenagers who used computers the most saw their test scores dip at the right hand side of a graph the authors describe as an “inverted U shape.”

As the authors report on page 6 of the article (page 8 of the pdf):

The PISA study tested the students with paper and pencil tests, which lasted two hours for each student.

Now, lets pretend we are examing a group of 15 year olds, some of whom spend none of their educational and recreational time on computers, some of whom spend some of their educational and recreational time on computers, and some of whom spend much of their educational time and recreational time on computers. Now, let’s imagine two scenarios in which we might measure the academic achievement of a sample of this group of 15 year olds. In one scenario, we will carry out the assessment on computers. In the other scenario, we will carry out the assessment in “more traditional” manner. Can we not form a strong hypothesis, ahead of data collection or analysis, about which sub-group will perform best in each scenario? Of course, the authors didn’t gather this data themselves, they used existing data from the 2000 administration of the PISA.

Some may argue that ETS’ willingness to administer very high stakes standardized tests via computer might indicate that no significant problems due to contextual interference could really exist. I am told there is ETS research data that implies this is the case. For the middle 95% of people I can believe that would be the case. But working from my own “sample of one” (insert snide remark here) I know that results from any two hour paper and pencil test I will ever take from this point on will be seriously distorted from a pure lack of writing stamina, due to the fact that the most paper and pencil writing I ever do is sign my name. I will be very anxious to see how the ETS research deals with individuals like me, who live on computers and lack the physical ability to write for two hours.

The part of the study I came closest to agreeing with:

the inverted U-shape of the relationship between computer and internet use at school and student performance may be due to a true causal effect of computer and internet use. In this case, some computerized instruction would be beneficial for student learning, constituting a valuable input in the students’ learning process. Only at higher intensities of computer and internet use at school would the negative effects of computer and internet use set in, in terms of crowding out more effective methods of teaching…

Of course, here we get the causailty / correlation issue arising again, but that’s not the point. The point I agree with is, of course, that interaction with computers should be only a part of the learning experience. Learning wants to be partly social as well, not 100% of interacting with instructional artifacts.