Instructional-Design

tl;dr - Go play around with generativetextbooks.org and let me know what you think. Earlier this year I began prototyping an open source tool for learning with AI in order to explore ways generative AI and OER could intersect. I’m specifically interested in trying to combine the technical power of generative AI with the participatory power of OER, in order to both increase access to educational opportunity and improve outcomes for those students who access it. I did some preliminary writing on this topic back in July of 2023, calling the artifacts that result from combining generative AI and OER “generative textbooks” and have continued to ruminate on the topic. ...

“Generative AI models don’t understand, they just predict the next token.” You’ve probably heard a dozen variations of this theme. I certainly have. But I recently heard a talk by Shuchao Bi that changed the way I think about the relationship between prediction and understanding. The entire talk is terrific, but the section that inspired this post is between 19:10 and 21:50. Saying a model can “just do prediction,” as if there were no relationship between understanding and prediction, is painting a woefully incomplete picture. Ask yourself: why do we expend all the time, effort, and resources we do on science? What is the primary benefit of, for example, understanding the relationship between force, mass, and acceleration? The primary benefit of understanding this relationship is being able to make accurate predictions about a huge range of events, from billiard balls colliding to planets crashing into each other. In fact, the relationship between understanding and prediction is so strong that the primary way we test people’s understanding of the relationship between force, mass, and acceleration is by asking them to make predictions. “A 100kg box is pushed to the right with a force of 500 N. What is its acceleration?” A student who understands the relationships will be able to predict the acceleration accurately; one who doesn’t, won’t. ...

All of the analysis, design, and development that happen in conjunction with retrieval augmented generation (RAG) and fine-tuning of large language models is instructional design. The instructional design process often begins with an analysis of the learner(s). What do they already know? What skills do they already have? What prior knowledge can we assume will be there, so we can design our instruction in a way that successfully builds on their understanding? In the case of LLMs, “prior knowledge” is the set of capabilities models get from their pre-training. When we begin the process of prompting, fine-tuning, or setting up retrieval augmented generation (RAG), we’re not starting from scratch. The first step is understanding the set of capabilities in your base model the same way you would want to understand the prior knowledge of your target learners. ...

Large language models (LLMs) make it possible for faculty to rapidly create a wide range of formative and summative assessments for their students. And, as we hear about so often, students can also use LLMs to write their essays and complete other assignments. (Apparently, when faculty use AI to create assignments, it’s a “productivity gain.” But when students use AI to complete assignments, it’s “cheating.” But that’s a topic for another day.) Reflecting on several conversations I’ve been part of at the SUNY CIT conference this week led me to realize an important principle about the symmetry of AI in assessment. LLMs are equally powerful tools for both faculty and students. Speaking solely about what is technologically possible (and not what is ethically appropriate), we might summarize this principle by saying: ...

It is absolutely critical that everyone who cares about technology-mediated learning understand this point. There is a seismic shift in perspective necessary from pre-generative AI technologies to generative AI technologies. It requires changes in the way we think about everything - from pedagogy to supporting infrastructure. I’ve been writing and speaking about this for months now, and I’m not alone. Here’s how the CEO of Groq put it: “Think about it this way: we were in an information age where you would make copies of data with high fidelity and you’d distribute it. That’s what the internet was. That’s what mobile was. But that’s also what the printing press was. They’re effectively the same type of technology, just at a different scale. And even though it was the same type of technology at a different scale, even that was hard for our intuitions to adapt to. But generative AI is not an information age technology - because you’re not making copies of something. You’re making something new in the moment. And the difference is when you’re making something new and in the moment you need *compute* to do that. It’s not about retrieving something from a hard drive, and doing a little bit of compute and sending it out. You’re creating it in response to a particular question.” ...

The earlier we all start thinking about this problem, the sooner we can start generating ideas and potential solutions. Given the magnitude of impact generative AI is having and will have in education (and many other aspects of life), I’m working with some diligence to keep up to date with developments in the field. Recently, I noticed how a couple of the emerging capabilities of generative AI will come together in the future in a way that will impact education much more dramatically than I am hearing anyone talking about currently (if I’m missing this conversation somewhere, please help me connect to it!). But before I give away the punch line, let me share the individual pieces. Maybe you’ll see what I saw. ...

As I’ve been (re-)reading OER adoption research through a more critical lens I’m seeing a recurring pattern of significant threats to validity in the designs of studies purporting to measure the impact of OER adoption on student outcomes. While there are numerous methodological issues to consider, in this essay I’ll discuss three. Specifically, I’ll share: three questions you should ask when you read research about the impact of OER adoption on student outcomes, the reasons why you should care about the answers to those questions, and the questions the study is really addressing when the answer to any of the three questions is “no.” Three questions to ask when you read research on the impact of OER adoption Ask yourself these three questions when you read OER adoption research - especially research that claims to find a positive impact on student outcomes. ...

Preparing to participate in a panel on generative AI and education at this week’s AECT convention gave me the excuse to carve out some dedicated time to think about the question, “how would you summarize the impact generative AI is going to have on education?” This question is impossible to answer over the medium to long-term, but maybe I could give an answer addressing the near-term? My approach to this question was to look for a different, comparable example and try to work my way into the question from that more familiar territory. The internet seems like the obvious choice here, as no other recent advance can even begin to compare to the potential impact generative AI will have. ...

I’ve limited myself to one hour of writing for this post, so it’s more a collection of thoughts than a coherent narrative. I expect I’ll have lots more to say on this topic in the future. For now, I just want to get the beginnings of this idea out into the world, together with some initial implications. Since ChatGPT’s meteoric rise to popularity, I’ve constantly been amazed by the creative ways people have imagined to make use of generative AI. For months now it feels like each week I read about some new way generative AI is being used that just completely blows my mind. For example, my son who is studying cybersecurity recently told me prompted ChatGPT along these lines: ‘You are the hiring manager for a cybersecurity position at a large company. Conduct a mock interview with me, asking me ten conceptual questions. After I have answered all the questions, give me feedback on my answers. Then ask me ten technical questions. After I have answered all the questions, give me feedback on my answers again.’ My son tells me this exercise was incredibly helpful and he plans to do it several more times before having a “real” interview. ...

2022 saw some significant advancements in artificial intelligence. My threshold for “significant” here being that the advances moved out of labs and arXiv.org preprints and into tools that many people were using and talking about. Lots of people thought text-to-image tools like DALL-E, Stable Diffusion, and Midjourney were fun. But Large Language Models (LLMs), and particularly the recent demo of ChatGPT, seem to have put the fear of God into everyone from middle school English teachers to the CEO of Google. The potential partnership between OpenAI (the makers of ChatGPT) and Microsoft may even present the first substantive challenge to Google’s search monopoly we’ve ever seen - and that’s saying something. While most of the dialog around AI and education seems to be focused on assessment, I think the implications for instructional designers are critically important, too. And, because you’ve got to play the hits, let’s look at what their impact will be on OER as well. ...