Learning

For those interested in issues around agentic AI and assessment, I’m excited to announce the launch of the CHEAT Benchmark (https://cheatbenchmark.org/). The CHEAT Benchmark is an AI benchmark like SWE-Bench Pro or GPQA Diamond, except this benchmark measures an agentic AI’s willingness to help students cheat. By measuring and publicizing the degree of dishonesty of various models, the goal of this work is to encourage model providers to create safer, better aligned models with stronger guardrails in support of academic integrity. ...

It is absolutely critical that everyone who cares about technology-mediated learning understand this point. There is a seismic shift in perspective necessary from pre-generative AI technologies to generative AI technologies. It requires changes in the way we think about everything - from pedagogy to supporting infrastructure. I’ve been writing and speaking about this for months now, and I’m not alone. Here’s how the CEO of Groq put it: “Think about it this way: we were in an information age where you would make copies of data with high fidelity and you’d distribute it. That’s what the internet was. That’s what mobile was. But that’s also what the printing press was. They’re effectively the same type of technology, just at a different scale. And even though it was the same type of technology at a different scale, even that was hard for our intuitions to adapt to. But generative AI is not an information age technology - because you’re not making copies of something. You’re making something new in the moment. And the difference is when you’re making something new and in the moment you need *compute* to do that. It’s not about retrieving something from a hard drive, and doing a little bit of compute and sending it out. You’re creating it in response to a particular question.” ...

If you’re not listening to the Latent Space podcast, you’re missing some of the best thinking on generative AI happening right now. The show notes for a recent episode begin, Stop me if you’ve heard this before: “GPT3 was trained on the entire Internet”. Blatantly, demonstrably untrue: the GPT3 dataset is a little over 600GB, primarily Wikipedia, Books corpuses, WebText and 2016-2019 CommonCrawl. The Macbook Air I am typing this on has more free disk space than that. In contrast, the “entire internet” is estimated to be 64 zetabytes, or 64 trillion GB. So it’s more accurate to say that GPT3 is trained on 0.0000000001% of the Internet. ...

Recently I’ve been thinking about the difference between an informational resource and an educational resource. I’ve had the sense that an educational resource is an informational resource with a little something extra and have enjoyed coming back to this thought again and again over the last several weeks, trying to reduce this “something extra” to its simplest form. Keeping the discussion informal, it seems that an informational resource is simply a compilation or collection of information - ideas, facts, processes, procedures, &c. I think of an encyclopedia as being the quintessential information resource - comprehensive, accurate, and well-organized. If you accept that definition (for sake of this argument), what would need to be added to an informational resource to make it an educational resource? ...

There’s a really terrific conversation happening on the cc-openedu listserv. It started out as a question about OER, but has moved on to a conversation about the purposes of open more generally. Dr. Chuck contributed over the weekend, and his contribution provides a great opportunity for me to respond with the first substantive post since I changed the name of the blog. All the pull quotes in this post are from Dr. Chuck. He writes: ...

My blog has changed names twice over the years. Today, after 15 years, it’s changing again. Before I had a “real” blog, I published random thoughts on whatever personal website my then-current university would give me access to, starting in 1993. Remember tilde accounts? (davidwiley.com / davidwiley.org has always (since 1995) been a personal home page, and not a place where I’ve published a lot of writing.) In the early 2000s, Brian Lamb introduced me to RSS and Moveable Type and I put down some blog roots at reusability.org. That first “real” blog was named Autounfocus, in recognition of the way it bounced back and forth between a pretty wide range of topics:A few years later, when Movable Type announced they were considering changing from open source to a proprietary licensing model, I briefly experimented with Plone as my blogging platform. But I quickly ended up moving to Wordpress - and opencontent.org - in February, 2005. At that point I renamed the blog Iterating Toward Openness, and later chose a new byline:Holy smokes! That’s 15 years ago now. How time flies. ...