I love the title of this article by Jessica Grose:

Every Tech Tool in the Classroom Should Be Ruthlessly Evaluated

It makes me think of a gardener ruthlessly pruning roses in a garden, pulling out weeds, making space for growing things. Ok, that’s the extent of my gardening knowledge.

Pruning Digital Tools and Failed Instructional Strategies from Classrooms

Whether it’s ineffective technology or failed instructional strategies, it’s time to stop doing what doesn’t work. Don’t you agree?

As we see a wave of AI tools finding their way into classrooms, schools and districts, you have to wonder, Where’s the evidence? Again and again, we keep coming back to digital tools to somehow accomplish something they weren’t designed for from the get-go: Ensuring deep learning happens, and that students are able to apply that learning to new situations.

How Can We…?

Kassandra Rios asks this question in a post on the Elon Musk owned network:

How can educators teach students to use AI in the classroom? #EdTech #teaching #learning #EdTechInTheClassroom #AI #education

At the time, my response included the following:

There are several different types of instructional strategies aligned to surface, deep, or transfer learning. For students in surface to deep learning, I don’t recommend AI since you want to build student critical thinking. I see definite applications during transfer learning, where students are applying what they have learned to new, novel situations for rapid prototyping (e.g. quickly trying out new solutions or getting AI to anticipate results).

Then, I mention an article I wrote on Flexible Grouping. But the more I think about it, the less convinced I am at AI’s utility to directly impact student learning. Most of the edtech tools I see are really productivity tools, information management focused. AI tools definitely fall into this category. I can see their use in the classroom by teachers, but for students, that means short-circuiting or denying them the learning and critical thinking they need to be engaged in.

A key quote from Jessica Grose’s article is this one:

And while there are certainly benefits to using tech as a classroom tool, I’m convinced that when it comes to the proliferation of tech in K-12 education, we need “a hard reset,” as Julia Freeland Fisher of the Christensen Institute put it….

We need a complete rethink of the ways that we’re evaluating and using tech in classrooms; the overall change that I want to see is that tech use in schools — devices and apps — should be driven by educators, not tech companies.

What if we COULD do a hard reset, STOP PAYMENT on all the recurring payment technology tools in schools. There’s certainly competing priorities for that funding. After all, how much would a district-wide LMS cost a 50K student school district? At least one full-time professional educator, no? And, what about the investment in computer labs and portable labs used for high-stakes assessments? From my own experience, the cost is considerable…from a few hundred thousand to millions every few years, depending on the size of the school district.

But the goal isn’t to get rid of technology everywhere. It IS to re-evaluate, “ruthlessly,” as Grose puts it, the technology that we allow in the classroom that directly impacts instructional efforts. Those tech tools, or digital tools, should be put on a list, and then evaluated for evidence. “What’s the scientific consensus on the use of XYZ popular tool in the classroom? Anecdotal evidence isn’t allowed.”

I suppose that the truth is, a better response to Kassandra’s question is, “What evidence is there that shows the efficacy of AI for use by students?”

The Artificial Intelligence Playbook

In a new book, The Artificial Intelligence Playbook, the authors say the following:

In this spirit, we invite you to consider AI as a virtual teaching assistant that can provide support to you. We hope that the tools in this playbook provide teachers with more opportunities to directly interact with their students, which not only is the reward of teaching but also is irreplaceable by a computer.

In the description, the focus is on AI as a virtual teaching assistant. I can definitely see the value of this for educators. In their virtual book jacket (a book I just started reading, BTW), the authors (or is it the publishers?) tout the book as highlighting:

  • Compose Writing Prompts and Avoid Plagiarism
  • Manage Content
  • Foster Student Engagement
  • Meet Students’ Instructional Needs
  • Assess Student Learning
  • Continue Lifelong Learning

Again, I can’t argue with those uses of AI. All of that is certainly made easier by AI (except the “Avoid plagiarism”…AI detectors are notoriously bad…see the scientific consensus with citations on them at the end of this post). The question isn’t whether teachers can use AI to make their jobs easier and less onerous.

Rather, the question is, “What evidence is there that supports using AI to impact student learning?”

And, so far, the evidence is lacking. Unless you consider that, so far, AI is being hyped as the cure to education ills. Yet, we already know what works in schools. More technology isn’t it. Instead, it’s high-effect sized instructional strategies that work.

Scientific Consensus on AI Writing Detectors

The following is taken from Perplexity Pro’s response to my prompt:

What is the scientific consensus on the use of AI detectors to stop students from having AI chatbots compose their writing?

As you read the response below, ask yourself, “Is my school or district investing in these ineffective technologies?”

Response

The scientific consensus on the use of AI detectors to stop students from having AI chatbots compose their writing suggests that these detectors are not entirely reliable.

According to the research, AI text classifiers, such as GPTZero and OpenAI’s own AI text classifier, have been found to misclassify a significant portion of human submissions as AI-generated and vice versa.

For instance, OpenAI’s Text Classifier misclassifies 5% of student submissions as AI-generated, and 49% of ChatGPT’s submissions as human-generated.

GPTZero has a higher false positive rate of 18%, but a lower false-negative rate of 32%[5].

Moreover, these detectors are susceptible to obfuscation attacks. A simple attack involving running the ChatGPT-generated text through a paraphrasing tool like Quillbot significantly increased the false-negative rates for both classifiers. After such an attack, the false-negative rate for OpenAI’s text classifier jumped from 49% to 98%, and for GPTZero, from 32% to 95%[5].

These findings indicate that while AI detectors can be a part of the strategy to identify AI-generated text, they cannot be solely relied upon due to their limitations in accurately detecting AI-written content and their vulnerability to simple manipulation techniques. The consensus suggests that a more nuanced approach, possibly combining AI detectors with critical human evaluation and changes in assessment strategies, might be necessary to effectively address the issue of AI-generated submissions in academic settings[2][3][5].

Citations: