Last week, OpenAI published Tips for Teachers in a promotional blog post showing how some teachers are using ChatGPT as an educational aid, along with suggested suggestions for getting started. In a related FAQ, they officially admit what we already know: AI writing detectors don’t work, despite being used to punish students with frequent false positives.
“Do AI detectors work?” In a section of the FAQ titled, OpenAI writes, “In short, no. Some (including OpenAI) have released tools intended to detect AI-generated content, but none of these have been proven to reliably distinguish between AI-generated and human-generated content. .”
In July, we delved into why AI write detectors like GPTZero don’t work, with experts calling them “mostly snake oil.” These detectors often produce false positives due to their reliance on unproven detection metrics. After all, AI-written text doesn’t always have anything special that distinguishes it from human-written text, and detectors can be defeated by rephrasing. That same month, OpenAI shut down its AI Classifier, an experimental tool designed to detect AI-written text. It had a whopping 26 percent accuracy rate.
OpenAI’s new FAQ clears up another big misconception, that ChatGPT itself can tell if text is AI-written. OpenAI writes, “Furthermore, ChatGPT has no ‘knowledge’ of what content can be AI-generated. It sometimes asks ‘Did you write this (essay)?’ or ‘Could this have been written by AI?’ These responses are random and have no basis.”
Along those lines, OpenAI also addresses the tendency of its AI models to produce inaccurate information, which we’ve covered in detail here at Ars. “Sometimes, ChatGPT sounds convincing, but it can give you false or misleading information (often called ‘illusions’ in the literature),” the company writes. “It can also make things like quotations or citations, so don’t use it as your only source for research.”
(In May, a lawyer ran into trouble doing just that—citing six nonexistent cases he pulled from ChatGPT.)
Even if automated AI detectors don’t work, that doesn’t mean a human can never detect AI writing. For example, a teacher familiar with a student’s particular writing style can tell when their style or ability suddenly changes. Also, some sloppy attempts to pass off AI-generated work as human-written can leave tell-tale signs, such as the phrase “AI language model” meaning someone carelessly copied and pasted the ChatGPT output. And recently, an article in the scientific journal Nature showed how humans noticed the phrase “regenerate response” in a scientific paper labeled a button in ChatGPT.
As technology stands today, it is safest to avoid automated AI detection tools altogether. “Until now, AI writing is undetectable and likely to remain that way,” frequent AI analyst and Wharton professor Ethan Mollick told Ars in July. “AI detectors have high false positive rates and consequently should not be used.”