Testing of Detection Tools for AI‑Generated Text

The article discusses the potential risks associated with the unfair use of AI-generated content in an academic environment and the efforts to detect such content. The authors examine the functionality of various detection tools for AI-generated text and evaluate their accuracy and error types. The study aims to determine if existing detection tools can effectively differentiate between human-written text and text generated by ChatGPT, an AI language model. It also investigates the impact of machine translation and content obfuscation techniques on the detection of AI-generated text.

The research covers 12 publicly available tools and two commercial systems (Turnitin and PlagiarismCheck) commonly used in academic settings. The authors conclude that the available detection tools are not accurate or reliable and tend to classify the output as human-written rather than detecting AI-generated text.

According to the article, the main limitations of the available detection tools for AI-generated text in academic settings are as follows:

1. Lack of accuracy and reliability: The researchers found that the existing detection tools are neither accurate nor reliable in distinguishing between human-written text and AI-generated text. These tools tend to classify the output as human-written rather than detecting AI-generated text, indicating a bias in their performance.
2. Bias towards human-written text: The detection tools show a bias towards classifying the text as human-written rather than identifying AI-generated content. This bias can lead to false negatives, where AI-generated text goes undetected, compromising academic integrity.
3. Negative impact of content obfuscation techniques: Content obfuscation techniques, which are used to make AI-generated text appear more like human-written text, significantly worsen the performance of the detection tools. These techniques make it even more challenging for the tools to accurately identify AI-generated content.
4. Limited coverage of tools: The study covers a total of 12 publicly available tools and two commercial systems commonly used in academic settings. However, this may not represent the full range of available detection tools, and there might be other tools with different capabilities and limitations that are not included in the study.
5. Usability issues: The article mentions usability issues with the detection tools, although specific details are not provided. These usability issues can affect the practicality and effectiveness of using the tools in academic settings.

testing-of-detection-tools-for-aie28091generatedDownload