Evaluating the Reliability of Generative AI in Distinguishing Machine from Human Text

Abstract

Introduction: The rapid progression of generative AI systems has facilitated the creation of human-like text with remarkable sophistication. Models such as GPT-4, Claude, and Gemini are capable of generating coherent content across a wide range of genres, thereby raising critical concerns regarding the differentiation between machine-generated and human-authored text. This capability presents significant challenges to academic integrity, content authenticity, and the development of reliable detection methodologies.
Objective: To evaluate the performance and reliability of current AI-based text detection tools in identifying machine-generated content across different text genres, AI models, and writing styles, establishing a comprehensive benchmark for detection capabilities.
Methodology: We systematically evaluated ten commercially available AI detection tools utilizing a curated dataset comprising 150 text samples, expanded from the original 50. This dataset included human-authored texts, both original and translated, as well as AI-generated content from six advanced models (GPT-3.5, GPT-4, Gemini, Bing, Claude, LLaMA2), along with paraphrased variants. Each tool underwent assessment through binary classification, employing metrics such as accuracy, precision, recall, F1 scores, and confusion matrices. Statistical significance was determined using McNemar's test with Bonferroni correction.
Results indicate that Content at Scale demonstrated the highest accuracy at 88% (95% CI: 84.2-91.8%), followed by Crossplag at 76% and Copyleaks at 70%. Notably, performance varied significantly across different text categories, with all tools exhibiting reduced accuracy for texts generated by more recent models, such as Claude and LLaMA2. False positive rates ranged from 4% to 32%, which raises concerns regarding their applicability in academic contexts. No tool achieved perfect accuracy, and a performance degradation of 12% was observed with models released subsequent to the initial study design.
Conclusions: Current AI text detection tools exhibit moderate to high levels of accuracy; however, they remain imperfect, displaying considerable variability across different AI models and text types. The ongoing challenge of achieving reliable detection, coupled with non-trivial false positive rates, necessitates cautious implementation in high-stakes environments. These tools should serve as a complement to, rather than a replacement for, human judgment in academic and professional contexts.