AI Writing Detection
Hello everyone,
I am seeking insight regarding a case involving AI writing detection results in Turnitin.
I have a student whose final submission is showing 100% AI writing detected. However, we have closely monitored the entire drafting process and have substantial evidence of authentic intellectual development, including:
Multiple draft submissions
Supervisor feedback and documented revisions
Clear incorporation of feedback over time
Observable progression in argumentation and structure
The submission history is as follows:
First draft: AI writing indicated
Revised draft resubmission: 0% AI detected
Final submission: 100% AI detected (student states they used the first draft as the base for the final version)
This dramatic fluctuation between versions is concerning and difficult to reconcile with the documented writing process.
We suspect the student may have used Grammarly or a similar writing-assistance tool at some stage, but we do not see evidence of externally generated content replacing the student’s work.
Has anyone encountered a similar situation where detection percentages varied so significantly between drafts?
Are there known technical factors (e.g., document formatting, copy/paste behavior, writing-assistance tools) that might explain this kind of result?
Any insight would be greatly appreciated, as this has significant academic integrity implications.
Thank you in advance.
1 reply
-
-
Thank you for sharing this feedback! We so appreciate and understand your concerns. I connected with our Product team and received some helpful insights for you. There are a few possible explanations for the results you are seeing:
- First, we updated our model in October to improve detection of AI content from newer LLMs (e.g., GPT-5 and Gemini 2.5). Depending on when the drafts were submitted, this may be the reason for the increase in AI score between the submissions.
- As generative AI capabilities evolve, we aim to follow quickly with AI detector updates that maintain a high level of efficacy. We regularly test our AI writing detection capabilities to maintain high accuracy and keep the false positive rate below 1%. This means that the AI score for a document may change as we adjust our model to keep up with generative AI changes.
- As you know, the AI score is just one data point and should not be used in isolation when evaluating a student's work for original writing. We continuously update our AI detection to maintain high accuracy and minimize our false positive rate.
- To maintain this standard, before every update or new model release, we perform additional tests on approximately 700,000+ academic papers that were written before the release of ChatGPT.
Based on customer feedback, we have re-tested our model on nearly one million human-written documents. We confirmed that for documents identified as having over 20% AI content after our October update, our false positive rate remains below 1%.
Second, it is worth noting that standard editors and grammar tools—such as Grammarly, MS Word, Google Docs, and other features that "refine" or "improve" writing—are now backed by LLMs and, as such, can leave AI signals when used, which is not always apparent to students. Our tests have shown that if the changes are minimal and restricted to simple corrections, our model will likely not flag these as AI-generated. However, if the rewrites are extensive and change a significant portion of the text, it is likely that the content will be flagged as AI.
Given the details you shared, particularly the use of Grammarly or other writing assistants, we're confident that the results you're seeing are accurate and expected. However, if you believe that we may be flagging human-written text as AI-generated, please share the submissions with us for further review. Please note that, given the volume of documents we receive, we may not always be able to provide individual responses.
And, as always, don't hesitate to fill out a formal ticket in our Help Center, so that you can surface your challenges and our team can work to support your specific needs and answer your questions.
Thank you!
Audrey, Turnitin Team