Jennifer Douglas Grammarly employs a combination of standard correction detection, which identifies frequently misused words based on patterns, and Chat GPT-4 to enhance its speed and accuracy. Similarly, the latest version of the Windows suite has integrated AI-powered correction tools, as has added the option Google Docs. It's worth noting that I've tested content I personally wrote years before AI tools were accessible to the public, and these detection tools still identified them as AI-generated. I've also noticed most bibliographies become flagged by the detection too aswell. The detection tools might not be advancing as rapidly as AI itself and are not very reliable.

Narelle Hunter I believe Google Translate may also cause similar behaviour. Are you able to message me direct with paper IDs so we are able to investigate in more detail? Or alternatively you can use the in-product feedback tool.

Jo-Anne Rondell I am suspecting what they are doing after logging in to Grammarly, is using Grammarly's AI function (called something like GrammarlyGo--you can see it on the right side of the window after you log in). The reason why I say that, is that I've taken texts that were NOT composed by AI, taken them to grammarly and ONLY made grammar suggestions by Grammarly. The edited document had a 0% or very low AI score. So for turnitin to give an AI score of 100% when the student claims they submitted it to Grammarly, tells me that they used Grammarly's AI function. See my post here: https://turnitin.forumbee.com/t/83h4qhz/grammarly-turnitin-coms-ai-score

Paul Rittman Thanks very much for the reply. Can you tell me if you tried using Grammarly's AI?

Jo-Anne Rondell At this point I don't remember. I've been pretty busy since that post. :(

Paul Rittman Thanks again. I'm pursuing this since I suspect the problems will multiply as we continue on. I'll post back here with whatever I discover, if anything.

Jo-Anne Rondell Sounds good. Look forward to your results.

Jo-Anne Rondell One more thing. If you get some concrete results about how turnitin detects Grammarly AI vs. Grammarly's normal grammar check, I would PM Gill Rowell, who has posted above here in this thread twice. She is a turnitin.com employee who has PMed me about issues I've discussed. I think she would be interested in looking at your files, if you can show how turnitin's AI generator is working.

Jo-Anne Rondell I am happy to pass on any feedback you have to our AI development team. Please DM me with any paper IDs that are exhibiting this behaviour. We are all learning about AI writing!

Paul Rittman I think this is part of the problem, that Grammarly has multiple services, some only available to paid accounts. Free accounts, as I understand it, offer suggestions, paid tiers will rewrite sentences to improve them. Much more of a grey area in terms of acceptability, I know our regulations wouldn't accept that from a human proofreader.

GrammarlyGo as their AI offering is an extra layer on top of that.

Jo-Anne Rondell I have just stumbled onto this thread because I've run into this same issue. I tried to test this theory (typing something from scratch in Grammarly, downloading it, and then submitting it in canvas), but I guess because I was in "student view" and not an actual student, the AI detection was unable to be processed. Literally, the box was gray and it never registered. I did this twice. I have since resorted to asking a student I know and trust (who is in my class) to test this as well so I can see if it is in fact Grammarly that's being detected. If so, the two essays written by another student that were detected to have been generated by AI (100%) will be graded as normal. I'm hoping Turnitin's AI detector will be able to discern Grammarly from the other AI generators, but who knows. Should we discourage our students from starting from scratch in Grammarly? 

Paul Rittman So, I figured the same thing (with GrammarlyGO), and I went to my Grammarly account to check. Our institution subscribes to Grammarly premium, so when I clicked on "features," the GrammarlyGO was disabled by admin. So if it's disabled on my end, it would have to be disabled on my students' Grammarlys right? That would mean that the AI detector is literally just detecting essays started and completed in Grammarly (but not using GrammarlyGO) as AI-generated. 

Fiona Heagerty Fiona, I have found two other websites that detect AI content in written work. One is GPTZero (https://app.gptzero.me/app/welcome), the other is Writer.com AI detector (https://writer.com/ai-content-detector/). If a student is objecting, I would run the written work past those two other sources as well.

Thanks Paul. I’ll take a look at those. Paul Rittman 

Annelize McKay It's worth confirming precisely which part of Grammarly the student used - if it was GrammarlyGo, that's their answer to ChatGPT.

If a student uses Grammarly's rewriting features (in the paid for version) excessively then it's not inconceivable that the output could look a lot like that from ChatGPT, so any detection mechanism is going to have trouble telling them apart.

Steve Bentley Thank you - this is most useful. I did not know that two versions of Grammarly existed so I certainly will look into that.

Patti Meyer Follow-up, created a paper in GrammarlyGo and ran it through Turnitin.  Came back 100% AI.  And 16% Similarity score stating they could not show me where the text came from because it was "outside the host institution." 

Purna Bose Thanks for that clarification. Can I confirm that your testing includes both the free tier of Grammarly and the paid-for product that has a "rewrite my sentence" feature?  Have you done similar testing with Quillbot and its paraphrasing capability?

Steve Bentley Yes, our testing has largely been based on the free version of Grammarly, So if the detector is flagging content as AI it may have been written using GrammarlyGo. Our investigations on Quillbot are ongoing. 

Gill Rowell Hi Gill, I think there's a bit of confusion here:  Grammarly has THREE services.

Free: Basic flagging of errors for the user to fix themselves
Paid: Has tools that can rewrite sentences/paragraphs to improve the English
GO: This is a ChatGPT clone/rebadge

It's the paid platform where there is ambiguity, whether the rewritten sentences/paragraphs look sufficiently like AI output that it is misdetected, which is what students are claiming.  (Obviously different institutions will have their own policy on whether that rewriting is acceptable or not)

Steve Bentley  I used the paid version of Grammarly for my own personal test. I accepted all the changes and made edits according to Grammarly's suggestions but still was not flagged for any AI use. I even rewrote/reworded the sentences it suggested I improve. I'm wondering how these children are claiming it's Grammarly when, on our end, GrammarlyGO is disabled and my own personal test proved the paid version didn't detect any AI. 

Steve Bentley All of my writing preferences are set to "on," too--I'm getting the full dose of what Grammarly Premium offers minus GrammarlyGo. The same goes for my students. 

Gill Rowell Could you please update about the Quillbot investigation?

Malik I don't have any specific updates on Quillbot. In our AI Innovation Lab, we have conducted tests using open sourced paraphrasing tools (including different LLMs) and in most cases, our detector has retained its effectiveness and is able to identify text as AI-generated even when a paraphrasing tool has been used to change the AI output.

Gill Rowell Thank you for your response. Actually, I wanted to confirm that if an AI paraphrasing tool is used on human generated content (to sound smarter or improve structure etc) does turnitin flag it as AI too? 

Hi Gill Rowell just to support Malik's comment, the concern is about avoiding false positive detections where tools like Quillbot and Grammarly have been used to improve a student's authentic writing. This is a common claim from students who are challenged about a high AI score from Turnitin.

I understand that there may be times when using these tools may trigger the AI indicator. You may find some of our resources on false positives useful. Firstly this blog post and video from our Product team. Additionally, some of our pedagogic resources from my colleagues on the TLI team. Using the AI detector, just like the similarity report, is just one indicator of possible misconduct and may be used a starting point for conversations with students about their use of these tools when completing their assessed work. 

Additionally you can always ask our Support team to take a look at any paper where there may be doubt. You can contact them by emailing tiisupport@turnitin.com.

lucy hunt I'm not sure if this recent blog post from our Chief Product Officer, Annie Chechitelli might help with your understanding of false positives. 

lucy hunt the % that gets flagged is only the chance that the text is AI written. Just because they are scoring some % AI does not mean they are using AI. The score is the % chance detectors think the article was written by AI not what % of the article was written with AI.

Chris Turner isn't 75 to 100% still pretty significant?

Chris Turner This is what Turnitin has on its page: "Using the average scores of all the segments within the document, the model then generates an overall prediction of how much text in the submission we believe has been generated by AI."

  lucy hunt 

AI vs Human score says the probability that AI thinks the content was generated by AI or generated by a person. A score of 20% AI 80% Human Means – The AI gives the content 80% probability of being Human and  20% probability of being AI generated; it doesn't mean – 80% of the article is human, and 20% is AI. My neighbour is the CTO of a large AI company and says that AI detection is not at a point where it can provide enough certainty and gives a high rate of false positives. I have run through pages of engineering textbooks through various AI detection including turnitin and each time the score had between 10- 20% approx AI content. This has to be incorrect as all the text I ran through detection was years old, well before AI writing tools came on the market. I'm interested to see how the detection tools develop, but they appear too inaccurate. 

TWC I understand, but I don’t count anything that’s been flagged under 30% from Turnitin. If a student turns in an essay and it comes back with a  70-100% AI detection, I ask them to redo it. If it came back with 30% or less, I ignore it.  

Karen Smith We're taking a view that the AI report alone is not sufficient to have that conversation. A "magic number" and no way to explain how that number has been reached is not a good basis to be making accusations - however gently they are framed. 

Our approach is to require some additional evidence, such as falsified references or differences in writing style or standard of English compared with previous work before we act on the report, mainly because we don't want to damage staff-student relationships with this kind of conversation based on something that we can't back up and where it's the student's word against the Turnitin algorithm.

Steve Bentley Yes, exactly. Many of our resources speak to the need for other data points. If I were still in the classroom, that number would nudge me to look more closely and have the conversations if it seems warranted. 

David Silver Thank you for joining us! I think there are a few things to unpack here. Firstly, it's important to say that the AI detection indicator, just like the Similarity score is just one part of the conversation regarding possible misconduct.

It seems like you have already engaged in a conversation with the student regarding their work. And I would say that if the student is adamant that this is their own work it might be worth asking our Support team to take a look at the paper. You can contact them by emailing tiisupport@turnitin.com.

Additionally you may find some of our resources on false positives useful. Firstly this blog post and video from our Product team. Additionally some of our pedagogic resources from my colleagues on the TLI team. 

Esther Jubb Potentially yes, but who knows as the detection tool doesn't show its working.

John Conner I recommend the following info on false positives: The first blog from March and the second blog from June of this year.

As for the student use, I would say to use the score as a discussion starter with students: How did they use the tool and why? Did it fit into the parameters of the paper, etc. From there it's easier to discuss what consequences are appropriate. Which may be a rewrite or it may require some disciplinary measures. But it's only a starting point, not a definitive or solo data point.