7

AI Detection and Grammarly

Hi everyone, We have faculty using the AI report and are appreciating that feature. An interesting question came up recently: the report showed 64% AI-generated text, but the student claimed he had only used Grammarly, which also helps to improve phrasing. Has anyone else experienced an AI report picking up on Grammarly, and is there a way of distinguishing between a "grammar help" tool and other generative AI?

Second, will the AI report be visible to students soon? Faculty assumed that students could see it, but the report is only visible to faculty in our LMS.

Thanks,

Jennifer Douglas, American Public University System

134 replies

null
    • Fiona_Heagerty
    • 1 yr ago
    • Reported - view

    So interesting to read this thread.  I had a student yesterday state that he only used Grammarly on an assignment that came up with a 73% AI score.  He also sent me a list of news articles about students who have been falsely accused of cheating based on Turnitin AI software detection.  I would be interested in any progress you make Gill Rowell in working out if it's picking up Grammerly or Grammerly AI (I agree with Paul Rittman - I think it will be the latter.  

      • Fiona_Heagerty
      • 1 yr ago
      • Reported - view

      Thanks Paul. I’ll take a look at those. Paul Rittman 

    • Daniel_Hernandez
    • 1 yr ago
    • Reported - view

    ok and so what

    • Amy_henderson
    • 1 yr ago
    • Reported - view

    Hi all, thank you for this forum. I have run into a similar predicament with a few of my students showing up with high AI scores. One of my students who I have personally seen writing during my office hours and asking questions about their essay turned in a paper with 90% AI detection score. After talking with them, she mentioned that she uses Grammarly for all of her writing and that this may be the cause. I would love to stay updated Gill Rowell about the prevalence of this situation. As it stands right now, I will not be giving any zeros based on AI scores until it is clear that Grammarly does not affect the report.

    • Customer Engagement Specialist, Turnitin
    • Gill_Rowell
    • 1 yr ago
    • Reported - view

    Amy Henderson and Fiona Heagerty I will update you as soon as I have further news on this. In the meantime, if you have specific papers which are demonstrating this behaviour, please message me directly with details and I will pass on to the AI development team to take a look. 

    • Annelize_McKay
    • 1 yr ago
    • Reported - view

    We are finding the same thing - a student swears he only used Grammarly, but that comes up as 55% AI use. It would be great if the TurnItIn could give us clarity on whether this is even likely that their detector will detect Grammarly.

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 1 yr ago
      • Reported - view

      Annelize McKay It's worth confirming precisely which part of Grammarly the student used - if it was GrammarlyGo, that's their answer to ChatGPT.

      If a student uses Grammarly's rewriting features (in the paid for version) excessively then it's not inconceivable that the output could look a lot like that from ChatGPT, so any detection mechanism is going to have trouble telling them apart.

      • Annelize_McKay
      • 1 yr ago
      • Reported - view

      Steve Bentley Thank you - this is most useful. I did not know that two versions of Grammarly existed so I certainly will look into that.

    • Patti_Meyer
    • 1 yr ago
    • Reported - view

    I also hear from students that they used Grammarly.  But I am wondering if they are using GrammarlyGo.  I did hear that GrammarlyGo uses GPT3 language so that is what ChatGPT is based off of, so would Turnitin AI then detect GrammarlyGo?

      • Patti_Meyer
      • 1 yr ago
      • Reported - view

      Patti Meyer Follow-up, created a paper in GrammarlyGo and ran it through Turnitin.  Came back 100% AI.  And 16% Similarity score stating they could not show me where the text came from because it was "outside the host institution." 

    • lucy_hunt
    • 1 yr ago
    • Reported - view

    Our institution uses Grammarly premium and the GrammarlyGo is disabled, so we can’t use it. I conducted three different tests to find out if Turnitin’s AI detector is in fact flagging Grammarly. Based on my findings, Grammarly isn’t the issue. I typed an essay from scratch in Grammarly, accepted the changes/edits, and downloaded it and turned it in to canvas. 0% AI detection. I then took a different essay I typed in Word, copied it into Grammarly, accepted the changes and edits and downloaded and turned it in. 0% AI detection. THEN I copied an essay from chatgpt, pasted it into Grammarly, accepted any changes, downloaded it and then turned it into canvas. 100% AI detection. My findings show that it’s not Grammarly. I’ve only gotten this far, but I mean it’s something to note.  

    • Purna_Bose
    • 1 yr ago
    • Reported - view

    Our detector is not tuned to target Grammarly generated content but rather, other AI content written by LLMs such as GPT-3.5.  Based on initial tests we conducted on human-written documents with no AI-generated content in them, in most cases, changes made by Grammarly and/or other grammar-checking tools were not flagged as AI-written by our detector.

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 1 yr ago
      • Reported - view

      Purna Bose Thanks for that clarification. Can I confirm that your testing includes both the free tier of Grammarly and the paid-for product that has a "rewrite my sentence" feature?  Have you done similar testing with Quillbot and its paraphrasing capability?

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 1 yr ago
      • Reported - view

      Steve Bentley Yes, our testing has largely been based on the free version of Grammarly, So if the detector is flagging content as AI it may have been written using GrammarlyGo. Our investigations on Quillbot are ongoing. 

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 1 yr ago
      • Reported - view

      Gill Rowell Hi Gill, I think there's a bit of confusion here:  Grammarly has THREE services.

      Free: Basic flagging of errors for the user to fix themselves
      Paid: Has tools that can rewrite sentences/paragraphs to improve the English
      GO: This is a ChatGPT clone/rebadge

      It's the paid platform where there is ambiguity, whether the rewritten sentences/paragraphs look sufficiently like AI output that it is misdetected, which is what students are claiming.  (Obviously different institutions will have their own policy on whether that rewriting is acceptable or not)

      • lucy_hunt
      • 1 yr ago
      • Reported - view

      Steve Bentley  I used the paid version of Grammarly for my own personal test. I accepted all the changes and made edits according to Grammarly's suggestions but still was not flagged for any AI use. I even rewrote/reworded the sentences it suggested I improve. I'm wondering how these children are claiming it's Grammarly when, on our end, GrammarlyGO is disabled and my own personal test proved the paid version didn't detect any AI. 

      • lucy_hunt
      • 1 yr ago
      • Reported - view

      Steve Bentley All of my writing preferences are set to "on," too--I'm getting the full dose of what Grammarly Premium offers minus GrammarlyGo. The same goes for my students. 

      • Malik.1
      • 1 yr ago
      • Reported - view

      Gill Rowell Could you please update about the Quillbot investigation?

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 1 yr ago
      • Reported - view

      Malik I don't have any specific updates on Quillbot. In our AI Innovation Lab, we have conducted tests using open sourced paraphrasing tools (including different LLMs) and in most cases, our detector has retained its effectiveness and is able to identify text as AI-generated even when a paraphrasing tool has been used to change the AI output.

      • Malik.1
      • 1 yr ago
      • Reported - view

      Gill Rowell Thank you for your response. Actually, I wanted to confirm that if an AI paraphrasing tool is used on human generated content (to sound smarter or improve structure etc) does turnitin flag it as AI too? 

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 1 yr ago
      • Reported - view

      Hi Gill Rowell just to support Malik's comment, the concern is about avoiding false positive detections where tools like Quillbot and Grammarly have been used to improve a student's authentic writing. This is a common claim from students who are challenged about a high AI score from Turnitin.

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 1 yr ago
      • Reported - view

      I understand that there may be times when using these tools may trigger the AI indicator. You may find some of our resources on false positives useful. Firstly this blog post and video from our Product team. Additionally, some of our pedagogic resources from my colleagues on the TLI team. Using the AI detector, just like the similarity report, is just one indicator of possible misconduct and may be used a starting point for conversations with students about their use of these tools when completing their assessed work. 

      Additionally you can always ask our Support team to take a look at any paper where there may be doubt. You can contact them by emailing tiisupport@turnitin.com.

    • lucy_hunt
    • 1 yr ago
    • Reported - view

    If an essay is flagged with 50% or over for AI, wouldn't that pretty much toss out the "false positive" theory? I get 20% or less can be less reliable, but 50%? 

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 1 yr ago
      • Reported - view

      lucy hunt I'm not sure if this recent blog post from our Chief Product Officer, Annie Chechitelli might help with your understanding of false positives. 

      • TWC
      • 1 yr ago
      • Reported - view

      lucy hunt the % that gets flagged is only the chance that the text is AI written. Just because they are scoring some % AI does not mean they are using AI. The score is the % chance detectors think the article was written by AI not what % of the article was written with AI.

      • lucy_hunt
      • 1 yr ago
      • Reported - view

      Chris Turner isn't 75 to 100% still pretty significant?

Content aside

  • 7 Likes
  • 2 mths agoLast active
  • 134Replies
  • 77127Views
  • 61 Following