7

AI Detection and Grammarly

Hi everyone, We have faculty using the AI report and are appreciating that feature. An interesting question came up recently: the report showed 64% AI-generated text, but the student claimed he had only used Grammarly, which also helps to improve phrasing. Has anyone else experienced an AI report picking up on Grammarly, and is there a way of distinguishing between a "grammar help" tool and other generative AI?

Second, will the AI report be visible to students soon? Faculty assumed that students could see it, but the report is only visible to faculty in our LMS.

Thanks,

Jennifer Douglas, American Public University System

134 replies

null
      • Fiona_Heagerty
      • 1 yr agoThu, May 18, 2023 at 5:18 PM UTC
      • Reported - view

      So interesting to read this thread.  I had a student yesterday state that he only used Grammarly on an assignment that came up with a 73% AI score.  He also sent me a list of news articles about students who have been falsely accused of cheating based on Turnitin AI software detection.  I would be interested in any progress you make Gill Rowell in working out if it's picking up Grammerly or Grammerly AI (I agree with Paul Rittman - I think it will be the latter.  

        • Fiona_Heagerty
        • 1 yr agoFri, May 19, 2023 at 6:47 PM UTC
        • Reported - view

        Thanks Paul. I’ll take a look at those. Paul Rittman 

      • Daniel_Hernandez
      • 1 yr agoFri, May 19, 2023 at 1:24 AM UTC
      • Reported - view

      ok and so what

      • Amy_henderson
      • 1 yr agoSat, May 20, 2023 at 10:35 PM UTC
      • Reported - view

      Hi all, thank you for this forum. I have run into a similar predicament with a few of my students showing up with high AI scores. One of my students who I have personally seen writing during my office hours and asking questions about their essay turned in a paper with 90% AI detection score. After talking with them, she mentioned that she uses Grammarly for all of her writing and that this may be the cause. I would love to stay updated Gill Rowell about the prevalence of this situation. As it stands right now, I will not be giving any zeros based on AI scores until it is clear that Grammarly does not affect the report.

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 1 yr agoMon, May 22, 2023 at 10:26 AM UTC
      • Reported - view

      Amy Henderson and Fiona Heagerty I will update you as soon as I have further news on this. In the meantime, if you have specific papers which are demonstrating this behaviour, please message me directly with details and I will pass on to the AI development team to take a look. 

      • Annelize_McKay
      • 1 yr agoThu, June 8, 2023 at 12:49 PM UTC
      • Reported - view

      We are finding the same thing - a student swears he only used Grammarly, but that comes up as 55% AI use. It would be great if the TurnItIn could give us clarity on whether this is even likely that their detector will detect Grammarly.

        • Turnitin Admin, University of Huddersfield, UK
        • s_d_bentley
        • 1 yr agoFri, June 9, 2023 at 11:43 AM UTC
        • Reported - view

        Annelize McKay It's worth confirming precisely which part of Grammarly the student used - if it was GrammarlyGo, that's their answer to ChatGPT.

        If a student uses Grammarly's rewriting features (in the paid for version) excessively then it's not inconceivable that the output could look a lot like that from ChatGPT, so any detection mechanism is going to have trouble telling them apart.

        • Annelize_McKay
        • 1 yr agoFri, June 9, 2023 at 4:22 PM UTC
        • Reported - view

        Steve Bentley Thank you - this is most useful. I did not know that two versions of Grammarly existed so I certainly will look into that.

      • Patti_Meyer
      • 1 yr agoTue, June 27, 2023 at 4:19 PM UTC
      • Reported - view

      I also hear from students that they used Grammarly.  But I am wondering if they are using GrammarlyGo.  I did hear that GrammarlyGo uses GPT3 language so that is what ChatGPT is based off of, so would Turnitin AI then detect GrammarlyGo?

        • Patti_Meyer
        • 1 yr agoWed, June 28, 2023 at 1:22 PM UTC
        • Reported - view

        Patti Meyer Follow-up, created a paper in GrammarlyGo and ran it through Turnitin.  Came back 100% AI.  And 16% Similarity score stating they could not show me where the text came from because it was "outside the host institution." 

      • lucy_hunt
      • 1 yr agoTue, June 27, 2023 at 4:34 PM UTC
      • Reported - view

      Our institution uses Grammarly premium and the GrammarlyGo is disabled, so we can’t use it. I conducted three different tests to find out if Turnitin’s AI detector is in fact flagging Grammarly. Based on my findings, Grammarly isn’t the issue. I typed an essay from scratch in Grammarly, accepted the changes/edits, and downloaded it and turned it in to canvas. 0% AI detection. I then took a different essay I typed in Word, copied it into Grammarly, accepted the changes and edits and downloaded and turned it in. 0% AI detection. THEN I copied an essay from chatgpt, pasted it into Grammarly, accepted any changes, downloaded it and then turned it into canvas. 100% AI detection. My findings show that it’s not Grammarly. I’ve only gotten this far, but I mean it’s something to note.  

      • Purna_Bose
      • 1 yr agoWed, June 28, 2023 at 4:21 PM UTC
      • Reported - view

      Our detector is not tuned to target Grammarly generated content but rather, other AI content written by LLMs such as GPT-3.5.  Based on initial tests we conducted on human-written documents with no AI-generated content in them, in most cases, changes made by Grammarly and/or other grammar-checking tools were not flagged as AI-written by our detector.

        • Turnitin Admin, University of Huddersfield, UK
        • s_d_bentley
        • 1 yr agoThu, June 29, 2023 at 7:56 AM UTC
        • Reported - view

        Purna Bose Thanks for that clarification. Can I confirm that your testing includes both the free tier of Grammarly and the paid-for product that has a "rewrite my sentence" feature?  Have you done similar testing with Quillbot and its paraphrasing capability?

        • Customer Engagement Specialist, Turnitin
        • Gill_Rowell
        • 1 yr agoThu, June 29, 2023 at 10:51 AM UTC
        • Reported - view

        Steve Bentley Yes, our testing has largely been based on the free version of Grammarly, So if the detector is flagging content as AI it may have been written using GrammarlyGo. Our investigations on Quillbot are ongoing. 

        • Turnitin Admin, University of Huddersfield, UK
        • s_d_bentley
        • 1 yr agoThu, June 29, 2023 at 10:56 AM UTC
        • Reported - view

        Gill Rowell Hi Gill, I think there's a bit of confusion here:  Grammarly has THREE services.

        Free: Basic flagging of errors for the user to fix themselves
        Paid: Has tools that can rewrite sentences/paragraphs to improve the English
        GO: This is a ChatGPT clone/rebadge

        It's the paid platform where there is ambiguity, whether the rewritten sentences/paragraphs look sufficiently like AI output that it is misdetected, which is what students are claiming.  (Obviously different institutions will have their own policy on whether that rewriting is acceptable or not)

        • lucy_hunt
        • 1 yr agoThu, June 29, 2023 at 5:02 PM UTC
        • Reported - view

        Steve Bentley  I used the paid version of Grammarly for my own personal test. I accepted all the changes and made edits according to Grammarly's suggestions but still was not flagged for any AI use. I even rewrote/reworded the sentences it suggested I improve. I'm wondering how these children are claiming it's Grammarly when, on our end, GrammarlyGO is disabled and my own personal test proved the paid version didn't detect any AI. 

        • lucy_hunt
        • 1 yr agoThu, June 29, 2023 at 5:06 PM UTC
        • Reported - view

        Steve Bentley All of my writing preferences are set to "on," too--I'm getting the full dose of what Grammarly Premium offers minus GrammarlyGo. The same goes for my students. 

        • Malik.1
        • 1 yr agoWed, July 26, 2023 at 4:37 PM UTC
        • Reported - view

        Gill Rowell Could you please update about the Quillbot investigation?

        • Customer Engagement Specialist, Turnitin
        • Gill_Rowell
        • 1 yr agoThu, July 27, 2023 at 3:38 PM UTC
        • Reported - view

        Malik I don't have any specific updates on Quillbot. In our AI Innovation Lab, we have conducted tests using open sourced paraphrasing tools (including different LLMs) and in most cases, our detector has retained its effectiveness and is able to identify text as AI-generated even when a paraphrasing tool has been used to change the AI output.

        • Malik.1
        • 1 yr agoThu, July 27, 2023 at 3:49 PM UTC
        • Reported - view

        Gill Rowell Thank you for your response. Actually, I wanted to confirm that if an AI paraphrasing tool is used on human generated content (to sound smarter or improve structure etc) does turnitin flag it as AI too? 

        • Turnitin Admin, University of Huddersfield, UK
        • s_d_bentley
        • 1 yr agoFri, July 28, 2023 at 7:33 AM UTC
        • Reported - view

        Hi Gill Rowell just to support Malik's comment, the concern is about avoiding false positive detections where tools like Quillbot and Grammarly have been used to improve a student's authentic writing. This is a common claim from students who are challenged about a high AI score from Turnitin.

        • Customer Engagement Specialist, Turnitin
        • Gill_Rowell
        • 1 yr agoTue, August 1, 2023 at 11:27 AM UTC
        • Reported - view

        I understand that there may be times when using these tools may trigger the AI indicator. You may find some of our resources on false positives useful. Firstly this blog post and video from our Product team. Additionally, some of our pedagogic resources from my colleagues on the TLI team. Using the AI detector, just like the similarity report, is just one indicator of possible misconduct and may be used a starting point for conversations with students about their use of these tools when completing their assessed work. 

        Additionally you can always ask our Support team to take a look at any paper where there may be doubt. You can contact them by emailing tiisupport@turnitin.com.

      • lucy_hunt
      • 1 yr agoSun, July 2, 2023 at 6:57 PM UTC
      • Reported - view

      If an essay is flagged with 50% or over for AI, wouldn't that pretty much toss out the "false positive" theory? I get 20% or less can be less reliable, but 50%? 

        • Customer Engagement Specialist, Turnitin
        • Gill_Rowell
        • 1 yr agoMon, July 3, 2023 at 4:11 PM UTC
        • Reported - view

        lucy hunt I'm not sure if this recent blog post from our Chief Product Officer, Annie Chechitelli might help with your understanding of false positives. 

        • TWC
        • 1 yr agoTue, July 11, 2023 at 12:59 AM UTC
        • Reported - view

        lucy hunt the % that gets flagged is only the chance that the text is AI written. Just because they are scoring some % AI does not mean they are using AI. The score is the % chance detectors think the article was written by AI not what % of the article was written with AI.

        • lucy_hunt
        • 1 yr agoTue, July 11, 2023 at 1:09 AM UTC
        • Reported - view

        Chris Turner isn't 75 to 100% still pretty significant?

            Login to reply

            Content aside

            • 7 Likes
            • 7 mths agoMon, October 7, 2024 at 12:39 PM UTCLast active
            • 134Replies
            • 90318Views
            • 62 Following