6

AI Detection and Grammarly

Hi everyone, We have faculty using the AI report and are appreciating that feature. An interesting question came up recently: the report showed 64% AI-generated text, but the student claimed he had only used Grammarly, which also helps to improve phrasing. Has anyone else experienced an AI report picking up on Grammarly, and is there a way of distinguishing between a "grammar help" tool and other generative AI?

Second, will the AI report be visible to students soon? Faculty assumed that students could see it, but the report is only visible to faculty in our LMS.

Thanks,

Jennifer Douglas, American Public University System

119 replies

null
    • Customer Engagement Specialist, Turnitin
    • Gill_Rowell
    • 10 mths ago
    • Reported - view

    Amy Henderson and Fiona Heagerty I will update you as soon as I have further news on this. In the meantime, if you have specific papers which are demonstrating this behaviour, please message me directly with details and I will pass on to the AI development team to take a look. 

    • Annelize_McKay
    • 10 mths ago
    • Reported - view

    We are finding the same thing - a student swears he only used Grammarly, but that comes up as 55% AI use. It would be great if the TurnItIn could give us clarity on whether this is even likely that their detector will detect Grammarly.

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 10 mths ago
      • Reported - view

      Annelize McKay It's worth confirming precisely which part of Grammarly the student used - if it was GrammarlyGo, that's their answer to ChatGPT.

      If a student uses Grammarly's rewriting features (in the paid for version) excessively then it's not inconceivable that the output could look a lot like that from ChatGPT, so any detection mechanism is going to have trouble telling them apart.

      • Annelize_McKay
      • 10 mths ago
      • Reported - view

      Steve Bentley Thank you - this is most useful. I did not know that two versions of Grammarly existed so I certainly will look into that.

    • Patti_Meyer
    • 9 mths ago
    • Reported - view

    I also hear from students that they used Grammarly.  But I am wondering if they are using GrammarlyGo.  I did hear that GrammarlyGo uses GPT3 language so that is what ChatGPT is based off of, so would Turnitin AI then detect GrammarlyGo?

      • Patti_Meyer
      • 9 mths ago
      • Reported - view

      Patti Meyer Follow-up, created a paper in GrammarlyGo and ran it through Turnitin.  Came back 100% AI.  And 16% Similarity score stating they could not show me where the text came from because it was "outside the host institution." 

    • lucy_hunt
    • 9 mths ago
    • Reported - view

    Our institution uses Grammarly premium and the GrammarlyGo is disabled, so we can’t use it. I conducted three different tests to find out if Turnitin’s AI detector is in fact flagging Grammarly. Based on my findings, Grammarly isn’t the issue. I typed an essay from scratch in Grammarly, accepted the changes/edits, and downloaded it and turned it in to canvas. 0% AI detection. I then took a different essay I typed in Word, copied it into Grammarly, accepted the changes and edits and downloaded and turned it in. 0% AI detection. THEN I copied an essay from chatgpt, pasted it into Grammarly, accepted any changes, downloaded it and then turned it into canvas. 100% AI detection. My findings show that it’s not Grammarly. I’ve only gotten this far, but I mean it’s something to note.  

    • Purna_Bose
    • 9 mths ago
    • Reported - view

    Our detector is not tuned to target Grammarly generated content but rather, other AI content written by LLMs such as GPT-3.5.  Based on initial tests we conducted on human-written documents with no AI-generated content in them, in most cases, changes made by Grammarly and/or other grammar-checking tools were not flagged as AI-written by our detector.

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 9 mths ago
      • Reported - view

      Purna Bose Thanks for that clarification. Can I confirm that your testing includes both the free tier of Grammarly and the paid-for product that has a "rewrite my sentence" feature?  Have you done similar testing with Quillbot and its paraphrasing capability?

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 9 mths ago
      • Reported - view

      Steve Bentley Yes, our testing has largely been based on the free version of Grammarly, So if the detector is flagging content as AI it may have been written using GrammarlyGo. Our investigations on Quillbot are ongoing. 

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 9 mths ago
      • Reported - view

      Gill Rowell Hi Gill, I think there's a bit of confusion here:  Grammarly has THREE services.

      Free: Basic flagging of errors for the user to fix themselves
      Paid: Has tools that can rewrite sentences/paragraphs to improve the English
      GO: This is a ChatGPT clone/rebadge

      It's the paid platform where there is ambiguity, whether the rewritten sentences/paragraphs look sufficiently like AI output that it is misdetected, which is what students are claiming.  (Obviously different institutions will have their own policy on whether that rewriting is acceptable or not)

      • lucy_hunt
      • 9 mths ago
      • Reported - view

      Steve Bentley  I used the paid version of Grammarly for my own personal test. I accepted all the changes and made edits according to Grammarly's suggestions but still was not flagged for any AI use. I even rewrote/reworded the sentences it suggested I improve. I'm wondering how these children are claiming it's Grammarly when, on our end, GrammarlyGO is disabled and my own personal test proved the paid version didn't detect any AI. 

      • lucy_hunt
      • 9 mths ago
      • Reported - view

      Steve Bentley All of my writing preferences are set to "on," too--I'm getting the full dose of what Grammarly Premium offers minus GrammarlyGo. The same goes for my students. 

      • Malik.1
      • 8 mths ago
      • Reported - view

      Gill Rowell Could you please update about the Quillbot investigation?

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 8 mths ago
      • Reported - view

      Malik I don't have any specific updates on Quillbot. In our AI Innovation Lab, we have conducted tests using open sourced paraphrasing tools (including different LLMs) and in most cases, our detector has retained its effectiveness and is able to identify text as AI-generated even when a paraphrasing tool has been used to change the AI output.

      • Malik.1
      • 8 mths ago
      • Reported - view

      Gill Rowell Thank you for your response. Actually, I wanted to confirm that if an AI paraphrasing tool is used on human generated content (to sound smarter or improve structure etc) does turnitin flag it as AI too? 

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 8 mths ago
      • Reported - view

      Hi Gill Rowell just to support Malik's comment, the concern is about avoiding false positive detections where tools like Quillbot and Grammarly have been used to improve a student's authentic writing. This is a common claim from students who are challenged about a high AI score from Turnitin.

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 8 mths ago
      • Reported - view

      I understand that there may be times when using these tools may trigger the AI indicator. You may find some of our resources on false positives useful. Firstly this blog post and video from our Product team. Additionally, some of our pedagogic resources from my colleagues on the TLI team. Using the AI detector, just like the similarity report, is just one indicator of possible misconduct and may be used a starting point for conversations with students about their use of these tools when completing their assessed work. 

      Additionally you can always ask our Support team to take a look at any paper where there may be doubt. You can contact them by emailing tiisupport@turnitin.com.

    • lucy_hunt
    • 9 mths ago
    • Reported - view

    If an essay is flagged with 50% or over for AI, wouldn't that pretty much toss out the "false positive" theory? I get 20% or less can be less reliable, but 50%? 

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 9 mths ago
      • Reported - view

      lucy hunt I'm not sure if this recent blog post from our Chief Product Officer, Annie Chechitelli might help with your understanding of false positives. 

      • TWC
      • 9 mths ago
      • Reported - view

      lucy hunt the % that gets flagged is only the chance that the text is AI written. Just because they are scoring some % AI does not mean they are using AI. The score is the % chance detectors think the article was written by AI not what % of the article was written with AI.

      • lucy_hunt
      • 9 mths ago
      • Reported - view

      Chris Turner isn't 75 to 100% still pretty significant?

      • lucy_hunt
      • 9 mths ago
      • Reported - view

      Chris Turner This is what Turnitin has on its page: "Using the average scores of all the segments within the document, the model then generates an overall prediction of how much text in the submission we believe has been generated by AI."

      • TWC
      • 9 mths ago
      • Reported - view

        lucy hunt 

      AI vs Human score says the probability that AI thinks the content was generated by AI or generated by a person. A score of 20% AI 80% Human Means – The AI gives the content 80% probability of being Human and  20% probability of being AI generated; it doesn't mean – 80% of the article is human, and 20% is AI. My neighbour is the CTO of a large AI company and says that AI detection is not at a point where it can provide enough certainty and gives a high rate of false positives. I have run through pages of engineering textbooks through various AI detection including turnitin and each time the score had between 10- 20% approx AI content. This has to be incorrect as all the text I ran through detection was years old, well before AI writing tools came on the market. I'm interested to see how the detection tools develop, but they appear too inaccurate. 

      • lucy_hunt
      • 9 mths ago
      • Reported - view

      TWC I understand, but I don’t count anything that’s been flagged under 30% from Turnitin. If a student turns in an essay and it comes back with a  70-100% AI detection, I ask them to redo it. If it came back with 30% or less, I ignore it.  

Content aside

  • 6 Likes
  • 7 days agoLast active
  • 119Replies
  • 36747Views
  • 51 Following