6

AI Detection and Grammarly

Hi everyone, We have faculty using the AI report and are appreciating that feature. An interesting question came up recently: the report showed 64% AI-generated text, but the student claimed he had only used Grammarly, which also helps to improve phrasing. Has anyone else experienced an AI report picking up on Grammarly, and is there a way of distinguishing between a "grammar help" tool and other generative AI?

Second, will the AI report be visible to students soon? Faculty assumed that students could see it, but the report is only visible to faculty in our LMS.

Thanks,

Jennifer Douglas, American Public University System

114 replies

null
    • Customer Engagement Specialist, Turnitin
    • Gill_Rowell
    • 11 mths ago
    • Reported - view

    Hi Jennifer Douglas Many thanks for your feedback on our new AI Detection feature. We have received several similar reports of this behaviour and are running some tests. It might be worth us investigating this further for you. I will message you directly for more details.

    Regarding student access to the report, this is something we will review in future. 

      • Dean, Graduate Studies and Research
      • Jennifer_Douglas
      • 10 mths ago
      • Reported - view

      Gill Rowell Here is a redacted sample of the Originality report and the AI report for the student in question, who then claimed that he had only used Grammarly and not an AI chatbot tool.

      • reesejohnson1
      • 7 mths ago
      • Reported - view

      Jennifer Douglas Grammarly employs a combination of standard correction detection, which identifies frequently misused words based on patterns, and Chat GPT-4 to enhance its speed and accuracy. Similarly, the latest version of the Windows suite has integrated AI-powered correction tools, as has added the option Google Docs. It's worth noting that I've tested content I personally wrote years before AI tools were accessible to the public, and these detection tools still identified them as AI-generated. I've also noticed most bibliographies become flagged by the detection too aswell. The detection tools might not be advancing as rapidly as AI itself and are not very reliable.

    • Narelle_Hunter
    • 11 mths ago
    • Reported - view

    Our faculty have experienced a similar issue with high AI likelihood but students stating they have used Grammarly, as well as language translation tools, such as Google Translate and not tools such as Chat GPT. I would be interested to know if the Turnitin AI detection is known to identify text from these tools as AI generated also.

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 10 mths ago
      • Reported - view

      Narelle Hunter I believe Google Translate may also cause similar behaviour. Are you able to message me direct with paper IDs so we are able to investigate in more detail? Or alternatively you can use the in-product feedback tool.

    • Paul_Rittman
    • 10 mths ago
    • Reported - view

    Same thing here. I submitted student work and Turnitin gave it a fairly low similarity score, but 100% AI score. The student told me that he had Grammarly edit his work. 

    Unfortunately, he composed his work on Grammarly (at least, that is what he says), so there isn't any original document that was corrected by Grammarly. 

    I have assigned a few zeros to different students, and this is the only one who has come back and insisted it wasn't AI. But this is a problem. I am not going to fault a student for using Grammarly, but man there really is IMO a big difference between this and AI. Assuming that my student was telling the truth, it would be nice for turnitin to know the difference.

    I think I will put in my syllabus for future classes, something to the effect that if you do choose to use Grammarly, to write out your text ahead of time, and keep the pre-edited version. 

      • Dean, Graduate Studies and Research
      • Jennifer_Douglas
      • 10 mths ago
      • Reported - view

      Paul Rittman Good idea to advise students to keep their pre-edited draft and be aware of the amount of editing that grammarly is doing.

    • JoAnne_Rondell
    • 10 mths ago
    • Reported - view

    I just encountered the same problem. According to Turnitin, a student's essay was created by AI. The student, however, claims he used Grammarly only. This is a serious problem, needless to say. Can we know by when Turnitin will be able to distinguish Grammarly from AI? 

      • Paul_Rittman
      • 10 mths ago
      • Reported - view

      Jo-Anne Rondell I am suspecting what they are doing after logging in to Grammarly, is using Grammarly's AI function (called something like GrammarlyGo--you can see it on the right side of the window after you log in). The reason why I say that, is that I've taken texts that were NOT composed by AI, taken them to grammarly and ONLY made grammar suggestions by Grammarly. The edited document had a 0% or very low AI score. So for turnitin to give an AI score of 100% when the student claims they submitted it to Grammarly, tells me that they used Grammarly's AI function. See my post here: https://turnitin.forumbee.com/t/83h4qhz/grammarly-turnitin-coms-ai-score

      • JoAnne_Rondell
      • 10 mths ago
      • Reported - view

      Paul Rittman Thanks very much for the reply. Can you tell me if you tried using Grammarly's AI?

      • Paul_Rittman
      • 10 mths ago
      • Reported - view

      Jo-Anne Rondell At this point I don't remember. I've been pretty busy since that post. :(

      • JoAnne_Rondell
      • 10 mths ago
      • Reported - view

      Paul Rittman Thanks again. I'm pursuing this since I suspect the problems will multiply as we continue on. I'll post back here with whatever I discover, if anything.

      • Paul_Rittman
      • 10 mths ago
      • Reported - view

      Jo-Anne Rondell Sounds good. Look forward to your results.

      • Paul_Rittman
      • 10 mths ago
      • Reported - view

      Jo-Anne Rondell One more thing. If you get some concrete results about how turnitin detects Grammarly AI vs. Grammarly's normal grammar check, I would PM Gill Rowell, who has posted above here in this thread twice. She is a turnitin.com employee who has PMed me about issues I've discussed. I think she would be interested in looking at your files, if you can show how turnitin's AI generator is working.

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 10 mths ago
      • Reported - view

      Jo-Anne Rondell I am happy to pass on any feedback you have to our AI development team. Please DM me with any paper IDs that are exhibiting this behaviour. We are all learning about AI writing!

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 9 mths ago
      • Reported - view

      Paul Rittman I think this is part of the problem, that Grammarly has multiple services, some only available to paid accounts. Free accounts, as I understand it, offer suggestions, paid tiers will rewrite sentences to improve them. Much more of a grey area in terms of acceptability, I know our regulations wouldn't accept that from a human proofreader.

       

      GrammarlyGo as their AI offering is an extra layer on top of that.

      • lucy_hunt
      • 9 mths ago
      • Reported - view

      Jo-Anne Rondell I have just stumbled onto this thread because I've run into this same issue. I tried to test this theory (typing something from scratch in Grammarly, downloading it, and then submitting it in canvas), but I guess because I was in "student view" and not an actual student, the AI detection was unable to be processed. Literally, the box was gray and it never registered. I did this twice. I have since resorted to asking a student I know and trust (who is in my class) to test this as well so I can see if it is in fact Grammarly that's being detected. If so, the two essays written by another student that were detected to have been generated by AI (100%) will be graded as normal. I'm hoping Turnitin's AI detector will be able to discern Grammarly from the other AI generators, but who knows. Should we discourage our students from starting from scratch in Grammarly? 

      • lucy_hunt
      • 9 mths ago
      • Reported - view

      Paul Rittman So, I figured the same thing (with GrammarlyGO), and I went to my Grammarly account to check. Our institution subscribes to Grammarly premium, so when I clicked on "features," the GrammarlyGO was disabled by admin. So if it's disabled on my end, it would have to be disabled on my students' Grammarlys right? That would mean that the AI detector is literally just detecting essays started and completed in Grammarly (but not using GrammarlyGO) as AI-generated. 

      • Emma_DukeWilliams
      • 5 mths ago
      • Reported - view

      lucy hunt COuld the student also have a free a/c with GO enabled? You get so many prompts for free ... 

      • lucy_hunt
      • 5 mths ago
      • Reported - view

      Emma Duke-Williams Super probable! I tell them they are only allowed to use their Grammarly accounts through the school, but of course, I'm sure some will use their personal accounts to sign up with Grammarly. I also tell my students that Turnitin will flag any and all AI sites/features--even if they're only using them to revise or clean up their writing. I just say avoid all AI sites and features on your essays that you have to submit to canvas. 

    • Fiona_Heagerty
    • 10 mths ago
    • Reported - view

    So interesting to read this thread.  I had a student yesterday state that he only used Grammarly on an assignment that came up with a 73% AI score.  He also sent me a list of news articles about students who have been falsely accused of cheating based on Turnitin AI software detection.  I would be interested in any progress you make Gill Rowell in working out if it's picking up Grammerly or Grammerly AI (I agree with Paul Rittman - I think it will be the latter.  

      • Paul_Rittman
      • 10 mths ago
      • Reported - view

      Fiona Heagerty Fiona, I have found two other websites that detect AI content in written work. One is GPTZero (https://app.gptzero.me/app/welcome), the other is Writer.com AI detector (https://writer.com/ai-content-detector/). If a student is objecting, I would run the written work past those two other sources as well.

      • Fiona_Heagerty
      • 10 mths ago
      • Reported - view

      Thanks Paul. I’ll take a look at those. Paul Rittman 

    • Daniel_Hernandez
    • 10 mths ago
    • Reported - view

    ok and so what

    • Amy_henderson
    • 10 mths ago
    • Reported - view

    Hi all, thank you for this forum. I have run into a similar predicament with a few of my students showing up with high AI scores. One of my students who I have personally seen writing during my office hours and asking questions about their essay turned in a paper with 90% AI detection score. After talking with them, she mentioned that she uses Grammarly for all of her writing and that this may be the cause. I would love to stay updated Gill Rowell about the prevalence of this situation. As it stands right now, I will not be giving any zeros based on AI scores until it is clear that Grammarly does not affect the report.

Content aside

  • 6 Likes
  • 21 hrs agoLast active
  • 114Replies
  • 31366Views
  • 47 Following