7

AI Detection and Grammarly

Hi everyone, We have faculty using the AI report and are appreciating that feature. An interesting question came up recently: the report showed 64% AI-generated text, but the student claimed he had only used Grammarly, which also helps to improve phrasing. Has anyone else experienced an AI report picking up on Grammarly, and is there a way of distinguishing between a "grammar help" tool and other generative AI?

Second, will the AI report be visible to students soon? Faculty assumed that students could see it, but the report is only visible to faculty in our LMS.

Thanks,

Jennifer Douglas, American Public University System

134 replies

null
    • lucy_hunt
    • 1 yr ago
    • Reported - view

    If an essay is flagged with 50% or over for AI, wouldn't that pretty much toss out the "false positive" theory? I get 20% or less can be less reliable, but 50%? 

      • lucy_hunt
      • 1 yr ago
      • Reported - view

      Chris Turner This is what Turnitin has on its page: "Using the average scores of all the segments within the document, the model then generates an overall prediction of how much text in the submission we believe has been generated by AI."

      • TWC
      • 1 yr ago
      • Reported - view

        lucy hunt 

      AI vs Human score says the probability that AI thinks the content was generated by AI or generated by a person. A score of 20% AI 80% Human Means – The AI gives the content 80% probability of being Human and  20% probability of being AI generated; it doesn't mean – 80% of the article is human, and 20% is AI. My neighbour is the CTO of a large AI company and says that AI detection is not at a point where it can provide enough certainty and gives a high rate of false positives. I have run through pages of engineering textbooks through various AI detection including turnitin and each time the score had between 10- 20% approx AI content. This has to be incorrect as all the text I ran through detection was years old, well before AI writing tools came on the market. I'm interested to see how the detection tools develop, but they appear too inaccurate. 

      • lucy_hunt
      • 1 yr ago
      • Reported - view

      TWC I understand, but I don’t count anything that’s been flagged under 30% from Turnitin. If a student turns in an essay and it comes back with a  70-100% AI detection, I ask them to redo it. If it came back with 30% or less, I ignore it.  

    • TWC
    • 1 yr ago
    • Reported - view

    Also don't forget there is Grammarly premium for desktop and browser use. Many folks use this application or similar writing tools if they have dyslexia or chronic fatigue that can affect concentration.  Grammarly premium significantly changes sentence structure or whole paragraphs. 

    • Senior Teaching & Learning Innovations Specialist
    • Karen_Turnitin
    • 1 yr ago
    • Reported - view

    It helps if you think of the percentage as only one data point to have a discussion with students; ideally, that would take place during drafting, rather than submission. Two resources I'd suggest for this are this one on approaching a student when you think AI misuse has occurred or discussion starters for that situation. 

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 1 yr ago
      • Reported - view

      Karen Smith We're taking a view that the AI report alone is not sufficient to have that conversation. A "magic number" and no way to explain how that number has been reached is not a good basis to be making accusations - however gently they are framed. 

      Our approach is to require some additional evidence, such as falsified references or differences in writing style or standard of English compared with previous work before we act on the report, mainly because we don't want to damage staff-student relationships with this kind of conversation based on something that we can't back up and where it's the student's word against the Turnitin algorithm.

      • Senior Teaching & Learning Innovations Specialist
      • Karen_Turnitin
      • 1 yr ago
      • Reported - view

      Steve Bentley Yes, exactly. Many of our resources speak to the need for other data points. If I were still in the classroom, that number would nudge me to look more closely and have the conversations if it seems warranted. 

    • Angelo_Marin
    • 1 yr ago
    • Reported - view

    a foremost authority on AI detection is Atheer Mahir. He has a copious thread on this subject on his Linkedin profile

      • Joseph_Jacob
      • 1 yr ago
      • Reported - view

      Angelo Marin I concur with you. His articles suggest that these apps often misjudge content. His tests show AI texts seem entirely human, while older texts appear 80% AI-generated. This highlights detection tools' inadequacy and potential for false accusations.

    • David_Silver
    • 1 yr ago
    • Reported - view

    I have just joined this network and have reviewed the posts below. I have had some instance where TII has flagged students as having 50% AI generated content. However, some students are adamant they did not use AI or Grammarly/Quillbot. Are they just lying to me or are there any other possibilities as to why their work might be flagged? I had a student initially get flagged at 58%. They made numerous edits to the essay (as seen via google docs) and the resubmission still got flagged at 60%. I am having trouble understanding what might be the issue and would love some guidance. 

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 1 yr ago
      • Reported - view

      David Silver Thank you for joining us! I think there are a few things to unpack here. Firstly, it's important to say that the AI detection indicator, just like the Similarity score is just one part of the conversation regarding possible misconduct.

      It seems like you have already engaged in a conversation with the student regarding their work. And I would say that if the student is adamant that this is their own work it might be worth asking our Support team to take a look at the paper. You can contact them by emailing tiisupport@turnitin.com.

      Additionally you may find some of our resources on false positives useful. Firstly this blog post and video from our Product team. Additionally some of our pedagogic resources from my colleagues on the TLI team. 

    • Esther_Jubb
    • 1 yr ago
    • Reported - view

    I have a similar case with a student who claims to have translated their work from their native language into English.  Is this likely to have produced a high indicator? ChatGPT does a great job of translation...better than google translate

      • Turnitin Admin, University of Huddersfield, UK
      • s_d_bentley
      • 1 yr ago
      • Reported - view

      Esther Jubb Potentially yes, but who knows as the detection tool doesn't show its working.

    • John_Conner
    • 1 yr ago
    • Reported - view

    Two of my own papers in progress tested positive for AI even though I used the basic Grammarly functions in editing the drafts. I did not use Grammarly's AI function. I'm not sure how I can trust Turnitin when it shows a false positive for me, much less send the paper out for review if it's going to come back as a false positive at a journal. This needs to be fixed. I certainly cannot use this to give a student a failing grade if it's going to open me to any legal repercussions.

      • Senior Teaching & Learning Innovations Specialist
      • Karen_Turnitin
      • 1 yr ago
      • Reported - view

      John Conner I recommend the following info on false positives: The first blog from March and the second blog from June of this year.

      As for the student use, I would say to use the score as a discussion starter with students: How did they use the tool and why? Did it fit into the parameters of the paper, etc. From there it's easier to discuss what consequences are appropriate. Which may be a rewrite or it may require some disciplinary measures. But it's only a starting point, not a definitive or solo data point.

    • lucy_hunt
    • 1 yr ago
    • Reported - view

    I don’t even glance at anything that is UNDER 50% AI detected. If it’s OVER 50%, they have to redo it. 

    • Lucinda_Dawes
    • 1 yr ago
    • Reported - view

    I do not rate Tii AI check at all. It is easy to make a 100% AI written document into a 0% AI written, simply by changing half a dozen words (max). Lancaster have turned it off.

    • Karen_Tham
    • 1 yr ago
    • Reported - view

    Hi.

    I'm just joining this thread as I have a student whose work has turned up 50% or above on MULTIPLE assignments, and I now also have in-class evidence of his work that clearly shows a discrepancy in voice and usage. English is not his native language and he admits to using Grammarly extensively--spending even more time "editing" his work than writing it. To me, this is not developing his own voice but merely becoming an adept 'sound-mixer'. His work is so incredibly academic-sounding that it's quite inaccessible to his high-school peers. I am compelled to have him complete all work in class without an internet-accessible computer or by hand. After he submits this AI-unassisted version, he can revise. 

    Note that our school uses TurnItIn and AI-detection does not always work for his work. I have cut and pasted it into ChatGPTZero instead (I have multiple samples in just a few weeks.)

    Does TurnItIn use ChatGPTZero-type tools? 

    • Debbie_Goldberg
    • 1 yr ago
    • Reported - view

    This just happened to one of the essays I was grading - The student said she used Grammarly

    • ren_casey
    • 1 yr ago
    • Reported - view

    Are students required to cite AI if they used Grammarly gen AI and asked it to "improve it"? This is an option on Grammarly now. The result is similar content-wise but often re-written for clarity and such. Will this get flagged on Turnitin or other plagiarism/AI detectors? I'm not sure what qualifies as plagiarism (assuming the student did not cite AI) in terms of using AI as an academic tool. Particularly I want to know if this is acceptable for the International Baccalaureate in writing portions of Arts exams. Is it acceptable if AI is cited for editing/paraphrasing? Is it not acceptable to use AI at all? 

     

    Please reply with your thoughts! I realize this is a tricky situation since AI is so new these days and we are all trying to adapt to these recent developments. Thanks!

      • Annelize_McKay
      • 1 yr ago
      • Reported - view

      ren casey 

      • Annelize_McKay
      • 1 yr ago
      • Reported - view

      ren casey There appears to be two versions of Grammarly - the free version which is AI, and the paid-for version which is not AI.

      The (free) AI version now can do what AI LLMs such as ChatGPT does - it generates text.

      That is what I learnt from students and the seminars I attended on the topic (and the people at TurnItIn. In discussions I have asked students who used the free (AI) version of Grammarly about the content of what they have written, and they do not have a clue - prompting me to think it has generated the text for them. It does not merely check grammar, spelling, style, etc. I therefore think that, indeed, the free version of Grammarly is a form of AI and students should declare its use.

      As well, we are supposed to teach students the skill of writing and we fail in doing that if a robot merely does the writing for them.

      But others may disagree with me?

      • Annelize_McKay
      • 1 yr ago
      • Reported - view

      Annelize McKay Oh, and yes, in my experience it seems TuirnItIn picks up Grammarly as AI.

      • ren_casey
      • 1 yr ago
      • Reported - view

      Annelize McKay One can use grammarly for spelling and grammar help without using their AI feature but in using the AI feature, there is the possibility of it entirely writing a piece for you based on whatever you prompt it with. Thank you for your response, but the instance which I'm referring to does not include text generated fully by AI, instead submitted and prompted to "improve" the work. 

      • Customer Engagement Specialist, Turnitin
      • Gill_Rowell
      • 1 yr ago
      • Reported - view

      ren casey I think this is something the IBO would need to give you guidance on. The following statement may be useful.

Content aside

  • 7 Likes
  • 2 mths agoLast active
  • 134Replies
  • 77011Views
  • 61 Following