Lunch seminar with Niklas Humble
Niklas Humble is a PhD in Computer and Systems Science and affiliated with RCR and the KKI department at Mid Sweden University. His lunch seminar is titled "Prompt Injections in Higher Education – And the Ethical Implications of AI-mediated Academic Work"
"The increasing integration of AI tools into academic work introduces both opportunities and new security risks, particularly concerning the integrity of evaluative practices. This study investigates the robustness of AI-mediated work in higher education by examining whether indirect prompt injection attacks can manipulate course assessment outcomes. The aim is to provide empirical evidence of how vulnerable AI-supported grading processes are to hidden adversarial inputs and to explore wider implications for academic trust.
The study applies a structured adversarial security evaluation, drawing on red-team testing methodologies. A synthetic student submission was generated and paired with five different indirect prompt injection attempts using combinations of different adversarial prompt techniques. Each attempt was iterated multiple times across different file formats against the same AI tool (Microsoft Copilot) under realistic usage conditions.
The results show that several prompt injection attempts successfully altered the output, with some attack strategies achieving success rates between 94-100%, while remaining undetected in the output. These findings highlight a discrepancy between security safeguards and system behaviour, raising concerns about the reliability of AI-mediated academic work. The study connects the results to a broader discussion about trust and integrity in academic work, arguing for development of clear guidelines and policies for AI use, improved professional development in AI use and safety, and standardised assessments for prompt injection resilience in organisations. The study concludes that while AI can support academic work, its susceptibility to manipulation poses significant challenges for maintaining academic trust and integrity."