Higher Education Examination

Develop examinations and learning

Examination and knowledge check-ups are an important part of the learning process and a way for you as a teacher to develop your teaching. A knowledge taxonomy, such as Benjamin Bloom's revised taxonomy, can be helpful. There, knowledge is arranged in hierarchical levels. Based on the levels, examinations and lessons can be created to get the desired effect. Bloom's revised taxonomy has six steps and is about theoretical knowledge. The lowest, easiest step is to remember and the highest, most complex is to create. More complex assignments and examinations can make it more difficult to plagiarize or cheat in other ways.

Another useful taxonomy is the John Biggs and Kevin Collis SOLO taxonomy. SOLO stands for Structure of the Observed Learning Outcomes and the taxonomy should describe the progression in the students' knowledge, from the simple (pre-structural/prestructural) to the complex (extended abstract).

Here are Bloom's revised and SOLO taxonomy next to each other, for comparison.

Open-ended questions, constructed with active verbs from the higher levels of the taxonomies, where students must connect course material/literature to their own opinions, thoughts, experiences and explain and prove their knowledge, provide both deeper learning and are more difficult to plagiarize or cheat in other ways.

In order for the student to succeed in the examination, clear and known assessment criteria are of the utmost importance. The examination, just like the course as a whole and the planned learning activities, must be designed so that they really measure the intended learning outcomes and also so that assessment is easy and legally. Read more here:

• Legally secure examination from UKÄ 
• Bergqvist, J Putting Practice on Paper: A Handbook in Writing Grading Criteria
• Laurillard, D. (2012). Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology. New York: Routledge.

Using quizzes or different types of check-ups the form of questions via digital tools that allow for real-time visualization gives a good idea on the level of knowledge in a group. Based on the answers, teaching and materials can then be adapted. Working in this way is also time-efficient for the teacher. Quizzes have also been shown to have a very good effect on learning (see, for example, Case, J and Kennedy, D: Using Quizzes Effectively: Understanding the Effects of Quiz Timing on Student Motivation and Knowledge Retention). Having students take a test early on in a course, for example in the form of a self-correcting quiz, on what the course covers gives students a good picture of what they are going to learn and the teacher a greater chance of planning their teaching. Quizzes can be self-correcting and even anonymous. An AI service can be very helpful in creating quiz questions. The best results are obtained if you can upload material that will form the basis for the quiz questions. Questions created in this way should always be reviewed for accuracy. 

Oral examination is a legally secure and effective way to examine up to 60-70 students, you  can ask follow-up questions and interact, which benefits the student and the learning. It is also cost-effective because more students pass the examination at the same time.

Oral examination can be carried out in a variety of ways. Individually, in pairs or groups. Such as presentations, panel debates, simulations, role-plays or in the form of seminar discussions. AI can be used to create documentation, questions, case studies or texts that are used in the examination. The oral element can be preceded by a written submission or written examination, or these elements can follow up on the oral part. In both cases, it becomes more difficult to simply reproduce what someone else (or an AI service) produced. Encouraging questions and feedback from other students in connection with the implementation can be a good idea to get a dynamic knowledge check. Feedback can be followed up by revisions or comments.

Initially, oral examination can be perceived as very time-consuming, but often you save the same amount of time on marking written examination assignments. To make it easier, it can be a good idea to make clear data for assessment that is used during the implementation, for example in the form of matrices based on the intended learning outcomes. Likewise, clear and equivalent instructions to students, for example in writing, usually give better results. Questions, cases, or assignments can be varied between individuals and groups for increased legal certainty. Oral examination can also be carried out digitally. Consider recording or, for example, allowing the student to supplement with a written compilation of sources that they may comment on during the examination. 

Project-based examination can also be carried out in many ways and is excellent for following a process and using formative assessment. Here, for example, AI can be part of the process by allowing students to use AI services to generate something (text, code, images, and so on) that is evaluated and improved at various stages. This can be carried out individually or in groups. AI can be used throughout the project or in isolated parts. Collaboration on specific problems and development of solutions, with documentation in the form of individual logs with meta-reflections and links to course literature or other sources can provide valuable insight into the level of knowledge. 

Other tips are to let the students write blogs/vlogs (video blogs), wikis, create films, e-portfolios, open-book examinations and to let the students create exam or quiz questions.

Some overall tips for examinations:

• Focus on intended learning outcomes and syllabus when you create the examination.
• Be specific in instructions and requirements. Clarify what is expected in all parts of the examination - including before it and what resources may be used in each step.
• Have the process at the center - work with formative assessment during the course instead of, or together with, a summative assessment at the end of the course, such as a written exam or an oral examination. During the process, get to know the student's own voice, in both text and speech, as far as possible.
• Create complex examinations with several different parts where you examine in different ways. It also creates a more accessible examination where more students will be able to demonstrate their knowledge in the best way.
• Create original questions that require original answers - bring in context and make the examination important (for example, through real recipients).
  • Personal reflections and credible descriptions of one's own experiences linked to literature and teaching.
  • Link examinations to current events or things that have happened during the course (study visits, seminars, named guest lecturers, etc.).
• Use oral examinations - as a partial examination or as a follow-up to other types of examinations. See more below.
• Require references (including page references, regardless of the reference system) and set requirements for the references that should be present. Requirements can be specific course literature, specific edition and the like. 
• In some situations, a written exam is the best choice. It can be when it is about presenting facts or basic knowledge or writing a text that in itself is what is being examined.

Oral examination can be used without changing the syllabus if it is used as a supplement to achieve a passing result after a written written examination or take-home examination, or as a supplement in case of suspicion of cheating, and regularly as a supplement even when there is no suspicion of cheating. In the most recent case, a random selection of a number of students is made. In all cases, it is important to inform students that this procedure exists and will be used.

AI and examinations

Examinations in higher education are often about writing texts, something that has caused concern about the development of generative AI. In this context, it is important to know that so-called detection services that claim to be able to identify AI-written text are not reliable. For example, read more in the study Testing of Detection Tools for AI-Generated Text. The study covers a number of tools and shows the same results as several other studies in the same area. The studies show that detection tools can give both false positives and false negatives. It is also not possible to ask any AI tool if it generated the text, as the answer is not to be trusted at all. 

In the context of "AI and cheating", it is important to clearly communicate your view of what constitutes academic honesty to the students.   Talking about cheating and what it is is the most important step to prevent it. To be able to do that, you need to think through what knowledge the student must possess and what knowledge, for example, an AI tool can assist with without the student losing anything in their education. We will probably all work with AI in a completely different way in the future. Based on this, it is a good idea to test your examinations in an AI service and see what kind of answers you get. The quality of the answers is very dependent on how you prompt, which can both lull into false security and frighten. In addition to planning examinations with the higher levels of the taxonomies above in mind and based on the tips under "Overall tips for examinations", it is worth keeping in mind that cheating is not something new that appeared with generative AI.   It is easier to directly use AI to generate text, or plagiarize from a source,  if the assignment is intended to account for or explain something, than a text that requires the student to connect context, specific course literature and/or other literature with justifications about why this particular source has been chosen, own experiences and examples and analyze, evaluate and develop based on this. However, it is easy to use any generative AI tool to generate parts or entire texts that are then changed, for example with other AI tools or through your own processing. It is not possible to design completely "cheat-proof" examinations, but it is more difficult to cheat on supervised written examinations, oral examinations on campus and practical examinations such as laboratory sessions, VFU or construction of some form of physical artefact. Online oral exams or online exams are also relatively difficult to cheat on, but not at all impossible. The same applies to examination seminars and process writing where the supervisor is involved in the writing process. It's important to remember that cheating involves many methods other than AI-generated text, and it's nothing new. The best way to avoid cheating is to have an open and clear dialogue with students about what constitutes academic honesty and cheating. Have a policy or guidelines on academic honesty where the use of AI services is a part. Also to vary their forms of examination and be careful with the design of these. Also consider adjusting the syllabus with regard to cheating with generative AI services, for example so that examinations can be supplemented. 

Text-generative AI has a harder time producing texts that contain idiomatic expressions, colloquialisms, dialectal expressions, spelling mistakes, and the type of linguistic errors that people with a mother tongue other than Swedish make (examples of such are prepositional errors, semantic errors, or errors in sentence structure). That being said, it is still important to know that the development of what generative AI tools are capable of producing is very fast and there is a big difference between free and paid versions of the tools. 

Do you want support in the management of generative AI and examination?

Contact Pedagogical Development at PUkontakt@miun.se or www.miun.se/boka-pu