Creating a valid and reliable assessment tool for simulation-based learning in allied health education is a challenging but rewarding process. The development of the SIMBA Assessment Tool for Simulation-Based Learning followed a systematic and structured approach. The first step involved a systematic review of the existing literature to identify approaches and methods used in simulation-based assessments in health education, to understand the psychometric properties of these assessments, and to evaluate their potential usefulness in OT education. This review of the literature provided a solid foundation for understanding the current state of simulation-based assessments and highlighted areas needing improvement or adaptation.
To complement the findings from the literature, two expert focus groups were conducted, which brought together students, teachers, and researchers with experience in simulation-based learning. Their input provided valuable insights into practical challenges when conducting and evaluating students’ performance during simulation-based learning. Focus group participants provided information about their diverse needs and expectations regarding assessment in simulation-based learning, which we used to ensure the SIMBA assessment tool would effectively address real-world requirements.
Using an iterative approach, the team developed a draft version of the assessment tool. This process involved three iteration loops that engaged educators, students and experts in SBL and occupational therapy and allowed for continuous refinement of the tool based on feedback and new insights. In June 2024, the draft assessment was ready for pilot implementation. To support this phase, the team provided a short explanatory video and a detailed presentation in English and conducted live “train-the-trainer” workshops for instructors at each SIMBA partner university. The SIMBA assessment was also translated into each of the respective languages of partner organizations – German, Dutch, Croatian, Turkish and Finnish. Engaging and considering different perspectives from numerous stakeholders within this iterative development process marks the first step towards assessing the tool’s validity.
Feedback from students and teachers provided early insights into how the assessment tool performs in practice. In the accompanying photo, you can see students engaged in a simulation-based learning scenario, during which they are evaluating their role-playing colleagues using the assessment tool. Following the simulation, these students provided feedback on the tool’s content validity via questionnaire. The results of pilot testing offered valuable insights in further refining the assessment.
The next steps in our evaluation of the assessment tool involves gathering reliability metrics to ensure the assessment’s consistency and accuracy. A study protocol has already been developed for this purpose and ethical approval for data collection has been granted. Data collection for the full validity and reliability study started in October 2024 and results are expected by April 2025. These results will be used to refine the tool and finalize the first version.
Developing a robust assessment tool for simulation-based learning requires careful planning, collaboration, and adaptability. By combining evidence from the literature, expert opinions, and real-world data, this project aims to create an effective and reliable tool to support assessment of occupational therapy competencies during simulation-based learning and thus enhance the quality of occupational therapy education. Stay tuned for the final results in April 2025!