Abstract Book of the 10th World Conference on Future of Education
Year: 2026
[PDF]
Methodology for applying generative artificial intelligence (AI) to create tasks for assessment of STEM-competencies
Nadezhda Angelova Kaloyanova, Zlatka Aleksandrova Dimitrova
ABSTRACT:
This study presents the development and expert validation of a methodology for constructing text instruction (input request) to a generative artificial intelligence for creating tasks for measuring STEM-competencies. The methodology is part of an experimental model for developing an adaptive metric system for assessment of STEM-competencies. The current methodology is based on an author’s taxonomic framework, developed as a part of the experimental model. Taxonomic levels are operationalized in mandatory fields of the text instruction to the AI and in a control “taxonomic matrix“, which is required at the output of the generated tasks. The methodology of this study includes the creation of a text instruction template (input request), generation of prototype tasks for 5th grade students and expert evaluation of the tasks received. As a result of the experimental verification of the developed methodology, 3 prototype tasks were generated. The quality of the tasks is assessed by four experts in the STEM field – 2 university professors and two teachers. The expert аassessment was carried out with two Likert-type scales – a scale for evaluate compliance with the taxonomic framework and a scale for compliance with educational requirements in Bulgaria. The Cronbach’s alpha was used to determine the internal consistency of the scales. The congruence of the assessments between the experts in each pair was analyzed by determining the Intraclass Correlation Coefficient – ICC. In conclusion, a comparative analysis was made between the assessments of the expert pairs and conclusions were formulated about the effectiveness of the text instruction.
Keywords: STEM-education; prompt engineering; educational measurement; STEM-tasks; taxonomy