Development of Gamified Higher-Order Thinking Skills Assessment Items for Phase D Informatics through Pedagogical Deep Learning
Keywords:
HOTS assessment, Phase D Informatics, pedagogical deep learning, gamification, digital assessment
Abstract
This study developed and evaluated gamified Higher-Order Thinking Skills (HOTS) assessment items for Phase D Informatics through a pedagogical deep-learning orientation. The research addressed the limited availability of valid, reliable, and practical digital assessment instruments capable of measuring students’ analytical and evaluative thinking in junior secondary Informatics. A Research and Development design was employed by integrating the ADDIE, 4-D, and Tessmer formative-evaluation models into five stages: needs analysis, product design, development and validation, implementation and evaluation, and finalization. The product consisted of 20 contextual multiple-choice items delivered through Quizizz and designed according to Phase D Informatics learning outcomes and the principles of mindful, meaningful, and joyful learning. Content, assessment, and language validation involved three experts, while field testing involved 60 Grade VIII students from three public junior secondary schools in Pinrang Regency, Indonesia. Data were collected using validation sheets, assessment responses, observation records, and teacher and student practicality questionnaires. The results showed an overall expert-validation score of 94.24%, indicating that the instrument was highly valid. Empirical item analysis identified 17 valid items, while three items required substantial revision. The instrument demonstrated acceptable internal consistency, with a Cronbach’s alpha of .735. Most items showed moderate difficulty and adequate-to-good discrimination. Teacher and student responses produced a combined practicality score of 86.81%, categorized as highly practical. These findings indicate that gamification can support the delivery of evaluated HOTS assessment in Phase D Informatics.
Downloads
Download data is not yet available.
References
Bai, S., Hew, K. F., & Huang, B. (2020). Does gamification improve student learning outcome? Evidence from a meta-analysis and synthesis of qualitative data in educational contexts. Educational Research Review, 30, 100322. https://doi.org/10.1016/j.edurev.2020.100322
Branch, R. M. (2009). Instructional design: The ADDIE approach. Springer. https://doi.org/10.1007/978-0-387-09506-6
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game-design elements to gamefulness: Defining gamification. In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (pp. 9–15). Association for Computing Machinery. https://doi.org/10.1145/2181037.2181040
Dichev, C., & Dicheva, D. (2017). Gamifying education: What is known, what is believed, and what remains uncertain—A critical review. International Journal of Educational Technology in Higher Education, 14, Article 9. https://doi.org/10.1186/s41239-017-0042-5
El-Hamamsy, L., Zapata-Cáceres, M., Marcelino, P., Bruno, B., Dehler Zufferey, J., Martín-Barroso, E., & Román-González, M. (2022a). Comparing the psychometric properties of two primary school computational thinking assessments for Grades 3 and 4: The Beginners’ CT Test and the Competent CT Test. Frontiers in Psychology, 13, 1082659. https://doi.org/10.3389/fpsyg.2022.1082659
El-Hamamsy, L., Zapata-Cáceres, M., Martín-Barroso, E., Mondada, F., Dehler Zufferey, J., & Bruno, B. (2022b). The competent computational thinking test: Development and validation of an unplugged computational thinking test for upper primary school. Journal of Educational Computing Research, 60(7), 1818–1866. https://doi.org/10.1177/07356331221081753
El-Hamamsy, L., Zapata-Cáceres, M., Martín-Barroso, E., Mondada, F., Dehler Zufferey, J., Bruno, B., & Román-González, M. (2025). The Competent Computational Thinking Test: A valid, reliable, and gender-fair test for longitudinal CT studies in Grades 3–6. Technology, Knowledge and Learning, 30, 1607–1661. https://doi.org/10.1007/s10758-024-09777-8
Ezeamuzie, N. O., & Leung, J. S. C. (2022). Computational thinking through an empirical lens: A systematic review of literature. Journal of Educational Computing Research, 60(2), 481–511. https://doi.org/10.1177/07356331211033158
Göksün, D. O., & Gürsoy, G. (2019). Comparing success and engagement in gamified learning experiences via Kahoot and Quizizz. Computers & Education, 135, 15–29. https://doi.org/10.1016/j.compedu.2019.02.015
Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–334. https://doi.org/10.1207/S15324818AME1503_5
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
Khaldi, A., Bouzidi, R., & Nader, F. (2023). Gamification of e-learning in higher education: A systematic literature review. Smart Learning Environments, 10, Article 10. https://doi.org/10.1186/s40561-023-00227-z
Kong, S. C., & Lai, M. (2022). Validating a computational thinking concepts test for primary education using item response theory: An analysis of students’ responses. Computers & Education, 187, 104562. https://doi.org/10.1016/j.compedu.2022.104562
Kwon, K., Ottenbreit-Leftwich, A. T., Brush, T. A., Jeon, M., & Yan, G. (2021). Integration of problem-based learning in elementary computer science education: Effects on computational thinking and attitudes. Educational Technology Research and Development, 69, 2761–2787. https://doi.org/10.1007/s11423-021-10034-3
Lewis, A., & Smith, D. (1993). Defining higher-order thinking. Theory Into Practice, 32(3), 131–137. https://doi.org/10.1080/00405849309543588
Li, L., Hew, K. F., & Du, J. (2024). Gamification enhances student intrinsic motivation, perceptions of autonomy and relatedness, but minimal impact on competency: A meta-analysis and systematic review. Educational Technology Research and Development, 72, 765–796. https://doi.org/10.1007/s11423-023-10337-7
Marton, F., & Säljö, R. (1976). On qualitative differences in learning: I—Outcome and process. British Journal of Educational Psychology, 46(1), 4–11. https://doi.org/10.1111/j.2044-8279.1976.tb02980.x
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Morris, R., Perry, T., & Wardle, L. (2021). Formative assessment and feedback for learning in higher education: A systematic review. Review of Education, 9(3), e3292. https://doi.org/10.1002/rev3.3292
OECD. (2023). PISA 2022 results: Volume I—The state of learning and equity in education. OECD Publishing. https://doi.org/10.1787/53f23881-en
Oliveira, W., Hamari, J., Joaquim, S., Toda, A. M., Palomino, P. T., Vassileva, J., & Isotani, S. (2022). The effects of personalized gamification on students’ flow experience, motivation, and enjoyment. Smart Learning Environments, 9, Article 16. https://doi.org/10.1186/s40561-022-00194-x
Pellegrino, J. W. (2014). Assessment as a positive influence on 21st-century teaching and learning: A systems approach to progress. Psicología Educativa, 20(2), 65–77. https://doi.org/10.1016/j.pse.2014.10.001
Redecker, C., & Johannessen, Ø. (2013). Changing assessment—Towards a new assessment paradigm using ICT. European Journal of Education, 48(1), 79–96. https://doi.org/10.1111/ejed.12018
Relkin, E., de Ruiter, L. E., & Bers, M. U. (2021). Learning to code and the acquisition of computational thinking by young children. Computers & Education, 169, 104222. https://doi.org/10.1016/j.compedu.2021.104222
Sailer, M., & Homner, L. (2020). The gamification of learning: A meta-analysis. Educational Psychology Review, 32, 77–112. https://doi.org/10.1007/s10648-019-09498-w
Scully, D. (2017). Constructing multiple-choice items to measure higher-order thinking. Practical Assessment, Research, and Evaluation, 22, Article 4. https://doi.org/10.7275/swgt-rj52
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003
Sun, L., Hu, L., & Zhou, D. (2021). Improving seventh-graders’ computational thinking skills through unplugged programming activities: A study on the influence of multiple factors. Thinking Skills and Creativity, 42, 100926. https://doi.org/10.1016/j.tsc.2021.100926
Taşkın, N., & Kılıç Çakmak, E. (2023). Effects of gamification on behavioral and cognitive engagement of students in the online learning environment. International Journal of Human–Computer Interaction, 39(17), 3334–3345. https://doi.org/10.1080/10447318.2022.2096190
Tessmer, M. (1993). Planning and conducting formative evaluations: Improving the quality of education and training. Kogan Page.
Thiagarajan, S., Semmel, D. S., & Semmel, M. I. (1974). Instructional development for training teachers of exceptional children: A sourcebook. Indiana University. https://eric.ed.gov/?id=ED090725
Tikva, C., & Tambouris, E. (2021). Mapping computational thinking through programming in K–12 education: A conceptual model based on a systematic literature review. Computers & Education, 162, 104083. https://doi.org/10.1016/j.compedu.2020.104083
Wang, A. I., & Tahir, R. (2020). The effect of using Kahoot! for learning—A literature review. Computers & Education, 149, 103818. https://doi.org/10.1016/j.compedu.2020.103818
Warburton, K. (2003). Deep learning and education for sustainability. International Journal of Sustainability in Higher Education, 4(1), 44–56. https://doi.org/10.1108/14676370310455332
Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2016). Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25(1), 127–147. https://doi.org/10.1007/s10956-015-9581-5
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215
Zainuddin, Z., Chu, S. K. W., Shujahat, M., & Perera, C. J. (2020). The impact of gamification on learning and instruction: A systematic review of empirical evidence. Educational Research Review, 30, 100326. https://doi.org/10.1016/j.edurev.2020.100326
Zhang, S., & Wong, G. K. W. (2023). Development and validation of a computational thinking test for lower primary school students. Educational Technology Research and Development, 71, 1595–1630. https://doi.org/10.1007/s11423-023-10231-2
Zhang, Z., & Crawford, J. (2024). EFL learners’ motivation in a gamified formative assessment: The case of Quizizz. Education and Information Technologies, 29, 6217–6239. https://doi.org/10.1007/s10639-023-12034-7
Branch, R. M. (2009). Instructional design: The ADDIE approach. Springer. https://doi.org/10.1007/978-0-387-09506-6
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game-design elements to gamefulness: Defining gamification. In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (pp. 9–15). Association for Computing Machinery. https://doi.org/10.1145/2181037.2181040
Dichev, C., & Dicheva, D. (2017). Gamifying education: What is known, what is believed, and what remains uncertain—A critical review. International Journal of Educational Technology in Higher Education, 14, Article 9. https://doi.org/10.1186/s41239-017-0042-5
El-Hamamsy, L., Zapata-Cáceres, M., Marcelino, P., Bruno, B., Dehler Zufferey, J., Martín-Barroso, E., & Román-González, M. (2022a). Comparing the psychometric properties of two primary school computational thinking assessments for Grades 3 and 4: The Beginners’ CT Test and the Competent CT Test. Frontiers in Psychology, 13, 1082659. https://doi.org/10.3389/fpsyg.2022.1082659
El-Hamamsy, L., Zapata-Cáceres, M., Martín-Barroso, E., Mondada, F., Dehler Zufferey, J., & Bruno, B. (2022b). The competent computational thinking test: Development and validation of an unplugged computational thinking test for upper primary school. Journal of Educational Computing Research, 60(7), 1818–1866. https://doi.org/10.1177/07356331221081753
El-Hamamsy, L., Zapata-Cáceres, M., Martín-Barroso, E., Mondada, F., Dehler Zufferey, J., Bruno, B., & Román-González, M. (2025). The Competent Computational Thinking Test: A valid, reliable, and gender-fair test for longitudinal CT studies in Grades 3–6. Technology, Knowledge and Learning, 30, 1607–1661. https://doi.org/10.1007/s10758-024-09777-8
Ezeamuzie, N. O., & Leung, J. S. C. (2022). Computational thinking through an empirical lens: A systematic review of literature. Journal of Educational Computing Research, 60(2), 481–511. https://doi.org/10.1177/07356331211033158
Göksün, D. O., & Gürsoy, G. (2019). Comparing success and engagement in gamified learning experiences via Kahoot and Quizizz. Computers & Education, 135, 15–29. https://doi.org/10.1016/j.compedu.2019.02.015
Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–334. https://doi.org/10.1207/S15324818AME1503_5
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
Khaldi, A., Bouzidi, R., & Nader, F. (2023). Gamification of e-learning in higher education: A systematic literature review. Smart Learning Environments, 10, Article 10. https://doi.org/10.1186/s40561-023-00227-z
Kong, S. C., & Lai, M. (2022). Validating a computational thinking concepts test for primary education using item response theory: An analysis of students’ responses. Computers & Education, 187, 104562. https://doi.org/10.1016/j.compedu.2022.104562
Kwon, K., Ottenbreit-Leftwich, A. T., Brush, T. A., Jeon, M., & Yan, G. (2021). Integration of problem-based learning in elementary computer science education: Effects on computational thinking and attitudes. Educational Technology Research and Development, 69, 2761–2787. https://doi.org/10.1007/s11423-021-10034-3
Lewis, A., & Smith, D. (1993). Defining higher-order thinking. Theory Into Practice, 32(3), 131–137. https://doi.org/10.1080/00405849309543588
Li, L., Hew, K. F., & Du, J. (2024). Gamification enhances student intrinsic motivation, perceptions of autonomy and relatedness, but minimal impact on competency: A meta-analysis and systematic review. Educational Technology Research and Development, 72, 765–796. https://doi.org/10.1007/s11423-023-10337-7
Marton, F., & Säljö, R. (1976). On qualitative differences in learning: I—Outcome and process. British Journal of Educational Psychology, 46(1), 4–11. https://doi.org/10.1111/j.2044-8279.1976.tb02980.x
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Morris, R., Perry, T., & Wardle, L. (2021). Formative assessment and feedback for learning in higher education: A systematic review. Review of Education, 9(3), e3292. https://doi.org/10.1002/rev3.3292
OECD. (2023). PISA 2022 results: Volume I—The state of learning and equity in education. OECD Publishing. https://doi.org/10.1787/53f23881-en
Oliveira, W., Hamari, J., Joaquim, S., Toda, A. M., Palomino, P. T., Vassileva, J., & Isotani, S. (2022). The effects of personalized gamification on students’ flow experience, motivation, and enjoyment. Smart Learning Environments, 9, Article 16. https://doi.org/10.1186/s40561-022-00194-x
Pellegrino, J. W. (2014). Assessment as a positive influence on 21st-century teaching and learning: A systems approach to progress. Psicología Educativa, 20(2), 65–77. https://doi.org/10.1016/j.pse.2014.10.001
Redecker, C., & Johannessen, Ø. (2013). Changing assessment—Towards a new assessment paradigm using ICT. European Journal of Education, 48(1), 79–96. https://doi.org/10.1111/ejed.12018
Relkin, E., de Ruiter, L. E., & Bers, M. U. (2021). Learning to code and the acquisition of computational thinking by young children. Computers & Education, 169, 104222. https://doi.org/10.1016/j.compedu.2021.104222
Sailer, M., & Homner, L. (2020). The gamification of learning: A meta-analysis. Educational Psychology Review, 32, 77–112. https://doi.org/10.1007/s10648-019-09498-w
Scully, D. (2017). Constructing multiple-choice items to measure higher-order thinking. Practical Assessment, Research, and Evaluation, 22, Article 4. https://doi.org/10.7275/swgt-rj52
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003
Sun, L., Hu, L., & Zhou, D. (2021). Improving seventh-graders’ computational thinking skills through unplugged programming activities: A study on the influence of multiple factors. Thinking Skills and Creativity, 42, 100926. https://doi.org/10.1016/j.tsc.2021.100926
Taşkın, N., & Kılıç Çakmak, E. (2023). Effects of gamification on behavioral and cognitive engagement of students in the online learning environment. International Journal of Human–Computer Interaction, 39(17), 3334–3345. https://doi.org/10.1080/10447318.2022.2096190
Tessmer, M. (1993). Planning and conducting formative evaluations: Improving the quality of education and training. Kogan Page.
Thiagarajan, S., Semmel, D. S., & Semmel, M. I. (1974). Instructional development for training teachers of exceptional children: A sourcebook. Indiana University. https://eric.ed.gov/?id=ED090725
Tikva, C., & Tambouris, E. (2021). Mapping computational thinking through programming in K–12 education: A conceptual model based on a systematic literature review. Computers & Education, 162, 104083. https://doi.org/10.1016/j.compedu.2020.104083
Wang, A. I., & Tahir, R. (2020). The effect of using Kahoot! for learning—A literature review. Computers & Education, 149, 103818. https://doi.org/10.1016/j.compedu.2020.103818
Warburton, K. (2003). Deep learning and education for sustainability. International Journal of Sustainability in Higher Education, 4(1), 44–56. https://doi.org/10.1108/14676370310455332
Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2016). Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25(1), 127–147. https://doi.org/10.1007/s10956-015-9581-5
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215
Zainuddin, Z., Chu, S. K. W., Shujahat, M., & Perera, C. J. (2020). The impact of gamification on learning and instruction: A systematic review of empirical evidence. Educational Research Review, 30, 100326. https://doi.org/10.1016/j.edurev.2020.100326
Zhang, S., & Wong, G. K. W. (2023). Development and validation of a computational thinking test for lower primary school students. Educational Technology Research and Development, 71, 1595–1630. https://doi.org/10.1007/s11423-023-10231-2
Zhang, Z., & Crawford, J. (2024). EFL learners’ motivation in a gamified formative assessment: The case of Quizizz. Education and Information Technologies, 29, 6217–6239. https://doi.org/10.1007/s10639-023-12034-7
Published
2026-05-30
How to Cite
Hasanah, U., Syahrul, & Arfandi, A. (2026). Development of Gamified Higher-Order Thinking Skills Assessment Items for Phase D Informatics through Pedagogical Deep Learning. ISEJ : Indonesian Science Education Journal, 7(2), 458-471. https://doi.org/10.62159/isej.v7i2.2730
Section
Articles
Copyright (c) 2026 Uswatun Hasanah, Syahrul, Anas Arfandi
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with ISEJ: Indonesian Science Education Journal agree to the following terms:
- Authors retain copyright and grant the ISEJ: Indonesian Science Education Journal right of first publication with the work simultaneously licensed under Creative Commons Attribution License (CC BY 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors can enter into separate, additional contractual arrangements for the non-exclusive distribution of the published version of the work (e.g., post it to an institutional repository or edit it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) before and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
.png)
_(3).png)
