Development of an Explainable AI Framework in Information Systems to Enhance User Trust and Effectiveness
Keywords:
Explainable AI, Information Systems, User Trust, Human-Centered AI, System EffectivenessAbstract
This study aims to develop an Explainable Artificial Intelligence framework in information systems to enhance user trust and system effectiveness. The study addresses the growing problem of limited user understanding of AI-based recommendations, especially when users must make decisions based on outputs that are difficult to interpret. A qualitative exploratory case study design was used to examine users’ experiences, meanings, and social processes when interacting with AI-based information systems in organizational settings. Data were collected through semi-structured interviews, non-participant observation, and documentation involving 15 to 20 informants, including end users, system administrators, developers or data analysts, unit managers, and decision-makers who interact with AI-supported systems. The data were analyzed using reflexive thematic analysis supported by the Miles and Huberman interactive model. The findings identified five main themes: limited understanding of AI decision logic, trust shaped by explanation clarity, contextual and role-based explanation needs, the importance of human control, and the contribution of explainability to work effectiveness. These findings show that XAI strengthens user trust when explanations are simple, relevant, transparent, and connected to real work practices. The study contributes to human-centered XAI and trust calibration literature by emphasizing the importance of explanation quality, user role, and organizational context. Practically, the study suggests that AI-based information systems should include layered explanations, human oversight, and feedback mechanisms to support responsible and effective system use.
References
Adeoye-Olatunde, O. A., & Olenik, N. L. (2021). Research and scholarly methods: Semi-structured interviews. Journal of the American College of Clinical Pharmacy, 4(10), 1358-1367. https://doi.org/10.1002/jac5.1441
Ahmed, S. K. (2024). The pillars of trustworthiness in qualitative research. Journal of Medicine, Surgery, and Public Health, 2, Article 100051. https://doi.org/10.1016/j.glmedi.2024.100051
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023). Explainable artificial intelligence (XAI): What we know and what is left to attain trustworthy artificial intelligence. Information Fusion, 99, Article 101805. https://doi.org/10.1016/j.inffus.2023.101805
Aoki, N. (2024). Explainable AI for government: Does the type of explanation matter to the accuracy, fairness, and trustworthiness of an algorithmic decision as perceived by those who are affected? Government Information Quarterly, 41(4), Article 101965. https://doi.org/10.1016/j.giq.2024.101965
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., & Herrera, F. (2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82-115. https://doi.org/10.1016/j.inffus.2019.12.012
Bach, T. A., Khan, A., Hallock, H., Beltrão, G., & Sousa, S. (2024). A systematic literature review of user trust in AI-enabled systems: An HCI perspective. International Journal of Human-Computer Interaction, 40(5), 1251-1266. https://doi.org/10.1080/10447318.2022.2138826
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-16). Association for Computing Machinery. https://doi.org/10.1145/3411764.3445717
Brasse, J., Broder, H. R., Förster, M., Klier, M., & Sigler, I. (2023). Explainable artificial intelligence in information systems: A review of the status quo and future research directions. Electronic Markets, 33, Article 26. https://doi.org/10.1007/s12525-023-00644-5
Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in reflexive thematic analysis? Qualitative Research in Psychology, 18(3), 328-352. https://doi.org/10.1080/14780887.2020.1769238
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), Article 188. https://doi.org/10.1145/3449287
Campbell, S., Greenwood, M., Prior, S., Shearer, T., Walkem, K., Young, S., Bywaters, D., & Walker, K. (2020). Purposive sampling: Complex or simple? Research case examples. Journal of Research in Nursing, 25(8), 652-661. https://doi.org/10.1177/1744987120927206
Choung, H., David, P., & Ross, A. (2023). Trust in AI and its role in the acceptance of AI technologies. International Journal of Human-Computer Interaction, 39(9), 1727-1739. https://doi.org/10.1080/10447318.2022.2050543
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021). Expanding explainability: Towards social transparency in AI systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-19). Association for Computing Machinery. https://doi.org/10.1145/3411764.3445188
Ferrario, A., & Loi, M. (2022). How explainability contributes to trust in AI. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 1457-1466). Association for Computing Machinery. https://doi.org/10.1145/3531146.3533202
Hase, P., & Bansal, M. (2020). Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 5540-5552). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.491
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021). Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AI. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 624-635). Association for Computing Machinery. https://doi.org/10.1145/3442188.3445923
Judijanto, L., Achmady, S., Fadhilah, S. A. N., & Vandika, A. Y. (2025). The impact of explainable AI on user trust and ethical AI adoption in Indonesia. The Eastasouth Journal of Information System and Computer Science, 2(03), 344-351. https://doi.org/10.58812/esiscs.v2i03.531
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020). Interpreting interpretability: Understanding data scientists’ use of interpretability tools for machine learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-14). Association for Computing Machinery. https://doi.org/10.1145/3313831.3376219
Kim, J., Maathuis, H., & Sent, D. (2024). Human-centered evaluation of explainable AI applications: A systematic review. Frontiers in Artificial Intelligence, 7, Article 1456486. https://doi.org/10.3389/frai.2024.1456486
Langer, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., Sesing, A., & Baum, K. (2021). What do we want from explainable artificial intelligence (XAI)? A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence, 296, Article 103473. https://doi.org/10.1016/j.artint.2021.103473
Leichtmann, B., Humer, C., Hinterreiter, A., Streit, M., & Mara, M. (2023). Effects of explainable artificial intelligence on trust and human behavior in a high-risk decision task. Computers in Human Behavior, 139, Article 107539. https://doi.org/10.1016/j.chb.2022.107539
Liao, Q. V., Gruen, D., & Miller, S. (2020). Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-15). Association for Computing Machinery. https://doi.org/10.1145/3313831.3376590
Lim, W. M. (2025). What is qualitative research? An overview and guidelines. Australasian Marketing Journal, 33(2), 199-229. https://doi.org/10.1177/14413582241264619
Ridley, M. (2025). Human-centered explainable artificial intelligence: An Annual Review of Information Science and Technology (ARIST) paper. Journal of the Association for Information Science and Technology, 76(1), 98-120. https://doi.org/10.1002/asi.24889
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024). Towards human-centered explainable AI: A survey of user studies for model explanations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4), 2104-2122. https://doi.org/10.1109/TPAMI.2023.3331846
Scharowski, N., Perrig, S. A. C., Svab, M., Opwis, K., & Brühlmann, F. (2023). Exploring the effects of human-centered AI explanations on trust and reliance. Frontiers in Computer Science, 5, Article 1151150. https://doi.org/10.3389/fcomp.2023.1151150
Shin, D. (2021). The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI. International Journal of Human-Computer Studies, 146, Article 102551. https://doi.org/10.1016/j.ijhcs.2020.102551
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021). Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-16). Association for Computing Machinery. https://doi.org/10.1145/3411764.3445088
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 295-305). Association for Computing Machinery. https://doi.org/10.1145/3351095.3372852
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Maya Pratama, Nadia Wibowo, Rizki Lestari (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
