Integrating Explainable Artificial Intelligence and Human-Centered Design into Information Systems to Improve Data-Driven Decision-Making
Keywords:
Explainable Artificial Intelligence, Human-Centered Design, Information Systems, Data-Driven Decision-MakingAbstract
This study aims to explore how Explainable Artificial Intelligence and Human-Centered Design can be integrated into information systems to improve data-driven decision-making. The study addresses the growing use of AI-supported information systems and the problem of limited user understanding, trust, and accountability in interpreting AI recommendations. A qualitative case study design was used to examine user experiences in three organizations in Indonesia that had implemented data-based information systems, analytics platforms, or AI-supported decision tools. Data were collected through semi-structured interviews, limited participant observation, and documentation involving information system users, unit managers, data analysts, system developers, and organizational decision-makers. The data were analyzed using reflexive thematic analysis. The findings reveal five main themes: the importance of understandable AI explanations, the role of trust and human judgment, the influence of user-centered interface design, organizational barriers related to data quality and culture, and the need for accountability in AI-supported decisions. These findings show that AI recommendations become valuable when users can interpret, verify, and adapt them to organizational contexts. The study contributes to information systems literature by strengthening a socio-technical understanding of XAI and Human-Centered Design. It also offers practical implications for designing transparent, usable, and accountable AI-supported information systems. Future research can examine different sectors or test the relationship between explanation quality, trust, usability, and decision performance.
References
Ahmed, S. K. (2024). The pillars of trustworthiness in qualitative research. Journal of Medicine, Surgery, and Public Health, 2, Article 100051. https://doi.org/10.1016/j.glmedi.2024.100051
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–16). Association for Computing Machinery. https://doi.org/10.1145/3411764.3445717
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., & Herrera, F. (2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82–115. https://doi.org/10.1016/j.inffus.2019.12.012
Berente, N., Gu, B., Recker, J., & Santhanam, R. (2021). Managing artificial intelligence. MIS Quarterly, 45(3), 1433–1450. https://doi.org/10.25300/MISQ/2021/16274
Brasse, J., Broder, H. R., Förster, M., Klier, M., & Sigler, I. (2023). Explainable artificial intelligence in information systems: A review of the status quo and future research directions. Electronic Markets, 33, Article 26. https://doi.org/10.1007/s12525-023-00644-5
Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in reflexive thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. https://doi.org/10.1080/14780887.2020.1769238
Campbell, S., Greenwood, M., Prior, S., Shearer, T., Walkem, K., Young, S., Bywaters, D., & Walker, K. (2020). Purposive sampling: Complex or simple? Research case examples. Journal of Research in Nursing, 25(8), 652–661. https://doi.org/10.1177/1744987120927206
Collins, C., Dennehy, D., Conboy, K., & Mikalef, P. (2021). Artificial intelligence in information systems research: A systematic literature review and research agenda. International Journal of Information Management, 60, Article 102383. https://doi.org/10.1016/j.ijinfomgt.2021.102383
Coussement, K., Abedin, M. Z., Kraus, M., Maldonado, S., & Topuz, K. (2024). Explainable AI for enhanced decision-making. Decision Support Systems, 184, Article 114276. https://doi.org/10.1016/j.dss.2024.114276
Ehsan, U., & Riedl, M. O. (2020). Human-centered explainable AI: Towards a reflective sociotechnical approach. In HCI International 2020: Late breaking papers: Multimodality and intelligence (pp. 449–466). Springer. https://doi.org/10.1007/978-3-030-60117-1_33
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021). Expanding explainability: Towards social transparency in AI systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–19). Association for Computing Machinery. https://doi.org/10.1145/3411764.3445188
Hennink, M., & Kaiser, B. N. (2022). Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social Science & Medicine, 292, Article 114523. https://doi.org/10.1016/j.socscimed.2021.114523
Janssen, M., Brous, P., Estevez, E., Barbosa, L. S., & Janowski, T. (2020). Data governance: Organizing data for trustworthy artificial intelligence. Government Information Quarterly, 37(3), Article 101493. https://doi.org/10.1016/j.giq.2020.101493
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020). Interpreting interpretability: Understanding data scientists’ use of interpretability tools for machine learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–14). Association for Computing Machinery. https://doi.org/10.1145/3313831.3376219
Kim, J., Maathuis, H., & Sent, D. (2024). Human-centered evaluation of explainable AI applications: A systematic review. Frontiers in Artificial Intelligence, 7, Article 1456486. https://doi.org/10.3389/frai.2024.1456486
Langer, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., Sesing, A., & Baum, K. (2021). What do we want from explainable artificial intelligence (XAI)? A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence, 296, Article 103473. https://doi.org/10.1016/j.artint.2021.103473
Liao, Q. V., Gruen, D., & Miller, S. (2020). Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–15). Association for Computing Machinery. https://doi.org/10.1145/3313831.3376590
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021). The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies. Journal of Biomedical Informatics, 113, Article 103655. https://doi.org/10.1016/j.jbi.2020.103655
Meske, C., Bunde, E., Schneider, J., & Gersch, M. (2022). Explainable artificial intelligence: Objectives, stakeholders, and future research opportunities. Information Systems Management, 39(1), 53–63. https://doi.org/10.1080/10580530.2020.1849465
Rai, A. (2020). Explainable AI: From black box to glass box. Journal of the Academy of Marketing Science, 48(1), 137–141. https://doi.org/10.1007/s11747-019-00710-5
Raisch, S., & Krakowski, S. (2021). Artificial intelligence and management: The automation-augmentation paradox. Academy of Management Review, 46(1), 192–210. https://doi.org/10.5465/amr.2018.0072
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024). Towards human-centered explainable AI: A survey of user studies for model explanations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4), 2104–2122. https://doi.org/10.1109/TPAMI.2023.3331846
Sayogo, D. S., Yuli, S. B. C., & Amalia, F. A. (2024). Data-driven decision-making challenges of local government in Indonesia. Transforming Government: People, Process and Policy, 18(1), 145–156. https://doi.org/10.1108/TG-05-2023-0058
Schemmer, M., Kühl, N., Benz, C., Bartos, A., & Satzger, G. (2023). Appropriate reliance on AI advice: Conceptualization and the effect of explanations. In Proceedings of the 28th International Conference on Intelligent User Interfaces (pp. 410–422). Association for Computing Machinery. https://doi.org/10.1145/3581641.3584066
Schoonderwoerd, T. A. J., Jorritsma, W., Neerincx, M. A., & van den Bosch, K. (2021). Human-centered XAI: Developing design patterns for explanations of clinical decision support systems. International Journal of Human-Computer Studies, 154, Article 102684. https://doi.org/10.1016/j.ijhcs.2021.102684
Shneiderman, B. (2020). Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Computer Interaction, 36(6), 495–504. https://doi.org/10.1080/10447318.2020.1741118
Zaitsava, M., Marku, E., & Di Guardo, M. C. (2022). Is data-driven decision-making driven only by data? When cognition meets data. European Management Journal, 40(5), 656–670. https://doi.org/10.1016/j.emj.2022.01.003
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Dewi Ningsi, Eko Permadi, Fajar Hermawan (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
