Differences in user perception of artificial intelligence-driven chatbots and traditional tools in qualitative data analysis

Boštjan Šumak (Author), Maja Pušnik (Author), Ines Kožuh (Author), Andrej Šorgo (Author), Saša Brdnik (Author)

Abstract

Qualitative data analysis (QDA) tools are essential for extracting insights from complex datasets. This study investigates researchers’ perceptions of the usability, user experience (UX), mental workload, trust, task complexity, and emotional impact of three tools: Taguette 1.4.1 (a traditional QDA tool), ChatGPT (GPT-4, December 2023 version), and Gemini (formerly Google Bard, December 2023 version). Participants (N = 85), Master’s students from the Faculty of Electrical Engineering and Computer Science with prior experience in UX evaluations and familiarity with AI-based chatbots, performed sentiment analysis and data annotation tasks using these tools, enabling a comparative evaluation. The results show that AI tools were associated with lower cognitive effort and more positive emotional responses compared to Taguette, which caused higher frustration and workload, especially during cognitively demanding tasks. Among the tools, ChatGPT achieved the highest usability score (SUS = 79.03) and was rated positively for emotional engagement. Trust levels varied, with Taguette preferred for task accuracy and ChatGPT rated highest in user confidence. Despite these differences, all tools performed consistently in identifying qualitative patterns. These findings suggest that AI-driven tools can enhance researchers’ experiences in QDA while emphasizing the need to align tool selection with specific tasks and user preferences.

Keywords

user experience;UX;usability;qualitative data analysis;QDA;chatbots;

Data

Language:	English
Year of publishing:	2025
Typology:	1.01 - Original Scientific Article
Organization:	UM FERI - Faculty of Electrical Engineering and Computer Science
Publisher:	MDPI
UDC:	004.5
COBISS:	222358787
ISSN:	2076-3417
Views:	0
Downloads:	0
Average score:	0 (0 votes)
Metadata:

Other data

Secondary language:	Slovenian
Secondary keywords:	uporabniška izkušnja;uporabnost;klepetalnice;
Type (COBISS):	Article
Pages:	37 str.
Volume:	ǂVol. ǂ15
Issue:	ǂiss. ǂ2, [article no.] 631
Chronology:	2025
DOI:	10.3390/app15020631
ID:	25859460