User Researcher
2021-06 - 2022-01 (6 months)
UX Design, Chatbot Scripting, User Research, Survey, Experiment, Statistical Analysis by using R
This study looked into how errors impact chatbot user satisfaction and whether warmth and competence can help mitigate negative perceptions. According to the findings, chatbots with high warmth and low competence had a negative influence on user satisfaction, but chatbots with high warmth and high competence had positive effects.
Annoyance increased with errors and decreased when they were fixed, but it increased again when the faults persisted. To avoid the double-edged effect of warmth, it is recommended that chatbots be designed with modest warmth.
The market for conversational agents is quickly expanding, with a CAGR of 30.2% estimated to reach USD 17.4 billion by 2023. Google, Amazon, Microsoft, and IBM are major players in the business, supporting areas such as e-commerce, healthcare, finance, and customer service. Conversational agents are being developed with an emphasis on emotional AI to perceive and respond to emotions, boosting interactions and user experiences.
Emotional intelligence is becoming increasingly important as conversational agents become more human-like. Conversational bots will be able to understand and respond to emotions using AI technology, which will improve the quality of interactions and user experiences.
This study explored the effect of warmth on competence in conversational agents, as well as how user emotions might help conversation designers create agents that can handle errors. An online experiment was proposed to examine the impact of warmth on user emotions during error circumstances using the stereotype content model based on social psychology.
How does warmth mitigate the perception of different levels of competent conversational agents?
The goal of this research is to explore the process of annoyance experienced by users while encountering errors in chatbots, as well as the impact of warmth on error recovery. We investigated the association between warmth and chatbot competence, the extent to which warmth might reduce bad user experiences, and the mechanisms driving changes in user emotions when chatbots face problems.
To address the research question, we propose the following hypotheses:
H1: Warmth mitigates the perception of competent conversational agents.
H2: Warmth mitigates the perception of incompetent conversational agents.
￭ Responsible for managing all aspects of this project, from initial planning and reviewing relevant papers, to conducting experiments, analyzing data, and publishing the resulting article.
￭ Writing the dialogue script and designing the interaction and functions for self-made conversational agents (chatbots).
￭ Worked closely with Professor Chen-Chao Tao and provided instruction to my juniors at the Communication and Cognition Lab in the Department of Communication and Technology at National Yang Ming Chiao Tung University.
Present at the international annual conference of communication.
This paper was presented at the 72nd Annual Meeting of the International Communication Association on May 26, 2022, under the title 'Can Warmth Breed Competence? The Double-Edged Sword Effect of Lexical Remediation on Incompetent Conversational Agents'.
150 participants were recruited from three of the most popular social media sites in Taiwan via an online survey. All participants were citizens in Taiwan aged over 20 years and received a NT$ 100 convenience store voucher for participating. The experiment was conducted from September 16th to 30th, 2021.
The experiment (and usability testing) employed a between-subjects factorial design with 2 warmth levels (low and high) x 2 competence levels (low and high) x 5 error situations (on tasks 2, 3, 4, 5, and all tasks failed). All participants were randomly assigned to one of the conditions.
The quantitative experimental data were analyzed and tested by using R. The analysis of variance (ANOVA) and structural equation modeling (SEM) techniques were utilized to examine the hypotheses and answer two of the research questions.
Through a series of e-commerce-related chats, participants interacted with a chatbot to make purchase decisions. Each task represents the purpose of each of the five stages of e-commerce customer purchasing decisions.
Different wordings were used to manipulate the warmth of the chatbot's conversational style and the capability of the chatbot's performance. The recovery of the error situation was manipulated through apologies and explanations.
The manipulations of conversational style and warmth conditions include using friendly and caring wording for high-warmth conditions and unconcerned and indifferent wording for low-warmth conditions, as well as the use of a less assertive message style for high-warmth conditions.
The chatbots' performance in executing commands was manipulated, with errors consisting of false recognition of input commands and product recommendations for different brands.
As a strategy for recovering from mistakes, we employed apologies and explanations and manipulated the wording and length of our statements in chatbots.
￭ Apologies were found to be shorter and more direct in low-warmth conditions and longer and more submissive in high-warmth conditions.
￭ Explanations were shorter in low competence conditions and more detailed in high competence conditions.
The dialogue script was developed for a chatbot operating within various experimental contexts. The conversation scripts were created by combining predetermined keywords and phrases.
Participants answered how they felt through 4-item emotion scales (Joy, Relief, Annoyance, Boredom) after each task, completed a post-experiment questionnaire (perceived warmth and competence toward chatbot, usability, Satisfaction, Continuance Usage Intention), and answered demographic information.
The study aimed to find out how user annoyance at error circumstances varied with chatbot warmth and level of competence and to determine whether user annoyance at incapable chatbots was mitigated by warmth.
￭ The surprising finding of the study was that warmth may not boost satisfaction with incompetent chatbots and may even be damaging to them. It's possible that people found the friendliness of incompetent chatbot interactions to be annoying. The answer to this study question was therefore unexpected.
￭ However, a skilled chatbot with a friendly conversational style had a positive synergistic effect on user satisfaction. To avoid unfavorable outcomes, conversational agents' designs for warmth should be employed with discretion and should vary depending on the level of competence.
￭ According to the study, users were more irritated by an incompetent chatbot with high warmth than low warmth, and recurrent errors made users feel more annoyed. We describe the mechanism of change in annoyance and explain how users experience less discomfort after overcoming obstacles followed by the approach-avoidance motivation system.
￭ To enhance user experience, it is crucial to comprehend the internal mechanisms of user reactions and emotions.
In conclusion, the study found that warmth and competence affect users' annoyance, satisfaction, and continuance usage intention when using chatbots.
The study suggests that designers and developers of chatbots should consider the effects of warmth and competence when designing and developing chatbots to enhance the user experience.
