Iterative Negotiation in Intent-Based Interactions: Bridging the User Interface Accessibility and Usability Gap with LLMs
Issues: Large Language Models (LLMs) offer advanced capabilities in natural language understanding, generation, and reasoning, with the potential to revolutionize user interfaces and interaction paradigms. They can significantly lower barriers for end-users, including those with intellectual disabilities, enabling them to operate complex interfaces. However, challenges remain: LLMs struggle to accurately interpret user intent due to natural language ambiguity and the difficulty non-experts face in crafting structured prompts. Additionally, these models may produce non-deterministic, flawed, or "hallucinated" outputs, which non-expert users cannot often evaluate and correct, posing risks to user experience, privacy, and security. These challenges are further exacerbated by conflicts between user knowledge, preferences, and security requirements. ; ; Research Goal: This doctoral research aims to explore how LLM-powered conversational agents can be integrated into user interfaces to enhance their accessibility and usability. By designing and leveraging the synergy between iterative negotiation and multimodal interaction paradigms, the study seeks to address the aforementioned issues and enable intent-based, contextually adaptive interactions that cater to diverse user needs. The research will further focus on users with cognitive impairments, investigating the potential of LLMs in language skill training, cognitive enhancement, and mental health promotion. ; ; Methodology: The research will adopt a Research through Design (RtD) and Participatory Design (PD) approach, engaging users in workshops, focus groups, and iterative cycles of prototyping, evaluation, and optimization. A mixed-methods approach will be employed for data collection and analysis, combining qualitative methods such as in-depth interviews, contextual observations, software logs, and multimodal data (e.g., eye-tracking, speech emotion analysis, gesture recognition) with quantitative measures such as task performance metrics, negotiation success rates (e.g., F1-score), and user experience evaluations (e.g., UEQ scale). Data analysis will involve thematic analysis, statistical analysis, and triangulation. The evaluation process will include multi-stage assessments (short-term usability testing and long-term user experience tracking), comparative studies with alternative methods, and third-party validation to comprehensively verify the effectiveness of the proposed solutions. A lessons-learned approach will then be applied to identify challenges and design opportunities from user studies, informing the design of a low-code/no-code platform that supports the development and configuration of intent-based user interfaces incorporating the notion of negotiation. ; ; Expected Results: The research is expected to deliver insights into the functional and architectural requirements of low-code/no-code platforms and provide relevant interface and interaction design patterns and toolkits. These findings will be synthesized into an iterative negotiation framework for human-computer interaction, encompassing user models and adaptive multi-round negotiation strategies. These contributions will drive the practical application and technological advancement of LLMs, fostering more intelligent, inclusive, and "fluid" interaction experiences. Furthermore, the research seeks to support cognitive impairment users and other vulnerable groups by enhancing their skill development, autonomy, and well-being, thereby bridging the digital divide. The research will extend its impact to socially assistive robots and mental well-being support, offering theoretical insights and practical guidance to related fields.
Back to Current Students