Users are increasingly empowered to personalize natural language interfaces (NLIs) by teaching how to handle new natural language (NL) inputs. However, our formative study found that when teach- ing new NL inputs, users require assistance in clarifying ambigui- ties that arise and want insight into which parts of the input the NLI understands. In this paper we introduce ONYX, an intelligent agent that interactively learns new NL inputs by combining NL programming and programming-by-demonstration, also known as multi-modal interactive task learning. To address the aforemen- tioned challenges, ONYX provides suggestions on how ONYX could handle new NL inputs based on previously learned concepts or user-defined procedures, and poses follow-up questions to clarify ambiguities in user demonstrations, using visual and textual aids to clarify the connections. Our evaluation shows that users provided with ONYX’s new features achieved significantly higher accuracy in teaching new NL inputs (median: 93.3%) in contrast to those without (median: 73.3%).

Governments and health organizations increasingly use dashboards to provide real-time information during natural disasters and pandemics. Although these dashboards aim to make crisis-related information accessible to the general public, the average user can have a hard time interacting with them and finding the information needed to make everyday decisions. To address this challenge, we draw on the theory of effective use to propose a theory-driven design for conversational dashboards in crisis response, which improves users’ transparent interaction and access to crisis-related information. We instantiate our proposed design in a conversational dashboard for the COVID-19 pandemic that enables natural language interaction in spoken or written form and helps users familiarize themselves with the use of natural language through conversational onboarding. The evaluation of our artifact shows that being able to use natural language improves users’ interaction with the dashboard and ultimately increases their efficiency and effectiveness in finding information. This positive effect is amplified when users complete the onboarding before interacting with the dashboard, particularly when they can use both natural language and mouse. Our findings contribute to research on dashboard design, both in general and in the specific context of crisis response, by providing prescriptive knowledge for extending crisis response dashboards with natural language interaction capabilities. In addition, our work contributes to the democratization of data science by proposing design guidelines for making information in crisis response dashboards more accessible to the general public.

Teams are crucial for organizations in making data-driven decisions. However, current business intelligence & analytics (BI&A) systems are primarily designed to support individuals and, therefore, cannot be used effectively in co-located team interactions. To address this challenge, we conduct a design science research (DSR) project to design a multimodal BI&A system providing touch and speech interactions that can be used effectively by teams. Drawing on the theory of effective use and existing guidelines for multimodal user interfaces, we propose three design principles and instantiate them in a software artifact. The results of a focus group evaluation indicate that enhancing the BI&A system with multimodal capabilities increases transparent interaction and facilitates effective use of the system in co-located team interactions. Our DSR project contributes novel design knowledge for multimodal BI&A systems with touch and speech modalities that facilitate effective use in co-located team interactions.