Usability and effectiveness evaluation of a course-advising chat bot
MetadataShow full item record
This research explored whether a course-advising chat bot could be used for academic advising in terms of its functionality and interface effectively. The research evaluated the usability and effectiveness of the chat bot with a usability test, comparing it with a menu-driven FAQs (frequently asked questions) system. For the test, both the course-advising chat bot (LISAbot), and the menu-driven FAQ system (LIS FAQ), have been created. The LISAbot was built a knowledge-base database implemented in AIML (Artificial Intelligence Markup Language). This knowledge base was customized for the Library and Information Studies (LIS) graduate program at University at Buffalo and covers various types of queries for course advisement. It also has an interactive interface, where the user can type questions in directly, and an avatar and voice response answers along with the textual response. The LIS FAQ was built based on a preferred FAQ interface and the content obtained from a FAQ interface web-survey and user interviews. The LIS FAQ is a text-based system to enable users to find information using a typical web page interface that includes organized text and hyperlinks. A total of sixteen LIS graduate students participated in the usability test. The functionality was assessed in terms of task completion, perceived effectiveness, ease of use, learnability, and ease of searching. LISAbot was ranked highly for its functionality. In addition, perceptions regarding interface preference, character preference, and features in the LISAbot have positive evaluations. The results of the usability test suggest that the LISAbot is a promising solution for supporting the course-related questions in terms of the functionality and interface for searching information. However, the analysis of LISAbot logs and user feedbacks shows that there are some challenge that need to be addressed to enhance functional effectiveness.