Evaluation of chatbots used in healthcare
Raulinaitis, Vytautas (2022)
2022
All rights reserved. This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2022120726961
https://urn.fi/URN:NBN:fi:amk-2022120726961
Tiivistelmä
Ubiquitous high-quality healthcare is one of the main goals of humanity. Delivery of adequate healthcare can be obstructed by many hurdles, some of which could be overcome by applying advances in technology. Artificial intelligence is providing many instruments for different industries and services that can improve efficiency as well as offer novel approaches to find solutions. One of them is conversational agents also known as chatbots.
Application of chatbots in healthcare has a far-reaching potential. Chatbots can help healthcare professionals, patients, and individuals who want to maintain their good health. The significance of health-related matters demands for highly reliable and credible means. To attain validation of highly reliable instruments and methods, well-established and proven protocols or frameworks are used. Chatbots entered healthcare field relatively recently and do not have such dependable and standardized procedures.
The research project of this thesis undertook a task to explore what methods are used for assessment of conversational agents employed in healthcare presently, whether a standardized framework is necessary, and what qualities for these agents are important. The study focused on chatbots used by patients. Investigation began by an examination of academic literature and was followed by a survey of experts in the field.
Literature analysis shows that quality assessment of healthcare chatbots is very discrepant. Methods that are being used are often generic and can be applied to chatbots used in any other area. The pursuit of standardized framework is lagging behind innovation and delivery of new healthcare chatbots. While the recognition of this shortcoming is present, the efforts to address the issue are unassertive and the search for consensus of earlier methods, rather than a search for fresh paradigms, dominates.
The survey of experts reveals that they consider chatbots used in healthcare to be significantly distinct from other chatbots and a standardized framework for evaluation is most necessary to ensure high quality healthcare services. The strength of these opinions appears to positively correlate with individual’s proximity to medical work. The most valued features for chatbots are personalization and accuracy, followed by privacy security and interactivity. Aesthetics has minimal importance of the considered metrics.
Consequently, the research of this study demonstrates expert-perceived distinction of conversational agents used in healthcare and the need for specialized and standardized evaluation framework(s). This should be communicated more confidently. This study can serve as a pilot project for a large study, which would convincingly exhibit misconceptions held about healthcare chatbots and would invigorate closer cooperation between all involved experts.
Application of chatbots in healthcare has a far-reaching potential. Chatbots can help healthcare professionals, patients, and individuals who want to maintain their good health. The significance of health-related matters demands for highly reliable and credible means. To attain validation of highly reliable instruments and methods, well-established and proven protocols or frameworks are used. Chatbots entered healthcare field relatively recently and do not have such dependable and standardized procedures.
The research project of this thesis undertook a task to explore what methods are used for assessment of conversational agents employed in healthcare presently, whether a standardized framework is necessary, and what qualities for these agents are important. The study focused on chatbots used by patients. Investigation began by an examination of academic literature and was followed by a survey of experts in the field.
Literature analysis shows that quality assessment of healthcare chatbots is very discrepant. Methods that are being used are often generic and can be applied to chatbots used in any other area. The pursuit of standardized framework is lagging behind innovation and delivery of new healthcare chatbots. While the recognition of this shortcoming is present, the efforts to address the issue are unassertive and the search for consensus of earlier methods, rather than a search for fresh paradigms, dominates.
The survey of experts reveals that they consider chatbots used in healthcare to be significantly distinct from other chatbots and a standardized framework for evaluation is most necessary to ensure high quality healthcare services. The strength of these opinions appears to positively correlate with individual’s proximity to medical work. The most valued features for chatbots are personalization and accuracy, followed by privacy security and interactivity. Aesthetics has minimal importance of the considered metrics.
Consequently, the research of this study demonstrates expert-perceived distinction of conversational agents used in healthcare and the need for specialized and standardized evaluation framework(s). This should be communicated more confidently. This study can serve as a pilot project for a large study, which would convincingly exhibit misconceptions held about healthcare chatbots and would invigorate closer cooperation between all involved experts.