Prompt Design for Enhanced NLP Performance : NER and IR

Abstract

Natural Language Processing (NLP) is crucial for enhancing machine learning-powered tools in business intelligence. NLP empowers these tools to grasp user queries phrased in natural language, streamlining data analysis and enhancing user experience. By improving the effectiveness of NLP capabilities, these tools can provide more accurate insights and support decision-making processes with greater precision. This research evaluates the relationship between prompt design and the effectiveness of two NLP capabilities: Named- Entity Recognition (NER) and Intent Recognition (IR). To evaluate the effectiveness of these NLP capabilities, the research utilised the F1 score as a metric. This statistical measure helped quantify the accuracy of the tool's responses in correctly identifying entities and understanding user intent, providing a balanced consideration of both precision and recall. The research mainly experimented with the Business Intelligence platform QuickSight Q, which is based on deep learning and machine learning and potentially leveraging some LLM capabilities; the platform worked well for straightforward prompts. It showed performance variation in complex prompts and struggled with ambiguous prompts. The other platform experimented with was Amazon Q, which leverages LLM capabilities to its full potential and potentially uses RAG to excel in the performance of NER and IR capabilities. However, it was a pilot experiment with the platform due to its preview nature of availability during the experimentation period. The NER and IR capabilities performed well across all the prompts in Amazon Q. The research findings would provide the foundation for future research in prompt engineering with LLMs and research aiming to explore the integration of Amazon Q in QuickSight and assess how these two platforms work together, enhancing user experience.