Proceedings of The 3rd International Academic Conference on Research in Engineering and Technology
Year: 2024
DOI:
[PDF]
IT Service Continuity: Fine-Tuning and Evaluating LLMs for Precise Intent Recognition and Entities Extraction
Mohammed Tou, and Adil Toumouh
ABSTRACT:
As part of our research in ensuring a minimum level of IT service continuity, this publication focuses on a critical aspect of the process: understanding the meaning (intent) of service requests and their associated informational elements or metadata (entities). We concentrate on a comparative analysis leading to the evaluation of several large language models (LLMs). Our approach is gradual. An initial analysis involves selecting LLMs based on usage licenses and costs, model size in terms of the number of parameters, linguistic coverage, and multimodal capability, which in our case is limited to text processing. Subsequent analysis steps are part of a more in-depth study, integral to the objectives of this document, aimed at selecting an optimal and appropriate model. Our study involves fine-tuning these models using specific datasets to enhance their performance in recognizing and classifying intent within the context of service continuity. MoCaaS, our ecosystem designed to ensure a minimum level of service continuity, is structured as a pipeline where the third stage involves: 1) recognizing and classifying user request intent, and 2) extracting the entities associated with the intent. From a general perspective, our research methodology consists of two main steps: 1) fine-tuning, using advanced techniques, on multiple domain-specific datasets for particular intents, and 2) contrastive learning to better differentiate similar intents. By meticulously evaluating different LLMs, our objective is to determine the model that best meets our requirements for intent recognition and entity extraction within the context of our research to complete the MoCaaS ecosystem.
keywords: IT service continuity, Intent recognition, Intent recognition, Intent classification, Entity extraction, LLM, Fine-tuning, Multimodal capability, MoCaaS