Novel utterance data augmentation for intent classification using large language models

Abstract

Data augmentation is a widely used strategy to enhance the predictive power of machine learning (ML) models. This is the case of intent classification problems, where the end goal of a utterance needs to be categorized using text mining techniques. Nevertheless, recent augmentation methods based on general off-the-shelf Large Language Models (LLMs) have room for improvement. They can struggle to effectively capture the nuances associated with domain-specific scenarios. This paper presents a novel utterance augmentation method that uses LLMs and word embedding models to address the issue, particularly in domain-specific problems. The proposed method starts from a given reference set. Then, paraphrases are generated using LLMs to obtain new utterances that are semantically similar to the original ones, but with different word choices and syntax. Next, synonym replacement is accomplished using a previously trained domain-specific word embedding model. This entails the incorporation of relevant vocabulary to a particular topic into the final augmented dataset, effectively capturing the nuances of domain-specific problems. Experiments were conducted to assess the quality of the proposal in two intent classification problems related to financial trading compliance. The proposal has an advantage over many well-known approaches in the first problem, comprising eight reference utterances and 375 challenging examples in general language. In the second problem, with 13 reference utterances and 525 challenging examples in general and financial trading vocabulary, the proposal outperforms the best-analyzed state-of-the-art methods by up to 15%.