Leveraging Multi-agent Systems for Domain-Pertinence Query Classification in Informative Chatbots

Large Language Models generate answers to any questions provided by users. Even though this is a positive characteristic, when they are integrated into a real-world domain-specific solution, the generation of an answer to a question that is not related to the domain is a weakness. This work addresses the challenge of classifying users’ questions as in-topic or out-of-topic to limit the capabilities of a Large Language Model. We propose a multi-agent approach, called “Pool of Experts”, which leverages a structured hierarchy of specialized agents to synthesize expert contributions into a final decision. To evaluate the effectiveness of the proposed approach, we tested our methodology by integrating two description-based frameworks for agent profile creation: User Design Persona and Cognitive Load Theory. We compared our approach against traditional Transformer-based Natural Language Inference models as a baseline. Experimental results, observed in a real-world scenario concerning a question-answering system supporting pregnant women, demonstrate the superiority of the proposed methodology.