Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations

Large pre-trained language models have exhibited unprecedented capabilities in producing high-quality text via prompting techniques. This fact introduces new possibilities for data collection and annotation, particularly in situations where such data are scarce, complex to gather, expensive, or even sensitive. In this paper, we explore the potential of pre-trained language models to generate and annotate goal-oriented dialogues, and conduct an in-depth analysis to evaluate their quality. Our experiments employ ChatGPT, and encompass three categories of goal-oriented dialogues (task-oriented, collaborative, and explanatory), two generation modes (interactive and one-shot), and two languages (English and Italian). Through extensive human-based evaluations, we demonstrate that the quality of generated dialogues is on par with those generated by humans. On the other side, we show that the complexity of dialogue annotation schema (e.g., for dialogue state tracking) exceeds the capacity of current language models, a task which still requires substantial human supervision.