Systematic reviews and meta-analyses are essential for advancing scientific research, but they are time-consuming, labor-intensive, and prone to human error during literature selection and data extraction. This paper proposes a framework that integrates Large Language Models (LLMs) and retrieval-augmented generation (RAG) into the PRISMA 2020 workflow to assist and standardize key review tasks. Our contribution is methodological rather than algorithmic: we operationalize its stages as a single reproducible, auditable pipeline built from established, openly available components, prioritizing transparency and re-execution over customized elements. The framework uses structured query templates to ensure reproducible bibliographic searches, an LLM-driven pipeline for title and abstract screening, and a RAG-based full-text eligibility assessment. Both automated stages rely on structured prompt templates that encode inclusion and exclusion criteria as machine-interpretable instructions. Beyond selecting relevant papers, it extracts specific information from documents, such as application details or datasets, and harmonizes it into a standardized metadata schema. To evaluate our proposal, we performed an experimental validation using an existing systematic literature review on Human Action Quality Assessment (AQA) as a reference. The framework achieved an extraction recall of 85.11% for domain-specific datasets and identified 25 additional data sources not reported in the original review. The pipeline achieved an overall article detection rate of 71.28%. This reduction was primarily a consequence of the strict, dataset-oriented Boolean query adopted in the identification phase, rather than failures in the automated LLM or RAG components. Supplementary analyses confirmed near-deterministic behavior across repeated runs and successful transfer to a different LLM family at negligible computational cost. These results suggest that the framework can effectively assist in identifying relevant studies and extracting information while maintaining alignment with PRISMA 2020 guidelines. The proposed approach is structured to facilitate adaptation to other research areas by declaring new query and prompt templates.

A retrieval-augmented LLM framework for PRISMA-aligned systematic reviews: a case study on action quality assessment

Silvestri, Romeo
;
Pincheira, Miguel;Vecchio, Massimo;Antonelli, Fabio
2026-01-01

Abstract

Systematic reviews and meta-analyses are essential for advancing scientific research, but they are time-consuming, labor-intensive, and prone to human error during literature selection and data extraction. This paper proposes a framework that integrates Large Language Models (LLMs) and retrieval-augmented generation (RAG) into the PRISMA 2020 workflow to assist and standardize key review tasks. Our contribution is methodological rather than algorithmic: we operationalize its stages as a single reproducible, auditable pipeline built from established, openly available components, prioritizing transparency and re-execution over customized elements. The framework uses structured query templates to ensure reproducible bibliographic searches, an LLM-driven pipeline for title and abstract screening, and a RAG-based full-text eligibility assessment. Both automated stages rely on structured prompt templates that encode inclusion and exclusion criteria as machine-interpretable instructions. Beyond selecting relevant papers, it extracts specific information from documents, such as application details or datasets, and harmonizes it into a standardized metadata schema. To evaluate our proposal, we performed an experimental validation using an existing systematic literature review on Human Action Quality Assessment (AQA) as a reference. The framework achieved an extraction recall of 85.11% for domain-specific datasets and identified 25 additional data sources not reported in the original review. The pipeline achieved an overall article detection rate of 71.28%. This reduction was primarily a consequence of the strict, dataset-oriented Boolean query adopted in the identification phase, rather than failures in the automated LLM or RAG components. Supplementary analyses confirmed near-deterministic behavior across repeated runs and successful transfer to a different LLM family at negligible computational cost. These results suggest that the framework can effectively assist in identifying relevant studies and extracting information while maintaining alignment with PRISMA 2020 guidelines. The proposed approach is structured to facilitate adaptation to other research areas by declaring new query and prompt templates.
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11582/374987
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
social impact