Abstract ID: 26-135
Compliance of Systematic Reviews and Meta-Analyses in Ophthalmology with the PRISMA Statement: An AI-Based Assessment and Longitudinal Comparison with 2017 Data
Author: Seonyoung Lee Base Hospital / Institution: James Cook University Hospital
Presentation Type: ePoster Presentation
Purpose
The aim of this study is to evaluate the reporting quality of systematic reviews and meta-analyses published in major ophthalmology journals between 2020 and 2024, based on the PRISMA 2020 checklist, and to compare human and AI assessments of compliance.
Methods
A total of 207 systematic reviews and meta-analyses published in 11 major ophthalmology journals were included in this study. Each article was independently assessed for adherence to the 2020 PRISMA checklist, first by two human reviewers, and subsequently by two distinct AI platforms (ChatGPT-4.0 and Gemini Pro 2.5). Compliance scores were calculated, and inter-observer agreement between human and AI evaluations was determined using Cohen’s kappa statistic. The Mann–Whitney U test was employed to compare these findings with those of a 2017 study.
Results
The mean compliance score, as assessed by human reviewers, was 36.28 out of 42 points (86.37%), indicating a substantial improvement in adherence to the PRISMA checklist compared with the level reported in the 2017 study (p < 0.00001). Compliance scores generated by the AI platforms demonstrated a moderate level of agreement with human assessments (Cohen’s κ = 0.63 for ChatGPT, 0.54 for Gemini). Strong compliance was observed for background and rationale, selection criteria, and limitations. Conversely, lower compliance was noted for risk of bias assessment, sensitivity analysis, and research registration.
Conclusion
This study demonstrates a marked improvement in the reporting quality of systematic reviews and meta-analyses in ophthalmology following adoption of the 2020 PRISMA statement. Nonetheless, persistent deficiencies remain, particularly in the reporting of bias, sensitivity analyses, and research registration. The application of AI models offers promising potential for enhancing the efficiency and effectiveness of reporting quality assessments; however, further refinement is required to ensure consistency and accuracy. Future iterations of the PRISMA guidelines should consider explicitly addressing the role of AI in research evaluation.
Additional Authors
| First name | Last name | Base Hospital / Institution |
|---|---|---|
| Jae Seon | Hong | Royal Victoria Infirmary |
| Sang Hyeok | Lee | North Tees University Hospital |
| Rajen | Gupta | Royal Victoria Infirmary |