Background: Approximately 90% of currently published clinical and public health research is in the form of observational studies. Having a detailed and registered study protocol prior to data collection is important in any empirical study. Without this, there is no reliable way to assess the occurrence of publication bias, outcome reporting bias, and other protocol deviations. However, there is currently no solid guidance available on the information that a protocol for an observational study should contain.
Objective: The aim of this study is to formulate the Standardized Protocol Items Recommendations for Observational Studies (SPIROS) reporting guidelines, which focus on 3 main study designs of analytical epidemiology: cohort, case-control, and cross-sectional studies.
Methods: A scoping review of published protocol papers of observational studies in epidemiology will identify candidate items for the SPIROS reporting guidelines. The list of items will be extended with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist items and recommendations from the SPIROS steering committee. This long list serves as the basis for a 2-round Delphi survey among experts to obtain consensus on which items to include. Each candidate item from the long list will be rated on a 5-point Likert scale to assess relevance for inclusion in the SPIROS reporting guidelines. Following the Delphi survey, an expert-driven consensus workshop will be convened to finalize the reporting guidelines.
Results: A scoping review of published observational study protocols has been completed, with 59 candidate items identified for inclusion into the Delphi survey, itself launched in early 2020.
Conclusions: This project aims to improve the timeliness, completeness, and clarity of study protocols of observational studies in analytical epidemiology by producing expert-based recommendations of items to be addressed. These reporting guidelines will facilitate and encourage researchers to prepare and register study protocols of sufficient quality prior to data collection in order to improve the transparency, reproducibility, and quality of observational studies.
International Registered Report Identifier (IRRID): PRR1-10.2196/17864
Crowdfunding campaign to support this specific research
We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.
Suggested contribution levels: $20/$50/$100
The protocol of any empirical study offers the researcher practical guidance in data collection, analysis, and reporting. It also forms the basis for subsequent replication of the study and detection of publication bias, outcome reporting bias, and other protocol deviations. A well-formulated study protocol uploaded prior to data collection through a time-stamped registry will allow a robust audit trail. This improves the chances that the study is appropriately planned, executed, and well documented, thus promoting optimal conduct, accountability, replicability, and overall research integrity. In summary, ensuring a high level of transparency in the research planning phase facilitates replicability of the study and fosters good research practices [, ].
Although the value of registered study protocols and reporting guidelines on how to write them are widely recognized in the field of randomized control trials (RCTs) [- ], reporting guidelines for study protocols of observational studies in analytical epidemiology (cohort, case-control studies, and cross-sectional studies) are currently absent. In many situations, observational studies may be the best or at least the only feasible way to answer important medical and public health questions in scenarios where RCTs are not possible for practical and ethical reasons. Therefore, the same rigorous standards that apply to RCTs must be maintained for observational studies.
An estimated 90% of currently published clinical or public health–related research is in the form of observational studies . In particular, most research focusing on understanding causes and distribution of diseases relies on cohort, case-control, or cross-sectional studies. Additionally, a substantial body of evidence for diagnostic or prognostic research takes the form of observational studies. The absence of reporting guidelines on how to write these study protocols is problematic, as by design, observational studies have a nonnegligible risk of bias [ ]. The risk of bias has been well recognized in experimental research, where guidelines for the development of RCT protocols are well established [ ]. Additionally, large observational data sets without any statistical analysis plan readily allow data-driven post hoc analyses, leading to an exceedingly large number of potential associations that can be tested [ , ]. In such situations of multiple comparisons, there is increased likelihood of type I error and “p-hacking” by doing multiple statistical tests on the data and only reporting those that come back with significant results, causing false-positive results [ ]. Therefore, it is important to ensure that study protocols are complete, well written, and registered prior to data collection in a suitable time-stamped repository (eg, ClinicalTrials.gov and Open Science Framework). This allows external bodies to consult the original study protocol and understand the specific hypotheses and the data analysis plan to which the researcher was committed, both for primary data analyses and for secondary analyses of existing data sets.
When protocols of observational studies do not provide sufficient details concerning the conduct of their preplanned study and are not preregistered or made openly available prior to data collection initiation, the scientific community is left with an incomplete picture of what the researchers intended to do and whether undisclosed protocol deviations and reporting biases occurred. Consequently, in the absence of registered protocols for observational studies, we are insufficiently able to judge the validity and reliability of the results. As such, interpretation of the results within the context of the study limitations becomes a potential major issue .
Overview of Objectives
The objective of this study is to develop reporting guidelines to be used in the design of observational study protocols. Our initiative has been given the acronym SPIROS (Standardized Protocol Items: Recommendations for Observational Studies). The project aims to improve the completeness and utility of study protocols for observational studies by producing expert-based recommendations for a minimum set of items these study protocols need to address. An item is defined as an important section, heading, or subheading within the study protocol, with each item describing an important methodological feature of the study. Our study is designed to meet three objectives.
Objective 1 is to conduct a scoping review of items addressed in study protocols of observational studies in epidemiology. The purpose of objective 1 is to develop a long list of candidate items that could a priori be incorporated in the draft reporting guidelines. This would serve as the basis for the first round of the Delphi survey (Objective 2).
Objective 2 is to conduct a Delphi survey with participants who are selected based on a prespecified range of expertise, with the intention of obtaining consensus on which items to include in the SPIROS reporting guidelines.
Objective 3 is to conduct a face-to-face consensus workshop to finalize the SPIROS reporting guidelines and then disseminate the results with an associated manual.
Methods to Conduct Scoping Review of Protocols (Objective 1)
A scoping review of published study protocols of observational studies will be conducted, in which every item mentioned more than once will be listed [, ]. The study will be based on a systematic selection of up to 100 protocols of observational studies identified via published study design papers indexed by Web of Science Core Collections until information threshold saturation is achieved (defined as the point where no further items are identified after 15 consecutive paper reviews). The list of items will then be extended with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist items, which are developed as reporting guidelines for the publication of observational studies [ ], the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklist items for the reporting of protocols of experimental studies [ ], and recommendations from an expert steering committee (the authors of this manuscript). The findings of the scoping review will be reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping reviews [ ].
Up to 100 protocols of observational studies (cohort studies, case-control studies, and cross-sectional studies) published in indexed journals from the Web of Science will be included in the present review. To ensure the most current state of affairs is assessed, only papers published between January 1, 2016, and May 1, 2018, will be included because, based on a preliminary assessment, the target number of studies is expected to be reached within this time range.
Data for this scoping review will be identified by searching the Web of Science Core Collection and references from relevant papers using the search terms “protocol” PLUS “observational study” AND/OR “cohort study” AND/OR “case control study” AND/OR “cross sectional study” AND/OR “prevalence study” AND/OR “survey” in the advanced search mode. A senior clinical librarian will be engaged to ensure that the search strategy is valid.
All the protocol items present in the headings, subheadings, or text of protocol papers will then be presented in the form of a table, following the structure of the STROBE checklist. Initially, all items will be categorized into the following themes: general information, introduction, methods, ethical considerations, reporting, dissemination, and others. However, if different themes emerge, this structure will be accordingly iteratively adapted.
Methods to Conduct Delphi Study (Objective 2)
The long list of candidate items for the protocols of observational studies identified in the scoping review will be evaluated by experts. For this purpose, we will conduct a 2-round Delphi study [- ]. We will aim for participation of 50 to 100 experts who represent the key stakeholders in observational research, including principal investigators, methodologists, journal editors, users of observational data, and representatives from research ethics review boards. Experts will need to have relevant knowledge and experience and be spread across geographical regions. Experts will be identified by a mixed approach, including nomination by other experts, PubMed and web search, ResearchGate, and participation in the development of previous reporting guidelines.
Round 1 of Delphi Survey
Each panelist will be asked to rate every candidate item from the long list on a 5-point Likert scale for relevance for inclusion in the reporting guidelines for protocols of observational studies ().
A field for adding free text will be provided for comments on comprehensiveness and comprehensibility for each item and to suggest alterations or additional items. Round 1 will also collect core demographic information (panelists’ working field and place of employment) and panelists’ self-rated level of expertise on the particular topic.
A threshold of 67% agreement among participants on a score of 4 or 5 for each candidate item will be considered to indicate consensus among the expert panel for consideration of inclusion in the SPIROS checklist. If agreement among participants is less than 67% scoring 4 or 5, the item will be considered discordant. The survey will be structured to last no more than 30 minutes and will be pretested and approved by the expert steering committee. The study protocol with any amendments will be uploaded in the Open Science Framework (OSF), as will the Delphi survey format, which will be updated, if needed, before the second round.
|Likert score||Implication on candidate item|
|1||Unimportant—should be dropped as an item to consider|
|2||Of little importance|
|5||Very important—should definitely be included|
Round 2 of Delphi Survey
Round 2 will contain all original candidate items plus additional items nominated or identified during the first round. For each item, panelists will be provided with percentage agreement from the first round, while the themes of each item´s free text comments will be merged into an anonymized executive summary.
The participants will be asked to rerate the items and consider the comment themes. Once collated, all items reaching a 67% consensus agreement ranking of 4 or 5 on the Likert score will be included in the short list for the reporting guidelines and advance to the next stage of the process.
Data Collection and Statistical Analysis
A preset online survey questionnaire will be developed on the Google Forms online survey platform and uploaded to OSF before being sent out to expert panelists via email. Medians, interquartile range, and percentage of agreement will be calculated for each item.
Consensus Workshop (Objective 3)
Following completion of the Delphi survey, the steering committee will convene a consensus workshop consisting of the steering committee plus 10 others previously involved in developing guidelines. Experts will review and adapt the draft reporting guidelines into a format appropriate for wider dissemination. A further workshop around dissemination strategy planning will also be convened, comprising experts involved with previous guideline dissemination, to ensure that a robust uptake mechanism is defined for widespread adoption of SPIROS, including choice of journal for publication of the final manuscript and content of a website exclusively for the initiative.
Dissemination, Implementation, and Ongoing Development (Objective 4)
The ultimate objective of this initiative is to facilitate widespread adoption of the SPIROS reporting guidelines. Therefore, all outputs of this initiative will be published as preprints and in open access journals or open choice access. This will include a detailed manual that explains and justifies the recommendations and includes examples of good reporting. We aim to have the SPIROS reporting guidelines adopted by journals and funders. The SPIROS protocol and final reporting guidelines will be published on a dedicated website (spiros-statement.org) and will be included in the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network.
Data Management and Registration of Study Protocol
The study will comply with the EU General Data Protection Regulation. Prior to data collection, the study protocol and all appendices will be registered and made openly available on the OSF, and links will be provided to reviewers and readers of the paper. This will include the complete data management and data analysis plan, and on completion of the study, all anonymized data will additionally be made available on OSF with no restrictions.
Ethics Approval and Consent to Participate
Ethics approval has been obtained from the Ethics Review Committee of Maastricht University. The study will not collect any sensitive information, and the identity of each panel member will be fully protected and blinded to other panel members and the research group (except for the corresponding researcher). The burden for the panelists includes a maximum of 30 minutes for each of the Delphi survey rounds.
Online informed consent will be obtained from all panel members, which will comprise an introductory page and completion of the survey as consent. Participants of the Delphi consensus survey will be provided with a summary of the overall aim of the SPIROS project and a link to the complete protocol. The consent for the Delphi survey will explain the specific aim of the survey and an outline of the procedures involved, as well as the benefits, risks, and burdens involved in participating. If consent is given, panels member will be acknowledged in the publication of the SPIROS reporting guidelines. However, the option to not be mentioned in the acknowledgments will also be given.
The scoping review of published protocols of observational studies has been completed, with 59 candidate items identified. This long list of candidate items has formed the basis of the 2-round Delphi survey launched in early 2020.
Over the last decade, there has been an increasing call for prospective registration of observational study protocols [, - ]. Without a registered protocol, it is impossible to detect p-hacking and any intentional amendments in protocols to match with preferred results. To conduct research without a detailed protocol registered prior to data collection is increasingly considered questionable research practice, limiting reviewers’ and readers’ ability to assess the occurrence of intentional or nonintentional bias. A detailed study protocol is essential for replicability as well. That said, even with strict prospective registration of observational study protocols, including digital object identifier records of any amendments, there remains the chance of post hoc protocol amendments at a later stage to match with preferred results. Ultimately, there is no foolproof way to prevent scientific malpractice in this domain.
There are several recent projects, such as the Reproducible Evidence: Practices to Enhance and Achieve Transparency (REPEAT) initiative, that aim to improve transparency, reproducibility, and validity of database research . At the same time, the need for one-size-fits-all guidance, or any guidance for protocol development of observational studies, remains a subject of debate [ , , , ]. For example, an industry-based epidemiological group have proposed a framework in which observational studies should be registered and can be exempted based on study design and study intent [ ].
However, the absence of guidelines to develop protocols has historically resulted in a large proportion of observational research being conducted without quality-assured study protocols. Existing protocols are often not fit for purpose, and the proportion of researchers ensuring registration of protocols for observational studies prior to data collection remains negligible. Therefore, our study aims to develop reporting guidelines for protocols of observational studies by launching SPIROS. We expect this initiative to increase transparency of methods used in observational studies, and through standardizing protocols, we hope that study designs will be improved and more often be registered and published in a journal or in a repository.
The main limitation of our study is that SPIROS reporting guidelines will be developed primarily for cohort, case-control, and cross-sectional study designs but not for other observational research designs. Because of the wide scope of observational study designs, it is difficult to prepare standard reporting guidelines that cover all observational study designs.
Nevertheless, as the first attempt to develop a guiding document for observational study protocols, our guidelines will fill an important gap. Once widely available, an impact assessment should be conducted after a 4- to 5-year period to determine whether more and higher-quality protocols have become available, similar to the approach adopted by other reporting guidelines .
The study is a PhD project of Maastricht University. No specific funding was obtained for this research.
RM, SB, and MPZ designed the final study protocol, had part in the technical design, and drafted the initial manuscript; MPZ conceived of the study. LMB, KS, AK, JK, PVD, SB, and MPZ had part in the technical design and are part of the expert steering committee of the SPIROS project. All authors reviewed and approved the final manuscript.
Conflicts of Interest
- Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, et al. A manifesto for reproducible science. Nat Hum Behav 2017 Jan 10;1(1). [CrossRef]
- Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci U S A 2018 Mar 13;115(11):2600-2606. [CrossRef] [Medline]
- Chan A, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med 2013 Feb 5;158(3):200-207. [CrossRef] [Medline]
- Tetzlaff JM, Chan A, Kitchen J, Sampson M, Tricco AC, Moher D. Guidelines for randomized clinical trial protocol content: a systematic review. Syst Rev 2012 Sep 24;1:43 [FREE Full text] [CrossRef] [Medline]
- Chan A, Tetzlaff JM, Altman DG, Dickersin K, Moher D. SPIRIT 2013: new guidance for content of clinical trial protocols. Lancet 2013 Jan 12;381(9861):91-92. [CrossRef] [Medline]
- Boccia S. Credibility of observational studies: why public health researchers should care? Eur J Public Health 2015 Aug;25(4):554-555. [CrossRef] [Medline]
- Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet 2002 Jan 19;359(9302):248-252. [CrossRef] [Medline]
- Bell S, Kivimäki M, Batty GD. Subgroup analysis as a source of spurious findings: an illustration using new data on alcohol intake and coronary heart disease. Addiction 2015 Jan;110(1):183-184 [FREE Full text] [CrossRef] [Medline]
- Kivimäki M, Singh-Manoux A, Ferrie JE, Batty GD. Post hoc decision-making in observational epidemiology--is there need for better research standards? Int J Epidemiol 2013 Apr;42(2):367-370 [FREE Full text] [CrossRef] [Medline]
- Teixeira PM. p-Hacking - A call for ethics. Pulmonology 2018;24(3):207-208 [FREE Full text] [CrossRef] [Medline]
- Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol 2018 Dec 19;18(1):143 [FREE Full text] [CrossRef] [Medline]
- Tricco AC, Lillie E, Zarin W, O'Brien K, Colquhoun H, Kastner M, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol 2016 Feb 09;16:15 [FREE Full text] [CrossRef] [Medline]
- von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg 2014 Dec;12(12):1495-1499 [FREE Full text] [CrossRef] [Medline]
- Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med 2018 Oct 02;169(7):467-473. [CrossRef] [Medline]
- Yousuf M. The Delphi technique. Essays Educ 2007;20:80-89. [CrossRef]
- Crisp J, Pelletier D, Duffield C, Adams A, Nagy S. The Delphi method? Nurs Res 1997;46(2):116-118. [CrossRef] [Medline]
- Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs 2000 Oct;32(4):1008-1015. [Medline]
- Hsu C, Sandford B. The Delphi technique: Making sense of consensus. Pract Assessment, Res Eval 2007;12(10):8. [CrossRef]
- Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Inf Manage 2004 Dec;42(1):15-29. [CrossRef]
- Dal-Ré R, Ioannidis JP, Bracken MB, Buffler PA, Chan AW, Franco EL, et al. Making prospective registration of observational research a reality. Sci Transl Med 2014 Feb 19;6(224):224cm1. [CrossRef] [Medline]
- Williams RJ, Tse T, Harlan WR, Zarin DA. Registration of observational studies: is it time? CMAJ 2010 Oct 19;182(15):1638-1642. [CrossRef] [Medline]
- PLOS Medicine Editors. Observational studies: getting clear about transparency. PLoS Med 2014 Aug;11(8):e1001711 [FREE Full text] [CrossRef] [Medline]
- Wang S, Rassen J, Pinheiro S, Berlin J, Schneeweiss S. What Does Replicable and Robust Non-Randomized Database Evidence Look Like? 2019 Presented at: Abstracts of the 35th International Conference on Pharmacoepidemiology & Therapeutic Risk Management; Aug 24-28, 2019; Philadelphia, PA. [CrossRef]
- Loder E, Groves T, Macauley D. Registration of observational studies. BMJ 2010 Feb 18;340:c950. [CrossRef] [Medline]
- Chavers S, Fife D, Wacholtz M, Stang P, Berlin J. Registration of Observational Studies: perspectives from an industry-based epidemiology group. Pharmacoepidemiol Drug Saf 2011 Oct;20(10):1009-1013. [CrossRef] [Medline]
- Chan A, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krle A-Jerić K, et al. SPIRIT 2013 Statement: defining standard protocol items for clinical trials. Rev Panam Salud Publica 2015 Dec;38(6):506-514 [FREE Full text] [Medline]
|EQUATOR: Enhancing the Quality and Transparency of Health Research|
|OSF: Open Science Framework|
|PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses|
|RCT: randomized control trial|
|REPEAT: Reproducible Evidence: Practices to Enhance and Achieve Transparency|
|SPIRIT: Standard Protocol Items: Recommendations for Interventional Trials|
|SPIROS: Standardized Protocol Items: Recommendations for Observational Studies|
|STROBE: Strengthening the Reporting of Observational Studies in Epidemiology|
Edited by G Eysenbach; submitted 17.01.20; peer-reviewed by J Berlin, T Aslanidis; comments to author 28.02.20; revised version received 18.03.20; accepted 21.03.20; published 21.10.20Copyright
©Raman Mahajan, Sakib Burza, Lex M Bouter, Klaas Sijtsma, André Knottnerus, Jos Kleijnen, Peter Van Dael, Maurice P Zeegers. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 21.10.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.