Abstract
Patient-reported outcomes (PROs) in randomized controlled trials pertaining to inflammatory bowel disease are important in identifying patients’ perspective of treatment. Incompletely reported PROs within trials could misrepresent information for clinicians and may contribute to treatment which lacks accommodation of patient input. Our study evaluates completeness of reporting of PROs and risk of bias (RoB) to identify how well trialists are adhering to known resources for trials. We used MEDLINE, Embase, and the Cochrane Central Register of Controlled Trials to identify eligible trials from 2006 to 2020 with at least 1 PRO measure related to inflammatory bowel disease. The trials were screened in duplicate using Rayyan. We then compared trial completion of reporting to the Consolidated Standards of Reporting Trials (CONSORT)-PRO adaptation, and assessed RoB using the Cochrane Collaboration RoB 2.0 tool. To measure trial and reporting characteristics, we performed bivariate regression analyses. Among a sample of 29 trials, the mean completion percentage for CONSORT-PRO was 46.77%. We found PROs as a secondary outcome had significantly lower CONSORT-PRO reporting (p<0.05). In addition, per cent completeness of reporting was significantly higher with both a ‘therapy’ intervention, and trials published following the development of CONSORT-PRO (p<0.05). Incomplete PRO reporting is common in trials focused on inflammatory bowel disease. This suboptimal reporting indicates the need for adherence to reporting guidelines. Trialists should use the CONSORT-PRO checklist, as endorsed by Patient-Reported Outcomes Tools: Engaging Users and Stakeholders, to assess their studies in order to enhance reporting adherence.
WHAT IS ALREADY KNOWN ON THIS TOPIC
Use of patient-reported outcomes (PROs) has shown to be beneficial for patient care across many specialties.
WHAT THIS STUDY ADDS
Reporting of PROs is commonly incomplete in randomized controlled trials studying inflammatory bowel disease, despite the existence of established methods of PRO reporting.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
Trials need better adherence to PRO reporting guidelines to help complete the patient perspective of treatment.
Introduction
Inflammatory bowel disease (IBD) affects over 3 million US adults,1 with an estimated financial burden of $26,555 per patient in the first year of diagnosis.2 Patients with IBD experience unexpected ‘flares’ of abdominal pain, fatigue, diarrhea, rectal bleeding, and unpredictable urges to defecate.3 In addition to physical symptomatology, several studies have shown that IBD can also increase psychiatric sequelae, such as anxiety and depression.4–8 Since no cure for IBD exists, management strategies primarily focus on patients’ symptoms and daily functioning.9 Therefore, due to the chronic nature, physical symptoms, psychiatric comorbidities, and financial burden of IBD, patients’ perspective may be increasingly helpful in guiding clinical decisions.
Patient-reported outcomes (PROs) are becoming increasingly used in randomized controlled trials (RCTs) as an individualized measure of patients’ experience. PROs are measurements reported directly by the patient in regard to their perceived health and function.10 PROs allow patients to express opinions related to their care that may otherwise be unknown to clinicians. Due to the importance of PROs, many variations have been included in IBD RCTs, including pain scales, the IBD disability index, and the Inflammatory Bowel Disease Questionnaire.11–16 Given the extensive variation of PROs and the subjectivity of patient reporting, RCTs could benefit from standardization of PROs to better establish reliability among outcomes.
The Consolidated Standards of Reporting Trials (CONSORT) set standards for the way authors report trial findings, facilitate transparent reporting, and aid in their interpretation.17 In 2013, the CONSORT-PRO adaptation was developed out of 5 items on the CONSORT 2010 checklist to specifically cater to authors reporting on PROs.18 High quality of reporting facilitates greater reproducibility of a trial, thus improving the comprehensive evidence supporting clinical decision-making. However, despite the CONSORT-PRO guidelines, many RCTs still lack proper reporting of PROs. For example, a 2021 systematic review found that, on average, oncological pharmaceutical RCTs published from 2011 to 2018 reported on less than 13 of 24 PROs reporting items.19 Another systematic review looking at the CONSORT-PRO checklist found immense variation in the reporting of PROs for patients with rheumatoid arthritis.20 The complexity and chronicity of IBD reveal a necessity for incorporation of the patient perspective regarding disease and treatment within RCT evaluation. Therefore, the aim of this study is to investigate the completeness of reporting of PROs in RCTs evaluating IBD treatment, so that patients and clinicians can make the most comprehensively assessed treatment decisions.
Materials and methods
Study design
We studied RCTs pertaining to IBD using a meta-epidemiological design. Data were collected by extracting details from published RCTs. Complete reporting was ensured by following reporting guidelines for meta-epidemiological studies.21
Search strategy
A medical research librarian assisted one investigator (RO) in accessing the OVID interface to search Cochrane Central Register of Controlled Trials, Embase, and MEDLINE for published RCTs pertaining to IBD. We used the Cochrane highly sensitive search strategy for identifying randomized trials which helped improve sensitivity and is a reputable filter for OVID interface.22 We uploaded our search string to Open Science Framework (OSF).23
Eligibility of studies
We included RCTs pertaining to IBD published between the years 2006 and 2020 with a minimum of 1 PRO measure as a primary or secondary outcome. Only studies in the English language were included. Observational studies, systematic reviews, meta-analyses, other reviews, letters to the editor, case reports, secondary analyses, cost-effectiveness studies, animal studies, protocols of clinical trials, open-label, single blind, and trials that lacked a PRO measure were excluded.
Selection process
Results of the literature search were combined and uploaded to Rayyan (https://rayyan.qcri.org/), a platform for screening. Two investigators (RM, PW) screened titles and abstracts in a masked, duplicate method in order to select RCTs and remove duplicates. Investigators reconciled differences of screening through discussion and adjudication through a third investigator (CH) was available.
Data collection process
A masked, duplicate abstraction of the CONSORT-PRO adaptation was carried out by two investigators (DT, PW) using a Google form. Investigators met to resolve disagreements once data were abstracted. The risk of bias (RoB) rating was performed in a similar fashion; two investigators (RM, JD) used a masked, duplicate method. A third investigator (CH) was available for adjudication.
With intent to train for data abstraction of the CONSORT-PRO adaptation, researchers (PW, DT) used work published by the guideline authors.24 25 In order to reach consensus, we carried out a masked, duplicate abstraction from 3 different RCTs that were not included in our sample, and resolved any disagreements. We used a pilot-tested Google form to extract data, and this was done by two investigators (PW, DT) in a masked, duplicate method. In order to increase accuracy of responses, we trained investigators RM and JD using videos supplied by Cochrane on their RoB tool.26 Two investigators (RM, JD) carried out the RoB evaluation in the same masked, duplicate fashion. Following the RoB evaluation and CONSORT-PRO extraction, all discrepancies were resolved by the investigators; a third investigator (CH) was available to settle any disagreements.
Data items
We assessed our sample of RCTs for completion of the CONSORT-PRO checklist adaptation—developed by Mercieca-Bebber et al— with our primary objective reported in terms of mean per cent completion (see scoring of CONSORT-PRO adaptation).25 Our secondary objective was evaluating relationships of characteristics in RCTs and their mean completeness of PRO reporting. Trial characteristics abstracted were: (1) publication year (before or after 2014, 1 year after the CONSORT-PRO reporting guidelines were published); (2) intervention of RCT (eg, device or therapy); (3) conflict of interest statement; (4) journal endorsement of CONSORT-PRO; (5) CONSORT-PRO citation within the publication; (6) whether an RCT used a PRO as a primary or secondary outcome; (7) RoB assessed by the Cochrane RoB 2.0 tool (see Rating RoB); (8) duration of time to PRO follow-up; (9) size of the sample in the trial; and (10) region by World Bank country economic status (www.worldbank.org).
The designations: not mentioned, recommended, or required were used to indicate endorsement of CONSORT guidelines within the journals in which our sample of RCTs were published. To complete this item, we screened for mention of CONSORT, CONSORT-PRO, or EQUATOR guidelines on the instructions to authors’ webpages.
Each RCT’s RoB was rated with the Cochrane RoB 2.0 tool. The domains evaluated via this tool are: (1) bias arising from the randomization process, (2) bias due to deviations from intended interventions, (3) bias due to missing endpoint data, (4) bias in measurement of the endpoint, (5) bias in selection of the reported result, and (6) overall risk of bias.
Scoring CONSORT-PRO
Methodology for scoring was adapted by Mercieca-Bebber et al.25 The difficulty in verifying criteria assessed by item 4a of CONSORT-PRO (the use of PROs in eligibility or stratification) led us to exclude it from the scoring in our study. Alternatively, we decided to record whether a study described adherence to this item as a ‘yes’ or ‘no’. We determined a maximum value of 0.5 or 1 for the presence of information on an item. Items which obtained the maximum value (1 or 0.5 if the item is double barreled) were considered ‘complete’, while items that failed to reach maximum value were reported as ‘not complete’. A score of ‘partially complete’ was given to item P1b if the PRO measure used in the study was reported by the RCT, but they did not specify the endpoint of the PRO. Therefore, item P1b could be scored as 0, 0.5, or 1, depending on what was provided by the RCT. Item 7a was conditional on the PRO measure being reported as a primary outcome. Because of this conditionality, primary PRO outcome RCTs had a maximum score of 15, whereas RCTs with a secondary PRO outcome had a maximum score of 14. The per cent completeness of reporting in each RCT was calculated by dividing the sum of the completed items by the total possible items.
Rating RoB
We used the Cochrane Collaboration’s decision algorithm to rate RoB. If partially divergent assessments on bias domains were present (eg, 1 investigator answered ‘yes’ and another investigator answered ‘partial yes’), we judged overall RoB as not altered for the trial outcome. RoB was reported as ‘high’ risk, ‘some concerns’, or ‘low’ risk for each domain and the overall RoB assessment was rated using Cochrane’s Excel tool.27
Data analysis
First, trial frequencies and percentages were reported in data items. We then calculated the mean per cent completion of the CONSORT-PRO adaptation to address our primary objective. Next, we reported the frequency and percentage of all RCTs’ incorporation of individual items from the CONSORT-PRO adaptation, of RCTs with primary outcome PROs, and of RCTs with secondary outcome PROs. Finally, we used bivariate regression models to determine the association between mean completion percentage of the CONSORT-PRO adaptation and the trial characteristics listed in data items. All analyses were performed using Stata V.16.1 (StataCorp, College Station, Texas, USA).
Reproducibility
We uploaded our study protocol, analysis scripts, data dictionary, extraction forms, and data sheets to OSF to foster the transparency, reproducibility, validity, and reliability of our study.23 We conducted this study jointly with other methodologically similar studies on completeness of reporting in other areas of medicine.
Results
General characteristics
The systematic search returned 5369 records with 3813 following deduplication. Following title and abstract review, 72 publications were included for full-text review. We included 29 RCTs for data extraction following screening of full-text manuscripts. Exclusion rationales can be found in figure 1.
Study screening and selection flow chart. PROs, patient-reported outcomes.
Of the 29 RCTs, 72.41% (21 of 29) were published during or after 2014. The most common region in which the RCTs took place was Europe and Central Asia at 37.93% (11 of 29), followed by Middle East and North Africa at 20.69% (6 of 29), North America at 20.69% (6 of 29), East Asia and Pacific at 10.34% (3 of 29), and multiregion at 10.34% (3 of 29). We found no statistically significant association between per cent complete and region. The most frequently studied intervention was drugs (24 of 29, 82.76%). Of the RCTs that include a conflict of interest statement, 39.13% (9 of 23) reported conflicts. Regarding the endorsement of CONSORT, 93.10% (27 of 29) recommended or required the guideline to be followed, while no RCTs discussed following CONSORT recommendations within the publication. Of the 29 RCTs in our sample, 7 included a primary PRO outcome and 22 included a secondary PRO outcome. The frequency of RCTs appraised in each overall RoB domain is as follows: 17.24% (5 of 29) had ‘high’ RoB, 55.17% (16 of 29) had ‘some concerns’, and 27.59% (8 of 29) had ‘low’ RoB. The characteristics of RCTs and associations of their completeness of reporting can be found in table 1.
Baseline characteristics of randomized controlled trials and associations by PROs being a primary or secondary endpoint
Completeness of reporting according to the CONSORT-PRO adaptation
The overall mean completeness percentage of the checklist adaptation was 46.77 (SD=17.95). Mean completeness for RCTs with a primary PRO outcome was 58.57 (SD=14.25) and was 43.02 (SD=17.62) for those with a secondary PRO outcome. The completion of CONSORT-PRO checklist items by primary and secondary outcomes can be seen in table 2.
Completion of CONSORT-PRO checklist by primary and secondary outcome designation
The most consistently reported items across all RCTs included both evidence of PRO validity (item P6ai; 26 of 29; 89.66%) and inclusion of baseline PRO in the demographics table (item 15; 26 of 29; 89.66%). No RCTs completely reported Item P2bii—PRO domains specified in the hypothesis.
Regarding RCTs with a primary PRO outcome, the following items were consistently reported: PRO identified as RCT endpoint in abstract (P1b), evidence of PRO validity (P6ai), results include an estimate of precision (17aii), and PRO study limitations (P20). Items not reported by RCTs with a primary PRO outcome included: PRO domains specified in the hypothesis (P2bi and P2bii) and mode of questionnaire administration (P6aiii).
For RCTs with a secondary PRO outcome, the most consistently reported item was item 15—inclusion of baseline PRO in the demographics table (20 of 22; 90.91%), and item P6ai—evidence of PRO validity—was the next most completely reported item (19 of 22; 86.36%). Aside from item P2bii, which no RCT reported, item P2bi—PRO present in the hypothesis—was the most inconsistently reported item among RCTs with a secondary PRO outcome (2 of 22; 9.09%).
Associations between trial characteristics and completeness of PRO reporting
RCTs published in 2014 or later have 15.89% (SE=6.95) better reporting than compared with RCTs published prior to 2014 (t-value=2.29; p=0.03). Additionally, we found that RCTs with secondary PRO outcomes reported 15.55% (SE=7.35) less completely than those with primary PRO outcomes (t-value=−2.12; p=0.04). Other significant associations are available in table 1.
Discussion
We found completeness of reporting for the CONSORT-PRO adaptation to be suboptimal—less than 50%—in our sample of RCTs. Our studies show a significant increase in completeness of reporting following the creation of CONSORT-PRO in 2014. Despite this increase in completeness, less than half of journals in our sample required adherence to CONSORT-PRO. Two of the most commonly omitted checklist items were the mode of PRO administration and the planned statistical methods to account for missing data. Our assessment of RoB revealed that nearly three-quarters of our studies had either ‘some concerns’ or ‘high’ RoB. Here, we will address our findings regarding CONSORT-PRO adherence and discuss recommendations that can improve reporting of PROs in clinical trials.
Over four-fifths of the RCTs in our sample did not completely report their statistical analysis plan to account for missing data. While statistical methods for missing data may not always relate to the PRO itself, it does affect the statistical power of the study and can produce potential bias.28 Several studies evaluate the efficacy of various statistical analyses for missing data. Dong and Peng looked at 68 studies in terms of statements of missing data and plans of how the authors handled missing data.29 They found that many studies either used biased ad-hoc methods or never mentioned implementation methods of missing data.29 Cro et al established that a sensitivity analysis is necessary in RCTs anytime there are missing data, in order to account for how it may affect the results.30 Without proper reporting of the evaluation of missing data, it is difficult to determine the influence on results, and thus, may alter the interpretation of a given PRO measurement. Because the reporting of missing data was often incomplete within IBD trials, we recommend authors improve reporting on how missing data are analyzed.
In addition to deficits of reporting missing data, RCTs with PROs as their secondary outcome had significantly worse per cent completion than RCTs with PROs as their primary outcome. This inconsistency can be identified in relation to PRO reporting in the abstract. For example, one study—an RCT with a secondary PRO outcome—stated in the abstract that ‘secondary and tertiary objectives included measures of efficacy, health-related quality of life (HRQoL), and effects on inflammatory markers’.31 While it was mentioned that HRQoL was an outcome, it did not state which PRO was specifically used, nor did they provide a clear endpoint for the PRO. In contrast, a primary PRO outcome RCT from our sample which reported this information well, stated in the abstract, ‘quality of life was assessed using the Short Inflammatory Bowel Disease Questionnaire (SIBDQ) at beginning and end of the trial’.32 While the primary outcomes are typically the motivation for clinical trials, secondary outcomes are useful in providing support for the primary outcome of a study and can potentially provide the most significant outcomes of a trial.33 34 This discrepancy between outcome reporting may pose an issue in interpreting RCT results. Given that secondary PRO outcomes are incompletely reported and are essential to the transparency of clinical trials, we recommend increased adherence to CONSORT-PRO, as well as guidance for trialists reporting PROs as secondary outcomes.
Despite these specific deficits in reporting, our results show that adherence to CONSORT-PRO has increased in more recent RCTs. In our sample, a significant increase was found in completion of reporting for RCTs published after the creation of CONSORT-PRO. This finding is consistent with previous studies analyzing CONSORT adherence before and after its creation. For example, Han et al 35 conducted a systematic review of 442 psychiatric articles for CONSORT adherence and found significant increases in completeness of reporting after CONSORT was developed. Another study compared 84 RCTs testing non-pharmacological treatments (NPTs) and found significant improvements in mean CONSORT-NPT scores in 2010 compared with 2004.36 Although these authors reported a low overall adherence to CONSORT, the completeness of reporting did improve over time. Given the significant improvement in PRO reporting over time, our results highlight the effects of implementing regulatory bodies to help govern trial conduct. Patient-Reported Outcomes Tools: Engaging Users and Stakeholders (PROTEUS) is a consortium which provides tools and resources for PRO reporting.37 The PROTEUS Consortium was created in 2019 with the primary goal of optimizing PRO data from clinical trials through development of methodological tools used to improve RCT reporting.37 38 Currently, CONSORT-PRO is 1 of 6 different resources promoted by PROTEUS to improve the use of PROs in clinical trials. We recommend IBD trialists using PROs adhere to CONSORT-PRO as a methodological tool given its promotion by PROTEUS, and continue to use resources as they are updated and developed.
Strengths and limitations
Our study is reinforced with strengths that permit increased reproducibility and internal validity. First, we published our protocol on OSF,23 allowing for complete transparency in our methodological practices. In addition, the double-masked, duplicate screening process is the gold standard for data extraction of meta-research and this extraction process increases the reliability of responses when reviewing our results from the RoB tool and the completeness of CONSORT-PRO reporting.39 Finally, in order to strengthen the reliability of our results, the authors involved in screening and data extraction received extensive training on CONSORT-PRO until a consensus was met, and were trained by Cochrane training videos for the Cochrane RoB 2.0 tool.26 Though our study has valid strengths, we do acknowledge limitations. First, we performed a systematic search using three different reputable databases, but there may have been additional RCTs related to IBD that were not included. In addition, our study cannot be generalized to all gastrointestinal literature, as it only applies to RCTs pertaining to IBD.
Our findings indicate a need for improvement in reporting of PROs in RCTs pertaining to IBD. Because IBD is an incurable condition, it is especially important to assess PROs in studies over IBD. Increased adherence to CONSORT-PRO can improve reporting of PROs and their implementation into the management of IBD.
Data availability statement
Data are available in a public, open access repository. Reference number 23 is the dataset provided via Open Science Framework.
Ethics statements
Patient consent for publication
Ethics approval
This study did not require Institutional Review Board oversight, as it did not include any human participants.
Acknowledgments
We are grateful to April Schweikhard who assisted in the development of our search string and the OSU medical library for their procurement of relevant literature.
Footnotes
Contributors All authors have contributed to the concept, design, protocol, screening, data extraction, statistical analysis, and writing and editing of the manuscript to varying degrees. Each author's contributions to the stated areas have earned authorship unless otherwise stated in the ‘Acknowledgments’ section. The guarantor of this study is the corresponding author, RM.
Funding This work was supported by the Oklahoma State University Center for Health Sciences Presidential Mentor-Mentee Research Fellowship Grant.
Competing interests No financial or other sources of support were provided during the development of this manuscript. MH reports receiving funding from the National Institute of Justice for work unrelated to the current subject. MV reports receipt of funding from the National Institute on Drug Abuse, the National Institute on Alcohol Abuse and Alcoholism, the US Office of Research Integrity, Oklahoma Center for Advancement of Science and Technology, and internal grants from Oklahoma State University Center for Health Sciences—all outside of the present work. All other authors have nothing to report.
Provenance and peer review Not commissioned; externally peer reviewed.
References
