Dissemination of research during the first year of the coronavirus disease 2019 pandemic ======================================================================================== * Justin S Brandt * Sonal Grover * Cande V Ananth * research * COVID-19 * publishing The rapid and effective dissemination of research during the coronavirus disease 2019 (COVID-19) pandemic is critical if healthcare providers and public health officials are to remain aware of new developments. Several organizations have collected relevant COVID-19 articles to facilitate data sharing, including the World Health Organization1 and the National Institutes of Health (NIH).2 Yet, in this time of remarkable research productivity and social media influence, how peer-reviewed research disseminates to the global community remains poorly understood. In order to ascertain how published research disseminated during the COVID-19 pandemic, we examined the world’s peer-reviewed literature on COVID-19 through the application of alternative metrics (altmetrics) based on social media engagement and evaluative bibliometrics using citation rates. We utilized almetrics and citation rates to analyze COVID-19 articles indexed in the NIH’s iSearch COVID-19 portfolio2 and the Almetric Explorer3 (study flow chart, figure 1). The iSearch COVID-19 Portfolio contains peer-reviewed COVID-19 articles from PubMed and preprints from several sources. We restricted our analysis to articles and used this dataset to ascertain citation rates. These data were merged by articles’ unique PubMed identification numbers with altmetrics from the Altmetric Explorer, where we ascertained articles’ Almetric Attention Scores (AAS; composite score of social interest that includes mentions in newsfeeds, Twitter, Facebook, and Google, among other sources).4 ![Figure 1](/https://d3hme472k3gd2d.cloudfront.net/content/jim/early/2021/05/28/jim-2021-001923/F1.medium.gif) [Figure 1](/content/early/2021/05/28/jim-2021-001923/F1) Figure 1 Study flow chart. WoS, ISI Web of Science. We analyzed COVID-19 articles in the ISI Web of Science (WoS) Core Collection5 to evaluate article characteristics, including study type, authorship, and funding sources. We queried the ISI WoS Core Collection using the search terms “coronavirus disease 2019”, “COVID-19”, “severe acute respiratory syndrome coronavirus 2”, “SARS-CoV-2”, and “novel coronavirus”. The ISI WoS query was limited to publications from January 2020 to February 2021 (with no language or article type restrictions). Although there are several search engines that can be used for bibliometric analysis, including Scopus, Medline, and Google Scholar,6 7 we selected the ISI WoS Core Collection because this database was classically used by Eugene Garfield, the developer of the impact factor metric, to identify “citation classics”.8 Research data were downloaded from iSearch and merged with Almetric Explorer on February 26, 2021. The results of the ISI WoS query were also downloaded on February 26, 2021. This analysis was based on a total of 87,643 articles in iSearch that were merged with data from the Altmetric Explorer (75,960 (86.7%) published in 2020 and 11,682 (13.3%) published in 2021) and 90,609 articles in the ISI WoS query (82,008 (90.5%) published in 2020 and 8601 (9.5%) in 2021). COVID-19 articles were published rapidly during the first months of the pandemic, peaking in April 2020, and then plateauing at persistently high rates (figure 2A). Trends in citation rates and AAS mirrored each other during the study period (figure 2B). There were 48 articles in iSearch with unique PubMed identification numbers that were retracted or were retraction notices, corresponding to 34 (0.04%) articles (geometric mean citation rate (95% CI) 7.7 (3.5 to 16.5) and arithmetic mean (SD) 30.3 (80.3) and geometric mean AAS (95% CI) 26.9 (8.5 to 84.9) and arithemetic mean AAS (SD) 1243.3 (3784.0)). Some of these articles received substantial social media attention. For example, among the top 10 articles with highest AAS, one of the articles was retracted and another article was the official retraction notice from the journal that published that article. The top 10 articles with the highest citation rates and highest AAS, after excluding retracted articles, are described in table 1. ![Figure 2](/https://d3hme472k3gd2d.cloudfront.net/content/jim/early/2021/05/28/jim-2021-001923/F2.medium.gif) [Figure 2](/content/early/2021/05/28/jim-2021-001923/F2) Figure 2 Monthly distribution of citation rates and altmetrics for articles indexed in the National Institutes of Health’s iSearch Coronavirus Disease 2019 Portfolio, January 2020 to February 2021. (A) Distribution of citation rates by month of publication. (B) Distribution of monthly article publication, mean citation rates, and mean Altmetric Attention Scores (AAS). View this table: [Table 1](/content/early/2021/05/28/jim-2021-001923/T1) Table 1 Top 10 articles with the highest citation rates and highest Almetric Attention Scores for articles indexed in the National Institutes of Health’s iSearch Coronavirus Disease 2019 Portfolio, January 2020 to February 2021 The ISI WoS query revealed the most common document types were “articles” (47,717; 52.7%), “editorials” (14,491; 16.0%), and “letters” (14,073; 15.5%). The most frequent WoS categories were “Medicine, General & Internal” (11,111; 12.2%), “Public, Environmental & Occupational Health” (7281; 8.0%), and “Infectious Disease” (4790; 5.2%). Over half of all articles originated from four countries: the US (25,312; 27.9%), China (10,535; 11.6%), Italy (8899; 9.8%), and England (8759; 9.7%). The top funding agencies were the NIH, the National Natural Science Foundation of China, and the European Commission. In this study of the world’s peer-reviewed COVID-19 literature in the first year of the pandemic, we observed a dramatic explosion of research output. With rapid publication of approximately 90,000 peer-reviewed articles that addressed all facets of COVID-19, the global community has been inundated with data. Notably, nearly 60% of the world’s research output originated from four countries that were hit first and hardest by COVID-19. Three countries are geographically distinct from the origination site of the virus, highlighting the global impact of the disease. We observed that mean citation rates and AAS mirrored each other. Although there is debate about whether altmetrics correlate with citation rates,9 10 this study suggests these metrics have correlated during the pandemic. While citation rates and altmetrics reflect influence, they do not provide insight into research quality. The explosion of COVID-19 publications has raised legitimate concerns about research quality11 as well as misconduct.12 Many journals, particularly top tier journals, prioritized submissions of COVID-19-related articles, potentially at the expense of other topics, and expedited their peer review and publication. The rush to publish on the part of investigators and journals may have encouraged suboptimal research designs and methods as well as suboptimal peer review. Retracted articles perhaps reflect the most egregious examples of how the push to publish during the pandemic promoted poor-quality research. While the proportion of retracted articles in this study was small, these articles received substantial social media attention as well as high citation rates. This study underscores the need for a novel metric that prioritizes research quality rather than quantity. ## Acknowledgments Almetric provided no-cost access to their data for this project, but were not involved in the study design, analysis, or drafting the manuscript nor did they review or approve any version of the manuscript. ## Footnotes * Twitter @DrJustinBrandt * Contributors JSB: conceptualization, methodology, formal analysis, investigation, data curation, writing - original draft, writing - review & editing. SG: writing - review & editing. CVA: writing - review & editing, supervision. * Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors. * Competing interests None declared. * Provenance and peer review Not commissioned; externally peer reviewed. This article is made freely available for use in accordance with BMJ’s website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained. [https://bmj.com/coronavirus/usage](https://bmj.com/coronavirus/usage) ## References 1. World Health Organization. Global research on coronavirus disease (COVID-19). Available: [https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov](https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov) [Accessed 27 Feb 2021]. 2. National Institutes of Health. iSearch COVID-19 Portfolio [Internet]. Available: [https://icite.od.nih.gov/covid19/search/](https://icite.od.nih.gov/covid19/search/) [Accessed 27 Feb 2021]. 3. Almetric. Almetric Explorer [Internet]. Available: [https://www.altmetric.com](https://www.altmetric.com) [Accessed 27 Feb 2021]. 4. Clarivate Analytics.. Web of Science Core Collection. [Internet]. Available: [https://clarivate.com/products/web-of-science/web-science-form/web-science-core-collection/](https://clarivate.com/products/web-of-science/web-science-form/web-science-core-collection/) [Accessed 27 Feb 2021]. 5. Trueger NS, Thoma B, Hsu CH, et al. The Altmetric score: a new measure for article-level dissemination and impact. Ann Emerg Med 2015;66:549–53.[doi:10.1016/j.annemergmed.2015.04.022](http://dx.doi.org/10.1016/j.annemergmed.2015.04.022)pmid:http://www.ncbi.nlm.nih.gov/pubmed/26004769 [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2Fearly%2F2021%2F05%2F28%2Fjim-2021-001923.atom) 6. Brandt JS, Hadaya O, Schuster M, et al. A bibliometric analysis of top-cited journal articles in obstetrics and gynecology. JAMA Netw Open 2019;2:e1918007. [doi:10.1001/jamanetworkopen.2019.18007](http://dx.doi.org/10.1001/jamanetworkopen.2019.18007)pmid:http://www.ncbi.nlm.nih.gov/pubmed/31860106 [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2Fearly%2F2021%2F05%2F28%2Fjim-2021-001923.atom) 7. Mitra AN, Aurora N, Grover S, et al. A bibliometric analysis of obstetrics and gynecology articles with highest relative citation ratios, 1980 to 2019. Am J Obstet Gynecol MFM 2021;3:100293. [doi:10.1016/j.ajogmf.2020.100293](http://dx.doi.org/10.1016/j.ajogmf.2020.100293)pmid:http://www.ncbi.nlm.nih.gov/pubmed/33451619 [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2Fearly%2F2021%2F05%2F28%2Fjim-2021-001923.atom) 8. Garfield E. What is a citation classic? Available: [http://garfield.library.upenn.edu/classics.html](http://garfield.library.upenn.edu/classics.html) [Accessed 3 Apr 2021]. 9. Giustini AJ, Axelrod DM, Lucas BP, et al. Association between citations, altmetrics, and article views in pediatric research. JAMA Netw Open 2020;3:e2010784. [doi:10.1001/jamanetworkopen.2020.10784](http://dx.doi.org/10.1001/jamanetworkopen.2020.10784)pmid:http://www.ncbi.nlm.nih.gov/pubmed/32687584 [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2Fearly%2F2021%2F05%2F28%2Fjim-2021-001923.atom) 10. Chang J, Desai N, Gosain A. Correlation between Altmetric score and citations in pediatric surgery core journals. J Surg Res 2019;243:52–8.[doi:10.1016/j.jss.2019.05.010](http://dx.doi.org/10.1016/j.jss.2019.05.010)pmid:http://www.ncbi.nlm.nih.gov/pubmed/31154133 [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2Fearly%2F2021%2F05%2F28%2Fjim-2021-001923.atom) 11. Weiner DL, Balasubramaniam V, Shah SI, et al. COVID-19 impact on research, lessons learned from COVID-19 research, implications for pediatric research. Pediatr Res 2020;88:148–50.[doi:10.1038/s41390-020-1006-3](http://dx.doi.org/10.1038/s41390-020-1006-3)pmid:http://www.ncbi.nlm.nih.gov/pubmed/32544922 [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2Fearly%2F2021%2F05%2F28%2Fjim-2021-001923.atom) 12. Bramstedt KA. The carnage of substandard research during the COVID-19 pandemic: a call for quality. J Med Ethics 2020;46:803–7.[doi:10.1136/medethics-2020-106494](http://dx.doi.org/10.1136/medethics-2020-106494)pmid:http://www.ncbi.nlm.nih.gov/pubmed/33004545 [Abstract/FREE Full Text](/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToibWVkZXRoaWNzIjtzOjU6InJlc2lkIjtzOjk6IjQ2LzEyLzgwMyI7czo0OiJhdG9tIjtzOjQyOiIvamltL2Vhcmx5LzIwMjEvMDUvMjgvamltLTIwMjEtMDAxOTIzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==)