April R. Miller, Samin Charepoo, Erik Yan, Ryan W. Frost, Zachary J. Sturgeon, Grace Gibbon, Patrick N. Balius, Cedonia S. Thomas, Melanie A. Schmitt, Daniel A. Sass, James B. Walters, Tracy L. Flood, Thomas A. Schmitt
Abstract
The rapid proliferation of COVID-19 has left governments scrambling, and several data aggregators are now assisting in the reporting of county cases and deaths. The different variables affecting reporting (e.g., time delays in reporting) necessitates a well-documented reliability study examining the data methods and discussion of possible causes of differences between aggregators.
Introduction
In the wake of the COVID-19 (2019 novel coronavirus) pandemic, death rates and spatial mapping, not dissimilar to methods used during the 19th century London cholera outbreak, have become talking points of the 21st century [1, 2]. As COVID-19 slowly gained momentum in late winter and early spring of 2020, governments and other organizations scrambled to collect and present temporo-spatial data. When governments, understandably, struggled with the proliferation of COVID-19, many non-governmental organizations and universities helped with the COVID-19 data collection by innovating with data aggregation techniques (e.g., web scraping, crowd-sourcing) [1–4].
Methods
Starting on March 16th, 2020, the Broadstreet team (consisting of approximately 120 volunteers) [31] began tracking diagnosed cumulative cases of, and deaths due to COVID-19 reported by state and county governments [26]. Broadstreet volunteers were recruited from a variety of universities through public health and other related undergraduate and graduate departments, and they were eligible to participate in this project if they had any interest or experience in a public health-related field.
Results
Table 3 results provide a LWCK reliability comparison across all the aggregators at the U.S. level. These results suggest that mean (M) reliability for JHU (MCases = 0.89, MDeath = 0.69), NYT (MCases = 0.89, MDeath = 0.69), USAF (MCases = 0.89, MDeath = 0.68), and CTP (MCases = 0.88, MDeath = 0.64) displayed the highest average inter-rater agreement among the five reported aggregators for both new cases and deaths. On average, BS (MCases = 0.75, MDeath = 0.59) yielded consistently lower inter-rater reliability when compared to other aggregators for both the number of cases and deaths. Interestingly, lower levels of inter-rater agreement were observed across aggregators associated with the number of deaths.
Discussion
This study compared the reliability of COVID-19 death and cases count data across national, state, and county-levels between data aggregators. As expected, given the larger sample sizes, reliability for both cases and deaths was higher at the national level across aggregators than at state and county levels. However, death count reliability was typically lower than reliability for reported cases. Variation in reliability remained across aggregators and suggests that aggregator choice could have a significant impact on any data analysis or subsequent action based on the data.
Citation: Miller AR, Charepoo S, Yan E, Frost RW, Sturgeon ZJ, Gibbon G, et al. (2022) Reliability of COVID-19 data: An evaluation and reflection. PLoS ONE 17(11): e0251470. https://doi.org/10.1371/journal.pone.0251470
Editor: Jagdish Khubchandani, New Mexico State University, UNITED STATES
Received: June 16, 2021; Accepted: December 10, 2021; Published: November 3, 2022
Copyright: © 2022 Miller et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All the data underlying the findings of the study are available at the provided URL (https://github.com/BroadStreet-Health/COVID-19-Cases-and-Mortalities).
Funding: Funders for this study include only BroadStreet Health, which provided support in the form of salaries for authors EY, RWF, and ZJS, but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The specific roles of these authors are articulated in the ‘author contributions’ section. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
