The Pediatrics Mask Study

A study recently making the Twitter rounds was published as a pre-publication release in the Journal Pediatrics: “School Masking Policies and Secondary SARS-CoV-2 Transmission.” The study’s results found that

“Districts that optionally masked throughout the study period had 3.6 times the rate of secondary transmission as universally masked districts. For every 100 community-acquired cases, universally masked districts had 7.3 predicted secondary infections, while optionally masked districts had 26.4.”

Unlike other papers on school masking, which typically only look at total cases detected within schools, regardless of where from (typically outside of school), this paper’s objective was to try to ascertain Sars-Cov-2 transmission within schools. To do that, they focused their efforts on secondary transmission. I think that objective is an important distinction and absolutely deserves treatment, as we already know that most transmissions detected in school were from outside of school, rather than from within.

In fact, almost a year ago, a previous study by the same group in Pediatrics found the following result:

“Among these students and staff, 773 community-acquired SARS-CoV-2 infections were documented by molecular testing. Through contact tracing, health department staff determined an additional 32 infections were acquired within schools. No instances of child-to-adult transmission of SARS-CoV-2 were reported within schools.”

The study design: to compare secondary transmission only as a means of determining effectiveness of mask policies, makes sense in light of their previous research. However, diving into the results, tables, conclusions and discussion, I found a few major issues that call their result into question.

The first issue is with the major imbalance in sample size between the two groups “Universal” Mask requirement, which you would not have realized had you read the headline in the NIH Press Release: “NIH-funded study compared more than 1.1 million students across nine states.”

One would immediately think that the the sample size of that magnitude means that the study had pretty serious statistical power. But looking deeper, at the sample size of each individual category (masked required, partial mask, or mask optional), shows a major imbalance.

How many optional masked students were part of that “more than 1.1 million students?” Well, Table 1 of the study shows their sample for that group was only 3950 students (including staff, N =4,576)! The “Masked” sample size was 1,112,899 students. Meaning the sample size of the mask optional districts was .35% the size of the universally masked students.

Here’s a visual breakdown of the sample size found in Table 2 of the paper.

Why is this imbalance an issue? Certainly when working with real world observations, we aren’t realistically going to conveniently get a perfectly equal sample size between intervention and controls. It’s not an RCT after all. Certainly as long as there was at least a decently large sample size to achieve statistical power, why would it matter? Let’s set aside the issue of the extremely wide confidence intervals that their predicted measures produced. How does a major imbalance in sample size between groups potentially impact the results? A paper that measured the impact of imbalance sample sized in the Journal Practical Assessment, Research, and Evaluation explains

“The overall power of the test is strongly influenced by the size of the sample, the amount of variability in the sample, and the size of the difference in the population.”

So the fact that these sample sizes are imbalanced by a factor of 280x, should be cause for serious concern.

The second issue with this paper is that while the secondary infection rate may have been lower in universal masked schools, the primary infections per 1000 students was higher! In fact, it was 3x! the rate of primary infections in mask optional schools.

See the comparisons of Primary Infection Rates on the left, and Secondary infection rates on the right, below. (from Table 3 of the study).