Op-ed: "Trump’s Desperate Plan to Cut Short the Census Could Backfire" (New York Times)

I had an opinion piece about the US census in the New York Times, dated September 23 (and in condensed form in print Sunday Review of September 27). It was published shortly before the census’s revised (and shortened) end date for non-response follow-up (NRFU). A federal court in California eventually extended that date by a couple of weeks.

There was a bit of luck in how this came together. I had already pitched the Times Opinion editors on the slow progress of NRFU, and how the Bureau’s own operational data could show that—a data story about the census being my perfect intersection. As we were iterating the idea, the American Statistical Association published a policy paper on the same topic. We joined forces, and I merged the work I had already done into the ASA model, which its authors Jonathan Auerbach and Steve Pierson were kind enough to share.

One of the output charts from my model.

I ended up swapping out nearly all the parts of the original model, with the largest changes in how I modelled the progress of enumeration itself. The ASA paper forecast the cumulative total NRFU directly, which led to two issues:

• it was clearly pessimistic

• the forecasts could be discontinuous with the observed data, or even show a fall from day to day, which is obviously impossible.¹ That wasn’t very noticeable in the cumulative total chart above (scaled 70-100%), but it really stood out on the remaining NRFU chart below (on a zoomed 0-10% axis). So I instead forecast the proportion of remaining NRFU that would be completed each day, which meant that the forecast would always smoothly (and nondecreasingly) pick up from observed data.

The same chart, transformed by Gus Wezerek.

Overall it was a very fun project. The pace of a news event meant that I had to understand the ASA model, modify if, check and then carefully comment everything for public release in the space of about two days.

The Github repo, with a very extensive README and very well-documented code is here.