Fotios Petropoulos, Spyros Makridakis and Neophytos Stylianou
Introduction
The COVID-19 pandemic presents humanity with a profound challenge, not confronted for more than 100 years, since the occurrence of the Spanish flu in1918. Policy decision-makers are faced with some tough decisions on how to deal with the pandemic, requiring forecasting its future behavior in terms of, among others, numbers infected with the virus and death rates so that they can better plan for healthcare and other resources. The choices are grim. Lockdowns to stop the spread of the virus and deaths, tolerating the serious economic consequences, or allowing business as usual to keep the economy going but enduring loss of life and human suffering.
In this paper, we will present a short-term forecasting model to predict the number of confirmed infections and deaths for the next ten days as well as estimating the uncertainty involved around these predictions. Although we could have extended the forecasting horizon, we did not do so because the range of uncertainty would have increased disproportionally, decreasing their practical value. At the same time, by repeating the forecasting process every ten days, we achieved the double objectives of evaluating the accuracy/uncertainty in an ex-post way many times while also observing the trajectory of infections and deaths over time. Evaluating our forecasts and prediction intervals can provide useful information to decision and policy makers about the trajectory of the pandemic as well as the uncertainty in the predictions, taking into account the enormous asymmetry in the risks and costs of over and underestimating the spread of the virus and the number of deaths.
Method
This study expands the study of Petropoulos and Makridakis and will report the results of a unique, live forecasting exercise that took place since early February 2020 (for the most up-to-date forecasts, please visit Fotios’ twitter account). We employed a simple time series model to predict the continuation of two important variables, confirmed cases and deaths. In particular, we used an exponential smoothing model with a multiplicative trend to capture the exponential behaviour of the data.
While we avoid making micro-assumptions on a large number of unknown variables (such as transmissibility or death rates), our approach is still based on some fundamental assumptions: First, we assume that the data is accurate; Second, our forecasts assume that past patterns will continue into the future. Both of these assumptions are necessary to obtain accurate predictions. Further, we argue that even if the confirmed cases and deaths are significantly under-reported, the forecasts still provide useful information for policy and decision-makers as long as the data is consistent.
Our first round of forecasts was made for the period 01-Feb-2020 up until 10-Feb-2020, using the first ten data points available (from 22-Jan-2020 to 31-Jan-2020) retrieved from the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University. We produced both point forecasts as well as prediction intervals at three confidence levels, 50, 70, and 90%. Once ten more data points became available, we produced a second round of forecasts, expanding our training data set to 20 days. We repeated this exercise, by producing sets of 10-days-ahead forecasts every ten days, until the end of April 2020. The results are summarised in the figure below, where the forecasts for the confirmed cases and deaths are depicted with blue and red respectively.
A comparison of forecasts with the recorded actual values gives us valuable insights into the trajectory of the virus. The significant over-forecasting in confirmed cases and deaths over the first couple of rounds suggested that COVID-19 was sufficiently addressed in China. The positive forecast errors (under-forecasting) in the fourth and fifth rounds of forecasts are linked with the spreading of the virus to countries outside China, and predominantly Europe and USA, and the need for more actions to restrict this spread. The negative forecast errors in the last three rounds, together with the decrease in the positive trend, suggest that we are, hopefully, reaching a plateau.
Another important factor of our forecasts is to illustrate the ability of simple, time-series models to capture the associated uncertainty. In all rounds but one, the actual values of the two variables are well-within the 50% prediction intervals. The only exception is the first round when the available data were very limited while the actual values were still within the 90% intervals. While one could argue that our intervals are too wide (statistically speaking, we would expect the actual values to exceed the 50% upper interval in ¼ of the cases), we believe that, given the asymmetric risks involved and the exponential nature of the challenge, it is preferable to be able to identify and prepare for more extreme scenarios.
Discussion
This live forecasting exercise and the results demonstrate the benefits of the approach. A common statistical aphorism attributed to George Box states that “all models are wrong, but some are useful”. Having this in mind, the model developed for this live forecasting exercise can be argued that it falls in the “useful” ones despite the possibility of being “wrong”. In times of crisis, such as a pandemic, when time is of essence, a simple but robust model can provide useful insight to policy decision-makers. Forecast performance, which in this case is high with all predictions within the intervals, is not the single criterion in these situations. Other criteria may be timeliness, usefulness for policymaking, ease in acquiring sufficient data, provision of sufficient detail, reasonableness of assumptions, internal consistency, cost of development, ease of explanation, and many more. Simple forecasting models which employ methods such as exponential smoothing are a quick, easy, and inexpensive way of developing a forecast that can be of use to various stakeholders. Clinicians and policy decision-makers are not statisticians. There is generally a reluctance in the use of “unknown” methodologies, especially anything that can be seen as a black box. More complex models or multivariable models are usually “data-hungry” and more time-consuming in their set-up, training, and validation. In times of crisis, time is a scarce resource. Finally, we should not forget that models are used to describe the direction of signal travel and they are by no means a substitution for experts; their purpose is to support decisions.