Press Release
19.06.2018
The M-Competitions and their far-reaching contributions to the Theory and Practice of Forecasting
The latest edition (M4) highlights that hybrid approaches and combinations of forecasting methods produce greater accuracy
The M4 (Makridakis 4) is the latest edition of the premier academic forecasting competition in the world, which recently came to a close, featuring a total of 248 individuals and teams from 48 countries across the globe. The revolutionary M-Competitions, which Professor Spyros Makridakis (whom the M-Competitions are also named after) launched four decades ago, have substantially contributed to improving forecasting accuracy and informing practitioners on the most appropriate forecasting method to use for their specific needs.
This year’s competition was organised by the Institute For the Future (IFF) at the University of Nicosia (UNIC), with the support of the Forecasting & Strategy Unit at the National Technical University of Athens (NTUA).
According to Rob Hyndman, the Editor-in-Chief of the International Journal of Forecasting: “The M-competitions have had an enormous influence on the field of forecasting. They focused attention on what models produced good forecasts, rather than on the mathematical properties of those models. For that, Spyros [Makridakis] deserves congratulations for changing the landscape of forecasting research through this series of competitions.”
The M4 attracted wide participation from universities as far as Brazil and Australia, as well as from multinational organisations and companies including UBER, Oracle and Wells Fargo Bank, with a large number of participants hailing from the USA, Malaysia, Greece and the UK.
Prof. Makridakis, considered by many the father of forecasting, remarked: “I am delighted with the overwhelming interest and support in this latest iteration of the M-Competition. Taken together, the major findings seem to indicate that single statistical or ML methods are of limited value and that hybrid approaches and combination methods are the way forward in improving upon forecasting accuracy and making forecasting more valuable.” Also noteworthy, according to Prof. Makridakis, is the fact that 50 out of the 248 participants completed and submitted forecasting and prediction intervals for all the 100,000 time series.
More specifically, the five major findings of the M4 Forecasting Competition are as follows:
- The combination of methods was the king of the M4. Of the 17 most accurate methods, 12 were “combinations” of mostly statistical approaches.
- The biggest surprise was a “hybrid” approach that utilised both statistical and ML features. This method produced both the most accurate forecasts and the most precise PIs, and was submitted by Slawek Smyl, a Data Scientist at Uber Technologies. According to sMAPE, it was close to 10% more accurate than the combination benchmark.
- The second most accurate method was a combination of seven statistical methods and an ML one, with the weights for the averaging calculated by an ML algorithm that was trained to minimise the forecasting error through holdout tests. This method was submitted jointly by Spain’s University of A Coruña and Australia’s Monash University.
- The most accurate and second most accurate methods also achieved an amazing success in specifying the 95% PIs correctly. These are the first methods we are aware of that have done so, rather than underestimating the uncertainty considerably.
- The six pure ML methods that were submitted in the M4 all performed poorly, with none of them being more accurate than Comb and only one being more accurate than Naïve2. This supports the findings of the latest PLOS ONE paper by Makridakis, Spiliotis and Assimakopoulos.
Prof. Makridakis and the forecasting community are looking forward to the launch of the next M-Competition – with preparations for the M5 underway – that will extend its coverage beyond time series methods to include additional exogenous (explanatory) variables, to improve forecasting accuracy used for financial (stock and commodity market) predictions.
For more information and for fuller findings and conclusions of the M4 Forecasting Competition, please contact Milton George at [email protected].