Are forecasting competitions data representative of the reality?

Evangelos Spiliotis, Andreas Kouloumos, Vassilios Assimakopoulos, Spyros Makridakis

    Research output: Contribution to journalArticlepeer-review


    Forecasters typically evaluate the performances of new forecasting methods by exploiting data from past forecasting competitions. Over the years, numerous studies have based their conclusions on such datasets, with mis-performing methods being unlikely to receive any further attention. However, it has been reported that these datasets might not be indicative, as they display many limitations. Since forecasting research is driven somewhat by data from forecasting competitions, it becomes vital to determine whether they are indeed representative of the reality or whether forecasters tend to over-fit their methods on a random sample of series. This paper uses the data from M4 as proportionate to the real world and compares its properties with those of past datasets commonly used in the literature as benchmarks in order to provide evidence on that question. The results show that many popular benchmarks of the past may indeed deviate from reality, and ways forward are discussed in response.

    Original languageEnglish
    JournalInternational Journal of Forecasting
    Publication statusPublished - 1 Jan 2019


    • Forecasting competitions
    • Forecasting methods
    • M4
    • Time series features
    • Time series visualization


    Dive into the research topics of 'Are forecasting competitions data representative of the reality?'. Together they form a unique fingerprint.

    Cite this