Are forecasting competitions data representative of the reality?

Evangelos Spiliotis, Andreas Kouloumos, Vassilios Assimakopoulos, Spyros Makridakis

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

Forecasters typically evaluate the performances of new forecasting methods by exploiting data from past forecasting competitions. Over the years, numerous studies have based their conclusions on such datasets, with mis-performing methods being unlikely to receive any further attention. However, it has been reported that these datasets might not be indicative, as they display many limitations. Since forecasting research is driven somewhat by data from forecasting competitions, it becomes vital to determine whether they are indeed representative of the reality or whether forecasters tend to over-fit their methods on a random sample of series. This paper uses the data from M4 as proportionate to the real world and compares its properties with those of past datasets commonly used in the literature as benchmarks in order to provide evidence on that question. The results show that many popular benchmarks of the past may indeed deviate from reality, and ways forward are discussed in response.

Original languageEnglish
JournalInternational Journal of Forecasting
DOIs
Publication statusPublished - 1 Jan 2019

Fingerprint

Benchmark
Forecasting competitions
Forecasting method

Keywords

  • Forecasting competitions
  • Forecasting methods
  • M4
  • Time series features
  • Time series visualization

Cite this

Spiliotis, Evangelos ; Kouloumos, Andreas ; Assimakopoulos, Vassilios ; Makridakis, Spyros. / Are forecasting competitions data representative of the reality?. In: International Journal of Forecasting. 2019.
@article{a25cdafa790b40b086b1a06a56e81360,
title = "Are forecasting competitions data representative of the reality?",
abstract = "Forecasters typically evaluate the performances of new forecasting methods by exploiting data from past forecasting competitions. Over the years, numerous studies have based their conclusions on such datasets, with mis-performing methods being unlikely to receive any further attention. However, it has been reported that these datasets might not be indicative, as they display many limitations. Since forecasting research is driven somewhat by data from forecasting competitions, it becomes vital to determine whether they are indeed representative of the reality or whether forecasters tend to over-fit their methods on a random sample of series. This paper uses the data from M4 as proportionate to the real world and compares its properties with those of past datasets commonly used in the literature as benchmarks in order to provide evidence on that question. The results show that many popular benchmarks of the past may indeed deviate from reality, and ways forward are discussed in response.",
keywords = "Forecasting competitions, Forecasting methods, M4, Time series features, Time series visualization",
author = "Evangelos Spiliotis and Andreas Kouloumos and Vassilios Assimakopoulos and Spyros Makridakis",
year = "2019",
month = "1",
day = "1",
doi = "10.1016/j.ijforecast.2018.12.007",
language = "English",
journal = "International Journal of Forecasting",
issn = "0169-2070",
publisher = "Elsevier",

}

Are forecasting competitions data representative of the reality? / Spiliotis, Evangelos; Kouloumos, Andreas; Assimakopoulos, Vassilios; Makridakis, Spyros.

In: International Journal of Forecasting, 01.01.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Are forecasting competitions data representative of the reality?

AU - Spiliotis, Evangelos

AU - Kouloumos, Andreas

AU - Assimakopoulos, Vassilios

AU - Makridakis, Spyros

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Forecasters typically evaluate the performances of new forecasting methods by exploiting data from past forecasting competitions. Over the years, numerous studies have based their conclusions on such datasets, with mis-performing methods being unlikely to receive any further attention. However, it has been reported that these datasets might not be indicative, as they display many limitations. Since forecasting research is driven somewhat by data from forecasting competitions, it becomes vital to determine whether they are indeed representative of the reality or whether forecasters tend to over-fit their methods on a random sample of series. This paper uses the data from M4 as proportionate to the real world and compares its properties with those of past datasets commonly used in the literature as benchmarks in order to provide evidence on that question. The results show that many popular benchmarks of the past may indeed deviate from reality, and ways forward are discussed in response.

AB - Forecasters typically evaluate the performances of new forecasting methods by exploiting data from past forecasting competitions. Over the years, numerous studies have based their conclusions on such datasets, with mis-performing methods being unlikely to receive any further attention. However, it has been reported that these datasets might not be indicative, as they display many limitations. Since forecasting research is driven somewhat by data from forecasting competitions, it becomes vital to determine whether they are indeed representative of the reality or whether forecasters tend to over-fit their methods on a random sample of series. This paper uses the data from M4 as proportionate to the real world and compares its properties with those of past datasets commonly used in the literature as benchmarks in order to provide evidence on that question. The results show that many popular benchmarks of the past may indeed deviate from reality, and ways forward are discussed in response.

KW - Forecasting competitions

KW - Forecasting methods

KW - M4

KW - Time series features

KW - Time series visualization

UR - http://www.scopus.com/inward/record.url?scp=85062656389&partnerID=8YFLogxK

U2 - 10.1016/j.ijforecast.2018.12.007

DO - 10.1016/j.ijforecast.2018.12.007

M3 - Article

JO - International Journal of Forecasting

JF - International Journal of Forecasting

SN - 0169-2070

ER -