Competition Instructions -
Register your email
Please enter your email to
receive the login information to download the dataset, access the
datasets and descriptions of previous competitions (NN3) and to receive
future announcements. (Please note,
that your login information for the NN3 competition will still allow you
to access the NN3 datasets and presentations, but not the new datasets
of NN5!):
-
Select a Dataset
The competition will offer 2 datasets. All time series of the
sets is drawn from a homogeneous population of cash withdrawals
at different cash machines located at different, unrelated
locations in England. Choose only one of the two
datasets! Those forecasting the complete set will automatically
be evaluated (and eligible to win awards and prices) on both
data sets. Consequently we particularly encourage submissions
for the dataset A with 111 time series.
- Dataset A is a
complete dataset of 111 daily time series from 111 different
cash machines .
- Dataset B is a sub
sample of 11 time series from the 111 time series, and is
therefore contained in the larger dataset.
-
Download the data
- Click on the
download link below and enter your login & password in
the dialog-box (case sensitive entry!) to
download the datasets. The login is provided in step 1 when
you register your email-address and personal details.
-
Forecast
- Develop a single
methodology to use on all time series - ideally in software
code or though exactly repeated steps & tests conducted
by a human expert (see the FAQ if this
is unclear).
- Forecast all of
the in-sample training data. For the in-sample data please
provide the 1-step ahead forecasts, as there is no room to
provide 56-step ahead forecasts for each time origin of the
training data. This data will be used to validate goodness
of fit, but will not be used to evaluate and rank the
performance of your submission. If you cannot provide this,
please leave it empty
- Forecast the last
56 observations as a
trace forecast for a forecasting horizon of 1, 2, ..., 56
for each of the 11 or 111 time series .
-
Write Description of your Method
-
Submit your forecasts
- Record your
forecasts in the original Microsoft Excel data files you
downloaded.
- Rename the
forecasting file to include your last name (of the main
author / contestant from a group).
- Create the PDF of
your method descriptions
- Create an email
which MUST:
- be addressed to
submission@neural-forecasting-competition.com
- have the text "NN5
submission" in the subject line
- Include your name & email
contact information in the main text
- inlcude the names &
contact emails of ALL co-workers
- Attach the
following two files to the email:
- attach the
Excel file of your preditions
- attach the
PDF-description of the methodology used
- Send the email by the
submission deadline 18 MAY 2008, 0:00 CET
If you encounter any problems in submitting please
contact sven.crone@neural-forecasting.com
immediately!
General Instructions -
Submissions are restricted to one entrance per competitor.
-
The competitors must certify upon submission that they didn’t try to
retrieve the original data.
-
As this is predominantly an academic competition, all advertising based
upon or referencing the results or participation in this competition
requires prior written consent from the organisers.
Submitting your predictions to us
will not automatically allow you to present your method at a conference. In
addition to submitting, we therefore encourage you to submit to one of the
conferences where we will host special sessions. This will allow you to Please check back here
regularly for information on submission deadlines & dates for theses
conferences.
Experimental Design
The competition design and dataset adhere to previously identified
requirements to derive valid and reliable results. -
Evaluation on multiple time series, using 11 and 111 daily time series
-
Representative time series structure for cash machine demand
-
No domain knowledge, no user intervention in the forecasting methodology
-
Ex ante (out-of-sample) evaluation
-
Single time series origin (1-fold cross validation) in order to limit
effort in computation & comparisons
-
Fixed time horizon of 56 days into the future t+1, t+2, ..., t+56
-
Evaluation using multiple, unbiased error measures
-
Evaluation of "novel" methods against established statistical methods &
software benchmarks
-
Evaluation of "novel" methods against standard Neural Networks software
packages
-
Testing of conditions under which NN & statistical methods perform well
(using multiple hypothesis)
Datasets
Two datasets are provided, which may be found [here].
Methods
The competition is open to all methods from Computational Intelligence,
listed below. The objective requires a single methodology, that is
implemented across all time series. This does not require you to build a
single neural network with a pre-specified input-, hidden and output-node
structure but allows you to develop a process in which to run tests and
determine a best setup for each time series. Hence you can come up with 111
different network architectures, fuzzy membership functions, mix of ensemble
members etc. for your submission. However, the process should always lead to
selecting the same final model structure as a rigorous process. -
Feed forward Neural Networks (MLP etc.)
-
Recurrent Neural Networks (TLRNN, ENN, ec.)
-
Fuzzy Predictors
-
Decision & Regression Trees
-
Particle Swarm Optimisation
-
Support Vector Regression (SVR)
- Evolutionary & Genetic
Algorithms
-
Composite & Hybrid approaches
- Others
These will be evaluated against established statistical forecasting methods -
Naďve
-
Single, Linear, Seasonal & Dampened Trend Exponential Smoothing
-
ARIMA-Methods
Statistical benchmarks will be calculated using the software AUTOBOX and ForecastPro,
two of the leading expert system software packages for automatic
forecasting (provided by courtesy of Dave Reilly and Eric Stellwagen
-THANKS!). We hope to also evaluate a number of additional
packages: SAS, NeuralWorks (pending), Alyuda Forecatser (peding),
NeuroDimensions (pending). In addition, the competition is open for
submissions from statistical benchmark methods. Although these can be
submitted and evaluated as benchmarks, only methods from computational
intelligence are eligible to "win".
Evaluation
We assume no particular
decision problem of the underlying forecasting competition and hence assume
symmetric cost of errors. To account for a different number of observations
in the individual data sub-samples of training and test set, and the
different scale between individual series we propose to use a mean
percentage error metric, which is also established best-practice in industry
and in previous competitions. All submissions will be evaluated using the
mean Symmteric Mean Absolute Percent Error (SMAPE) across al time series.
The SMAPE calculates the symmetric absolute error in percent between the
actuals X and the forecast F across all
observations t of the test set of size n for
each time series s with
The SMAPE
of each series will then be averaged over all time series in the dataset for
a mean SMAPE. To determine a winner, all submissions will be ranked by mean
SMAPE across all series. However, biases may be introduced in selecting a
“best” method based upon a single metric, particularly in the lack of a true
objective or loss function. Therefore, while our primary means of ranking
forecasting approaches is mean SMAPE, alternative metrics will be used so as
to guarantee the integrity of the presented results. All submitted forecasts
will also be evaluated on a number of additional statistical error measures
in order to analyse sensitivity to different error metrics. Additional
Metrics for reporting purposes include:
-
Average SMAPE (main metric to determine winner)
- Median SMAPE
-
Median absolute percentage error (MdAPE)
- Median relative absolute error (MdRAE)
-
Average Ranking based upon the error measures
- …
Publication &
Non-Disclosure of Results
We respect the decision of individuals to withhold their name should they
feel unsatisfied with their results. Therefore each contestant reserves the
right to withdraw their name and software package used after they have
learned their relative rank on the datasets. However, we reserve the right
to publish an anonymised version of the descriptions of themethod and methodology used, i.e. MLP, SVR etc
without the name of the contributor. |
Important Dates
|
18
February 2008 |
Start of the NN5 daily
time series forecasting competition |
18 May 2008
|
Submission deadline
for predictions of 11 and 111 time series |
1-6 June 2008
|
NN5 special session at
the World Congress on Computational Intelligence (WCCI'08),
Hong Kong, China |
23-26 June 2008 |
NN5 special session at
the International
Symposium on Forecasting (ISF'08),
Nice, France |
14-17
July 2008 |
NN5 special session at
the International
Conference on Data Mining (DMIN'08)
Las Vegas, USA |
|