In addition, we encourage you to submit to one of the conferences where we will host special sessions. This will allow you to
Please check back here regularly for information on submission deadlines & dates for theses conferences.
The competition design and dataset adhere to previously identified requirements to derive valid and reliable results.
Two datasets are provided, which may be found [here].
The competition is open to all methods from Computational Intelligence, listed below. The objective requires a single methodology, that is implemented across all time series. This does not require you to build a single neural network with a pre-specified input-, hidden and output-node structure but allows you to develop a process in which to run tests and determine a best setup for each time series. Hence you can come up with 111 different network architectures, fuzzy membership functions, mix of ensemble members etc. for your submission. However, the process should always lead to selecting the same final model structure as a rigorous process.
These will be evaluated against established statistical forecasting methods
Statistical benchmarks will be calculated using the software ForecastPro, one of the leading expert system software packages for automatic forecasting. ForecastPro comparisons are provided by courtesy of Eric Stellwagen. Thank you Eric! We hope to also evaluate a number of additional packages: Autobox (pending), NeuralWorks (pending), Alyuda Forecatser (peding), NeuroDimensions (pending).
We assume no particular decision problem of the underlying forecasting competition and hence assume symmetric cost of errors. To account for a different number of observations in the individual data sub-samples of training and test set, and the different scale between individual series we propose to use a mean percentage error metric, which is also established best-practice in industry and in previous competitions. All submissions will be evaluated using the mean Symmteric Mean Absolute Percent Error (SMAPE) across al time series. The SMAPE calculates the symmetric absolute error in percent between the actuals X and the forecast F across all observations t of the test set of size n for each time series s with
The SMAPE of each series will then be averaged over all time series in the dataset for a mean SMAPE. To determine a winner, all submissions will be ranked by mean SMAPE across all series. However, biases may be introduced in selecting a “best” method based upon a single metric, particularly in the lack of a true objective or loss function. Therefore, while our primary means of ranking forecasting approaches is mean SMAPE, alternative metrics will be used so as to guarantee the integrity of the presented results. All submitted forecasts will also be evaluated on a number of additional statistical error measures in order to analyse sensitivity to different error metrics. Additional Metrics for reporting purposes include:
Publication & Non-Disclosure of Results
We respect the decision of individuals to withhold their name should they feel unsatisfied with their results. Therefore we will ask each & every contestant's permission to publish his name and software package used AFTER he learns his relative rank on the datasets. (However, we reserve the right to indicate the type of method and methodology used, i.e. MLP, SVR etc without the name).
© 2006 BI3S-lab - Hamburg, Germany - All rights reserved - Questions, Comments and Enquiries via eMail - [Impressum & Disclaimer]
The Knowledge Portal on Forecasting with Neural Networks @ www.neural-forecasting.com - last update: 18.10.2006