TimeGPT vs Statistical Models for Forecasting SPY
In this post, we compare TimeGPT's performance against traditional statistical forecasting methods.
In the previous post of this TimeGPT series, we used TimeGPT to forecast the price of SPY in the next 7 days. In this post, we compare TimeGPT's performance against traditional statistical forecasting methods, namely the AutoARIMA Model and Optimized Theta Model, which are two well-known and versatile time series models.
We will be using the same dataset - last year's SPY adjusted closing prices, and apply each algorithm to forecast the next 7 days' prices. Here's a sample of the dataset:
ds | y | unique_id | |
---|---|---|---|
0 | 2023-07-24 | 447.935 | SPY |
1 | 2023-07-25 | 449.157 | SPY |
2 | 2023-07-26 | 449.227 | SPY |
3 | 2023-07-27 | 446.248 | SPY |
4 | 2023-07-28 | 450.617 | SPY |
![](https://www.signalstalk.com/content/images/2024/07/timegpt-data-2.png)
Setup Models
AutoARIMA and Optimized Theta Model are available from StatsForecast
package. First install the package:
$ pip install statsforecast
Then set up the models:
from statsforecast.models import AutoARIMA, OptimizedTheta
season_length = 7 # Daily price data
horizon = 7 # Prediction horizon
models = [AutoARIMA(season_length=season_length), OptimizedTheta(season_length=season_length)]
sf = StatsForecast(df=df, models=models, freq='C') # 'C' for custom business days
Cross Validation
For each model, we will perform a 5-window cross validation, in each window we split the data for training and predict the remaining 7-day prices.
cv_sf = sf.cross_validation(df=df, h=horizon, step_size=horizon, n_windows=5, level=[90])
cv_timegpt = nixtla_client.cross_validation(
df,
h=horizon,
n_windows=5,
freq='C',
level=[90]
)
Evaluation Metrics and Result
For each model and their predictions, we compute Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE) and Symmetric Mean Absolute Percentage Error (SMAPE) as evaluation metrics. The lower the metric, the better the prediction performance.
model | MSE | RMSE | MAE | MAPE | SMAPE | |
---|---|---|---|---|---|---|
0 | TimeGPT | 52.6241 | 6.92912 | 6.18127 | 0.0112929 | 0.00566702 |
1 | AutoARIMA | 53.5264 | 6.74953 | 5.78576 | 0.0105836 | 0.00532166 |
2 | OptimizedTheta | 46.3959 | 6.50248 | 5.5702 | 0.0101794 | 0.00510243 |
Optimized Theta Model is the clear winner across all evaluation metrics. AutoARIMA beats TimeGPT in 4 out of 5 metrics.
Summary
In this post, we compared TimeGPT, AutoARIMA and Optimized Theta on forecasting SPY's closing prices in the next 7 days. I encourage you to try different algorithms on your time series data and share your thoughts!