I have an already existing ARIMA (p,d,q) model fit to a time series data (for ex,
data[0:100]) using python. I would like to do forecasts (
forecast[100:120]) with this model. However, given that I also have the future true data (eg:
data[100:120]), how do i ensure that the multi-step forecast takes into account the future true data that I have instead of using the data it forecasted?
In essence, when forecasting i would like
forecast to be computed using
data instead of
I would like to avoid refitting the entire ARIMA model at every time step with the updated "history".
I fit the ARIMAX model as follows:
train, test = data[:100], data[100:] ext_train, ext_test = external[:100], external[100:] model = ARIMA(train, order=(p, d, q), exog=ext_train) model_fit = model.fit(displ=False)
Now, the following code allows me to predict values for the entire dataset, including the test
forecast = model_fit.predict(end=len(data)-1, exog=external, dynamic=False)
However in this case after 100 steps, the ARIMAX predicted values quickly converge to the long-run mean (as expected, since after 100 time steps it is using the forecasted values only). I would like to know if there is a way to provide the "future" true values to give better online predictions. Something along the lines of:
forecast = model_fit.predict_fn(end = len(data)-1, exog=external, true=data, dynamic=False)
I know I can always keep refitting the ARIMAX model by doing
historical = train historical_ext = ext_train predictions =  for t in range(len(test)): model = ARIMA(historical, order=(p,d,q), exog=historical_ext) model_fit = model.fit(disp=False) output = model_fit.forecast(exog=ext_test[t]) predictions.append(output) observed = test[t] historical.append(observed) historical_ext.append(ext_test[t])
but this leads to me training the ARIMAX model again and again which doesn't make a lot of sense to me. It leads to using a lot of computational resources and is quite impractical. It further makes it difficult to evaluate the ARIMAX model cause the fitted params keep on changing every iteration.
Is there something incorrect about my understanding/use of the ARIMAX model?
You are right, if you want to do online forecasting using new data you will need to estimate the parameters over and over again which is computationally inefficient. One thing to note is that for the ARIMA model mainly the estimation of the parameters of the MA part of the model is computationally heavy, since these parameters are estimated using numerical optimization, not using ordinary least squares. Since after calculating the parameters once for the initial model you know what is expected for future models, since one observation won't change them much, you might be able to initialize the search for the parameters to improve computational efficiency.
Also, there may be a method to do the estimation more efficiently, since you have your old data and parameters for the model, the only thing you do is add one more datapoint. This means that you only need to calculate the theta and phi parameters for the combination of the new datapoint with all the others, while not computing the known combinations again, which would save quite some time. I very much like this book: Heij, Christiaan, et al. Econometric methods with applications in business and economics. Oxford University Press, 2004.
And this lecture might give you some idea of how this might be feasible: lecture on ARIMA parameter estimation
You would have to implement this yourself, I'm afraid. As far as I can tell, there is nothing readily available to do this.
Hope this gives you some new ideas!
- interaction of time.sleep with threading.Lock
- Python os.path.join() on a list
- How to get numbers after decimal point?
- how to subquery in queryset in django?
- Unable to run django custom command
- How to create a Function in python for detecting missing value?
- Jupyter notebook Memory limit
- Python code to convert from objectSid to SID representation
- Whoosh - accessing search_page result items throws ReaderClosed exception
- Fix Error converting data type varchar to numeric. (8114) (SQLExecDirectW)')
- Add –cleanup option to python3 -m test to remove `test_pyt
- thon 2.7 at https://docs.python.org. If you want to understa
- GI scripts * mod_python * FastCGI and SCGI
- utils SIG at https://www.python.org/community/sigs/current/d
- sourceforge.net/projects/python/. The most important of the
- w recognises the word “python” as an instruction to star
- !/usr/local/bin/python as the very first line
- d. > > —https://mail.python.org/pipermail/python-3000/20
- here in the world is www.python.org located? * Why is
- ww.activestate.com/activepython). Boa Constructor is an IDE