Evaluating time series forecasts

Photo by Jonathan Borba on Unsplash

See other stories in this series

So far I have looked into understanding time series and then forecasting with Prophet, SARIMA, Greykite and Auto-ML with AWS and GCP. This is now the final part to understand how to evaluate and explain accuracy of forecasts.

As with all predictions, explaining and establishing trust in forecasts is important. Understanding this in detail is beyond my scope but similar to an adjusted R-squared value for regression or AUC for a classification problem, what can we use here?

AWS provides a good page of what metrics they run and also Facebook Prophet lists the kinds of metrics you can get easily. Let us see what Amazon Forecast showed for its winner.

AWS forecast metrics

An intuitive way to think about this is to see how closely predicted values (ŷ) match with actual observed values (y) from a test set. This is indeed what I have eyeballed and commented on in all the previous posts. This difference between y and ŷ is the “residual” and RMSE (root mean square error) is by far the easiest to explain and apply (see more details here). RMSE is the square root of the average of squared residuals = √( ∑ (ŷ - y)² )/n. Since we squared and then did a square root, the scale of the number is same as your values and the larger the number the less accurate the predicted values. Ofcourse as with averages, outliers will get amplified. So, the 84.79 indicates that on average the forecast values are off by that much from observed values. If you see my sample data, values vary from 200 to 1400 so an average error of 85 is ok?

That is it for now. As I get opportunities to apply this to real problems in production I can share more meaningful learnings!

See other stories in this series

If these topics interest you then reach out to me, and I will appreciate any feedback. If you would like to work on such problems you will generally find open roles as well! Please refer to LinkedIn.





Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Simple Matplotlib Graphs

Once data at choice assume throw.

Anomaly Detection Using PyCaret!!!

Who should Manchester United scout?

5 things to consider when doing a city data inventory

How to Navigate Analytics Job Search During COVID-19

The Prerequisites for Data Disruption

Data theory: file format (part 2)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Rahul Agarwal

Rahul Agarwal


More from Medium

Cracking the Secret Behind Time Series Forecasting

To predict the sales of weather sensitive products of Walmart during major weather events using…

Time series prediction with Machine Learning

An easy start into Time-series Forecasting: A practical example using Darts library