Advanced Forecasting with Python: Mastering Modern Forecasting Techniques with Machine Learning and Cloud Tools. 2 Ed

Advanced Forecasting with Python....2 Introduction....5 Table of Contents....7 About the Author....19 About the Technical Reviewer....20 Part I: Machine Learning for Forecasting....21 1. Models for Forecasting....22 Reading Guide for This Book....23 Machine Learning Landscape....23 Univariate Time Series Models....23 A Quick Example of the Time Series Approach....24 Supervised Machine Learning Models....29 A Quick Example of the Supervised Machine Learning Approach....29 Correlation Coefficient....36 Other Distinctions in Machine Learning Models....37 Supervised vs. Unsupervised Models....37 Classification vs. Regression Models....38 Univariate vs. Multivariate Models....38 Key Takeaways....39 2. Model Evaluation for Forecasting....40 Evaluation with an Example Forecast....40 Error Metrics....43 Error Metric 1: MSE....43 Error Metric 2: RMSE....44 Error Metric 3: R2....44 Error Metric 4: MAE....45 Error Metric 5: MAPE....46 Model Evaluation Strategies....46 Overfit and the Out of Sample Error....47 Strategy 1: Train-Test Split....47 Strategy 2: Train-Validation-Test Split....49 Strategy 3: Cross-Validation for Forecasting....51 K-Fold Cross-Validation....51 Time Series Cross-Validation....53 Rolling Time Series Cross-Validation....55 Backtesting....57 Which Strategy to Use for Safe Forecasts?....58 Final Considerations on Model Evaluation....58 Key Takeaways....59 3. Model Management and Benchmarking Using MLflow....61 Introduction to MLflow....61 Local vs. Hosted MLflow....62 Setting Up MLflow Locally....62 Experiment Tracking Using MLflow....63 MLflow Data....64 Artifacts....65 Metrics....65 Params....66 Tags....66 Inspecting the Model Logs Through Python....66 The MLflow UI....67 MLflow Throughout This Book....68 Key Takeaways....69 Part II: Univariate Time Series Models....70 4. The AR Model....71 Autocorrelation: The Past Influences the Present....72 Compute Autocorrelation in Earthquake Counts....72 Positive and Negative Autocorrelation....77 Stationarity and the ADF Test....78 Differencing a Time Series....79 Lags in Autocorrelation....81 Partial Autocorrelation....84 How Many Lags to Include?....86 AR Model Definition....86 Estimating the AR Using Yule–Walker Equations....87 The Yule–Walker Method....87 Train, Test, Evaluation, and Tuning....92 Saving the Final Model Using MLflow....97 Key Takeaways....99 5. The MA Model....100 The Model Definition....101 Fitting the MA Model....102 Stationarity....103 Choosing Between an AR and an MA Model....103 Application of the MA Model....105 Multistep Forecasting with Model Retraining....113 Grid Search to Find the Best MA Order....116 Saving This Model in MLflow....118 Key Takeaways....119 6. The ARMA Model....121 The Idea Behind the ARMA Model....121 The Mathematical Definition of the ARMA Model....122 An Example: Predicting Births Using ARMA....122 Fitting an ARMA(1,1) Model....128 More Model Evaluation KPIs....129 Automated Hyperparameter Tuning....133 Grid Search: Tuning for Predictive Performance....134 Saving This Model in MLflow....138 Key Takeaways....140 7. The ARIMA Model....141 ARIMA Model Definition....141 Model Definition....142 ARIMA on the CO2 Example....142 Key Takeaways....149 8. The SARIMA Model....151 Univariate Time Series Model Breakdown....151 The SARIMA Model Definition....152 Example: SARIMA on Walmart Sales....153 Key Takeaways....158 Part III: Multivariate Time Series Models....159 9. The SARIMAX Model....160 Time Series Building Blocks....160 Model Definition....161 Supervised Models vs. SARIMAX....161 Example of SARIMAX on the Walmart Dataset....162 Key Takeaways....166 10. The VAR Model....168 The Model Definition....168 Order: Only One Hyperparameter....169 Stationarity....169 Estimation of the VAR Coefficients....169 One Multivariate Model vs. Multiple Univariate Models....170 An Example: VAR for Forecasting Walmart Sales....170 Key Takeaways....174 11. The VARMAX Model....175 Model Definition....176 Multiple Time Series with Exogenous Variables....176 Key Takeaways....179 Part IV: Supervised Models....180 12. The Linear Regression....181 Linear Regression....182 Model Definition....182 Example: Linear Model to Forecast CO2 Levels....184 Key Takeaways....190 13. The Decision Tree Model....192 Mathematics....193 Splitting....193 Pruning and Reducing Complexity....194 Example....194 Key Takeaways....202 14. The kNN Model....203 Intuitive Explanation....203 Mathematical Definition of Nearest Neighbors....204 Combining k Neighbors into One Forecast....205 Deciding on the Number of Neighbors k....206 Predicting Traffic Using kNN....207 Grid Search on kNN....210 Random Search: An Alternative to Grid Search....211 Key Takeaways....212 15. The Random Forest....213 Intuitive Idea Behind Random Forests....213 Random Forest Concept 1: Ensemble Learning....214 Bagging Concept 1: Bootstrap....214 Bagging Concept 2: Aggregation....215 Random Forest Concept 2: Variable Subsets....216 Predicting Births Using a Random Forest....216 Grid Search on the Two Main Hyperparameters of the Random Forest....218 Random Search CV Using Distributions....220 Distribution for max_features....220 Distribution for n_estimators....222 Fitting the RandomizedSearchCV....223 Interpretation of Random Forests: Feature Importance....224 Key Takeaways....227 16. Gradient Boosting with XGBoost, LightGBM, and CatBoost....228 Boosting: A Different Way of Ensemble Learning....228 Gradient Boosting....229 The Difference Between XGBoost and LightGBM....230 Adding CatBoost to the Mix....232 Forecasting Traffic Volume with XGBoost....232 Forecasting Traffic Volume with LightGBM....234 Forecasting Traffic Volume with CatBoost....234 Inspecting the Differences in Predictions....235 Hyperparameter Tuning Using Bayesian Optimization....236 The Theory of Bayesian Optimization....236 Bayesian Optimization Using scikit-optimize....237 Conclusion....243 Key Takeaways....243 17. Bayesian Models with pyBATS....245 Intuitive Idea Behind Bayesian Models....245 Bayesian vs. Frequentist Statistics....246 pyBATS As an Implementation of Bayesian Forecasting....247 Forecasting Sales Using pyBATS....248 Sales Forecast: Exploratory Data Analysis....248 Sales Forecast: Univariate Poisson DGLM....253 Sales Forecast: Normal DGLM with X Variable....256 Key Takeaways....258 Part V: Neural Networks....260 18. Neural Networks....261 Fully Connected Neural Networks....262 Activation Functions....262 The Weights: Backpropagation....263 Optimizers....264 Learning Rate of the Optimizer....264 Hyperparameters at Play in Developing a NN....264 Introducing the Example Data....265 Specific Data Prep Needs for NN....267 Scaling and Standardization....267 Principal Component Analysis (PCA)....267 The Neural Network Using Keras....270 Conclusion....276 Key Takeaways....277 19. RNNs Using SimpleRNN and GRU....278 What Are RNNs: Architecture....278 Inside the SimpleRNN Unit....279 The Example....280 Predicting a Sequence Rather Than a Value....280 Univariate Model Rather Than Multivariable....280 Preparing the Data....281 A Simple SimpleRNN....283 SimpleRNN with Hidden Layers....285 Simple GRU....287 GRU with Hidden Layers....290 Key Takeaways....292 20. LSTM RNNs....293 What Is LSTM....293 The LSTM Cell....293 Example....294 LSTM with 1 Layer of 8....296 LSTM with 3 Layers of 64....299 Conclusion....301 Key Takeaways....301 Part VI: Black Box and Cloud-Based Models....303 21. The N-BEATS Model with Darts....304 Intuition and Mathematics of N-BEATS....304 Interpretability....304 Stacking....305 Automatic Feature Engineering and Decomposition....305 The Darts Package and Its Implementation of N-BEATS....305 Forecasting Sales Using N-BEATS in Darts....306 Sales Forecast: Preparing the Data....306 Sales Forecast: Create Default N-BEATS Model....310 Sales Forecast: Create Multivariate N-BEATS Model....312 Sales Forecast: Tuning the Multivariate N-BEATS Model....315 Benchmark Results....319 Key Takeaways....319 22. The Transformer Model with Darts....321 Intuition and Mathematics of Transformers....321 Attention Is All You Need....322 Transformers Move Away from Recurrent Units....322 Positional Encoding....322 Self-Attention....323 The Darts Package and Its Implementation of Transformers....323 Forecasting Sales Using Transformer in Darts....323 Sales Forecast: Preparing the Data....324 Sales Forecast: Create a Default N-BEATS Model....326 Sales Forecast: Create the Multivariate Transformer Model....329 Sales Forecast: Tuning the Multivariate Transformer Model....330 Benchmark Results....336 Key Takeaways....337 23. The NeuralProphet Model....339 The NeuralProphet Model....340 Predicting Heat Waves with NeuralProphet....340 Wikipedia Pageview Tool....340 Preparing the Data in Python....341 Time Series Decomposition Using NeuralProphet....342 Advanced Decomposition Setting....345 Train-Test Split....348 Default Model....349 Tuned Model....352 MLflow....356 Key Takeaways....358 24. The DeepAR Model and AWS SageMaker AI....360 About DeepAR....360 Forecasting Heat Wave Page Views Using gluonts DeepAR Locally....360 Tuning the DeepAR Model Using Ray HyperOptSearch....362 What Is Ray HyperOptSearch?....362 DeepAR vs. AWS SageMaker AI....364 Forecasting Heat Wave Page Views Using DeepAR Inside an AWS SageMaker AI Training Job....365 Step 1: Prepare the Data File....365 Step 2: Upload the Data to S3....365 Step 3: SageMaker AI....365 Key Takeaways....371 25. Uber’s Orbit Model....372 About Orbit....372 Forecasting Heat Wave Page Views Using Orbit’s Local-Global Trend Model....372 Tuning Orbit’s GLT....376 Forecasting Heat Wave Page Views Using Orbit’s Damped Local Trend Model....379 Tuning Orbit’s DLT....382 Key Takeaways....385 26. AutoML with Microsoft Azure....387 Cloud Computing....387 From Cloud-Based Maths to AutoML....388 Microsoft Azure Cloud....388 Forecasting Heat Wave Page Views with Microsoft Azure Cloud AutoML....389 Step 1: Create an Azure Machine Learning Studio....389 Step 2: Create an Automated ML Run....390 Optional: Download MLflow Artifact of the Model....401 Key Takeaways....402 27. AutoML with Vertex AI on Google Cloud Platform....403 GCP and BigQuery....403 Vertex AI....404 Forecasting Heat Wave Page Views with Google Cloud Platform’s Vertex AI AutoML....404 Step 1: Data Preparation for GCP....404 Step 2: Create a Bucket....406 Step 3: Uploading the Data....408 Optional Next Steps....420 Key Takeaways....422 28. Nixtla Suite and TimeGPT....423 The NixtlaVerse....423 Simple Use Case with the NixtlaVerse....424 The GPT in TimeGPT....429 Nixtla’s TimeGPT API....430 Key Takeaways....431 29. Model Selection....432 Model Selection Based on Metrics....432 Model Structure and Inputs....433 One-Step Forecasts vs. Multistep Forecasts....434 Model Complexity vs. Gain....434 Model Complexity vs. Interpretability....435 Model Stability and Variation....436 Open Source vs. Cloud Services Black Box....436 Conclusion....437 Key Takeaways....437 Index....439