Introduction to Deep Learning Using R: A Step-by-Step Guide to Learning and Implementing Deep Learning Models Using R

 Contents at a Glance....4 Contents....5 About the Author....13 About the Technical Reviewer....14 Acknowledgments....15 Introduction....16 Chapter 1: Introduction to Deep Learning....17 Deep Learning Models....19 Single Layer Perceptron Model (SLP)....19 Multilayer Perceptron Model (MLP)....20 Convolutional Neural Networks (CNNs)....21 Recurrent Neural Networks (RNNs)....21 Restricted Boltzmann Machines (RBMs)....22 Deep Belief Networks (DBNs)....22 Other Topics Discussed....23 Experimental Design....23 Feature Selection....23 Applied Machine Learning and Deep Learning....23 History of Deep Learning....23 Summary....25 Chapter 2: Mathematical Review....26 Statistical Concepts....26 Probability....26 And vs. Or....27 Bayes’ Theorem....29 Random Variables....29 Variance....30 Standard Deviation....31 Coefficient of Determination (R Squared)....32 Mean Squared Error (MSE)....32 Linear Algebra....32 Scalars and Vectors....32 Properties of Vectors....33 Addition....33 Subtraction....33 Element Wise Multiplication....34 Axioms....34 Associative Property....34 Commutative Property....34 Identity Element of Addition....34 Inverse Elements of Addition....34 Identity Element of Scalar Multiplication....35 Distributivity of Scalar Multiplication with Respect to Vector Addition....35 Distributivity of Scalar Multiplication with Respect to Field Addition....35 Subspaces....35 Matrices....35 Matrix Properties....36 Addition....36 Scalar Multiplication....36 Transposition....36 Types of Matrices....36 Matrix Multiplication....37 Scalar Multiplication....38 Matrix by Matrix Multiplication....38 Row and Column Vector Multiplication....39 Column Vector and Square Matrix....39 Square Matrices....40 Row Vector, Square Matrix, and Column Vector....40 Rectangular Matrices....41 Matrix Multiplication Properties (Two Matrices)....41 Not Commutative....41 Distributive over Matrix Addition....42 Scalar Multiplication Is Compatible with Matrix Multiplication....42 Transpose....43 Trace....43 Norms....44 Euclidean Norm....44 L2 Norm....44 L1 Norm....44 P-norm....45 Matrix Norms....46 Inner Products....47 Norms on Inner Product Spaces....47 Proofs....48 Orthogonality....49 Outer Product....49 Eigenvalues and Eigenvectors....49 Linear Transformations....51 Quadratic Forms....52 Sylvester’s Criterion....52 Orthogonal Projections....53 Range of a Matrix....53 Nullspace of a Matrix....54 Hyperplanes....54 Sequences....55 Properties of Sequences....55 Limits....56 Derivatives and Differentiability....57 Partial Derivatives and Gradients....57 Hessian Matrix....58 Summary....58 Chapter 3: A Review of Optimization and Machine Learning....59 Unconstrained Optimization....59 Local Minimizers....61 Global Minimizers....61 Conditions for Local Minimizers....62 Neighborhoods....63 Interior and Boundary Points....64 Machine Learning Methods: Supervised Learning....64 History of Machine Learning....64 What Is an Algorithm?....65 Regression Models....65 Linear Regression....65 Ordinary Least Squares (OLS)....65 Gradient Descent Algorithm....67 Multiple Linear Regression via Gradient Descent....68 Learning Rates....68 Choosing An Appropriate Learning Rate....69 Newton’s Method....74 Levenberg-Marquardt Heuristic....75 What Is Multicollinearity?....76 Testing for Multicollinearity....76 Variance Inflation Factor (VIF)....76 Ridge Regression....76 Least Absolute Shrinkage and Selection Operator (LASSO)....77 Comparing Ridge Regression and LASSO....78 Evaluating Regression Models....78 Coefficient of Determination (R 2)....79 Mean Squared Error (MSE)....79 Standard Error (SE)....79 Classification....79 Logistic Regression....80 Receiver Operating Characteristic (ROC) Curve....81 Confusion Matrix....82 Limitations to Logistic Regression....83 Support Vector Machine (SVM)....84 Types of Kernels....86 Sub-Gradient Method Applied to SVMs....86 Extensions of Support Vector Machines....87 Limitations Associated with SVMs....87 Machine Learning Methods: Unsupervised Learning....88 K-Means Clustering....88 Assignment Step....88 Update Step....89 Limitations of K-Means Clustering....89 Expectation Maximization (EM) Algorithm....90 Expectation Step....91 Maximization Step....91 Limitations to Expectation Maximization Algorithm....91 Decision Tree Learning....92 Classification Trees....93 Regression Trees....94 Limitations of Decision Trees....95 Ensemble Methods and Other Heuristics....96 Gradient Boosting....96 Gradient Boosting Algorithm....96 Random Forest....97 Limitations to Random Forests....97 Bayesian Learning....97 Naïve Bayes Classifier....98 Limitations Associated with Bayesian Classifiers....98 Final Comments on Tuning Machine Learning Algorithms....99 50/25/25 Cross-Validation....99 Tune One Parameter at a Time....99 Using Search Algorithms to Tune Machine Learning Parameters....100 Reinforcement Learning....100 Summary....101 Chapter 4: Single and Multilayer Perceptron Models....102 Single Layer Perceptron (SLP) Model....102 Training the Perceptron Model....103 Widrow-Hoff (WH) Algorithm....103 Limitations of Single Perceptron Models....104 Summary Statistics....107 Multi-Layer Perceptron (MLP) Model....107 Converging upon a Global Optimum....108 Back-propagation Algorithm for MLP Models:....108 Limitations and Considerations for MLP Models....110 How Many Hidden Layers to Use and How Many Neurons Are in It....112 Summary....113 Chapter 5: Convolutional Neural Networks (CNNs)....114 Structure and Properties of CNNs....114 Components of CNN Architectures....116 Convolutional Layer....116 Pooling Layer....118 Rectified Linear Units (ReLU) Layer....119 Fully Connected (FC) Layer....119 Loss Layer....120 Tuning Parameters....121 Notable CNN Architectures....121 Regularization....124 Summary....125 Chapter 6: Recurrent Neural Networks (RNNs)....126 Fully Recurrent Networks....126 Training RNNs with Back-Propagation Through Time (BPPT)....127 Elman Neural Networks....128 Neural History Compressor....129 Long Short-Term Memory (LSTM)....129 Traditional LSTM....131 Training LSTMs....131 Structural Damping Within RNNs....132 Tuning Parameter Update Algorithm....132 Practical Example of RNN: Pattern Detection....133 Summary....137 Chapter 7: Autoencoders, Restricted Boltzmann Machines, and Deep Belief Networks....138 Autoencoders....138 Linear Autoencoders vs. Principal Components Analysis (PCA)....139 Restricted Boltzmann Machines....140 Contrastive Divergence (CD) Learning....142 Momentum Within RBMs....145 Weight Decay....146 Sparsity....146 No. and Type Hidden Units....146 Deep Belief Networks (DBNs)....147 Fast Learning Algorithm (Hinton and Osindero 2006)....148 Algorithm Steps....149 Summary....149 Chapter 8: Experimental Design and Heuristics....150 Analysis of Variance (ANOVA)....150 One-Way ANOVA....150 Two-Way (Multiple-Way) ANOVA....150 Mixed-Design ANOVA....151 Multivariate ANOVA (MANOVA)....151 F-Statistic and F-Distribution....151 Fisher’s Principles....157 Plackett-Burman Designs....159 Space Filling....160 Full Factorial....160 Halton, Faure, and Sobol Sequences....161 A/B Testing....161 Simple Two-Sample A/B Test....162 Beta-Binomial Hierarchical Model for A/B Testing....162 Feature/ Variable Selection Techniques....164 Backwards and Forward Selection....164 Principal Component Analysis (PCA)....165 Factor Analysis....167 Limitations of Factor Analysis....168 Handling Categorical Data....168 Encoding Factor Levels....169 Categorical Label Problems: Too Numerous Levels....169 Canonical Correlation Analysis (CCA)....169 Wrappers, Filters, and Embedded (WFE) Algorithms....170 Relief Algorithm....170 Algorithm....170 Other Local Search Methods....170 Hill Climbing Search Methods....171 Genetic Algorithms (GAs)....171 Algorithm....171 Simulated Annealing (SA)....172 Algorithm....172 Ant Colony Optimization (ACO)....172 Algorithm....173 Variable Neighborhood Search (VNS)....173 Algorithm....174 Reactive Search Optimization (RSO)....174 Reactive Prohibitions....175 Fixed Tabu Search....176 Reactive Tabu Search (RTS)....177 WalkSAT Algorithm....178 K-Nearest Neighbors (KNN)....178 Summary....179 Chapter 9: Hardware and Software Suggestions....180 Processing Data with Standard Hardware....180 Solid State Drives and Hard Drive Disks (HDD)....180 Graphics Processing Unit (GPU)....181 Central Processing Unit (CPU)....182 Random Access Memory (RAM)....182 Motherboard....182 Power Supply Unit (PSU)....183 Optimizing Machine Learning Software....183 Summary....183 Chapter 10: Machine Learning Example Problems....184 Problem 1: Asset Price Prediction....184 Problem Type: Supervised Learning—Regression....185 Description of the Experiment....186 Feature Selection....188 Model Evaluation....189 Ridge Regression....189 Support Vector Regression (SVR)....191 Problem 2: Speed Dating....193 Problem Type: Classification....194 Preprocessing: Data Cleaning and Imputation....195 Feature Selection....198 Model Training and Evaluation....199 Method 1: Logistic Regression....199 Method 3: K-Nearest Neighbors (KNN)....202 Method 2: Bayesian Classifier....204 Summary....207 Chapter 11: Deep Learning and Other Example Problems....208 Autoencoders....208 Convolutional Neural Networks....215 Preprocessing....217 Model Building and Training....219 Collaborative Filtering....227 Summary....231 Chapter 12: Closing Statements....232 Index....234