Practical Deep Learning: A Python-Based Introduction. 2 Ed

Cover Page....2 Title Page....3 Copyright Page....4 Dedication Page....6 About the Author....7 About the Technical Reviewer....8 BRIEF CONTENTS....9 CONTENTS IN DETAIL....11 FOREWORD TO THE FIRST EDITION....24 ACKNOWLEDGMENTS....27 INTRODUCTION....28 Who This Book Is For....30 What You Can Expect to Learn....30 About This Book....31 Terminology....32 What’s New in the Second Edition....34 Synopsis....35 0 ENVIRONMENT AND MATHEMATICAL PRELIMINARIES....39 The Operating Environment....39 NumPy....40 scikit-learn....40 TensorFlow with Keras....40 Installing the Toolkits....41 Basic Linear Algebra....43 Vectors....43 Matrices....44 Vector and Matrix Multiplication....44 Statistics and Probability....46 Descriptive Statistics....46 Probability Distributions....47 Statistical Tests....48 Graphics Processing Units....49 Summary....50 PART I DATA IS EVERYTHING....51 1 IT’S ALL ABOUT THE DATA....52 Classes and Labels....52 Features and Feature Vectors....53 Types of Features....54 Feature Selection and the Curse of Dimensionality....57 Qualities of a Good Dataset....60 Interpolation and Extrapolation....61 The Parent Distribution....63 Prior Class Probabilities....64 Confusers....66 Dataset Size....66 Data Preparation....67 Scaling Features....68 Dealing with Missing Features....74 Training, Validation, and Test Data....75 The Three Subsets....76 Dataset Partitioning....77 k-Fold Cross-Validation....84 Data Analysis....86 How to Find Problems in the Data....87 Cautionary Tales....92 Summary....93 2 BUILDING THE DATASETS....94 Irises....94 Breast Cancer....97 MNIST Digits....100 CIFAR-10....104 Data Augmentation....107 Reasoning....108 Methods....110 The Iris Dataset....111 The CIFAR-10 Dataset....118 Summary....123 PART II CLASSICAL MACHINE LEARNING....125 3 INTRODUCTION TO MACHINE LEARNING....126 Nearest Centroid....127 k-Nearest Neighbors....132 Naive Bayes....134 Tree Classifiers....139 Recursion Primer....143 Decision Trees....144 Random Forests....146 Support Vector Machines....148 Margins....148 Support Vectors....151 Optimization....152 Kernels....152 Summary....154 4 EXPERIMENTS WITH CLASSICAL MODELS....155 Experiments with the Iris Dataset....155 Testing the Classical Models....156 Implementing a Nearest-Centroid Classifier....160 Experiments with the Breast Cancer Dataset....162 Running Two Initial Tests....163 Testing the Effect of Random Splits....166 Adding k-Fold Validation....168 Searching for Hyperparameters....175 Experiments with the MNIST Dataset....182 Testing the Classical Models....182 Analyzing Runtimes....190 Experimenting with PCA Components....193 Scrambling the Dataset....196 Classical Model Summary....198 Nearest Centroid....198 k-Nearest Neighbors....199 Naive Bayes....199 Decision Trees....200 Random Forests....200 Support Vector Machines....201 When to Use Classical Models....202 Handling Small Datasets....202 Dealing with Reduced Computational Requirements....202 Having Explainable Models....203 Working with Vector Inputs....203 Summary....204 PART III NEURAL NETWORKS....205 5 INTRODUCTION TO NEURAL NETWORKS....206 Anatomy of a Neural Network....207 The Neuron....208 Activation Functions....210 The Architecture of a Network....215 Output Layers....217 Weight and Bias Representation....220 A Simple Neural Network Implementation....221 Building the Dataset....222 Implementing the Neural Network....224 Training and Testing the Neural Network....227 Summary....230 6 TRAINING A NEURAL NETWORK....231 A High-Level Overview....231 Gradient Descent....233 Finding Minimums....235 Updating the Weights....237 Stochastic Gradient Descent....238 Batches and Minibatches....238 Convex vs. Nonconvex Functions....240 When to Stop Training....242 The Learning Rate....244 Momentum....245 Backpropagation....245 A Simple Example....247 An Abstract Example....251 Loss Functions....256 Absolute and Mean Squared Error Loss....256 Cross-Entropy Loss....257 Weight Initialization....259 Managing Model Complexity and Generalization....261 Overfitting....262 Regularization....265 L2 Regularization....266 Dropout....268 Summary....270 7 EXPERIMENTS WITH NEURAL NETWORKS....273 The Dataset....273 The MLPClassifier Class....274 Architecture and Activation Functions....275 The Code....275 The Results....280 Batch Size....284 Base Learning Rate....289 Training-Set Size....292 L2 Regularization....293 Momentum....296 Weight Initialization....298 Feature Ordering....303 Summary....305 8 EVALUATING MODELS....306 Definitions and Assumptions....306 Why Accuracy Is Not Enough....307 The 2×2 Confusion Matrix....310 Metrics Derived from the 2×2 Confusion Matrix....314 Deriving Metrics from the 2×2 Table....314 Using Metrics to Interpret Models....318 More-Advanced Metrics....320 Informedness and Markedness....321 F1 Score....322 Cohen’s Kappa....322 Matthews Correlation Coefficient....323 Metric Implementation....324 The Receiver Operating Characteristics Curve....326 Gathering the Models....326 Plotting the Metrics....328 Exploring the ROC Curve....330 Comparing Models with ROC Analysis....333 Generating an ROC Curve....336 Handling Multiple Classes....339 Extending the Confusion Matrix....340 Calculating Weighted Accuracy....344 Considering the Multiclass Matthews Correlation Coefficient....347 Summary....348 PART IV CONVOLUTIONAL NEURAL NETWORKS....350 9 INTRODUCTION TO CONVOLUTIONAL NEURAL NETWORKS....351 Why Convolutional Neural Networks?....352 Convolution....353 Scanning with the Kernel....353 Using Convolution for Image Processing....357 Anatomy of a Convolutional Neural Network....359 Exploring the Types of Layers....360 Passing Data Through the CNN....362 Convolutional Layers....363 How They Work....364 In Action....367 Multiple Layers....370 Initialization....372 Pooling Layers....372 Fully Connected Layers....374 Fully Convolutional Layers....376 How the CNN Operates....378 Summary....384 10 EXPERIMENTS WITH KERAS AND MNIST....386 Building CNNs in Keras....386 Loading the MNIST Data....387 Building the Model....389 Training and Evaluating the Model....392 Plotting the Error....395 Basic Experiments....398 Architecture Experiments....399 Training-Set Size, Minibatches, and Epochs....403 Optimizers....408 Fully Convolutional Networks....410 Building and Training the Model....411 Making the Test Images....414 Testing the Model....416 Scrambled MNIST Digits....427 Summary....428 11 EXPERIMENTS WITH CIFAR-10....430 A CIFAR-10 Refresher....430 The Full CIFAR-10 Dataset....432 Building the Models....432 Analyzing the Models....437 Animal or Vehicle?....439 Binary or Multiclass?....446 Using the Augmented CIFAR-10 Dataset....451 Summary....457 12 A CASE STUDY: CLASSIFYING AUDIO SAMPLES....459 Building the Dataset....459 Augmenting the Dataset....461 Preprocessing the Data....466 Classifying the Audio Features....468 With Classical Models....468 With a Traditional Neural Network....471 With a Convolutional Neural Network....472 Spectrograms....479 Classifying Spectrograms....484 Ensembles....489 Summary....495 PART V ADVANCED NETWORKS AND GENERATIVE AI....497 13 ADVANCED CNN ARCHITECTURES....498 The Keras Functional API....499 VGG....504 Standardizing with Batch Normalization....505 Applying Dropout After Convolutional Layers....506 Building the VGG8 Model....507 Testing the VGG8 Model....512 ResNet....517 Mitigating the Vanishing Gradient Problem....519 Exploring ResNet Configurations....521 Building the ResNet-18 Model....523 Testing the ResNet-18 Model....526 MobileNet....528 Implementing Depthwise Convolutions....528 Building the MobileNet Model....532 Testing the MobileNet Model....533 Building an Ensemble....534 Summary....537 14 FINE-TUNING AND TRANSFER LEARNING....539 Fine-Tuning a Pretrained Model....540 VGG16 and MobileNet with CIFAR-10....541 Experiments....546 A Study in Fine-Tuning....550 Creating New Features with Transfer Learning....555 Extracting CIFAR-10 Features with VGG16....555 Training Classical Models with CIFAR-10 Features....560 Detecting Anomalies with CIFAR-10 Features....562 Retrieving Images with CIFAR-10 Features....565 Summary....572 15 FROM CLASSIFICATION TO LOCALIZATION....574 Detection Experiments with MNIST....575 Building the Dataset....576 Building the Model....578 Running the Model....581 Running an Advanced Detection Model....590 Semantic Segmentation with U-Net....593 Implementing the U-Net Model....595 Running the Model and Interpreting Its Output....599 Multilabel Classification....605 Dataset....606 Performance....608 Summary....611 16 SELF-SUPERVISED LEARNING....613 Building the Unlabeled Dataset....614 Rotation Prediction....615 Building and Testing RotNet....616 Exploring Variations on the RotNet Theme....622 Examining RotNet Features....626 Fine-Tuning Experiments....630 Siamese Networks....633 Building and Testing the Siamese Networks....634 Examining Siamese Network Features....640 Fine-Tuning Experiments....642 For Further Exploration....644 Summary....644 17 GENERATIVE ADVERSARIAL NETWORKS....646 How GANs Work....646 Unconditional GANs....647 Building a GAN with Multilayer Perceptrons....647 Building a GAN with Convolutional Layers....650 Experimenting with Unconditional GANs....653 Conditional GANs....659 Implementing a Conditional GAN....659 Experimenting with Conditional GANs....662 Exploring the Latent Space....663 Intrinsic Dimensionality....663 Latent Space Interpolation....665 Summary....667 18 LARGE LANGUAGE MODELS....668 Understanding Large Language Models....670 Evaluating the LLM Block Diagram....671 Tokenizing and Embedding....672 Exploring the Transformer Layer....675 Predicting the Next Token....680 In-Context Learning....683 Configuring the Experiment....683 Running the Experiment....685 Testing Another LLM....689 Running LLMs Locally....690 Representing Parameters with Quantization....691 Installing Ollama....692 Installing Open Source LLMs....693 Chatting with a Simple Chatbot....694 LLM Embeddings....697 Embeddings Encode Meaning....698 Semantic Search and Retrieval-Augmented Generation....702 Sentiment Analysis....708 LLMs and Images....713 Describing Images....713 Classifying Images with an LLM?....717 Are LLMs Creative?....719 The Divergent Association Task....719 DAT Scores as a Function of Temperature....722 Creative Writing as a Function of Temperature....725 Summary....728 AFTERWORD....730 INDEX....733