Deep Learning with PyTorch Step-by-Step: A Beginner’s Guide

Deep Learning with PyTorch Step-by-Step: A Beginner’s Guide....2 Table of Contents....5 Preface....23 Acknowledgements....24 About the Author....25 Frequently Asked Questions (FAQ)....26 Why PyTorch?....26 Why This Book?....27 Who Should Read This Book?....28 What Do I Need to Know?....29 How to Read This Book....29 What’s Next?....32 Setup Guide....33 Official Repository....33 Environment....33 Google Colab....33 Binder....34 Local Installation....35 1. Anaconda....36 2. Conda (Virtual) Environments....36 3. PyTorch....38 4. TensorBoard....40 5. GraphViz and Torchviz (optional)....41 6. Git....42 7. Jupyter....44 Moving On....44 Part I: Fundamentals....46 Chapter 0: Visualizing Gradient Descent....47 Spoilers....47 Jupyter Notebook....47 Imports....48 Visualizing Gradient Descent....48 Model....49 Data Generation....50 Synthetic Data Generation....50 Train-Validation-Test Split....52 Step 0 - Random Initialization....53 Step 1 - Compute Model’s Predictions....54 Step 2 - Compute the Loss....55 Loss Surface....57 Cross-Sections....61 Step 3 - Compute the Gradients....62 Visualizing Gradients....64 Backpropagation....65 Step 4 - Update the Parameters....66 Learning Rate....68 Low Learning Rate....69 High Learning Rate....71 Very High Learning Rate....72 "Bad" Feature....73 Scaling / Standardizing / Normalizing....76 Step 5 - Rinse and Repeat!....80 The Path of Gradient Descent....81 Recap....83 Chapter 1: A Simple Regression Problem....85 Spoilers....85 Jupyter Notebook....85 Imports....86 A Simple Regression Problem....86 Data Generation....87 Synthetic Data Generation....87 Gradient Descent....88 Step 0 - Random Initialization....89 Step 1 - Compute Model’s Predictions....89 Step 2 - Compute the Loss....89 Step 3 - Compute the Gradients....90 Step 4 - Update the Parameters....91 Step 5 - Rinse and Repeat!....92 Linear Regression in Numpy....92 PyTorch....96 Tensor....96 Loading Data, Devices, and CUDA....101 Creating Parameters....106 Autograd....110 backward....110 grad....112 zero_....113 Updating Parameters....114 no_grad....117 Dynamic Computation Graph....117 Optimizer....121 step / zero_grad....122 Loss....124 Model....128 Parameters....130 state_dict....131 Device....132 Forward Pass....132 train....134 Nested Models....134 Sequential Models....137 Layers....138 Putting It All Together....140 Data Preparation....141 Model Configuration....142 Model Training....143 Recap....146 Chapter 2: Rethinking the Training Loop....148 Spoilers....148 Jupyter Notebook....148 Imports....148 Rethinking the Training Loop....149 Training Step....155 Dataset....159 TensorDataset....161 DataLoader....161 Mini-Batch Inner Loop....167 Random Split....170 Evaluation....172 Plotting Losses....176 TensorBoard....177 Running It Inside a Notebook....177 Running It Separately (Local Installation)....179 Running It Separately (Binder)....180 SummaryWriter....180 add_graph....182 add_scalars....183 Saving and Loading Models....189 Model State....189 Saving....189 Resuming Training....190 Deploying / Making Predictions....193 Setting the Model’s Mode....194 Putting It All Together....195 Recap....198 Chapter 2.1: Going Classy....200 Spoilers....200 Jupyter Notebook....200 Imports....200 Going Classy....201 The Class....201 The Constructor....202 Arguments....202 Placeholders....203 Variables....205 Functions....205 Training Methods....212 Saving and Loading Models....216 Visualization Methods....217 The Full Code....218 Classy Pipeline....219 Model Training....222 Making Predictions....224 Checkpointing....224 Resuming Training....225 Putting It All Together....227 Recap....229 Chapter 3: A Simple Classification Problem....231 Spoilers....231 Jupyter Notebook....231 Imports....231 A Simple Classification Problem....232 Data Generation....233 Data Preparation....234 Model....235 Logits....236 Probabilities....237 Odds Ratio....237 Log Odds Ratio....239 From Logits to Probabilities....240 Sigmoid....242 Logistic Regression....243 Loss....246 BCELoss....248 BCEWithLogitsLoss....250 Imbalanced Dataset....253 Model Configuration....256 Model Training....257 Decision Boundary....261 Classification Threshold....266 Confusion Matrix....268 Metrics....270 True and False Positive Rates....270 Precision and Recall....273 Accuracy....274 Trade-offs and Curves....275 Low Threshold....275 High Threshold....277 ROC and PR Curves....278 The Precision Quirk....280 Best and Worst Curves....281 Comparing Models....282 Putting It All Together....284 Recap....286 Part II: Computer Vision....289 Chapter 4: Classifying Images....290 Spoilers....290 Jupyter Notebook....290 Imports....290 Classifying Images....291 Data Generation....292 Shape (NCHW vs NHWC)....296 Torchvision....299 Datasets....299 Models....299 Transforms....299 Transforms on Images....303 Transforms on Tensor....303 Normalize Transform....304 Composing Transforms....306 Data Preparation....308 Dataset Transforms....308 SubsetRandomSampler....310 Data Augmentation Transforms....313 WeightedRandomSampler....314 Seeds and more (seeds)....318 Putting It Together....320 Pixels as Features....321 Shallow Model....323 Notation....324 Model Configuration....325 Model Training....326 Deep-ish Model....326 Model Configuration....329 Model Training....329 Show Me the Math!....331 Show Me the Code!....333 Weights as Pixels....336 Activation Functions....337 Sigmoid....337 Hyperbolic Tangent (TanH)....339 Rectified Linear Unit (ReLU)....340 Leaky ReLU....342 Parametric ReLU (PReLU)....344 Deep Model....345 Model Configuration....346 Model Training....347 Show Me the Math Again!....349 Putting It All Together....351 Recap....355 Bonus Chapter: Feature Space....356 Two-Dimensional Feature Space....356 Transformations....357 A Two-Dimensional Model....358 Decision Boundary, Activation Style!....360 More Functions, More Boundaries....363 More Layers, More Boundaries....365 More Dimensions, More Boundaries....366 Recap....368 Chapter 5: Convolutions....369 Spoilers....369 Jupyter Notebook....369 Imports....369 Convolutions....370 Filter / Kernel....370 Convolving....372 Moving Around....373 Shape....376 Convolving in PyTorch....377 Striding....381 Padding....383 A REAL Filter....387 Pooling....389 Flattening....391 Dimensions....392 Typical Architecture....392 LeNet-5....393 A Multiclass Classification Problem....396 Data Generation....396 Data Preparation....397 Loss....400 Logits....400 Softmax....400 LogSoftmax....403 Negative Log-Likelihood Loss....403 Cross-Entropy Loss....407 Classification Losses Showdown!....409 Model Configuration....409 Model Training....412 Visualizing Filters and More!....413 Visualizing Filters....416 Hooks....419 Visualizing Feature Maps....428 Visualizing Classifier Layers....431 Accuracy....432 Loader Apply....434 Putting It All Together....435 Recap....439 Chapter 6: Rock, Paper, Scissors....441 Spoilers....441 Jupyter Notebook....441 Imports....441 Rock, Paper, Scissors…​....442 Rock Paper Scissors Dataset....443 Data Preparation....444 ImageFolder....444 Standardization....445 The Real Datasets....449 Three-Channel Convolutions....450 Fancier Model....453 Dropout....456 Two-Dimensional Dropout....462 Model Configuration....463 Optimizer....463 Learning Rate....464 Model Training....464 Accuracy....465 Regularizing Effect....465 Visualizing Filters....467 Learning Rates....469 Finding LR....471 Adaptive Learning Rate....479 Moving Average (MA)....479 EWMA....480 EWMA Meets Gradients....486 Adam....487 Visualizing Adapted Gradients....488 Stochastic Gradient Descent (SGD)....495 Momentum....496 Nesterov....499 Flavors of SGD....500 Learning Rate Schedulers....503 Epoch Schedulers....504 Validation Loss Scheduler....505 Schedulers in StepByStep — Part I....507 Mini-Batch Schedulers....510 Schedulers in StepByStep — Part II....512 Scheduler Paths....514 Adaptive vs Cycling....517 Putting It All Together....517 Recap....520 Chapter 7: Transfer Learning....523 Spoilers....523 Jupyter Notebook....523 Imports....524 Transfer Learning....524 ImageNet....525 ImageNet Large Scale Visual Recognition Challenge (ILSVRC)....525 ILSVRC-2012....526 AlexNet (SuperVision Team)....526 ILSVRC-2014....526 VGG....527 Inception (GoogLeNet Team)....527 ILSVRC-2015....527 ResNet (MSRA Team)....528 Comparing Architectures....528 Transfer Learning in Practice....530 Pre-Trained Model....530 Adaptive Pooling....532 Loading Weights....533 Model Freezing....534 Top of the Model....535 Model Configuration....538 Data Preparation....538 Model Training....540 Generating a Dataset of Features....541 Top Model....544 Auxiliary Classifiers (Side-Heads)....546 1x1 Convolutions....549 Inception Modules....552 Batch Normalization....557 Running Statistics....560 Evaluation Phase....566 Momentum....567 BatchNorm2d....569 Other Normalizations....570 Small Summary....570 Residual Connections....571 Learning the Identity....571 The Power of Shortcuts....575 Residual Blocks....576 Putting It All Together....579 Fine-Tuning....580 Feature Extraction....581 Recap....583 Extra Chapter: Vanishing and Exploding Gradients....586 Spoilers....586 Jupyter Notebook....586 Imports....586 Vanishing and Exploding Gradients....587 Vanishing Gradients....587 Ball Dataset and Block Model....588 Weights, Activations, and Gradients....590 Initialization Schemes....592 Batch Normalization....595 Exploding Gradients....596 Data Generation & Preparation....596 Model Configuration & Training....597 Gradient Clipping....599 Value Clipping....600 Norm Clipping (or Gradient Scaling)....601 Model Configuration & Training....605 Clipping with Hooks....608 Recap....609 Part III: Sequences....611 Chapter 8: Sequences....612 Spoilers....612 Jupyter Notebook....612 Imports....613 Sequences....613 Data Generation....614 Recurrent Neural Networks (RNNs)....616 RNN Cell....619 RNN Layer....626 Shapes....629 Stacked RNN....632 Bidirectional RNN....636 Square Model....640 Data Generation....640 Data Preparation....641 Model Configuration....641 Model Training....643 Visualizing the Model....644 Transformed Inputs....644 Hidden States....645 The Journey of a Hidden State....647 Can We Do Better?....649 Gated Recurrent Units (GRUs)....650 GRU Cell....651 GRU Layer....659 Square Model II — The Quickening....660 Model Configuration & Training....661 Visualizing the Model....662 Hidden States....662 The Journey of a Gated Hidden State....663 Can We Do Better?....665 Long Short-Term Memory (LSTM)....665 LSTM Cell....666 LSTM Layer....674 Square Model III — The Sorcerer....675 Model Configuration & Training....676 Visualizing the Hidden States....677 Variable-Length Sequences....678 Padding....679 Packing....682 Unpacking (to padded)....686 Packing (from padded)....688 Variable-Length Dataset....689 Data Preparation....689 Collate Function....691 Square Model IV — Packed....692 Model Configuration & Training....694 1D Convolutions....695 Shapes....696 Multiple Features or Channels....697 Dilation....699 Data Preparation....701 Model Configuration & Training....701 Visualizing the Model....703 Putting It All Together....704 Fixed-Length Dataset....704 Variable-Length Dataset....705 There Can Be Only ONE …​ Model....706 Model Configuration & Training....707 Recap....708 Chapter 9 — Part I: Sequence-to-Sequence....711 Spoilers....711 Jupyter Notebook....711 Imports....711 Sequence-to-Sequence....712 Data Generation....712 Encoder-Decoder Architecture....714 Encoder....714 Decoder....716 Teacher Forcing....721 Encoder + Decoder....723 Data Preparation....726 Model Configuration & Training....728 Visualizing Predictions....729 Can We Do Better?....729 Attention....730 "Values"....733 "Keys" and "Queries"....733 Computing the Context Vector....735 Scoring Method....739 Attention Scores....741 Scaled Dot Product....742 Attention Mechanism....748 Source Mask....751 Decoder....753 Encoder + Decoder + Attention....755 Model Configuration & Training....757 Visualizing Predictions....758 Visualizing Attention....759 Multi-Headed Attention....760 Chapter 9 — Part II: Sequence-to-Sequence....765 Spoilers....765 Self-Attention....765 Encoder....766 Cross-Attention....771 Decoder....773 Subsequent Inputs and Teacher Forcing....775 Attention Scores....776 Target Mask (Training)....777 Target Mask (Evaluation/Prediction)....779 Encoder + Decoder + Self-Attention....783 Model Configuration & Training....787 Visualizing Predictions....788 Sequential No More....789 Positional Encoding (PE)....790 Encoder + Decoder + PE....803 Model Configuration & Training....805 Visualizing Predictions....806 Visualizing Attention....807 Putting It All Together....809 Data Preparation....809 Model Assembly....810 Encoder + Decoder + Positional Encoding....812 Self-Attention "Layers"....813 Attention Heads....815 Model Configuration & Training....817 Recap....818 Chapter 10: Transform and Roll Out....821 Spoilers....821 Jupyter Notebook....821 Imports....821 Transform and Roll Out....822 Narrow Attention....822 Chunking....823 Multi-Headed Attention....826 Stacking Encoders and Decoders....832 Wrapping "Sub-Layers"....833 Transformer Encoder....836 Transformer Decoder....841 Layer Normalization....846 Batch vs Layer....851 Our Seq2Seq Problem....853 Projections or Embeddings....854 The Transformer....856 Data Preparation....859 Model Configuration & Training....860 Visualizing Predictions....863 The PyTorch Transformer....863 Model Configuration & Training....869 Visualizing Predictions....870 Vision Transformer....871 Data Generation & Preparation....871 Patches....874 Rearranging....874 Embeddings....876 Special Classifier Token....878 The Model....882 Model Configuration & Training....884 Putting It All Together....886 Data Preparation....886 Model Assembly....886 1. Encoder-Decoder....888 2. Encoder....891 3. Decoder....892 4. Positional Encoding....893 5. Encoder "Layer"....894 6. Decoder "Layer"....895 7. "Sub-Layer" Wrapper....896 8. Multi-Headed Attention....898 Model Configuration & Training....900 Recap....901 Part IV: Natural Language Processing....904 Chapter 11: Down the Yellow Brick Rabbit Hole....905 Spoilers....905 Jupyter Notebook....905 Additional Setup....906 Imports....906 "Down the Yellow Brick Rabbit Hole"....908 Building a Dataset....908 Sentence Tokenization....910 HuggingFace’s Dataset....916 Loading a Dataset....917 Attributes....918 Methods....919 Word Tokenization....921 Vocabulary....925 HuggingFace’s Tokenizer....931 Before Word Embeddings....939 One-Hot Encoding (OHE)....939 Bag-of-Words (BoW)....940 Language Models....941 N-grams....943 Continuous Bag-of-Words (CBoW)....944 Word Embeddings....944 Word2Vec....944 What Is an Embedding Anyway?....949 Pre-trained Word2Vec....952 Global Vectors (GloVe)....953 Using Word Embeddings....956 Vocabulary Coverage....956 Tokenizer....959 Special Tokens' Embeddings....960 Model I — GloVE + Classifier....962 Data Preparation....962 Pre-trained PyTorch Embeddings....964 Model Configuration & Training....966 Model II — GloVe + Transformer....967 Visualizing Attention....970 Contextual Word Embeddings....973 ELMo....974 BERT....982 Document Embeddings....984 Model III — Preprocessed Embeddings....987 Data Preparation....987 Model Configuration & Training....989 BERT....990 Tokenization....993 Input Embeddings....995 Pre-training Tasks....1000 Masked Language Model (MLM)....1000 Next Sentence Prediction (NSP)....1003 Outputs....1004 Model IV — Classifying Using BERT....1009 Data Preparation....1011 Model Configuration & Training....1013 Fine-Tuning with HuggingFace....1014 Sequence Classification (or Regression)....1014 Tokenized Dataset....1017 Trainer....1019 Predictions....1024 Pipelines....1026 More Pipelines....1027 GPT-2....1029 Putting It All Together....1033 Data Preparation....1033 "Packed" Dataset....1034 Model Configuration & Training....1037 Generating Text....1039 Recap....1041 Thank You!....1043