Essential Math for AI: Next-Level Mathematics for Efficient and Successful AI Systems

Preface xvii1. Why Learn the Mathematics of AI?....1What Is AI?....2Why Is AI So Popular Now?....3What Is AI Able to Do?....3An AI Agent’s Specific Tasks....4What Are AI’s Limitations?....6What Happens When AI Systems Fail?....8Where Is AI Headed?....8Who Are the Current Main Contributors to the AI Field?....10What Math Is Typically Involved in AI?....10Summary and Looking Ahead....112. Data, Data, Data....13Data for AI....14Real Data Versus Simulated Data....16Mathematical Models: Linear Versus Nonlinear....16An Example of Real Data....18An Example of Simulated Data....21Mathematical Models: Simulations and AI....25Where Do We Get Our Data From?....27The Vocabulary of Data Distributions, Probability, and Statistics....29Random Variables....30Probability Distributions....30Marginal Probabilities....31The Uniform and the Normal Distributions....31Conditional Probabilities and Bayes’ Theorem....31Conditional Probabilities and Joint Distributions....31Prior Distribution, Posterior Distribution, and Likelihood Function....32Mixtures of Distributions....32Sums and Products of Random Variables....32Using Graphs to Represent Joint Probability Distributions....33Expectation, Mean, Variance, and Uncertainty....33Covariance and Correlation....33Markov Process....34Normalizing, Scaling, and/or Standardizing a Random Variable or Data Set....34Common Examples....34Continuous Distributions Versus Discrete Distributions (Density VersusMass)....35The Power of the Joint Probability Density Function....37Distribution of Data: The Uniform Distribution....38Distribution of Data: The Bell-Shaped Normal (Gaussian) Distribution....40Distribution of Data: Other Important and Commonly Used Distributions....43The Various Uses of the Word “Distribution”....47A/B Testing....48Summary and Looking Ahead....483. Fitting Functions to Data....51Traditional and Very Useful Machine Learning Models....53Numerical Solutions Versus Analytical Solutions....55Regression: Predict a Numerical Value....56Training Function....58Loss Function....60Optimization....71Logistic Regression: Classify into Two Classes....85Training Function....85Loss Function....86Optimization....88Softmax Regression: Classify into Multiple Classes....88Training Function....90Loss Function....92Optimization....92Incorporating These Models into the Last Layer of a Neural Network....93Other Popular Machine Learning Techniques and Ensembles of Techniques....94Support Vector Machines....94Decision Trees....98Random Forests....107k-means Clustering....108Performance Measures for Classification Models....109Summary and Looking Ahead....1104. Optimization for Neural Networks....113The Brain Cortex and Artificial Neural Networks....113Training Function: Fully Connected, or Dense,Feed Forward Neural Networks....115A Neural Network Is a Computational Graph Representation of theTraining Function....117Linearly Combine, Add Bias, Then Activate....117Common Activation Functions....122Universal Function Approximation....125Approximation Theory for Deep Learning....131Loss Functions....131Optimization....133Mathematics and the Mysterious Success of Neural Networks....134Gradient Descent ω i + 1 = ω i − η∇L ω i....135Explaining the Role of the Learning Rate Hyperparameter η....137Convex Versus Nonconvex Landscapes....140Stochastic Gradient Descent....143Initializing the Weights ω 0 for the Optimization Process....144Regularization Techniques....145Dropout....145Early Stopping....146Batch Normalization of Each Layer....146Control the Size of the Weights by Penalizing Their Norm....147Penalizing the l2 Norm Versus Penalizing the l1 Norm....150Explaining the Role of the Regularization Hyperparameter α....151Hyperparameter Examples That Appear in Machine Learning....152Chain Rule and Backpropagation: Calculating ∇L ω i....153Backpropagation Is Not Too Different from How Our Brain Learns....154Why Is It Better to Backpropagate?....155Backpropagation in Detail....155Assessing the Significance of the Input Data Features....157Summary and Looking Ahead....1585. Convolutional Neural Networks and Computer Vision....161Convolution and Cross-Correlation....163Translation Invariance and Translation Equivariance....167Convolution in Usual Space Is a Product in Frequency Space....167Convolution from a Systems Design Perspective....168Convolution and Impulse Response for Linear and Translation InvariantSystems....169Convolution and One-Dimensional Discrete Signals....171Convolution and Two-Dimensional Discrete Signals....172Filtering Images....174Feature Maps....178Linear Algebra Notation....179The One-Dimensional Case: Multiplication by a Toeplitz Matrix....182The Two-Dimensional Case: Multiplication by a Doubly BlockCirculant Matrix....182Pooling....183A Convolutional Neural Network for Image Classification....184Summary and Looking Ahead....1866. Singular Value Decomposition: Image Processing, Natural Language Processing,and Social Media....187Matrix Factorization....188Diagonal Matrices....191Matrices as Linear Transformations Acting on Space....193Action of A on the Right Singular Vectors....194Action of A on the Standard Unit Vectors and the Unit Square Determinedby Them....195Action of A on the Unit Circle....196Breaking Down the Circle-to-Ellipse Transformation According to theSingular Value Decomposition....197Rotation and Reflection Matrices....198Action of A on a General Vector x....199Three Ways to Multiply Matrices....200The Big Picture....201The Condition Number and Computational Stability....203The Ingredients of the Singular Value Decomposition....204Singular Value Decomposition Versus the Eigenvalue Decomposition....204Computation of the Singular Value Decomposition....206Computing an Eigenvector Numerically....207The Pseudoinverse....208Applying the Singular Value Decomposition to Images....209Principal Component Analysis and Dimension Reduction....212Principal Component Analysis and Clustering....214A Social Media Application....214Latent Semantic Analysis....215Randomized Singular Value Decomposition....216Summary and Looking Ahead....2167. Natural Language and Finance AI: Vectorization and Time Series....219Natural Language AI....222Preparing Natural Language Data for Machine Processing....223Statistical Models and the log Function....226Zipf ’s Law for Term Counts....226Various Vector Representations for Natural Language Documents....227Term Frequency Vector Representation of a Document or Bag of Words....227Term Frequency-Inverse Document Frequency Vector Representation of aDocument....228Topic Vector Representation of a Document Determined byLatent Semantic Analysis....228Topic Vector Representation of a Document Determined byLatent Dirichlet Allocation....232Topic Vector Representation of a Document Determined byLatent Discriminant Analysis....233Meaning Vector Representations of Words and of Documents Determinedby Neural Network Embeddings....234Cosine Similarity....241Natural Language Processing Applications....243Sentiment Analysis....243Spam Filter....244Search and Information Retrieval....244Machine Translation....246Image Captioning....247Chatbots....247Other Applications....247Transformers and Attention Models....247The Transformer Architecture....248The Attention Mechanism....251Transformers Are Far from Perfect....255Convolutional Neural Networks for Time Series Data....255Recurrent Neural Networks for Time Series Data....257How Do Recurrent Neural Networks Work?....258Gated Recurrent Units and Long Short-Term Memory Units....260An Example of Natural Language Data....261Finance AI....261Summary and Looking Ahead....2628. Probabilistic Generative Models....263What Are Generative Models Useful For?....264The Typical Mathematics of Generative Models....265Shifting Our Brain from Deterministic Thinking to Probabilistic Thinking....268Maximum Likelihood Estimation....270Explicit and Implicit Density Models....272Explicit Density-Tractable: Fully Visible Belief Networks....273Example: Generating Images via PixelCNN and Machine Audiovia WaveNet....273Explicit Density-Tractable: Change of Variables Nonlinear IndependentComponent Analysis....276Explicit Density-Intractable: Variational Autoencoders Approximation viaVariational Methods....277Explicit Density-Intractable: Boltzman Machine Approximation viaMarkov Chain....279Implicit Density-Markov Chain: Generative Stochastic Network....279Implicit Density-Direct: Generative Adversarial Networks....280How Do Generative Adversarial Networks Work?....281Example: Machine Learning and Generative Networks for High EnergyPhysics....283Other Generative Models....285Naive Bayes Classification Model....286Gaussian Mixture Model....287The Evolution of Generative Models....288Hopfield Nets....290Boltzmann Machine....290Restricted Boltzmann Machine (Explicit Density and Intractable)....291The Original Autoencoder....292Probabilistic Language Modeling....293Summary and Looking Ahead....2959. Graph Models....297Graphs: Nodes, Edges, and Features for Each....299Example: PageRank Algorithm....302Inverting Matrices Using Graphs....307Cayley Graphs of Groups: Pure Algebra and Parallel Computing....308Message Passing Within a Graph....309The Limitless Applications of Graphs....310Brain Networks....311Spread of Disease....312Spread of Information....312Detecting and Tracking Fake News Propagation....312Web-Scale Recommendation Systems....314Fighting Cancer....314Biochemical Graphs....315Molecular Graph Generation for Drug and Protein Structure Discovery....316Citation Networks....316Social Media Networks and Social Influence Prediction....316Sociological Structures....317Bayesian Networks....317Traffic Forecasting....317Logistics and Operations Research....318Language Models....318Graph Structure of the Web....320Automatically Analyzing Computer Programs....321Data Structures in Computer Science....321Load Balancing in Distributed Networks....322Artificial Neural Networks....323Random Walks on Graphs....324Node Representation Learning....326Tasks for Graph Neural Networks....327Node Classification....327Graph Classification....328Clustering and Community Detection....329Graph Generation....329Influence Maximization....329Link Prediction....330Dynamic Graph Models....330Bayesian Networks....331A Bayesian Network Represents a Compactified ConditionalProbability Table....333Making Predictions Using a Bayesian Network....334Bayesian Networks Are Belief Networks, Not Causal Networks....334Keep This in Mind About Bayesian Networks....335Chains, Forks, and Colliders....336Given a Data Set, How Do We Set Up a Bayesian Network for theInvolved Variables?....337Table of Contents | xiGraph Diagrams for Probabilistic Causal Modeling....338A Brief History of Graph Theory....340Main Considerations in Graph Theory....341Spanning Trees and Shortest Spanning Trees....341Cut Sets and Cut Vertices....342Planarity....342Graphs as Vector Spaces....343Realizability....343Coloring and Matching....344Enumeration....344Algorithms and Computational Aspects of Graphs....344Summary and Looking Ahead....34510. Operations Research....347No Free Lunch....349Complexity Analysis and O() Notation....350Optimization: The Heart of Operations Research....353Thinking About Optimization....356Optimization: Finite Dimensions, Unconstrained....357Optimization: Finite Dimensions, Constrained Lagrange Multipliers....357Optimization: Infinite Dimensions, Calculus of Variations....360Optimization on Networks....365Traveling Salesman Problem....365Minimum Spanning Tree....366Shortest Path....367Max-Flow Min-Cut....368Max-Flow Min-Cost....369The Critical Path Method for Project Design....369The n-Queens Problem....370Linear Optimization....371The General Form and the Standard Form....372Visualizing a Linear Optimization Problem in Two Dimensions....373Convex to Linear....374The Geometry of Linear Optimization....377The Simplex Method....379Transportation and Assignment Problems....386Duality, Lagrange Relaxation, Shadow Prices, Max-Min, Min-Max,and All That....386Sensitivity....401Game Theory and Multiagents....402Queuing....404Inventory....405Machine Learning for Operations Research....405Hamilton-Jacobi-Bellman Equation....406Operations Research for AI....407Summary and Looking Ahead....40711. Probability....411Where Did Probability Appear in This Book?....412What More Do We Need to Know That Is Essential for AI?....415Causal Modeling and the Do Calculus....415An Alternative: The Do Calculus....417Paradoxes and Diagram Interpretations....420Monty Hall Problem....420Berkson’s Paradox....422Simpson’s Paradox....422Large Random Matrices....424Examples of Random Vectors and Random Matrices....424Main Considerations in Random Matrix Theory....427Random Matrix Ensembles....429Eigenvalue Density of the Sum of Two Large Random Matrices....430Essential Math for Large Random Matrices....430Stochastic Processes....432Bernoulli Process....433Poisson Process....433Random Walk....434Wiener Process or Brownian Motion....435Martingale....435Levy Process....436Branching Process....436Markov Chain....436Itô’s Lemma....437Markov Decision Processes and Reinforcement Learning....438Examples of Reinforcement Learning....438Reinforcement Learning as a Markov Decision Process....439Reinforcement Learning in the Context of Optimal Control andNonlinear Dynamics....441Python Library for Reinforcement Learning....441Theoretical and Rigorous Grounds....441Which Events Have a Probability?....442Can We Talk About a Wider Range of Random Variables?....443A Probability Triple (Sample Space, Sigma Algebra, Probability Measure)....443Where Is the Difficulty?....444Random Variable, Expectation, and Integration....445Distribution of a Random Variable and the Change of Variable Theorem....446Next Steps in Rigorous Probability Theory....447The Universality Theorem for Neural Networks....448Summary and Looking Ahead....44812. Mathematical Logic....451Various Logic Frameworks....452Propositional Logic....452From Few Axioms to a Whole Theory....455Codifying Logic Within an Agent....456How Do Deterministic and Probabilistic Machine Learning Fit In?....456First-Order Logic....457Relationships Between For All and There Exist....458Probabilistic Logic....460Fuzzy Logic....460Temporal Logic....461Comparison with Human Natural Language....462Machines and Complex Mathematical Reasoning....462Summary and Looking Ahead....46313. Artificial Intelligence and Partial Differential Equations....465What Is a Partial Differential Equation?....466Modeling with Differential Equations....467Models at Different Scales....468The Parameters of a PDE....468Changing One Thing in a PDE Can Be a Big Deal....469Can AI Step In?....471Numerical Solutions Are Very Valuable....472Continuous Functions Versus Discrete Functions....472PDE Themes from My Ph.D. Thesis....474Discretization and the Curse of Dimensionality....477Finite Differences....478Finite Elements....484Variational or Energy Methods....489Monte Carlo Methods....490Some Statistical Mechanics: The Wonderful Master Equation....493Solutions as Expectations of Underlying Random Processes....495Transforming the PDE....495Fourier Transform....495Laplace Transform....498Solution Operators....499Example Using the Heat Equation....499Example Using the Poisson Equation....501Fixed Point Iteration....503AI for PDEs....509Deep Learning to Learn Physical Parameter Values....509Deep Learning to Learn Meshes....510Deep Learning to Approximate Solution Operators of PDEs....512Numerical Solutions of High-Dimensional Differential Equations....519Simulating Natural Phenomena Directly from Data....520Hamilton-Jacobi-Bellman PDE for Dynamic Programming....522PDEs for AI?....528Other Considerations in Partial Differential Equations....528Summary and Looking Ahead....53014. Artificial Intelligence, Ethics, Mathematics, Law, and Policy....531Good AI....533Policy Matters....534What Could Go Wrong?....536From Math to Weapons....536Chemical Warfare Agents....537AI and Politics....538Unintended Outcomes of Generative Models....539How to Fix It?....539Addressing Underrepresentation in Training Data....539Addressing Bias in Word Vectors....540Addressing Privacy....540Addressing Fairness....541Injecting Morality into AI....542Democratization and Accessibility of AI to Nonexperts....543Prioritizing High Quality Data....543Distinguishing Bias from Discrimination....544The Hype....545Final Thoughts....546Index....549