Exploratory Data Analysis with Python Cookbook: Over 50 recipes to analyze, visualize, and extract insights from structured and unstructured data

Exploratory Data Analysis with Python Cookbook....2 Contributors....7 About the author....7 About the reviewers....7 Preface....24 Who this book is for....25 What this book covers....26 To get the most out of this book....28 Download the example code files....28 Download the color images....28 Conventions used....28 Get in touch....29 Share Your Thoughts....30 Download a free PDF copy of this book....30 Chapter 1: Generating Summary Statistics....31 Technical requirements....31 Analyzing the mean of a dataset....31 Getting ready....32 How to do it....32 How it works.......33 Theres more.......33 Checking the median of a dataset....33 Getting ready....33 How to do it....34 How it works.......34 Theres more.......34 Identifying the mode of a dataset....34 Getting ready....35 How to do it....35 How it works.......35 Theres more.......36 Checking the variance of a dataset....36 Getting ready....36 How to do it....36 How it works.......36 Theres more....37 Identifying the standard deviation of a dataset....37 Getting ready....37 How to do it....37 How it works.......38 Theres more.......38 Generating the range of a dataset....38 Getting ready....38 How to do it....38 How it works.......39 Theres more.......39 Identifying the percentiles of a dataset....39 Getting ready....39 How to do it....39 How it works.......40 Theres more.......40 Checking the quartiles of a dataset....40 Getting ready....40 How to do it....40 How it works.......41 Theres more.......41 Analyzing the interquartile range (IQR) of a dataset....41 Getting ready....42 How to do it....42 How it works.......42 Chapter 2: Preparing Data for EDA....43 Technical requirements....43 Grouping data....43 Getting ready....43 How to do it....43 How it works.......44 Theres more.......44 See also....44 Appending data....44 Getting ready....44 How to do it....45 How it works.......45 Theres more.......46 Concatenating data....46 Getting ready....46 How to do it....46 How it works.......47 Theres more.......47 See also....47 Merging data....47 Getting ready....48 How to do it....48 How it works.......49 Theres more.......49 See also....49 Sorting data....49 Getting ready....49 How to do it....49 How it works.......50 Theres more.......50 Categorizing data....50 Getting ready....50 How to do it....50 How it works.......51 Theres more.......51 Removing duplicate data....51 Getting ready....51 How to do it....52 How it works.......52 Theres more.......52 Dropping data rows and columns....52 Getting ready....52 How to do it....53 How it works.......53 Theres more.......53 Replacing data....53 Getting ready....53 How to do it....54 How it works.......54 Theres more.......54 See also....54 Changing a data format....54 Getting ready....55 How to do it....55 How it works.......55 Theres more.......55 See also....55 Dealing with missing values....55 Getting ready....56 How to do it....56 How it works.......56 Theres more.......56 See also....56 Chapter 3: Visualizing Data in Python....58 Technical requirements....58 Preparing for visualization....58 Getting ready....58 How to do it....59 How it works.......59 Theres more.......59 Visualizing data in Matplotlib....60 Getting ready....60 How to do it....60 How it works.......63 Theres more.......64 See also....64 Visualizing data in Seaborn....64 Getting ready....64 How to do it....64 How it works.......67 Theres more.......68 See also....68 Visualizing data in GGPLOT....68 Getting ready....68 How to do it....69 How it works.......71 Theres more.......71 See also....71 Visualizing data in Bokeh....71 Getting ready....72 How to do it....72 How it works.......75 There's more.......76 See also....76 Chapter 4: Performing Univariate Analysis in Python....77 Technical requirements....77 Performing univariate analysis using a histogram....77 Getting ready....77 How to do it....77 How it works.......79 Performing univariate analysis using a boxplot....79 Getting ready....79 How to do it....79 How it works.......81 Theres more.......81 Performing univariate analysis using a violin plot....81 Getting ready....82 How to do it....82 How it works.......83 Performing univariate analysis using a summary table....83 Getting ready....83 How to do it....83 How it works.......84 Theres more.......84 Performing univariate analysis using a bar chart....84 Getting ready....85 How to do it....85 How it works.......86 Performing univariate analysis using a pie chart....86 Getting ready....86 How to do it....86 How it works.......87 Chapter 5: Performing Bivariate Analysis in Python....89 Technical requirements....89 Analyzing two variables using a scatter plot....89 Getting ready....90 How to do it....90 How it works.......91 Theres more.......92 See also.......92 Creating a crosstabtwo-way table on bivariate data....92 Getting ready....92 How to do it....92 How it works.......93 Analyzing two variables using a pivot table....93 Getting ready....93 How to do it....93 How it works.......94 There is more.......94 Generating pairplots on two variables....94 Getting ready....95 How to do it....95 How it works.......95 Analyzing two variables using a bar chart....96 Getting ready....96 How to do it....96 How it works.......97 There is more.......98 Generating box plots for two variables....98 Getting ready....98 How to do it....98 How it works.......99 Creating histograms on two variables....99 Getting ready....99 How to do it....100 How it works.......101 Analyzing two variables using a correlation analysis....101 Getting ready....101 How to do it....102 How it works.......103 Chapter 6: Performing Multivariate Analysis in Python....104 Technical requirements....104 Implementing Cluster Analysis on multiple variables using Kmeans....104 Getting ready....104 How to do it....105 How it works.......106 There is more.......106 See also.......107 Choosing the optimal number of clusters in Kmeans....107 Getting ready....107 How to do it....107 How it works.......108 There is more.......109 See also.......109 Profiling Kmeans clusters....109 Getting ready....109 How to do it....109 How it works.......111 Theres more.......112 Implementing principal component analysis on multiple variables....112 Getting ready....112 How to do it....112 How it works.......113 There is more.......113 See also.......114 Choosing the number of principal components....114 Getting ready....114 How to do it....114 How it works.......115 Analyzing principal components....116 Getting ready....116 How to do it....116 How it works.......117 Theres more.......118 See also.......118 Implementing factor analysis on multiple variables....118 Getting ready....118 How to do it....118 How it works.......120 There is more.......121 Determining the number of factors....121 Getting ready....121 How to do it....121 How it works.......122 Analyzing the factors....123 Getting ready....123 How to do it....123 How it works.......126 Chapter 7: Analyzing Time Series Data in Python....128 Technical requirements....129 Using line and boxplots to visualize time series data....129 Getting ready....129 How to do it....129 How it works.......131 Spotting patterns in time series....132 Getting ready....132 How to do it....132 How it works.......134 Performing time series data decomposition....134 Getting ready....136 How to do it....136 How it works.......140 Performing smoothing – moving average....141 Getting ready....141 How to do it....141 How it works....144 See also.......145 Performing smoothing – exponential smoothing....145 Getting ready....145 How to do it....145 How it works.......148 See also.......148 Performing stationarity checks on time series data....148 Getting ready....149 How to do it....149 How it works.......150 See also....150 Differencing time series data....150 Getting ready....151 How to do it....151 How it works.......152 Getting ready....153 How to do it....153 How it works.......156 See also.......157 Chapter 8: Analysing Text Data in Python....158 Technical requirements....158 Preparing text data....158 Getting ready....159 How to do it....159 How it works.......161 Theres more....162 See also....162 Dealing with stop words....162 Getting ready....162 How to do it....162 How it works.......165 Theres more....166 Analyzing part of speech....167 Getting ready....167 How to do it....167 How it works.......169 Performing stemming and lemmatization....170 Getting ready....170 How to do it....170 How it works.......174 Analyzing ngrams....175 Getting ready....175 How to do it....175 How it works.......177 Creating word clouds....177 Getting ready....177 How to do it....178 How it works.......179 Checking term frequency....179 Getting ready....180 How to do it....180 How it works.......182 Theres more....182 See also....183 Checking sentiments....183 Getting ready....183 How to do it....183 How it works.......186 Theres more....186 See also....186 Performing Topic Modeling....187 Getting ready....187 How to do it....187 How it works.......190 Choosing an optimal number of topics....190 Getting ready....190 How to do it....190 How it works.......192 Chapter 9: Dealing with Outliers and Missing Values....193 Technical requirements....193 Identifying outliers....193 Getting ready....194 How to do it....194 How it works.......195 Spotting univariate outliers....195 Getting ready....196 How to do it....196 How it works.......197 Finding bivariate outliers....198 Getting ready....198 How to do it....198 How it works.......200 Identifying multivariate outliers....200 Getting ready....200 How to do it....200 How it works.......204 See also....205 Flooring and capping outliers....205 Getting ready....205 How to do it....205 How it works.......207 Removing outliers....207 Getting ready....207 How to do it....208 How it works.......209 Replacing outliers....209 Getting ready....209 How to do it....209 How it works.......211 Identifying missing values....211 Getting ready....212 How to do it....212 How it works.......214 Dropping missing values....214 Getting ready....215 How to do it....215 How it works.......215 Replacing missing values....216 Getting ready....216 How to do it....216 How it works.......217 Imputing missing values using machine learning models....217 Getting ready....218 How to do it....218 How it works.......219 Chapter 10: Performing Automated Exploratory Data Analysis in Python....220 Technical requirements....220 Doing Automated EDA using pandas profiling....220 Getting ready....221 How to do it....221 How it works.......226 See also....227 Performing Automated EDA using dtale....227 Getting ready....227 How to do it....227 How it works.......231 See also....232 Doing Automated EDA using AutoViz....232 Getting ready....232 How to do it....232 How it works.......236 See also....236 Performing Automated EDA using Sweetviz....237 Getting ready....237 How to do it....237 How it works.......239 See also....239 Implementing Automated EDA using custom functions....239 Getting ready....240 How to do it....240 How it works.......243 Theres more....244 Index....245 Why subscribe?....259 Other Books You May Enjoy....259 Packt is searching for authors like you....263 Share Your Thoughts....263 Download a free PDF copy of this book....263