Python Polars: The Definitive Guide: Transforming, Analyzing, and Visualizing Data with a Fast and Expressive DataFrame API

Foreword xviiPreface xxiPart I. Begin1. Introducing Polars....3What Is This Thing Called Polars?....4Key Features....4Key Concepts....4Advantages....5Why You Should Use Polars....5Performance....6Usability....6Popularity....7Sustainability....8Polars Compared to Other Data Processing Packages....8Why We Focus on Python Polars....10How This Book Is Organized....10An ETL Showcase....11Extract....12Bonus: Visualizing Neighborhoods and Stations....17Transform....21Bonus: Visualizing Daily Trips per Borough....26Load....28Bonus: Becoming Faster by Being Lazy....29Takeaways....322. Getting Started....33Setting Up Your Environment....33Downloading the Project....34Installing uv....35Installing the Project....35Working with the Virtual Environment....35Verifying Your Installation....36Crash Course in JupyterLab....37Keyboard Shortcuts....38Installing Polars on Other Projects....39All Optional Dependencies....40Optional Dependencies for Interoperability....40Optional Dependencies for Working with Spreadsheets....40Optional Dependencies for Working with Databases....41Optional Dependencies for Working with Remote Filesystems....41Optional Dependencies for Other I/O Formats....41Optional Dependencies for Extra Functionality....42Installing Optional Dependencies....42Configuring Polars....42Temporary Configuration Using a Context Manager....43Local Configuration Using a Decorator....46Compiling Polars from Scratch....46Edge Case: Very Large Datasets....47Edge Case: Processors Lacking AVX Support....48Takeaways....483. Moving from pandas to Polars....49Animals....50Similarities to Recognize....50Appearances to Appreciate....51Differences in Code....51Differences in Display....52Concepts to Unlearn....57Index....57Axes....58Indexing and Slicing....59Eagerness....61Relaxedness....63Syntax to Forget....64Common Operations Side By Side....65To and From pandas....69Takeaways....70Part II. Form4. Data Structures and Data Types....73Series, DataFrames, and LazyFrames....73Data Types....75Nested Data Types....77Missing Values....79Data Type Conversion....84Takeaways....865. Eager and Lazy APIs....87Eager API: DataFrame....87Lazy API: LazyFrame....89Performance Differences....90Functionality Differences....91Attributes....92Aggregation Methods....92Computation Methods....93Descriptive Methods....93GroupBy Methods....94Exporting Methods....94Manipulation and Selection Methods....95Miscellaneous Methods....97Tips and Tricks....98Going from LazyFrame to DataFrame and Vice Versa....98Joining a DataFrame with a LazyFrame....99Caching Intermittent Results....100Takeaways....1016. Reading and Writing Data....103Format Overview....104Reading CSV Files....105Parsing Missing Values Correctly....107Reading Files with Encodings Other Than UTF-8....108Reading Excel Spreadsheets....110Working with Multiple Files....111Reading Parquet....114Reading JSON and NDJSON....115JSON....115NDJSON....118Other File Formats....120Querying Databases....121Writing Data....123CSV Format....123Excel Format....124Parquet Format....124Other Considerations....125Takeaways....125Part III. Express7. Beginning Expressions....129Methods and Namespaces....131Expressions by Example....131Selecting Columns with Expressions....132Creating New Columns with Expressions....133Filtering Rows with Expressions....135Aggregating with Expressions....135Sorting Rows with Expressions....136The Definition of an Expression....137Properties of Expressions....139Creating Expressions....141From Existing Columns....142From Literal Values....143From Ranges....145Other Functions to Create Expressions....146Renaming Expressions....147Expressions Are Idiomatic....149Takeaways....1518. Continuing Expressions....153Types of Operations....154Example A: Element-Wise Operations....155Example B: Operations That Summarize to One....155Example C: Operations That Summarize to One or More....156Example D: Operations That Extend....156Element-Wise Operations....157Operations That Perform Mathematical Transformations....157Operations Related to Trigonometry....159Operations That Round and Categorize....160Operations for Missing or Infinite Values....161Other Operations....163Nonreducing Series-Wise Operations....164Operations That Accumulate....164Operations That Fill and Shift....166Operations Related to Duplicate Values....167Operations That Compute Rolling Statistics....168Operations That Sort....170Other Operations....171Series-Wise Operations That Summarize to One....172Operations That Are Quantifiers....173Operations That Compute Statistics....174Operations That Count....176Other Operations....178Series-Wise Operations That Summarize to One or More....179Operations Related to Unique Values....179Operations That Select....180Operations That Drop Missing Values....181Other Operations....182Series-Wise Operations That Extend....185Takeaways....1859. Combining Expressions....187Inline Operators Versus Methods....188Arithmetic Operations....190Comparison Operations....191Boolean Algebra Operations....195Bitwise Operations....197Using Functions....199When, Then, Otherwise....202Takeaways....204Part IV. Transform10. Selecting and Creating Columns....209Selecting Columns....211Introducing Selectors....212Selecting Based on Name....213Selecting Based on Data Type....214Selecting Based on Position....216Combining Selectors....218Creating Columns....220Related Column Operations....225Dropping....225Renaming....225Stacking....226Adding Row Indices....227Takeaways....22711. Filtering and Sorting Rows....229Filtering Rows....230Filtering Based on Expressions....230Filtering Based on Column Names....231Filtering Based on Constraints....232Sorting Rows....233Sorting Based on a Single Column....234Sorting in Reverse....235Sorting Based on Multiple Columns....235Sorting Based on Expressions....236Sorting Nested Data Types....237Related Row Operations....239Filtering Missing Values....239Slicing....240Top and Bottom....241Sampling....241Semi-Joins....241Takeaways....24212. Working with Textual, Temporal, and Nested Data Types....245String....246String Methods....246String Examples....248Categorical....252Categorical Methods....253Categorical Examples....253Enum....256Temporal....257Temporal Methods....257Temporal Examples....259List....263List Methods....263List Examples....265Array....267Array Methods....267Array Examples....268Struct....270Struct Methods....270Struct Examples....271Takeaways....27413. Summarizing and Aggregating....275Split, Apply, and Combine....276GroupBy Context....276The Descriptives....279Advanced Methods....284Row-Wise Aggregations....289Window Functions in Selection Context....291Dynamic Grouping....293Rolling Aggregations....294Upsampling....297Takeaways....29914. Joining and Concatenating....301Joining....301Join Strategies....302Joining on Multiple Columns....306Validation....306Inexact Joining....308Inexact Join Strategies....310Additional Fine-Tuning....312Use Case: Marketing Campaign Attribution....312Vertical and Horizontal Concatenation....316Vertical....317Horizontal....318Diagonal....318Align....319Relaxed....322Stacking....323Appending....324Extending....324Takeaways....32515. Reshaping....327Wide Versus Long DataFrames....327Pivot to a Wider DataFrame....330Unpivot to a Longer DataFrame....335Transposing....337Exploding....339Partition into Multiple DataFrames....342Takeaways....345Part V. Advance16. Visualizing Data....349NYC Bike Trips....351Built-In Plotting with Altair....353Introducing Altair....353Methods in the Plot Namespaces....354Plotting DataFrames....355Too Large to Handle....357Plotting Series....359pandas-Like Plotting with hvPlot....363Introducing hvPlot....363A First Plot....364Methods in the hvPlot Namespace....365pandas as Backup....366Manual Transformations....367Changing the Plotting Backend....368Plotting Points on a Map....369Composing Plots....369Adding Interactive Widgets....371Publication-Quality Graphics with plotnine....372Introducing plotnine....373Plots for Exploration....373Plots for Communication....377Styling DataFrames With Great Tables....381Takeaways....38617. Extending Polars....387User-Defined Functions in Python....387Applying a Function to Elements....388Applying a Function to a Series....390Applying a Function to Groups....391Applying a Function to an Expression....394Applying a Function to a DataFrame or LazyFrame....395Registering Your Own Namespace....396Polars Plugins in Rust....397Prerequisites....398The Anatomy of a Plugin Project....398The Plugin....398Compiling the Plugin....401Performance Benchmark....401Register Arguments....402Using a Rust Crate....405Use Case: geo....405Takeaways....41618. Polars Internals....417Polars’ Architecture....417Arrow....419Multithreaded Computations and SIMD Operations....421The String Data Type in Memory....422ChunkedArrays in Series....423Query Optimization....424LazyFrame Scan-Level Optimizations....425Other Optimizations....427Checking Your Expressions....429meta Namespace Overview....429meta Namespace Examples....430Profiling Polars....432Tests in Polars....434Comparing DataFrames and Series....435Common Antipatterns....437Using Brackets for Column Selection....437Misusing Collect....437Using Python Code in your Polars Queries....438Takeaways....439Appendix: Accelerating Polars with the GPU....441Index....461