CPython: A Complete Guide to CPython’s Architecture and Performance

Table of Contents....4 About the Author....12 Chapter 1: Reading the CPython Source Code....13 What Is CPython?....13 Why Read the Source Code?....13 Where to Begin?....14 Obtaining the Source Code....14 Development Tools....15 Can I Understand This Without Knowing C?....16 Chapter 2: Overview of the CPython Project Structure....17 Project Structure....17 Building the Project....19 Greet CPython!....20 A (Very) Basic Module....23 Saying Goodbye on Exit....25 Chapter 3: Everything Is an Object: Part 1....27 What Is an “Object”?....27 The Previous and Next Object....29 Garbage Collection Mechanism....30 Immortal Objects....33 PyTypeObject....34 Summary....36 Chapter 4: How Objects Are Created in CPython....37 Running the Program!....37 Step 0: Code Analysis....37 Step 1: Transformation to AST....38 Step 2: Compilation to Bytecode....39 Step 3: Instantiating an Object....41 Summary....43 Chapter 5: Everything Is an Object: Part 2....45 PyTypeObject....45 Basic Members....46 Methods and Operators....46 Access Methods....47 Other....47 The List Type....48 Printing a List....49 Using the Bracket Operator....51 List Methods....53 List Addition....54 Number of Elements....56 List Comparison....57 Summary....59 Chapter 6: Defining a Custom Built-in Type....60 Creating a New Type....60 Defining Methods....61 Implementing the Type....62 Making It a Built-in Type....64 Building and Running....65 Parameterized Initialization....66 Chapter 7: What Happens During Module Import....72 Different Ways of Importing....72 The import Instruction....73 The from .. import .. Instruction....77 Crazy Side-Effects!....78 The Behind-the-Scenes Hero: meta_path....80 Summary....82 Chapter 8: The Internal Representation of Integers....83 How Are Numbers Created?....83 Integer Objects....85 Astronomical Numbers!....87 Small Integers....88 Chapter 9: Floating Point Numbers in CPython....91 What Is a “Floating Point Number”?....91 The Structure of Floating Point Numbers....92 About Floating Point Numbers....93 Floating Point Arithmetic....94 Infinity!....96 Not a Number!....97 Comparing Floating Point Numbers....98 Floating Point Performance....99 Chapter 10: Inside the String Object: Part 1....101 Creating a String....101 String Objects....102 The Fundamental String Structure....105 String Operations....106 Encoding Conversion....106 Strings Are Immutable....108 Chapter 11: Inside the String Object: Part 2....110 String Operations....110 Copying Strings....110 String Slicing....113 Performance....116 String Interning....116 Chapter 12: What Happens When Python Starts....119 Using a Debugger....119 Program Entry Point....120 Reading the Program File....123 Building the Abstract Syntax Tree....125 Creating the Code Object....126 Ready for Liftoff!....127 Chapter 13: From Source to Bytecode: How .py Becomes .pyc....129 Having a .pyc Is All You Need....129 “Maybe” a .pyc File?....131 Magic Number, Magic!....133 Unpacking a .pyc File....137 Chapter 14: The List Object and Its Internal Management....140 Internal Structure of Lists....140 Creating and Initializing a List....141 Memory Management....144 When Is More Memory Needed?....145 1. Adding Elements, But Within Current Capacity....146 2. Adding Elements, Now Exceeding Capacity....146 3. Removing Some Elements, But Still Above Half the Capacity....146 4. Sharply Decreasing Size....146 5. Clearing the List....146 The Over-allocation Formula....147 Common List Operations....148 Appending Elements....148 Inserting Elements....149 Removing Elements....150 Chapter 15: The Dictionary Object: Part 1....152 The Internal Structure of Dictionaries....152 Creating a Dictionary....154 Adding Elements....156 Handling Hash Collisions....161 Looking Up Elements....164 Chapter 16: The Dictionary Object: Part 2....166 Dictionary Memory Management Techniques....166 Adding More Elements....166 Should We Request More Capacity?....169 How Much Space to Allocate?....173 Returning Memory: Does It Happen?....174 Chapter 17: The Tuple Object and Its Immutability....176 Tuple Design....176 Creating a Tuple....177 Empty Tuples....179 Non-empty Tuples....180 Deallocation Mechanism....181 Common Tuple Operations....184 Modifying Tuples....184 Tuple Unpacking....185 Chapter 18: Inside the Python VM: Code Objects....188 Functions Are Also Objects....188 Preparing to Create a Function....191 Code Object....192 Chapter 19: Inside the Python VM: Function Objects....198 Creating Function Objects....198 What Do the Parameters Look Like?....200 Accessing Function Attributes....201 Calling a Function....202 What Is “Vectorcall”?....203 Chapter 20: Inside the Python VM: Frame Objects....206 Frame Object....206 The Life Cycle of a Frame Object....208 Chapter 21: Inside the Python VM: Namespaces and Scopes....212 Variable Scope....212 Local Variables (L)....213 Global and Built-in Variables (G, B)....214 Enclosing Variables (E)....218 Chapter 22: Inside the Python VM: Cells and Closures....221 Creating a Cell Object....221 Closures....223 Free Variables....225 From the Python Perspective....226 Chapter 23: Classes and Where They Come From....229 Creating a Class....229 The Mastermind Behind the Scenes....232 Selecting the Metaclass....233 Preparing the Namespace....234 The Birth of a Class!....235 Chicken or the Egg?....237 Chapter 24: Class Inheritance in CPython....239 Classes and Inheritance....239 Creating a Class....239 How Does Inheritance Work?....241 Method Lookup....242 Chapter 25: Method Resolution Order and C3 Linearization....248 The C3 Linearization Algorithm....248 Whose Method Gets Called?....249 MRO Calculation....250 Single Inheritance....250 Multiple Inheritance....254 A More Complex Inheritance Example....255 What If It Can’t Be Calculated?....258 Chapter 26: The Role of super() in Multiple Inheritance....261 Algorithm Implementation....261 Preparation Before Merging....261 Merging....264 Family Feuds....266 Super!....267 Whose Child Is It?....269 Solving Family Feuds....273 Specifying the Superclass....274 Quick Quiz: Who Am I?....275 Chapter 27: The Generator Object and the Yield Statement....276 The Generator Class....277 yield, Please!....281 Next, Please!....282 Chapter 28: How Iterators Work Internally....286 The Iterator Protocol....286 Halt! Password, Please!....288 Different Types of Iterators?....294 Chapter 29: Understanding Descriptors in Python....297 When Calling Methods....297 Attribute Lookup Process....298 Process Summary....302 Method Descriptors....303 Chapter 30: Exception Handling Internals in CPython....306 Exception Handling....306 Stacking Up....307 Exception Table....308 Entry Portals....309 Exception Type Matching....309 Handling the Exception....311 Finally!....312 Index....314