Dead Simple Python: Idiomatic Python for the Impatient Programmer

Title Page....25 Copyright....26 Dedication....28 About the Author....29 Foreword....31 Acknowledgments....33 Introduction....35 Who Is This Book For?....37 What Does “Simple” Mean?....37 What’s in This Book?....38 What’s NOT in This Book....39 How to Read This Book....40 About the Vocabulary....40 Theory Recaps....41 Objective or Opinionated?....41 The Examples....42 What About a Project?....42 Prerequisites....43 Part I: THE PYTHON ENVIRONMENT....45 Chapter 1: The Python Philosophy....46 What Is Python, Exactly?....46 Myths: What Python Isn’t....47 Myth #1: Python Is Merely a Scripting Language....47 Myth #2: Python Is Slow....48 Myth #3: Python Cannot Be Compiled....48 Myth #4: Python Gets Compiled Behind the Scenes....49 Myth #5: Python Is Unsuitable for Large Projects....50 Python 2 vs. Python 3....51 Defining “Pythonic” Code....52 The Zen of Python....53 Documentation, PEPs, and You....54 Who Calls the Shots?....54 The Python Community....55 The Pursuit of the One Obvious Way....56 Wrapping Up....58 Chapter 2: Your Workbench....59 Installing Python....59 Installing on Windows....60 Installing on macOS....60 Installing on Linux....61 Installing from Source....61 Meet the Interpreter....64 Interactive Session....64 Running Python Files....65 Packages and Virtual Environments....66 Creating a Virtual Environment....67 Activating a Virtual Environment....68 Leaving a Virtual Environment....69 Introducing pip....69 System-Wide Packages....69 Installing Packages....70 requirements.txt....71 Upgrading Packages....71 Removing Packages....72 Finding Packages....72 One Warning About pip . . .....72 Virtual Environments and Git....73 The Whole Shebang....74 File Encoding....76 A Few Extra Virtual Environment Tricks....77 Using a Virtual Environment Without Activating....77 The Alternatives....77 The Line Limit Debate....79 Tabs or Spaces?....80 Quality Control: Static Analyzers....81 Pylint....81 Flake8....84 Mypy....87 Style Janitors: Autoformatting Tools....87 autopep8....87 Black....88 Testing Frameworks....88 An Exhibition of Code Editors....88 IDLE....89 Emacs and Vim....89 PyCharm....89 Visual Studio Code....90 Sublime Text....91 Spyder....91 Eclipse + PyDev/LiClipse....91 The Eric Python IDE....92 Wrapping Up....92 Chapter 3: Syntax Crash Course....94 Hello, World!....94 Statements and Expression....95 The Importance of Whitespace....96 Doing Nothing....98 Comments and Docstrings....98 Docstrings....99 Declaring Variables....100 What About Constants?....102 Mathematics....102 Meet the Number Types....102 Operators....104 The math Module....106 Logic....107 Conditionals....107 Comparison Operators....108 Boolean, None, and Identity Operators....109 Truthiness....110 Logical Operators....111 The Walrus Operator....112 Ellipsis....113 Strings....113 String Literals....114 Raw Strings....116 Formatted Strings....117 Template Strings....123 String Conversion....124 A Note on String Concatenation....125 Functions....126 Classes and Objects....127 Error Handling....129 Tuples and Lists....129 Loops....131 while Loop....131 Loop Control....132 for Loop....132 Structural Pattern Matching....133 Literal Patterns and Wildcards....134 Or Patterns....134 Capture Patterns....135 Guard Statements....137 More About Structural Pattern Matching....137 Wrapping Up....138 Chapter 4: Project Structure and Imports....139 Setting Up the Repository....140 Modules and Packages....140 PEP 8 and Naming....142 Project Directory Structure....142 How import Works....144 Import Dos and Don’ts....145 Importing Functions from Modules....146 The Problem of Shadowing....147 The Trouble with Nested Packages....149 Beware of Importing All....150 Importing Within Your Project....152 Absolute Imports....152 Relative Imports....153 Importing from the Same Package....154 Entry Points....155 Module Entry Points....155 Package Entry Points....157 Controlling Package Imports....158 Program Entry Points....160 The Python Module Search Path....161 What Really Happens....163 Wrapping Up....165 Part II: ESSENTIAL STRUCTURES....166 Chapter 5: Variables and Types....167 Variables According to Python: Names and Values....167 Assignment....168 Data Types....171 The type() Function....172 Duck Typing....173 Scope and Garbage Collection....174 Local Scope and the Reference-Counting Garbage Collector....174 Interpreter Shutdown....176 Global Scope....176 The Dangers of Global Scope....179 The nonlocal Keyword....179 Scope Resolution....180 The Curious Case of the Class....181 Generational Garbage Collector....182 The Immutable Truth....182 Passing by Assignment....183 Collections and References....186 Shallow Copy....189 Deep Copy....191 Coercion and Conversion....193 A Note About Systems Hungarian Notation....195 Terminology Review....196 Wrapping Up....198 Chapter 6: Functions and Lambdas....200 Python Function Essentials....203 Recursion....205 Default Argument Values....208 Keyword Arguments....211 On Overloaded Functions....213 Variadic Arguments....214 Keyword Variadic Arguments....216 Keyword-Only Parameters....217 Positional-Only Parameters....218 Argument Types: All Together Now!....219 Nested Functions....220 Closures....221 Recursion with Closures....222 Stateful Closures....223 Lambdas....225 Why Lambdas Are Useful....226 Lambdas as Sorting Keys....229 Decorators....230 Type Hints and Function Annotations....234 Duck Typing and Type Hints....237 Should You Use Type Hinting?....237 Wrapping Up....238 Chapter 7: Objects and Classes....239 Declaring a Class....241 The Initializer....242 The Constructor....243 The Finalizer....244 Attributes....246 Instance Attributes....246 Class Attributes....247 Scope-Naming Conventions....248 Nonpublic....248 Public....249 Name Mangling....249 Public, Nonpublic, or Name Mangled?....250 Methods....251 Instance Methods....251 Class Methods....252 Static Methods....253 Properties....254 Setting Up the Scenario....254 Defining a Property....256 Property with Decorators....260 When Not to Use Properties....262 Special Methods....263 Scenario Setup....264 Conversion Methods....265 Comparison Methods....270 Binary Operator Support....272 Unary Operator Support....273 Making Callable....274 More Special Methods: Looking Ahead....276 Class Decorators....276 Structural Pattern Matching with Objects....278 Functional Meets Object Oriented....281 When to Use Classes....282 Classes Aren’t Modules....282 Single Responsibility....283 Sharing State....283 Are Objects Right for You?....284 Wrapping Up....284 Chapter 8: Errors and Exceptions....286 Exceptions in Python....286 Reading Tracebacks....289 Catching Exceptions: LBYL vs. EAFP....291 Multiple Exceptions....293 Beware the Diaper Anti-pattern....295 Raising Exceptions....298 Using Exceptions....300 Exceptions and Logging....302 Bubbling Up....308 Exception Chaining....309 Else and Finally....311 Else: “If All Goes Well”....312 Finally: “After Everything”....314 Creating Exceptions....318 A Gallery of Exceptions....320 Wrapping Up....323 Part III: DATA AND FLOW....324 Chapter 9: Collections and Iteration....325 Loops....325 while Loops....326 for Loops....328 Collections....329 Tuples....329 Named Tuples....330 Lists....331 Deques....333 Sets....334 frozenset....336 Dictionaries....337 Check or Except?....338 Dictionary Variants....340 Unpacking Collections....341 Starred Expressions....343 Unpacking Dictionaries....344 Structural Pattern Matching on Collections....345 Accessing by Index or Key....348 Slice Notation....349 Start and Stop....350 Negative Indices....351 Steps....352 Copy with Slice....354 Slice Objects....354 Slicing on Custom Objects....355 Using islice....355 The in Operator....356 Checking Collection Length....357 Iteration....358 Iterables and Iterators....359 Manually Using Iterators....359 Iterating with for Loops....363 Sorting Collections in Loops....365 Enumerating Loops....366 Mutation in Loops....367 Loop Nesting and Alternatives....370 Iteration Tools....372 Basic Built-in Tools....372 Filter....373 Map....374 Zip....376 Itertools....377 Custom Iterable Classes....379 Wrapping Up....384 Chapter 10: Generators and Comprehensions....385 Lazy Evaluation and Eager Iterables....385 Infinite Iterators....386 Generators....387 Generators vs. Iterator Classes....390 Closing Generators....394 Behavior on Close....396 Throwing Exceptions....397 yield from....400 Generator Expressions....402 Generator Objects Are Lazy....404 Generator Expressions with Multiple Loops....406 Conditionals in Generator Expressions....407 Nested Generator Expressions....411 List Comprehensions....412 Set Comprehensions....414 Dictionary Comprehensions....414 Hazards of Generator Expressions....415 They Quickly Become Unreadable....416 They Don’t Replace Loops....417 They Can Be Hard to Debug....418 When to Use Generator Expressions....419 Simple Coroutines....420 Returning Values from a Coroutine....423 Sequence of Behavior....424 What About Async?....425 Wrapping Up....426 Chapter 11: Text IO and Context Managers....427 Standard Input and Output....427 Revisiting print()....428 Revisiting input()....434 Streams....435 Context Manager Basics....437 File Modes....438 Reading Files....441 The read() Method....441 The readline() Method....442 The readlines() Method....443 Reading with Iteration....444 Stream Position....445 Writing Files....446 The write() Method....447 The writelines() Method....449 Writing Files with print()....451 Line Separators....452 Context Manager Details....453 How Context Managers Work....453 Using Multiple Context Managers....455 Implementing the Context Management Protocol....455 The __enter__() Method....457 The __exit__() Method....458 Using the Custom Class....459 Paths....462 Path Objects....463 Parts of a Path....465 Creating a Path....470 Relative Paths....473 Paths Relative to Package....474 Path Operations....478 Out-of-Place File Writes....480 The os Module....482 File Formats....482 JSON....483 Other Formats....487 Wrapping Up....492 Chapter 12: Binary and Serialization....493 Binary Notation and Bitwise....493 Number Systems Refresher....494 Python Integers and Binary....501 Bitwise Operations....502 Bytes Literals....505 Bytes-Like Objects....507 Creating a bytes Object....508 Using int.to_bytes()....510 Sequence Operations....512 Converting bytes to int....513 struct....513 struct Format String and Packing....514 Unpacking with struct....518 struct objects....520 Bitwise on Bytes-Like Objects....521 Bitwise Operations via Integers....521 Bitwise Operations via Iteration....523 memoryview....526 Reading and Writing Binary Files....529 Organizing the Data....530 Writing to a File....535 Reading from a Binary File....536 Seek with Binary Stream....540 BufferedRWPair....541 Serialization Techniques....543 Forbidden Tools: pickle, marshal, and shelve....545 Serialization Formats....547 Wrapping Up....550 Part IV: ADVANCED CONCEPTS....552 Chapter 13: Inheritance and Mixins....553 When to Use Inheritance....556 Crimes of Inheritance....557 Basic Inheritance in Python....558 Multiple Inheritance....561 Method Resolution Order....562 Ensuring Consistent Method Resolution Order....567 Explicit Resolution Order....571 Resolving Base Class in Multiple Inheritance....572 Mixins....576 Wrapping Up....579 Chapter 14: Metaclasses and ABCs....581 Metaclasses....581 Creating Classes with type....582 Custom Metaclasses....584 Type Expectations with Duck Typing....587 EAFP: Catching Exceptions....588 LBYL: Checking for Attributes....588 Abstract Classes....591 Built-in ABCs....592 Deriving from ABCs....593 Implementing Custom ABCs....597 Virtual Subclasses....601 Setting Up the Example....601 Using Virtual Subclassing....604 Wrapping Up....608 Chapter 15: Introspection and Generics....609 Special Attributes....609 Inside Object Attribute Access: The __dict__ Special Attribute....610 Listing Attributes....613 Getting an Attribute....614 Checking for an Attribute....617 Setting an Attribute....618 Deleting an Attribute....620 Function Attributes....620 The Wrong Way to Use Function Attributes....621 Mutability and Function Attributes....623 Descriptors....625 The Descriptor Protocol....625 Writing a Descriptor Class (the Slightly Wrong Way)....626 Using a Descriptor....628 Writing a Descriptor Class the Right Way....630 Using Multiple Descriptors in the Same Class....632 Slots....635 Binding Attribute Names to Values....636 Using Arbitrary Attributes with Slots....637 Slots and Inheritance....638 Immutable Classes....638 Single-Dispatch Generic Functions....642 Registering Single-Dispatch Functions with Type Hints....644 Registering Single-Dispatch Functions with Explicit Type....645 Registering Single-Dispatch Functions with the register() Method....646 Using the Element Class....647 Arbitrary Execution....649 Wrapping Up....651 Chapter 16: Asynchrony and Concurrency....653 Asynchrony in Python....655 The Example Scenario: Collatz Game, Synchronous Version....658 Asynchrony....662 Native Coroutines....663 Tasks....666 The Event Loop....667 Making It (Actually) Asynchronous....669 Scheduling and Asynchronous Execution Flow....671 Simplifying the Code....673 Asynchronous Iteration....674 Asynchronous Context Managers....678 Asynchronous Generators....678 Other Asynchrony Concepts....678 Wrapping Up....679 Chapter 17: Threading and Parallelism....681 Threading....682 Concurrency vs. Parallelism....682 Basic Threading....683 Timeouts....687 Daemonic Threads....688 Futures and Executors....689 Timeouts with Futures....693 Race Conditions....695 A Race Condition Example....697 Creating Multiple Threads with ThreadPoolExecutor....699 Locks....701 Deadlock, Livelock, and Starvation....702 Passing Messages with Queue....705 Futures with Multiple Workers....707 Achieving Parallelism with Multiprocessing....709 Pickling Data....710 Speed Considerations and ProcessPoolExecutor....712 The Producer/Consumer Problem....714 Importing the Modules....716 Monitoring the Queue....717 Subprocess Cleanup....717 Consumers....718 Checking for an Empty Queue....719 Producers....720 Starting the Processes....721 Performance Results....722 Logging with Multiprocessing....723 Wrapping Up....723 Part V: BEYOND THE CODE....726 Chapter 18: Packaging and Distribution....727 Planning Your Packaging....728 The Dangers of Cargo Cult Programming....729 A Note on Packaging Opinions....730 Determining Your Packaging Goals....730 Project Structure: src or src-less....733 Packaging a Distribution Package with setuptools....734 Project Files and Structure....735 Where Metadata Belongs....735 The README.md and LICENSE Files....736 The setup.cfg File....737 The setup.py File....746 The MANIFEST.in File....747 The requirements.txt File....748 The pyproject.toml File....750 Testing the Setup Configuration....751 Building Your Package....753 Publishing on pip (Twine)....754 Uploading to Test PyPI....754 Installing Your Uploaded Package....756 Uploading to PyPI....757 Alternative Packaging Tools....757 Poetry....758 Flit....758 Distributing to End Users....758 PEX....759 Freezers....760 Images and Containers....762 A Note on Native Linux Packaging....766 Documentation....767 Wrapping Up....768 Chapter 19: Debugging and Logging....770 Warnings....771 Types of Warnings....772 Filtering Warnings....774 Converting Warnings into Exceptions....776 Logging....777 Logger Objects....778 Handler Objects....780 Logging with Levels....781 Controlling the Log Level....783 Running the Example....785 Filter, Formatter, and Configuration....787 Assert Statements....787 Proper Use of assert....788 Wrong Use of assert....790 Seeing assert in Action....792 The inspect Module....793 Using pdb....793 A Debugging Example....794 Starting the Debugger....796 Debugger Shell Commands....797 Stepping Through the Code....798 Setting a Breakpoint and Stepping into a Function....799 Moving Through the Execution Stack....800 Inspecting the Source....802 Checking a Solution....804 Postmortem Debugging....805 Using faulthandler....806 Evaluating Your Program’s Security with Bandit....809 Reporting Bugs to Python....812 Wrapping Up....813 Chapter 20: Testing and Profiling....814 What About TDD?....815 Test Frameworks....815 The Example Project....817 Testing and Project Structure....818 Testing Basics....820 Starting the Example....821 Unit Testing....822 Executing the Tests with pytest....823 Testing for Exceptions....824 Test Fixtures....826 Continuing the Example: Using the API....829 Sharing Data Between Test Modules....831 Flaky Tests and Conditionally Skipping Tests....832 Advanced Fixtures: Mocking and Parametrizing....834 Continuing the Example: Representing a Typo....834 Parametrizing....837 Indirect Parametrization of Fixtures....838 Mocking Inputs with Monkeypatch....841 Marking....842 Capturing from Standard Streams....845 GUI Testing....847 Continuing the Example: Connecting the API to Typo....848 Autouse Fixtures....849 Mixed Parametrization....852 Fuzzing....853 Wrapping Up the Example....853 Code Coverage....855 Automating Testing with tox....858 Benchmarking and Profiling....861 Benchmarking with timeit....861 Profiling with cProfile or profile....863 tracemalloc....868 Wrapping Up....869 Chapter 21: The Parting of the Ways....870 About the Future....871 Where Do You Go from Here?....872 Application Development in Python....872 Game Development in Python....874 Web Development in Python....874 Client-Side Python....875 Data Science in Python....876 Machine Learning in Python....878 Security....880 Embedded Development in Python....881 Scripting....882 Python Flavors....882 Developing for Python....884 Developing Python Packages and Tools....884 Developing Python Extensions....885 Contributing to Python....887 Getting Involved with Python....888 Asking Questions....889 Answering Questions....890 User Groups....892 PyLadies....892 Conferences....892 Joining the Python Software Foundation....893 And the Road Goes Ever On . . .....894 Appendix A: Special Attributes and Methods....896 Special Methods....896 Special Attributes....907 Appendix B: Python Debugger (pdb) Commands....911 Appendix : Glossary....914 A....914 B....917 C....920 D....925 E....928 F....930 G....933 H....935 I....936 K....939 L....939 M....940 N....943 O....944 P....945 Q....950 R....950 S....951 T....957 U....959 V....960 W....960 Y....961 Z....961 Index....963