Python in a Nutshell: A Desktop Quick Reference. 4 Ed

Preface....5 How To Use This Book....5 Part I, Getting Started with Python....6 Part II, Core Python Language and Built-ins....7 Part III, Python Library and Extension Modules....9 Part IV, Network and Web Programming....11 Part V, Extending, Distributing, and Version Upgrade and Migration....12 Conventions Used in This Book....12 Reference Conventions....12 Version Conventions....13 Typographic Conventions....13 Using Code Examples....14 OReilly Online Learning....15 How to Contact Us....15 Acknowledgments....16 1. Introduction to Python....18 The Python Language....18 The Python Standard Library and Extension Modules....19 Python Implementations....20 CPython....20 PyPy....21 Choosing Between CPython, PyPy, and Other Implementations....21 Other Developments, Implementations, and Distributions....22 Jython and IronPython....22 Numba....23 Pyjion....23 IPython....23 MicroPython....24 Anaconda and Miniconda....25 pyenv: Simple support for multiple versions....26 Transcrypt: Convert your Python to JavaScript....26 Licensing and Price Issues....27 Python Development and Versions....28 Python Resources....29 Documentation....29 Python documentation for nonprogrammers....29 Extension modules and Python sources....30 Books....30 Community....31 Python Software Foundation....31 Workgroups....31 Python conferences....32 User groups and organizations....32 Mailing lists....32 Social media....32 Installation....33 Installing Python from Binaries....33 Installing Python from Source Code....35 Microsoft Windows....35 Uncompressing and unpacking the Python source code....35 Building the Python source code....36 Unix-Like Platforms....36 Uncompressing and unpacking the Python source code....36 Configuring, building, and testing....36 Installing after the build....37 2. The Python Interpreter....39 The python Program....39 Environment Variables....39 Command-Line Syntax and Options....40 The Windows py Launcher....42 The PyPy Interpreter....43 Interactive Sessions....44 Python Development Environments....45 IDLE....46 Other Python IDEs....46 Free Text Editors with Python Support....47 Tools for Checking Python Programs....47 Running Python Programs....48 Running Python in the Browser....49 PyScript....50 Jupyter....50 3. The Python Language....52 Lexical Structure....52 Lines and Indentation....52 Character Sets....53 Tokens....54 Identifiers....54 Keywords....57 Operators....57 Delimiters....58 Literals....58 Statements....59 Simple statements....59 Compound statements....60 Data Types....60 Numbers....61 Integer numbers....61 Floating-point numbers....62 Complex numbers....63 Underscores in numeric literals....64 Sequences....64 Iterables....64 Strings....64 bytes objects....68 bytearray objects....68 Tuples....68 Lists....69 Sets....70 Dictionaries....71 None....73 Ellipsis (...)....73 Callables....73 Boolean Values....74 Variables and Other References....75 Variables....75 Object attributes and items....75 Accessing nonexistent references....76 Assignment Statements....76 Plain assignment....77 Augmented assignment....79 del Statements....80 Expressions and Operators....81 Comparison Chaining....82 Short-Circuiting Operators....82 The conditional operator....83 Assignment Expressions....83 : in an ifelif statement....83 : in a while statement....84 : in a list comprehension filter....84 Numeric Operations....85 Numeric Conversions....85 Arithmetic Operations....86 Division....86 Exponentiation....87 Comparisons....87 Bitwise Operations on Integers....87 Sequence Operations....87 Sequences in General....88 Sequence conversions....88 Concatenation and repetition....88 Membership testing....88 Indexing a sequence....89 Slicing a sequence....89 Strings....91 Tuples....91 Lists....91 Modifying a list....91 In-place operations on a list....92 List methods....92 Sorting a list....93 Set Operations....94 Set Membership....95 Set Methods....95 Dictionary Operations....97 Dictionary Membership....97 Indexing a Dictionary....97 Dictionary Methods....98 Control Flow Statements....100 The if Statement....100 The match Statement....102 Building patterns....103 Literal patterns....104 The wildcard pattern....104 Capture patterns....105 Value patterns....105 OR patterns....106 Group patterns....107 Sequence patterns....107 as patterns....108 Mapping patterns....109 Class patterns....110 Guards....111 Configuring classes for positional matching....112 The while Statement....113 The for Statement....113 Iterators....116 Iterables versus iterators....117 range....118 List comprehensions....118 Set comprehensions....121 Dictionary comprehensions....121 The break Statement....121 The continue Statement....122 The else Clause on Loop Statements....123 The pass Statement....123 The try and raise Statements....124 The with Statement....124 Functions....124 Defining Functions: The def Statement....125 Parameters....127 Positional parameters....127 Named parameters....127 Positional-only marker....128 Positional argument collector....128 Named argument collector....129 Parameter sequence....129 Mutable default parameter values....129 Argument collector parameters....130 Attributes of Function Objects....131 Docstrings....131 Other attributes of function objects....133 Function Annotations....133 The return Statement....134 Calling Functions....134 Positional and named arguments....135 Keyword-only parameters....137 Matching arguments to parameters....138 The semantics of argument passing....140 Namespaces....140 The global statement....141 Nested functions and nested scopes....142 lambda Expressions....144 Generators....145 yield from....147 Generators as near-coroutines....147 Generator expressions....147 Recursion....149 4. Object-Oriented Python....153 Classes and Instances....153 Python Classes....153 The class Statement....154 The Class Body....155 Attributes of class objects....155 Function definitions in a class body....157 Class-private variables....157 Class documentation strings....158 Descriptors....158 Overriding and nonoverriding descriptors....159 Instances....159 __init__....159 Attributes of instance objects....160 The factory function idiom....162 __new__....162 Attribute Reference Basics....163 Getting an attribute from a class....164 Getting an attribute from an instance....165 Setting an attribute....166 Bound and Unbound Methods....166 Inheritance....170 Method resolution order....170 Overriding attributes....171 Delegating to superclass methods....171 Cooperative superclass method calling....173 Dynamic class definition using the type built-in function....174 Deleting class attributes....175 The Built-in object Type....176 Class-Level Methods....176 Static methods....177 Class methods....177 Properties....178 Why properties are important....180 Properties and inheritance....181 __slots__....182 __getattribute__....183 Per Instance Methods....183 Inheritance from Built-in Types....184 Special Methods....184 General-Purpose Special Methods....185 Special Methods for Containers....189 Sequences....189 Mappings....190 Sets....190 Container slicing....191 Container methods....191 Abstract Base Classes....192 The abc module....193 ABCs in the collections module....193 ABCs in the numbers module....195 Special Methods for Numeric Objects....196 Decorators....198 Metaclasses....199 Alternatives to Custom Metaclasses for Simple Class Customization....199 How Python Determines a Classs Metaclass....201 How a Metaclass Creates a Class....201 Defining and using your own metaclasses....202 A substantial custom metaclass example....203 Data Classes....205 Enumerated Types (Enums)....208 5. Type Annotations....213 History....213 Type-Checking Utilities....214 mypy....215 Other Type Checkers....215 Type Annotation Syntax....216 The typing Module....219 Types....219 Type Expression Parameters....221 Abstract Base Classes....221 Protocols....222 Utilities and Decorators....224 Defining Custom Types....226 Generics and TypeVars....226 Restricting TypeVar to specific types....229 NamedTuple....230 TypedDict....230 TypeAlias....233 NewType....234 Using Type Annotations at Runtime....235 How to Add Type Annotations to Your Code....237 Adding Type Annotations to New Code....237 Adding Type Annotations to Existing Code (Gradual Typing)....238 Using .pyi Stub Files....239 Summary....240 6. Exceptions....241 The try Statement....241 tryexcept....242 tryfinally....244 tryexceptfinally....246 The raise Statement....246 The with Statement and Context Managers....248 Generators and Exceptions....251 Exception Propagation....251 Exception Objects....253 The Hierarchy of Standard Exceptions....253 Standard Exception Classes....254 OSError subclasses....255 Exceptions wrapping other exceptions or tracebacks....256 Custom Exception Classes....258 Custom Exceptions and Multiple Inheritance....260 Other Exceptions Used in the Standard Library....260 ExceptionGroup and except....261 Error-Checking Strategies....262 LBYL Versus EAFP....262 Handling Errors in Large Programs....264 Logging Errors....266 The logging module....266 Configuring logging....267 The assert Statement....269 7. Modules and Packages....271 Module Objects....271 The import Statement....272 The module body....272 Attributes of module objects....273 Python built-ins....275 Module documentation strings....276 Module-private variables....276 The from Statement....276 from...import ....277 from versus import....278 Handling import failures....278 Module Loading....279 Built-in Modules....279 Searching the Filesystem for a Module....279 The Main Program....281 Reloading Modules....282 Circular Imports....283 Custom Importers....284 Rebinding __import__....284 Import hooks....284 Packages....286 Special Attributes of Package Objects....287 Absolute Versus Relative Imports....287 Distribution Utilities (distutils) and setuptools....288 Python Environments....290 Enter the Virtual Environment....291 What Is a Virtual Environment?....292 Creating and Deleting Virtual Environments....292 Working with Virtual Environments....294 Managing Dependency Requirements....296 Other Environment Management Solutions....297 Best Practices with Virtualenvs....298 8. Core Built-ins and Standard Library Modules....301 Built-in Types....301 Built-in Functions....304 The sys Module....311 The copy Module....314 The collections Module....315 ChainMap....316 Counter....316 OrderedDict....318 defaultdict....318 deque....320 The functools Module....321 The heapq Module....323 The Decorate–Sort–Undecorate Idiom....324 The argparse Module....326 The itertools Module....327 9. Strings and Things....332 Methods of String Objects....332 The string Module....336 String Formatting....337 Formatted String Literals (F-Strings)....338 Debug printing with f-strings....339 Formatting Using format Calls....340 Value Conversion....342 Value Formatting: The Format Specifier....343 Fill and alignment....343 Sign indication....344 Zero normalization (z)....345 Radix indicator ()....346 Leading zero indicator (0)....346 Field width....346 Grouping option....347 Precision specification....348 Format type....348 Nested Format Specifications....350 Formatting of User-Coded Classes....351 Legacy String Formatting with %....352 Format Specifier Syntax....354 Text Wrapping and Filling....355 The pprint Module....356 The reprlib Module....356 Unicode....357 The codecs Module....357 The unicodedata Module....358 10. Regular Expressions....360 Regular Expressions and the re Module....360 REs and bytes Versus str....360 Pattern String Syntax....361 Common Regular Expression Idioms....363 Sets of Characters....365 Alternatives....366 Groups....367 Optional Flags....367 Match Versus Search....370 Anchoring at String Start and End....370 Regular Expression Objects....371 Match Objects....374 Functions of the re Module....375 REs and the : Operator....376 The Third-Party regex Module....377 11. File and Text Operations....378 The io Module....379 Creating a File Object with open....380 mode....381 Binary and text modes....381 Buffering....381 Sequential and nonsequential (random) access....382 Attributes and Methods of File Objects....382 Iteration on File Objects....384 File-Like Objects and Polymorphism....384 The tempfile Module....385 Auxiliary Modules for File IO....387 The fileinput Module....387 The struct Module....389 In-Memory Files: io.StringIO and io.BytesIO....391 Archived and Compressed Files....392 The tarfile Module....392 The TarFile class....394 The TarInfo class....395 The zipfile Module....396 The ZipFile class....397 The ZipInfo class....399 The os Module....400 Filesystem Operations....400 Path-string attributes of the os module....400 Permissions....401 File and directory functions of the os module....402 File descriptor operations....406 The os.path Module....408 OSError Exceptions....410 The errno Module....411 The pathlib Module....411 The stat Module....417 The filecmp Module....418 The fnmatch Module....420 The glob Module....421 The shutil Module....421 Text Input and Output....423 Standard Output and Standard Error....423 The print Function....423 Standard Input....424 The getpass Module....425 Richer-Text IO....426 The readline Module....426 Console IO....427 curses....428 The msvcrt module....428 Internationalization....428 The locale Module....429 The gettext Module....434 Using gettext for localization....434 Essential gettext functions....436 More Internationalization Resources....436 12. Persistence and Databases....438 Serialization....439 The csv Module....439 csv functions and classes....439 A csv example....440 The json Module....441 json functions....441 A json example....443 The pickle Module....443 pickle functions and classes....445 A pickling example....446 Pickling instances....447 Pickling customization with the copyreg module....448 The shelve Module....449 A shelving example....450 DBM Modules....451 The dbm Package....452 Examples of DBM-Like File Use....453 The Python Database API (DBAPI)....454 Exception Classes....454 Thread Safety....455 Parameter Style....455 Factory Functions....457 Type Description Attributes....457 The connect Function....458 Connection Objects....458 Cursor Objects....459 DBAPI-Compliant Modules....460 SQLite....461 The sqlite3.Connection class....463 The sqlite3.Row class....464 A sqlite3 example....464 13. Time Operations....466 The time Module....466 The datetime Module....469 The date Class....470 The time Class....471 The datetime Class....472 The timedelta Class....475 The tzinfo Abstract Class....476 The timezone Class....476 The zoneinfo Module....476 The dateutil Module....479 The sched Module....480 The calendar Module....481 14. Customizing Execution....484 Per-Site Customization....484 Termination Functions....485 Dynamic Execution and exec....485 Avoiding exec....486 Expressions....487 compile and Code Objects....487 Never exec or eval Untrusted Code....489 Internal Types....490 Type Objects....490 The Code Object Type....491 The Frame Type....491 Garbage Collection....491 The gc Module....492 Instrumenting garbage collection....495 The weakref Module....495 15. Concurrency: Threads and Processes....498 Threads in Python....500 The threading Module....501 Thread Objects....502 Thread Synchronization Objects....503 Timeout parameters....503 Lock and RLock objects....504 Condition objects....506 Event objects....507 Semaphore and BoundedSemaphore objects....508 Timer objects....509 Barrier objects....509 Thread Local Storage....511 The queue Module....512 The multiprocessing Module....514 Differences Between multiprocessing and threading....515 Structural differences....515 The Process class....515 Differences in queues....516 Sharing State: Classes Value, Array, and Manager....516 The Value class....517 The Array class....517 The Manager class....518 Process Pools....520 The Pool class....520 The AsyncResult class....522 The ThreadPool class....522 The concurrent.futures Module....523 Threaded Program Architecture....525 Process Environment....530 Running Other Programs....531 Using the Subprocess Module....531 What to run, and how....532 Subprocess files....532 Other, advanced arguments....532 Attributes of subprocess.Popen instances....533 Methods of subprocess.Popen instances....533 Running Other Programs with the os Module....534 The mmap Module....536 Methods of mmap Objects....537 Using mmap Objects for IPC....539 16. Numeric Processing....540 Floating-Point Values....540 The math and cmath Modules....544 The statistics Module....548 The operator Module....549 Random and Pseudorandom Numbers....551 The random Module....551 Crypto-Quality Random Numbers: The secrets Module....553 The fractions Module....554 The decimal Module....556 Array Processing....559 The array Module....559 Extensions for Numeric Array Computation....561 NumPy....561 Creating a NumPy array....562 Shape, indexing, and slicing....565 Matrix operations in NumPy....568 SciPy....570 Additional numeric packages....570 17. Testing, Debugging, and Optimizing....572 Testing....573 Unit Testing and System Testing....573 The doctest Module....576 The unittest Module....579 The TestCase class....582 Unit tests dealing with large amounts of data....584 Testing with nose2....585 Testing with pytest....586 Debugging....587 Before You Debug....588 The inspect Module....588 Introspecting callables....590 An example of using inspect....590 The traceback Module....592 The pdb Module....592 Other Debugging Modules....595 ipdb....596 pudb....596 The warnings Module....597 Classes....597 Objects....598 Filters....598 Functions....600 Optimization....601 Developing a Fast-Enough Python Application....602 Benchmarking....603 Large-Scale Optimization....604 List operations....607 String operations....607 Dictionary operations....607 Set operations....608 Summary of big-O times for operations on Python built-in types....608 Profiling....609 The profile module....609 Calibration....609 The pstats module....610 Small-Scale Optimization....611 The timeit module....612 Memoizing....613 Precomputing a lookup table....615 Building up a string from pieces....616 Searching and sorting....617 Avoid exec and from ... import ....618 Short-circuiting of Boolean expressions....618 Short-circuiting of iterators....619 Optimizing loops....620 Using multiprocessing for heavy CPU work....623 Optimizing IO....623 18. Networking Basics....626 The Berkeley Socket Interface....626 Socket Addresses....628 ClientServer Computing....628 Connectionless client and server structures....629 Connection-oriented client and server structures....630 The socket Module....632 Socket Objects....634 A Connectionless Socket Client....638 A Connectionless Socket Server....639 A Connection-Oriented Socket Client....640 A Connection-Oriented Socket Server....641 Transport Layer Security....642 SSLContext....643 19. Client-Side Network Protocol Modules....646 Email Protocols....646 The poplib Module....647 The smtplib Module....648 HTTP and URL Clients....649 URL Access....649 The urllib Package....650 The urllib.parse module....650 The urllib.request module....652 The Third-Party requests Package....653 Sending requests....653 requests optional named parameters....654 The files argument (and other ways to specify the requests body)....654 How to interpret requests examples....655 The Response class....656 Other Network Protocols....657 20. Serving HTTP....660 http.server....660 WSGI....661 WSGI Servers....662 ASGI....662 Python Web Frameworks....662 Full-Stack Versus Lightweight Frameworks....663 A Few Popular Full-Stack Frameworks....663 Considerations When Using Lightweight Frameworks....663 A Few Popular Lightweight Frameworks....664 Flask....664 Flask request objects....666 Flask response objects....666 FastAPI....668 21. Email, MIME, and Other Network Encodings....673 MIME and Email Format Handling....673 Functions in the email Package....673 The email.message Module....674 The email.Generator Module....677 Creating Messages....677 The email.encoders Module....678 The email.utils Module....678 Example Uses of the email Package....679 Encoding Binary Data as ASCII Text....680 The base64 Module....681 The quopri Module....682 The uu Module....683 22. Structured Text: HTML....684 The html.entities Module....685 The BeautifulSoup Third-Party Package....685 The BeautifulSoup Class....686 Which parser BeautifulSoup uses....686 BeautifulSoup, Unicode, and encoding....687 The Navigable Classes of bs4....688 Indexing instances of Tag....688 Getting an actual string....689 Attribute references on instances of BeautifulSoup and Tag....691 contents, children, and descendants....692 parent and parents....692 next_sibling, previous_sibling, next_siblings, and previous_siblings....693 next_element, previous_element, next_elements, and previous_elements....693 bs4 find Methods (aka Search Methods)....694 Arguments of search methods....695 Filters....695 name....696 string....697 attrs....697 Other named arguments....698 bs4 CSS Selectors....698 An HTML Parsing Example with BeautifulSoup....698 Generating HTML....699 Editing and Creating HTML with bs4....699 Building and adding new nodes....700 Replacing and removing nodes....701 Building HTML with bs4....702 Templating....703 The jinja2 Package....704 The jinja2.Environment class....704 The jinja2.Template class....705 Building HTML with jinja2....705 23. Structured Text: XML....708 ElementTree....708 The Element Class....709 The ElementTree Class....712 Functions in the ElementTree Module....713 Parsing XML with ElementTree.parse....714 Selecting Elements from an ElementTree....715 Editing an ElementTree....716 Building an ElementTree from Scratch....717 Parsing XML Iteratively....717 24. Packaging Programs and Extensions....720 What We Dont Cover in This Chapter....721 A Brief History of Python Packaging....722 Online Material....723 25. Extending and Embedding Classic Python....725 Online Material....726 26. v3.7 to v3.n Migration....728 Significant Changes in Python Through 3.11....728 Planning a Python Version Upgrade....731 Choosing a Target Version....732 Scoping the Work....732 Applying the Code Changes....733 Upgrade Automation Using pyupgrade....734 Multiversion Testing....735 Use a Controlled Deployment Process....735 How Often Should You Upgrade?....735 Summary....735 A. New Features and Changes in Python 3.7 Through 3.11....737 Python 3.7....737 Python 3.8....740 Python 3.9....743 Python 3.10....745 Python 3.11....748 Index....751