Expert Python Programming: Master Python by learning the best coding practices and advanced programming concepts. 4 Ed

Cover....1 Copyright....3 Contributors....4 Table of Contents....6 Preface....14 Chapter 1: Current Status of Python....20 Where are we now and where are we going?....21 What to do with Python 2....22 Keeping up to date....24 PEP documents....25 Active communities....27 Other resources....30 Summary....31 Chapter 2: Modern Python Development Environments....34 Technical requirements....35 Python's packaging ecosystem....36 Installing Python packages using pip....36 Isolating the runtime environment....38 Application-level isolation versus system-level isolation....42 Application-level environment isolation....43 Poetry as a dependency management system....46 System-level environment isolation....51 Containerization versus virtualization....53 Virtual environments using Docker....55 Writing your first Dockerfile....56 Running containers....60 Setting up complex environments....62 Useful Docker and Docker Compose recipes for Python....65 Virtual development environments using Vagrant....75 Popular productivity tools....78 Custom Python shells....78 Using IPython....80 Incorporating shells in your own scripts and programs....84 Interactive debuggers....85 Other productivity tools....87 Summary....89 Chapter 3: New Things in Python....90 Technical requirements....91 Recent language additions....91 Dictionary merge and update operators....92 Alternative – Dictionary unpacking....95 Alternative – ChainMap from the collections module....95 Assignment expressions....98 Type-hinting generics....102 Positional-only parameters....103 zoneinfo module....106 graphlib module....107 Not that new, but still shiny....112 breakpoint() function....112 Development mode....113 Module-level __getattr__() and __dir__() functions....116 Formatting strings with f-strings....117 Underscores in numeric literals....119 secrets module....119 What may come in the future?....120 Union types with the | operator....121 Structural pattern matching....122 Summary....127 Chapter 4: Python in Comparison with Other Languages....128 Technical requirements....129 Class model and object-oriented programming ....129 Accessing super-classes....131 Multiple inheritance and Method Resolution Order....133 Class instance initialization....139 Attribute access patterns....143 Descriptors....144 Real-life example – lazily evaluated attributes....147 Properties....151 Dynamic polymorphism....157 Operator overloading....159 Dunder methods (language protocols)....160 Comparison to C++....164 Function and method overloading....166 Single-dispatch functions....168 Data classes....170 Functional programming....174 Lambda functions....176 The map(), filter(), and reduce() functions....178 Partial objects and partial functions....181 Generators....182 Generator expressions....184 Decorators....185 Enumerations....187 Summary....190 Chapter 5: Interfaces, Patterns, and Modularity....192 Technical requirements....193 Interfaces....194 A bit of history: zope.interface....196 Using function annotations and abstract base classes....205 Using collections.abc....210 Interfaces through type annotations....211 Inversion of control and dependency injection....214 Inversion of control in applications....216 Using dependency injection frameworks....225 Summary....231 Chapter 6: Concurrency....232 Technical requirements....233 What is concurrency?....233 Multithreading....235 What is multithreading?....236 How Python deals with threads....240 When should we use multithreading?....242 Application responsiveness....242 Multiuser applications....243 Work delegation and background processing....244 An example of a multithreaded application....245 Using one thread per item....248 Using a thread pool....250 Using two-way queues....255 Dealing with errors in threads....257 Throttling....260 Multiprocessing....264 The built-in multiprocessing module....266 Using process pools....270 Using multiprocessing.dummy as the multithreading interface....273 Asynchronous programming....274 Cooperative multitasking and asynchronous I/O....275 Python async and await keywords....276 A practical example of asynchronous programming....281 Integrating non-asynchronous code with async using futures....284 Executors and futures....286 Using executors in an event loop....287 Summary....288 Chapter 7: Event-Driven Programming....290 Technical requirements....291 What exactly is event-driven programming?....291 Event-driven != asynchronous....292 Event-driven programming in GUIs....293 Event-driven communication....296 Various styles of event-driven programming....298 Callback-based style....299 Subject-based style....300 Topic-based style....305 Event-driven architectures....307 Event and message queues....309 Summary....312 Chapter 8: Elements of Metaprogramming....314 Technical requirements....315 What is metaprogramming?....315 Using decorators to modify function behavior before use ....316 One step deeper: class decorators....318 Intercepting the class instance creation process....323 Metaclasses....326 The general syntax....328 Metaclass usage....331 Metaclass pitfalls....334 Using the __init__subclass__() method as an alternative to metaclasses....336 Code generation....338 exec, eval, and compile....338 The abstract syntax tree ....340 Import hooks....342 Notable examples of code generation in Python....342 Falcon's compiled router....343 Hy....344 Summary....345 Chapter 9: Bridging Python with C and C++....346 Technical requirements....348 C and C++ as the core of Python extensibility....348 Compiling and loading Python C extensions....349 The need to use extensions....351 Improving performance in critical code sections....352 Integrating existing code written in different languages....353 Integrating third-party dynamic libraries....354 Creating efficient custom datatypes....354 Writing extensions....355 Pure C extensions....356 A closer look at the Python/C API....360 Calling and binding conventions....364 Exception handling....368 Releasing GIL....370 Reference counting....372 Writing extensions with Cython....375 Cython as a source-to-source compiler....375 Cython as a language....379 Downsides of using extensions....381 Additional complexity....382 Harder debugging....383 Interfacing with dynamic libraries without extensions....384 The ctypes module....384 Loading libraries....384 Calling C functions using ctypes....386 Passing Python functions as C callbacks....388 CFFI....391 Summary....393 Chapter 10: Testing and Quality Automation....396 Technical requirements....397 The principles of test-driven development....398 Writing tests with pytest....400 Test parameterization....408 pytest's fixtures....411 Using fakes....421 Mocks and the unittest.mock module....424 Quality automation....429 Test coverage....430 Style fixers and code linters....434 Static type analysis....438 Mutation testing....439 Useful testing utilities....446 Faking realistic data values....446 Faking time values....448 Summary....449 Chapter 11: Packaging and Distributing Python Code....452 Technical requirements....453 Packaging and distributing libraries....453 The anatomy of a Python package....454 setup.py....457 setup.cfg....459 MANIFEST.in....459 Essential package metadata....461 Trove classifiers....462 Types of package distributions....464 sdist distributions....464 bdist and wheel distributions....466 Registering and publishing packages....469 Package versioning and dependency management....472 The SemVer standard for semantic versioning....474 CalVer for calendar versioning....475 Installing your own packages....476 Installing packages directly from sources....476 Installing packages in editable mode....477 Namespace packages....478 Package scripts and entry points....480 Packaging applications and services for the web....484 The Twelve-Factor App manifesto....485 Leveraging Docker....486 Handling environment variables....489 The role of environment variables in application frameworks....494 Creating standalone executables....499 When standalone executables are useful....500 Popular tools....500 PyInstaller....501 cx_Freeze....505 py2exe and py2app....507 Security of Python code in executable packages....509 Summary....510 Chapter 12: Observing Application Behavior and Performance....512 Technical requirements....513 Capturing errors and logs....513 Python logging essentials....514 Logging system components....516 Logging configuration....524 Good logging practices....528 Distributed logging....530 Capturing errors for later review....533 Instrumenting code with custom metrics....537 Using Prometheus....539 Distributed application tracing....549 Distributed tracing with Jaeger....553 Summary....559 Chapter 13: Code Optimization....560 Technical requirements....561 Common culprits for bad performance....561 Code complexity....562 Cyclomatic complexity....563 The big O notation....564 Excessive resource allocation and leaks....567 Excessive I/O and blocking operations....568 Code profiling....568 Profiling CPU usage....570 Macro-profiling....570 Micro-profiling....576 Profiling memory usage....579 Using the objgraph module....581 C code memory leaks....589 Reducing complexity by choosing appropriate data structures....590 Searching in a list....590 Using sets....592 Using the collections module....593 deque....593 defaultdict....595 namedtuple....597 Leveraging architectural trade-offs....599 Using heuristics and approximation algorithms....599 Using task queues and delayed processing....600 Using probabilistic data structures....604 Caching....605 Deterministic caching....606 Non-deterministic caching....609 Summary....614 Why subscribe?....616 Packt Page....616 Other Books You May Enjoy....618 Index....620