Fast Python: High performance techniques for large datasets

contentsprefaceacknowledgmentsabout this bookWho should read this book?How this book is organized: A road mapAbout the codeliveBook discussion forumHardware and softwareabout the authorabout the cover illustrationPart 1: Foundational ApproachesChapter 1: An urgent need for efficiency in data processing1.1 How bad is the data deluge?1.2 Modern computing architectures and high-performance computing1.2.1 Changes inside the computer1.2.2 Changes in the network1.2.3 The cloud1.3 Working with Python’s limitations1.3.1 The Global Interpreter Lock1.4 A summary of the solutionsChapter 2: Extracting maximum performance from built-in features2.1 Profiling applications with both IO and computing workloads2.1.1 Downloading data and computing minimum temperatures2.1.2 Python’s built-in profiling module2.1.3 Using local caches to reduce network usage2.2 Profiling code to detect performance bottlenecks2.2.1 Visualizing profiling information2.2.2 Line profiling2.2.3 The takeaway: Profiling code2.3 Optimizing basic data structures for speed: Lists, sets, and dictionaries2.3.1 Performance of list searches2.3.2 Searching using sets2.3.3 List, set, and dictionary complexity in Python2.4 Finding excessive memory allocation2.4.1 Navigating the minefield of Python memory estimation2.4.2 The memory footprint of some alternative representations2.4.3 Using arrays as a compact representation alternative to lists2.4.4 Systematizing what we have learned: Estimating memory usage of Python objects2.4.5 The takeaway: Estimating memory usage of Python objects2.5 Using laziness and generators for big-data pipelining2.5.1 Using generators instead of standard functionsChapter 3: Concurrency, parallelism, and asynchronous processing3.1 Writing the scaffold of an asynchronous server3.1.1 Implementing the scaffold for communicating with clients3.1.2 Programming with coroutines3.1.3 Sending complex data from a simple synchronous client3.1.4 Alternative approaches to interprocess communication3.1.5 The takeaway: Asynchronous programming3.2 Implementing a basic MapReduce engine3.2.1 Understanding MapReduce frameworks3.2.2 Developing a very simple test scenario3.2.3 A first attempt at implementing a MapReduce framework3.3 Implementing a concurrent version of a MapReduce engine3.3.1 Using concurrent.futures to implement a threaded server3.3.2 Asynchronous execution with futures3.3.3 The GIL and multithreading3.4 Using multiprocessing to implement MapReduce3.4.1 A solution based on concurrent.futures3.4.2 A solution based on the multiprocessing module3.4.3 Monitoring the progress of the multiprocessing solution3.4.4 Transferring data in chunks3.5 Tying it all together: An asynchronous multithreaded and multiprocessing MapReduce server3.5.1 Architecting a complete high-performance solution3.5.2 Creating a robust version of the serverChapter 4: High-performance NumPy4.1 Understanding NumPy from a performance perspective4.1.1 Copies vs. views of existing arrays4.1.2 Understanding NumPy’s view machinery4.1.3 Making use of views for efficiency4.2 Using array programming4.2.1 The takeaway4.2.2 Broadcasting in NumPy4.2.3 Applying array programming4.2.4 Developing a vectorized mentality4.3 Tuning NumPy’s internal architecture for performance4.3.1 An overview of NumPy dependencies4.3.2 How to tune NumPy in your Python distribution4.3.3 Threads in NumPyPart 2: HardwareChapter 5: Re-implementing critical code with Cython5.1 Overview of techniques for efficient code re-implementation5.2 A whirlwind tour of Cython5.2.1 A naive implementation in Cython5.2.2 Using Cython annotations to increase performance5.2.3 Why annotations are fundamental to performance5.2.4 Adding typing to function returns5.3 Profiling Cython code5.3.1 Using Python’s built-in profiling infrastructure5.3.2 Using line_profiler5.4 Optimizing array access with Cython memoryviews5.4.1 The takeaway5.4.2 Cleaning up all internal interactions with Python5.5 Writing NumPy generalized universal functions in Cython5.5.1 The takeaway5.6 Advanced array access in Cython5.6.1 Bypassing the GIL’s limitation on running multiple threads at a time5.6.2 Basic performance analysis5.6.3 A spacewar example using Quadlife5.7 Parallelism with CythonChapter 6: Memory hierarchy, storage, and networking6.1 How modern hardware architectures affect Python performance6.1.1 The counterintuitive effect of modern architectures on performance6.1.2 How CPU caching affects algorithm efficiency6.1.3 Modern persistent storage6.2 Efficient data storage with Blosc6.2.1 Compress data; save time6.2.2 Read speeds (and memory buffers)6.2.3 The effect of different compression algorithms on storage performance6.2.4 Using insights about data representation to increase compression6.3 Accelerating NumPy with NumExpr6.3.1 Fast expression processing6.3.2 How hardware architecture affects our results6.3.3 When NumExpr is not appropriate6.4 The performance implications of using the local network6.4.1 The sources of inefficiency with REST calls6.4.2 A naive client based on UDP and msgpack6.4.3 A UDP-based server6.4.4 Dealing with basic recovery on the client side6.4.5 Other suggestions for optimizing network computingPart 3: Applications and Libraries for Modern Data ProcessingChapter 7: High-performance pandas and Apache Arrow7.1 Optimizing memory and time when loading data7.1.1 Compressed vs. uncompressed data7.1.2 Type inference of columns7.1.3 The effect of data type precision7.1.4 Recoding and reducing data7.2 Techniques to increase data analysis speed7.2.1 Using indexing to accelerate access7.2.2 Row iteration strategies7.3 pandas on top of NumPy, Cython, and NumExpr7.3.1 Explicit use of NumPy7.3.2 pandas on top of NumExpr7.3.3 Cython and pandas7.4 Reading data into pandas with Arrow7.4.1 The relationship between pandas and Apache Arrow7.4.2 Reading a CSV file7.4.3 Analyzing with Arrow7.5 Using Arrow interop to delegate work to more efficient languages and systems7.5.1 Implications of Arrow’s language interop architecture7.5.2 Zero-copy operations on data with Arrow’s Plasma serverChapter 8: Storing big data8.1 A unified interface for file access: fsspec8.1.1 Using fsspec to search for files in a GitHub repo8.1.2 Using fsspec to inspect zip files8.1.3 Accessing files using fsspec8.1.4 Using URL chaining to traverse different filesystems transparently8.1.5 Replacing filesystem backends8.1.6 Interfacing with PyArrow8.2 Parquet: An efficient format to store columnar data8.2.1 Inspecting Parquet metadata8.2.2 Column encoding with Parquet8.2.3 Partitioning with datasets8.3 Dealing with larger-than-memory datasets the old-fashioned way8.3.1 Memory mapping files with NumPy8.3.2 Chunk reading and writing of data frames8.4 Zarr for large-array persistence8.4.1 Understanding Zarr’s internal structure8.4.2 Storage of arrays in Zarr8.4.3 Creating a new array8.4.4 Parallel reading and writing of Zarr arraysPart 4: Advanced TopicsChapter 9: Data analysis using GPU computing9.1 Making sense of GPU computing power9.1.1 Understanding the advantages of GPUs9.1.2 The relationship between CPUs and GPUs9.1.3 The internal architecture of GPUs9.1.4 Software architecture considerations9.2 Using Numba to generate GPU code9.2.1 Installation of GPU software for Python9.2.2 The basics of GPU programming with Numba9.2.3 Revisiting the Mandelbrot example using GPUs9.2.4 A NumPy version of the Mandelbrot code9.3 Performance analysis of GPU code: The case of a CuPy application9.3.1 GPU-based data analysis libraries9.3.2 Using CuPy: A GPU-based version of NumPy9.3.3 A basic interaction with CuPy9.3.4 Writing a Mandelbrot generator using Numba9.3.5 Writing a Mandelbrot generator using CUDA C9.3.6 Profiling tools for GPU codeChapter 10: Analyzing big data with Dask10.1 Understanding Dask’s execution model10.1.1 A pandas baseline for comparison10.1.2 Developing a Dask-based data frame solution10.2 The computational cost of Dask operations10.2.1 Partitioning data for processing10.2.2 Persisting intermediate computations10.2.3 Algorithm implementations over distributed data frames10.2.4 Repartitioning the data10.2.5 Persisting distributed data frames10.3 Using Dask’s distributed scheduler10.3.1 The dask.distributed architecture10.3.2 Running code using dask.distributed10.3.3 Dealing with datasets larger than memoryAppendicesAppendix A: Setting up the environmentA.1 Setting up Anaconda PythonA.2 Installing your own Python distributionA.3 Using DockerA.4 Hardware considerationsAppendix B: Using Numba to generate efficient low-level codeB.1 Generating optimized code with NumbaB.2 Writing explicitly parallel functions in NumbaB.3 Writing NumPy-aware code in NumbaindexSymbolsA-Z (по алфавиту)