System Programming Essentials with Go: System calls, networking, efficiency, and security practices with practical projects in Golang

Cover....1 Title Page....2 Copyright and Credits....3 Contributors....5 Table of Contents....8 Preface....16 Part 1: Introduction....22 Chapter 1: Why Go?....24 Choosing Go....25 Concurrency and goroutines....25 Concurrency....26 Goroutines....26 CSP-inspired model....26 Share by communication....27 Interacting with the OS....27 Tooling....28 go build....28 go test....28 go run....29 go vet....30 go fmt....30 Cross-platform development with Go....31 Summary....33 Chapter 2: Refreshing Concurrency and Parallelism....34 Technical requirements....34 Understanding goroutines....34 WaitGroup....36 Changing shared state....37 Managing data races....41 Atomic operations....42 Mutexes....43 Making sense of channels....45 How to use channels....45 An unbuffered channel....45 Buffered channels....51 The guarantee of delivery....53 Latency....53 State and signaling....55 State....55 Signaling....56 Choosing your synchronization mechanism....57 Summary....57 Part 2: Interation with the OS....58 Chapter 3: Understanding System Calls....60 Technical requirements....60 Introduction to system calls....61 The catalog of services and identification....61 Information exchange....62 The syscall package....62 A closer look at the os and x/sys packages....63 x/sys package – low-level system calls....63 Operating system functionality....65 Portability....67 Everyday system calls....68 Tracing system calls....68 Tracing specific system calls....69 Developing and testing a CLI program....70 Standard streams....70 File descriptors....72 Creating a CLI application....73 Redirections and standard streams....75 Making it testable....75 Summary....80 Chapter 4: File and Directory Operations....82 Technical requirements....82 Identifying unsafe file and directory permissions....82 Files and permissions....84 Scanning directories in Go....87 Understanding file paths....87 Using the path/filepath package....87 Traversing directories....89 Symbolic links and unlinking files....92 Symbolic links – the shortcut of the file world....92 Unlinking files – the great escape act....93 Calculating directory size....96 Finding duplicate files....98 Optimizing filesystem operations....100 Summary....102 Chapter 5: Working with System Events....104 Managing system events....104 What are signals?....104 The os/signal package....105 Task scheduling in Go....107 Why schedule?....107 Basic scheduling....108 Handling timer signals....110 File monitoring....111 Inotify....111 fsnotify....114 File rotation....115 Process management....120 Execution and timeouts....120 Execute and control process execution time....120 Building a distributed lock manager in Go....122 Summary....124 Chapter 6: Understanding Pipes in Inter-Process Communication....126 Technical requirements....126 What are pipes in IPC?....127 Why are pipes important?....127 Pipes in Golang....129 The mechanics of anonymous pipes....129 Navigating named pipes (Mkfifo())....134 Best practices – guidelines for using pipes....137 Efficient data handling....137 Error handling and resource management....139 Security considerations....140 Performance optimization....141 Developing a log processing tool....141 Summary....143 Chapter 7: Unix Sockets....144 Introduction to Unix sockets....144 Creating a Unix socket....145 Going a little deeper into socket creation....148 Creating the client....148 Inspecting the socket with lsof....150 Building a chat server....150 The complete chat client....157 Serving HTTP under UNIX domain sockets....158 Client....160 HTTP request line....162 HTTP request header....163 Empty line signifying end of headers....163 The textproto package....163 Performance....164 Other common use cases....164 Summary....165 Part 3: Performance....166 Chapter 8: Memory Management....168 Technical requirements....168 Garbage collection....168 Stack and heap allocation....169 The GC algorithm....170 GOGC....171 GC pacer....172 GODEBUG....173 Memory ballast....175 GOMEMLIMIT....176 Memory arenas....177 Using memory arenas....178 Summary....180 Chapter 9: Analyzing Performance....182 Escape analysis....182 Stack and pointers....183 Pointers....184 Stack....186 Heap....186 How can we analyze?....187 Benchmarking your code....191 Writing your first benchmark....192 Memory allocations....194 Common pitfalls....197 CPU profiling....200 Memory profiling....205 Profiling memory over time....206 Preparing to explore the trade-offs....207 Summary....209 Part 4: Connected Apps....210 Chapter 10: Networking....212 The net package....212 TCP sockets....214 HTTP servers and clients....217 HTTP verbs....218 HTTP status codes....219 Putting it all together....219 Securing the connection....221 Certificates....222 Advanced networking....228 UDP versus TCP....228 Summary....240 Chapter 11: Telemetry....242 Technical requirements....242 Logs....243 Zap versus slog....245 Logging for debugging or monitoring?....246 What to log?....249 What not to log?....250 Traces....251 Effective tracing....254 Distributed tracing....255 Metrics....256 What metric should we use?....260 The OTel project....262 OTel....263 Summary....266 Chapter 12: Distributing Your Apps....268 Technical requirements....268 Go Modules....268 The routine using modules....270 CI....278 Caching....279 Static analysis....281 Releasing your application....282 Summary....284 Part 5: Going Beyond....286 Chapter 13: Capstone Project – Distributed Cache....288 Technical requirements....288 Understanding distributed caching....289 System requirements....289 Requirements....290 Design and trade-offs....292 Creating the project....292 Thread safety....293 Choosing the right approach....294 Adding thread safety....295 The interface....295 TCP....296 HTTP....296 Others....297 Eviction policies....300 Sharding....314 Summary....324 Chapter 14: Effective Coding Practices....326 Technical requirements....326 Reusing resources....326 Using sync.Pool in a network server....329 Using sync.Pool for JSON marshaling....330 Executing tasks once....333 singleflight....335 Effective memory mapping....338 API usage....340 Advanced usage with protection and mapping flags....340 Avoiding common performance pitfalls....341 Leaking with time.After....342 Defer in for loops....344 Maps management....345 Resource management....346 Handling HTTP bodies....348 Channel mismanagement....349 Summary....351 Chapter 15: Stay Sharp with System Programming....352 Real-world applications....352 Dropbox’s leap of faith....352 HashiCorp – Go from day one....353 Grafana Labs – visualizing success with Go....354 Docker – building a container revolution with Go....355 SoundCloud – from Ruby to Go....356 Navigating the system programming landscape....357 Go release notes and blog....357 Community....357 Contribution....358 Experimentation....358 Resources for continued learning....358 Advanced Programming in the UNIX Environment by W. Richard Stevens....359 Learn C Programming - Second Edition: A beginner’s guide to learning the most powerful and general-purpose programming language with ease....359 Linux Kernel Programming - Second Edition: A comprehensive and practical guide to kernel internals, writing modules, and kernel synchronization....360 Linux System Programming Techniques: Become a proficient Linux system programmer using expert recipes and techniques....360 Operating Systems: Design and Implementation by Andrew S. Tanenbaum....361 Unix Network Programming by W. Richard Stevens....361 Linux System Programming Techniques: Become a proficient Linux system programmer using expert recipes and techniques....362 Mastering Embedded Linux Programming - Third Edition: Create fast and reliable embedded solutions with Linux 5.4 and the Yocto Project 3.1 (Dunfell)....363 Modern Operating Systems by Andrew S. Tanenbaum....363 The Art of UNIX Programming by Eric S. Raymond....364 Your system programming journey....365 Appendix ....366 Hardware Automation....366 Automation in system programming....366 USB....367 Application....367 The goal....368 The /proc/mounts file....370 Reading the files on the flash drive....371 Partitions versus blocks versus devices versus disks....374 Open source to the rescue!....375 Interacting with USB events....379 Bluetooth....384 Detecting the smartwatch....385 Locking the screen....390 XDG dilemma....391 The Wayland conundrum....391 Summary....391 Index....394