Scripting: Automation with Bash, PowerShell, and Python

Notes on Usage....4 Table of Contents....6 Preface....31 Bash, PowerShell, or Python?....31 About This Book....32 Part I Scripting Languages....35 1 Scripting: Doing One Thing....36 1.1 What Does Scripting Mean?....36 1.1.1 Scripting versus Programming....37 1.1.2 Glue Languages and Glue Code....38 1.1.3 Do One Thing and Do It Well....39 1.2 Scripting Languages....41 1.2.1 Bash and Zsh....42 1.2.2 PowerShell....44 1.2.3 Python....45 1.2.4 Many Similarities, Even More Differences....46 1.3 The Agony of Choice....47 2 Ten Times Ten Lines....50 2.1 Markdown Spell Checker (Bash)....51 2.2 Sorting Images by Date (PowerShell)....53 2.3 Converting a JSON File to XML Format (Python)....54 2.4 Daily Server Backups (Bash)....56 2.5 Web Scraping (Python)....57 2.6 Logging Weather Data (Python)....59 2.7 Microsoft Hyper-V Cleanup (PowerShell)....60 2.8 Statistical Analysis of a Logging File (Bash)....61 2.9 Uploading Files to the Cloud (PowerShell)....62 2.10 Cloning Virtual Machines (Bash)....63 3 Bash and Zsh....64 3.1 Terminal, Shell, and Bash....64 3.1.1 Determining the Current Shell....65 3.2 Installation....67 3.2.1 Using Bash in a Virtual Machine....67 3.2.2 Windows Subsystem for Linux....68 3.2.3 Git Bash....70 3.2.4 Docker....70 3.2.5 Bash Configuration (/etc/profile and .bashrc)....70 3.3 Running Commands Interactively....72 3.3.1 Operating the Terminal....74 3.3.2 Online Help....75 3.3.3 Command Reference....75 3.4 Zsh as an Alternative to Bash....77 3.4.1 Installation....77 3.4.2 Zsh Configuration (/etc/zshrc and .zshrc)....78 3.4.3 Oh My Zsh....78 3.5 The First Bash Script....80 3.5.1 Hash Bang (Shebang)....80 3.5.2 Shebang Variants....80 3.5.3 Making the Script Executable (chmod +x)....81 3.5.4 Hello, World!....82 3.5.5 Elementary Syntax Rules....83 3.5.6 A “Real” Example: Backup Script....84 3.5.7 Running Scripts without ./....86 3.6 Running Commands....88 3.6.1 Serial Execution....88 3.6.2 Conditional Command Execution....88 3.6.3 Running Commands in the Background....90 3.6.4 Running Commands in a Subshell....90 3.7 Standard Input and Standard Output....92 3.7.1 Redirecting Input and Output....92 3.7.2 The Pipe Operator....94 3.7.3 Redirecting and Displaying Outputs at the Same Time....95 3.8 Globbing, Brace Extension, and Handling File and Directory Names....96 3.8.1 Recursive Globbing....98 3.8.2 Access to Important Directories....98 3.8.3 Brace Extension....98 3.8.4 Filenames and Directory Names with Spaces....99 3.9 Variables....101 3.9.1 Initializing and Deleting Variables....101 3.9.2 Declaring Variables....102 3.9.3 Performing Calculations in Bash....102 3.9.4 Arrays....103 3.9.5 Predefined Variables....105 3.9.6 Environment Variables....106 3.10 Strings....108 3.10.1 Single versus Double Quotation Marks....108 3.10.2 Outputting Strings (echo)....109 3.10.3 Colors....110 3.10.4 Entering or Reading Character Strings (read)....111 3.10.5 Substitution and Expansion Mechanisms....112 3.10.6 Parameter Substitution....113 3.10.7 Heredocs and Herestrings....116 3.10.8 The Backslash....117 3.11 Branches....119 3.11.1 if Short Notation with && or ||....119 3.11.2 Conditions....120 3.11.3 Branches with case....123 3.11.4 Parameter Analysis via case....124 3.12 Loops....127 3.12.1 Processing Filenames with Spaces....128 3.12.2 while and until....129 3.12.3 Loops over Text Files....130 3.13 Functions....132 3.13.1 Local Variables....132 3.14 Error Protection....134 3.14.1 Detecting Errors....134 3.14.2 Canceling in Case of Errors....134 3.14.3 exit....135 3.14.4 Responding to Signals (trap)....135 3.14.5 Timeout....136 4 PowerShell....138 4.1 Installation....139 4.1.1 Installation on Linux....141 4.1.2 Installation on macOS....142 4.1.3 Limitations of PowerShell on Windows versus Linux/macOS....142 4.1.4 Configuration....143 4.2 Windows Terminal....145 4.2.1 Configuration....146 4.2.2 Operation....147 4.3 Calling cmdlets and Functions....149 4.3.1 cmdlets....151 4.3.2 The Verb-Noun Terminology....151 4.3.3 Aliases....152 4.3.4 Parameters and Options....153 4.3.5 Splatting....153 4.3.6 Functions....154 4.3.7 Running Conventional Commands....154 4.3.8 Online Help....155 4.4 Combining Commands....157 4.4.1 Chains of Commands....158 4.4.2 Multiline Statements....159 4.4.3 Dealing with Special Characters....159 4.5 The First Script....161 4.5.1 Hello, World!....161 4.5.2 Trouble with the Execution Policy....162 4.5.3 Setting Up a Custom Script Directory....164 4.5.4 Running Scripts on Linux and macOS....165 4.5.5 Example: Cleaning Up the Downloads Directory....166 4.5.6 Differentiating Between PowerShell Versions....167 4.6 Variables, Strings, and Objects....169 4.6.1 Data Types....170 4.6.2 Calculating and Comparing....172 4.6.3 Strings....173 4.6.4 Command Substitution....176 4.6.5 Objects....177 4.6.6 Assembling Custom Objects....178 4.6.7 Predefined Variables....179 4.6.8 Environment Variables....180 4.7 Arrays and Hash Tables....182 4.7.1 Creating and Processing Custom Arrays....182 4.7.2 Hash Tables....184 4.8 Output Redirection....185 4.8.1 Streams....185 4.8.2 Output Redirection....186 4.8.3 Duplicating Output (Tee-Object)....188 4.8.4 (Not) Formatting the Output....188 4.8.5 No Input Redirection....189 4.9 Loops....191 4.9.1 ForEach-Object....191 4.9.2 for, while, do-while, and do-until....192 4.10 Branches....194 4.10.1 Case Distinctions Using switch....196 4.11 Functions and Parameters....197 4.11.1 Function Results....197 4.11.2 return....199 4.11.3 Parameters....199 4.11.4 Calling Functions....200 4.11.5 Processing Standard Input....201 4.11.6 Syntax Variants....202 4.12 Modules....204 4.12.1 Module Directories....205 4.12.2 Modules for Experts....206 4.13 Error Protection....208 4.13.1 Set-StrictMode....208 4.13.2 Aborting Scripts on Errors....209 4.13.3 Separating Error Messages from Other Output....209 4.13.4 Error Protection Using try/catch....209 4.13.5 Terminating the Script Execution Prematurely (Exit)....211 5 Python....212 5.1 Installing Python....213 5.1.1 Windows....213 5.1.2 macOS....215 5.2 Getting to Know Python in a Terminal Window....217 5.3 Programming Custom Scripts....219 5.3.1 Shebang....219 5.4 Elementary Syntax Rules....222 5.4.1 Block Elements....222 5.4.2 Comments....223 5.5 Numbers....225 5.5.1 Floats....226 5.5.2 Random Numbers....226 5.6 Strings....228 5.6.1 Raw Strings....229 5.6.2 Processing Strings....229 5.6.3 Slicing....231 5.6.4 print and input....232 5.6.5 Formatting Outputs....233 5.7 Lists....237 5.7.1 List Comprehension....238 5.7.2 map and filter....239 5.8 Tuples, Sets, and Dictionaries....240 5.8.1 Sets....240 5.8.2 Dictionaries....241 5.9 Variables....244 5.9.1 Assignments....245 5.9.2 Data Types....246 5.9.3 Mutable or Immutable?....247 5.10 Operators....250 5.10.1 Combining Assignment and Evaluation....252 5.11 Branches (if)....253 5.11.1 Conditions....253 5.11.2 Short Notation with if....254 5.12 Loops (for and while)....255 5.12.1 break, continue, and else....255 5.12.2 Loops over Number Ranges (range)....256 5.12.3 Loops over the Characters of a String....257 5.12.4 Loops over Lists, Tuples, and Sets....257 5.12.5 Loops over Dictionaries....258 5.12.6 Loops over the Parameters of a Script....259 5.12.7 Globbing on Windows....260 5.12.8 Loops over the Lines of a Text File....260 5.12.9 Loops over All Files in a Directory....261 5.13 Functions....262 5.13.1 Global and Local Variables....263 5.13.2 Parameters....264 5.13.3 Optional Parameters....264 5.13.4 Variable Number of Parameters....265 5.13.5 Lambda Functions....265 5.14 Processing Text Files....267 5.14.1 Example: Analyzing a CSV File....268 5.14.2 Example: Creating SQL Commands....269 5.15 Error Protection....271 5.15.1 try/except....271 5.16 System Functions....273 5.16.1 Access to the Program Parameters....273 5.16.2 Access to Standard Input and Standard Output....273 5.16.3 Exiting a Program....273 5.16.4 Calling Linux Commands....274 5.16.5 Processing Results....274 5.16.6 Running Commands through the Shell....275 5.16.7 Error during a Command Call....276 5.16.8 Waiting (sleep)....276 5.17 Modules....277 5.17.1 import....277 5.17.2 Distributing Custom Scripts across Multiple Files....278 5.18 Installing Additional Modules Using pip....279 5.18.1 Installing pip....279 5.18.2 Applying pip....279 5.18.3 pip Problems with Windows....280 5.18.4 pip Problems with Linux....281 5.18.5 requirements.txt....283 5.18.6 pipenv....284 Part II Work Techniques and Tools....287 6 Linux Toolbox....288 6.1 Directories and Files....289 6.1.1 Listing, Copying, Moving, and Deleting Files....290 6.2 Finding Files....292 6.2.1 Text Search Using grep....292 6.3 Compressing and Archiving Files....294 6.3.1 Archiving Files Using tar....294 6.3.2 ZIP Files....296 6.4 Using Root Privileges....297 6.4.1 sudo Privileges for Selected Accounts Only....297 6.4.2 Issues with Output Redirection....298 6.4.3 Access Rights....298 6.4.4 Managing Processes....300 6.4.5 Terminating Processes....301 6.4.6 Background Processes and System Services....301 6.4.7 Logging Files....302 6.4.8 Determining Free Memory....303 6.4.9 Determining Other System Information....304 6.5 Software Installation....305 6.5.1 Updating Software....305 6.5.2 Installing Additional Packages....306 6.6 Other Commands....308 7 cmdlets for PowerShell....311 7.1 Directories and Files....311 7.1.1 Listing, Copying, Moving, and Deleting Files....312 7.1.2 Reading and Writing Text....314 7.1.3 Inputting and Outputting Text....315 7.2 Finding Files....317 7.2.1 Searching Text Files....317 7.2.2 Text versus Objects....319 7.3 Compressing and Archiving Files....322 7.3.1 Compressing Search Results....322 7.4 Process Management....324 7.4.1 Launching Other PowerShell Scripts....325 7.4.2 Running Commands in the Background....325 7.4.3 Managing Services....326 7.4.4 Working on Other Computers....327 7.5 Registration Database and System Information....329 7.5.1 Determining System Information....330 7.5.2 Evaluating the Logging System....331 7.6 Processing cmdlet Results....332 7.6.1 Select-Object and Sort-Object....332 7.6.2 Where-Object....333 7.6.3 Group-Object....333 7.6.4 ForEach-Item....334 7.6.5 Measure-Object....334 7.6.6 Formatting and Exporting cmdlet Results....335 7.7 Other cmdlets....338 7.8 Installing Additional Modules....340 7.8.1 Listing Modules, Commands, and Package Sources....341 7.8.2 NuGet and winget....342 7.9 Standard Aliases....344 8 Analyzing Texts with Filters and Pipes....346 8.1 grep, sort, cut, and uniq....347 8.1.1 grep....348 8.1.2 wc....349 8.1.3 cut....350 8.1.4 sort....350 8.1.5 head and tail....351 8.1.6 uniq....352 8.1.7 tr....353 8.1.8 awk and sed....354 8.2 Example: Statistical Data Analysis....356 8.2.1 Data Extraction via Script....356 8.3 Example: ping Analysis....358 8.3.1 ping Call via Script....359 8.4 Example: Apache Log Analysis....360 8.4.1 Extracting IP Addresses....361 8.4.2 Identifying Popular Pages....362 8.5 CSV Files....364 8.5.1 Processing CSV Files Using Python....364 8.5.2 Processing CSV Files in PowerShell....366 9 Regular Expressions....368 9.1 Syntax Rules for Regular Expressions....369 9.1.1 Characters....370 9.2 Groups and Alternatives....373 9.2.1 Quantifiers....374 9.2.2 On Greed (Greedy versus Lazy)....375 9.2.3 Alpha and Omega....377 9.2.4 Exercise Pages and Cheat Sheets....378 9.2.5 Example: Recognizing a Date....379 9.2.6 Example: Recognizing IPv4 Addresses....380 9.2.7 Example: Recognizing Email Addresses....381 9.3 Regular Expressions in Bash (grep, sed)....382 9.3.1 Bash Comparison Operator for Regular Expressions....382 9.3.2 Filtering Text Using grep....383 9.3.3 The "sed" Stream Editor....384 9.3.4 Searching and Replacing Using sed....385 9.3.5 sed Example: Manipulating Paths of Image Files....386 9.4 Regular Expressions in PowerShell....388 9.4.1 The match and replace Operators....388 9.4.2 Select-String....389 9.4.3 Example: Input Validation....390 9.4.4 Example: List of Figures....390 9.5 Regular Expressions in Python (re Module)....393 9.5.1 Example: Verifying a MAC Address....394 9.5.2 Example: Anonymizing a Logging File....395 10 JSON, XML, and INI....397 10.1 JSON in PowerShell....397 10.1.1 Example: Saving Event Log Entries in JSON Format....398 10.1.2 Example: Analyzing Domain Queries....399 10.1.3 Example: Converting between CSV and JSON....400 10.2 JSON and Python....402 10.2.1 Example: Collecting Birthdays....403 10.2.2 Example: Determining Holidays....404 10.3 JSON in Bash....407 10.3.1 Viewing JSON Files Interactively Using fx....409 10.4 XML in PowerShell....411 10.4.1 XML Data Type....411 10.4.2 Select-Xml....412 10.4.3 ConvertTo-Xml, Export-Clixml, and Import-Clixml....414 10.5 XML and Python....416 10.5.1 Example: Creating a Dictionary for Country Codes....417 10.5.2 Example: Analyzing an RSS Feed....418 10.5.3 Example: Extracting MAC Addresses from Virtual Machine Files....419 10.6 XML in Bash....421 10.6.1 xmllint....421 10.6.2 XMLStarlet....422 10.7 INI Files....424 10.7.1 Python....424 10.7.2 PowerShell....424 10.7.3 Bash....425 11 Running Scripts Automatically....426 11.1 cron....426 11.1.1 /etc/crontab....427 11.1.2 Personal crontab Files....429 11.1.3 Hourly, Daily, Weekly, and Monthly Directories....430 11.1.4 Alternatives to cron....430 11.1.5 Starting Jobs Automatically on macOS....431 11.2 Example: Web Server Monitoring....432 11.2.1 Testing and Troubleshooting....433 11.2.2 Real Monitoring....434 11.3 Microsoft Windows Task Scheduler....435 11.3.1 Troubleshooting....438 11.3.2 Setting Up Tasks via cmdlets....438 11.4 Example: Saving Exchange Rates....441 11.5 Tracking File System Changes....443 11.5.1 inotify....444 11.5.2 Alternatives to inotify....445 12 SSH....447 12.1 Installing the SSH Client and Server....448 12.1.1 Linux....448 12.1.2 macOS....449 12.1.3 Windows....449 12.1.4 Using an Editor in an SSH Session....451 12.1.5 Securing an SSH Server....452 12.2 Working with SSH....453 12.2.1 Host Verification....454 12.2.2 Potential Issues and Their Cause....454 12.2.3 Running Linux and macOS Commands....455 12.2.4 Running Windows Commands....456 12.2.5 SSH Remoting in PowerShell....456 12.3 scp and rsync....459 12.3.1 Copy-Item with SSH Remoting....459 12.3.2 rsync....460 12.4 SSH Authentication with Keys....462 12.4.1 Generating a Key Pair....462 12.4.2 Storing the Public Component of the Key on the Server (macOS and Linux)....464 12.4.3 Storing the Public Component of the Key on the Server (Windows)....464 12.5 Example: Uploading Images to a Linux Web Server....466 12.5.1 Preparatory Measures....466 12.5.2 Bash Script....467 12.5.3 PowerShell Script....468 12.6 Example: Analyzing Virtual Machines....470 13 Visual Studio Code....472 13.1 Introduction....473 13.1.1 Comparing VS Code, VSCodium, and Visual Studio....473 13.1.2 Think in Terms of Directories, Not Files!....474 13.2 Language-Specific VS Code Extensions....476 13.2.1 PowerShell Extension....476 13.2.2 Python Extension....477 13.2.3 Bash/Shell Extensions....478 13.3 Remote – SSH Extension....479 13.3.1 Applying the Remote – SSH Extension....479 13.3.2 Limitations....480 14 Git....482 14.1 Git Crash Course....483 14.1.1 Preparation Tasks....483 14.1.2 The First Repository....484 14.1.3 The First Commit....486 14.1.4 Additional Commits....487 14.1.5 Setting Up the Repository on a Second Computer....488 14.1.6 Git Status....490 14.1.7 Excluding Files from Git (.gitignore)....491 14.1.8 Transferring Existing Code to a New Repository....491 14.2 Handling Settings and Passwords Correctly....493 14.2.1 What’s a Better Method?....493 14.2.2 Example....495 14.3 Git Automation....497 14.3.1 Controlling GitHub Remotely Using gh....499 14.4 Git Hooks....500 14.4.1 Example: Detecting Unversioned Files Before the Commit....501 Part III Applications and Examples....502 15 Backups....503 15.1 Synchronizing Directories to External Storage....504 15.1.1 PowerShell Script with robocopy....505 15.1.2 Ideas for Improvement....507 15.1.3 Bash Script with rsync....508 15.2 WordPress Backup....510 15.3 SQL Server Backup....513 15.3.1 Backing Up and Compressing all Databases via Script....514 16 Image Processing....516 16.1 Manipulating Image Files....516 16.1.1 Installing ImageMagick....517 16.1.2 Trying Out ImageMagick....518 16.1.3 Example: convert2eps (Bash Variant)....520 16.1.4 Example: convert2eps (PowerShell Variant)....521 16.2 Sorting Photos by Date Taken....523 16.2.1 Installing and Trying Out ExifTool....523 16.2.2 Bash Example....525 16.2.3 PowerShell Example....526 16.3 Converting Exif Metadata to SQL Commands....528 16.3.1 PyExifTool....528 16.3.2 EXIF2SQL....530 16.3.3 Options for Enhancement....532 17 Web Scraping....534 17.1 Limitations....535 17.2 Web Scraping, Web Crawling, and Data Mining....537 17.3 Downloading Websites Using wget....538 17.3.1 Example 1: Downloading Directly Linked Images....538 17.3.2 Example 2: Downloading all PDF Files Linked on the Website....539 17.3.3 Example 3: Creating a Local Copy of the Website....540 17.4 Web Scraping with Regular Expressions....542 17.5 Web Scraping with Beautiful Soup....544 17.5.1 Beautiful Soup and Requests....544 17.5.2 Hello, Beautiful Soup!....544 17.5.3 Example: Determining a List of Top Titles from Rheinwerk Publishing....546 17.6 Web Scraping with Requests-HTML....549 17.6.1 Example: Analyzing the GitHub Status....550 17.7 Web Scraping with PowerShell....553 17.7.1 PowerHTML Module....554 18 Using REST APIs....556 18.1 Tools....556 18.2 Sample APIs to Try Out....558 18.3 Implementing Custom REST APIs....559 18.4 curl and wget....560 18.4.1 curl....560 18.4.2 wget....563 18.4.3 Using REST APIs in Bash Scripts....564 18.5 Using REST APIs in PowerShell....566 18.5.1 Options....567 18.6 Example: Determining the Current Weather....569 18.7 Using REST APIs in Python....571 18.8 Example: Determining Electricity Prices and Displaying Them Graphically....573 18.8.1 aWATTar API....574 18.8.2 Analysis of the Data....575 18.8.3 Matplotlib....576 18.8.4 Controlling the Energy Consumption....577 19 Databases....578 19.1 Updating and Maintaining Databases....579 19.1.1 PowerShell and sqlcmd....580 19.2 Creating a New Customer Account....581 19.2.1 Account Data....581 19.2.2 Structure of the Script....582 19.3 Storing Exif Metadata in a Database....585 19.3.1 PyMySQL....585 19.3.2 Saving Exif Metadata....587 19.4 Importing JSON Data into a Table....589 20 Scripting in the Cloud....591 20.1 AWS CLI....592 20.1.1 Installation on Linux and macOS....592 20.1.2 Configuration....593 20.1.3 Getting Started....593 20.1.4 Encrypting Files....595 20.2 Example: Uploading Encrypted Backup Files to the Cloud....598 20.3 AWS PowerShell Module....600 20.3.1 Getting Started....600 20.3.2 Copying Files....602 20.4 Example: Offloading Large Files from a Website to the Cloud....604 20.4.1 Preparations....604 20.4.2 Script....605 20.4.3 Limitations....606 21 Virtual Machines....608 21.1 Setting Up and Running Virtual Machines (KVMs)....609 21.1.1 Cloning Virtual Machines....609 21.1.2 Starting and Shutting Down Virtual Machines....610 21.1.3 Running Scripts on Multiple Virtual Machines....611 21.2 Automating the Network Configuration (KVMs)....612 21.2.1 Starting Point....612 21.3 Controlling Hyper-V....616 21.3.1 Cloning Virtual Machines....618 The Author....621 Index....622 Service Pages....664 Legal Notes....667