Let's Go Further: Advanced patterns for building, managing and deploying RESTful JSON APIs and web applications in Go

Contents....2 Introduction....6 Conventions....7 About the author....8 Copyright and disclaimer....8 Prerequisites....9 Background knowledge....9 Go 1.16....9 Other software....9 Getting Started....11 Project Setup and Skeleton Structure....12 Generating the skeleton directory structure....13 Hello world!....14 A Basic HTTP Server....16 Creating the healthcheck handler....17 Demonstration....18 Additional Information....20 API versioning....20 API Endpoints and RESTful Routing....21 Choosing a router....22 Encapsulating the API routes....23 Adding the new handler functions....25 Creating a helper to read ID parameters....28 Additional Information....29 Conflicting routes....29 Customizing httprouter behavior....30 Sending JSON Responses....31 Fixed-Format JSON....33 Additional Information....36 JSON charset....36 JSON Encoding....38 Creating a writeJSON helper method....40 Additional Information....42 How different Go types are encoded....42 Using json.Encoder....43 Performance of json.Encoder and json.Marshal....44 Additional JSON encoding nuances....45 Encoding Structs....46 Changing keys in the JSON object....49 Hiding struct fields in the JSON object....50 Additional Information....52 The string struct tag directive....52 Formatting and Enveloping Responses....54 Relative performance....55 Enveloping responses....56 Additional Information....59 Response structure....59 Advanced JSON Customization....60 Customizing the Runtime field....61 Additional Information....64 Alternative #1 - Customizing the Movie struct....64 Alternative #2 - Embedding an alias....66 Sending Error Messages....68 Routing errors....70 Additional Information....72 System-generated error responses....72 Parsing JSON Requests....74 JSON Decoding....76 Zero values....78 Additional Information....79 Supported destination types....79 Using the json.Unmarshal function....79 Additional JSON decoding nuances....80 Managing Bad Requests....81 Triaging the Decode error....82 Making a bad request helper....85 Additional Information....86 Panicking vs returning errors....86 Restricting Inputs....88 Custom JSON Decoding....92 The json.Unmarshaler interface....92 Validating JSON Input....96 Creating a validator package....96 Performing validation checks....98 Making validation rules reusable....100 Database Setup and Configuration....104 Setting up PostgreSQL....105 Installing PostgreSQL....105 Connecting to the PostgreSQL interactive terminal....105 Creating databases, users, and extensions....107 Connecting as the new user....108 Additional Information....108 Optimizing PostgreSQL settings....108 Connecting to PostgreSQL....110 Establishing a connection pool....110 Decoupling the DSN....113 Additional Information....115 Using the DSN with psql....115 Configuring the Database Connection Pool....116 Configuring the pool....116 The SetMaxOpenConns method....116 The SetMaxIdleConns method....117 The SetConnMaxLifetime method....118 The SetConnMaxIdleTime method....119 Putting it into practice....119 Configuring the connection pool....120 SQL Migrations....123 An Overview of SQL Migrations....124 Installing the migrate tool....124 Working with SQL Migrations....126 Executing the migrations....129 Additional Information....130 Migrating to a specific version....130 Executing down migrations....130 Fixing errors in SQL migrations....131 Remote migration files....132 Running migrations on application startup....132 CRUD Operations....134 Setting up the Movie Model....135 Additional Information....138 Mocking models....138 Creating a New Movie....141 Executing the SQL query....141 Hooking it up to our API handler....143 Creating additional records....145 Additional Information....147 $N notation....147 Executing multiple statements....147 Fetching a Movie....149 Updating the API handler....150 Additional Information....152 Why not use an unsigned integer for the movie ID?....152 Updating a Movie....154 Executing the SQL query....154 Creating the API handler....155 Using the new endpoint....158 Deleting a Movie....160 Adding the new endpoint....161 Advanced CRUD Operations....164 Handling Partial Updates....165 Performing the partial update....166 Demonstration....168 Additional Information....169 Null values in JSON....169 Optimistic Concurrency Control....170 Preventing the data race....171 Implementing optimistic locking....172 Additional Information....176 Round-trip locking....176 Locking on other fields or types....176 Managing SQL Query Timeouts....178 Mimicking a long-running query....178 Adding a query timeout....180 Timeouts outside of PostgreSQL....183 Updating our database model....184 Additional Information....186 Using the request context....186 Filtering, Sorting, and Pagination....188 Parsing Query String Parameters....189 Creating helper functions....190 Adding the API handler and route....191 Creating a Filters struct....193 Validating Query String Parameters....196 Listing Data....199 Updating the application....199 Filtering Lists....204 Dynamic filtering in the SQL query....204 Full-Text Search....209 Adding indexes....211 Additional Information....212 Non-simple configuration and more information....212 Using STRPOS and ILIKE....213 Sorting Lists....215 Implementing sorting....216 Paginating Lists....221 The LIMIT and OFFSET clauses....221 Updating the database model....223 Returning Pagination Metadata....226 Calculating the total records....226 Updating the code....227 Structured Logging and Error Handling....233 Structured JSON Log Entries....234 Creating a custom logger....235 Additional Information....240 Integration with the http.Server error log....241 Third-party logging packages....241 Panic Recovery....243 Additional Information....245 Panic recovery in other goroutines....245 Rate Limiting....247 Global Rate Limiting....248 Enforcing a global rate limit....249 IP-based Rate Limiting....253 Deleting old limiters....254 Additional Information....257 Distributed applications....257 Configuring the Rate Limiters....258 Graceful Shutdown....262 Sending Shutdown Signals....263 Intercepting Shutdown Signals....266 Catching SIGINT and SIGTERM signals....268 Executing the Shutdown....271 User Model Setup and Registration....275 Setting up the Users Database Table....276 Setting up the Users Model....278 Adding Validation Checks....280 Creating the UserModel....282 Registering a User....286 Additional Information....290 Email case-sensitivity....290 User enumeration....291 Sending Emails....293 SMTP Server Setup....294 Setting up Mailtrap....294 Creating Email Templates....298 Sending a Welcome Email....300 Creating an email helper....300 Using embedded file systems....302 Using our mail helper....303 Checking the email in Mailtrap....306 Additional Information....307 Retrying email send attempts....307 Sending Background Emails....309 Recovering panics....311 Using a helper function....312 Graceful Shutdown of Background Tasks....316 An introduction to sync.WaitGroup....316 Fixing our application....317 User Activation....322 Setting up the Tokens Database Table....324 Creating Secure Activation Tokens....326 Creating the TokenModel and Validation Checks....328 Additional Information....330 The math/rand package....330 Sending Activation Tokens....331 Additional Information....336 A standalone endpoint for generating tokens....336 Activating a User....337 Creating the activateUserHandler....337 The UserModel.GetForToken method....340 Additional Information....344 Web application workflow....344 SQL query timing attack....346 Authentication....347 Authentication Options....348 HTTP basic authentication....348 Token authentication....349 Stateful token authentication....350 Stateless token authentication....350 API-key authentication....352 OAuth 2.0 / OpenID Connect....352 What authentication approach should I use?....353 Generating Authentication Tokens....355 Building the endpoint....356 Additional Information....360 The Authorization header....360 Authenticating Requests....362 Creating the anonymous user....362 Reading and writing to the request context....363 Creating the authentication middleware....365 Demonstration....368 Permission-based Authorization....370 Requiring User Activation....371 Demonstration....373 Splitting up the middleware....375 Additional Information....376 In-handler checks....376 Setting up the Permissions Database Table....378 Relationship between permissions and users....378 Creating the SQL migrations....380 Setting up the Permissions Model....382 Checking Permissions....385 Demonstration....387 Granting Permissions....390 Updating the permissions model....390 Updating the registration handler....391 Cross Origin Requests....394 An Overview of CORS....395 Demonstrating the Same-Origin Policy....397 Demonstration....399 Simple CORS Requests....403 Restricting Origins....406 Supporting multiple dynamic origins....406 Additional Information....411 Partial origin matches....411 The null origin....412 Authentication and CORS....412 Preflight CORS Requests....413 Demonstrating a preflight request....413 Responding to preflight requests....418 Updating our middleware....419 Additional Information....421 Caching preflight responses....421 Preflight wildcards....422 Metrics....424 Exposing Metrics with Expvar....425 Creating Custom Metrics....429 Dynamic metrics....431 Additional Information....437 Protecting the metrics endpoint....437 Removing default metrics....437 Request-level Metrics....438 Additional Information....444 Calculating additional metrics....444 Recording HTTP Status Codes....446 Additional Information....450 Visualizing and analyzing metrics....450 Building, Versioning and Quality Control....452 Creating and Using Makefiles....453 A simple makefile....453 Environment variables....454 Passing arguments....455 Namespacing targets....457 Prerequisite targets and asking for confirmation....458 Displaying help information....460 Phony targets....461 Managing Environment Variables....465 Using a .envrc file....466 Quality Controlling Code....468 Additional Information....471 Testing....471 Module Proxies and Vendoring....473 Module proxies....473 Vendoring....475 Vendoring new dependencies....480 Additional Information....483 The ./… pattern....483 Building Binaries....485 Reducing binary size....486 Cross-compilation....487 Additional Information....488 Build caching....488 Managing and Automating Version Numbers....490 Displaying the version number....490 Including the build time....492 Automated version numbering with Git....495 Using Git tags....498 Deployment and Hosting....500 Creating a Digital Ocean Droplet....502 Creating a SSH key....503 Adding the SSH key to Digital Ocean....505 Creating a droplet....507 Server Configuration and Installing Software....515 Connecting as the greenlight user....518 Connecting to the droplet....521 Additional Information....522 Future changes to the droplet configuration....522 Deployment and Executing Migrations....523 Running the API....526 Running the API as a Background Service....529 Restarting after reboot....532 Disable port 4000....533 Additional Information....534 Listening on a restricted port....534 Viewing logs....534 Configuring the SMTP provider....535 Additional unit file options....535 Using Caddy as a Reverse Proxy....536 Blocking access to application metrics....538 Using a domain name....541 Enabling HTTPS....543 Appendices....546 Managing Password Resets....547 Additional Information....553 Web application workflow....553 Creating Additional Activation Tokens....554 Authentication with JSON Web Tokens....559 JSON Encoding Nuances....568 Nil and empty slices are encoded differently....568 Using omitempty on a zero-valued struct doesn’t work....568 Using omitempty on a zero-value time.Time doesn’t work....569 Non-ASCII punctuation characters aren’t supported in struct tags....569 Integer, time.Time and net.IP values can be used as map keys....570 Angle brackets and ampersands in strings are escaped....570 Trailing zeroes are removed from floats....571 Working with pre-computed JSON....571 The MarshalText fallback....572 The receiver matters when using MarshalJSON....573 JSON Decoding Nuances....574 Decoding into Go arrays....574 Partial JSON decoding....574 Decoding into interface{} types....575 Decoding a JSON number to an interface{}....576 Struct tag directives....578 Request Context Timeouts....579