Practical Event-Driven Microservices Architecture: Building Sustainable and Highly Scalable Event-Driven Microservices

Table of Contents....5 About the Author....12 About the Technical Reviewer....13 Acknowledgments....14 Introduction....15 Chapter 1: Embracing Event-Driven Architectures....17 1.1 The Truth About Monoliths....20 1.1.1 Anatomy of a Typical Monolith....20 Patchwork Monoliths....21 Modular Monoliths....23 1.1.2 What They Don’t Tell You About Monoliths; It’s Not All Bad....25 Business Flow Is Visible....25 No Network Overhead and Limited External Dependencies....26 Local Validation....26 Code Reuse....26 Monitoring and Troubleshooting....27 End-to-End Testing....27 Simpler Deployment Strategy....28 Data Is Centralized....28 Possible to Scale....28 Consistency....28 Concurrency....29 1.1.3 When Monoliths Become the Business Constrictor Knot....29 Coupling and Lack of Boundaries....29 Team’s Autonomy....30 Release Cycle....30 Scaling....30 Outdated Technology Stack....31 Reliability....32 1.1.4 Using Event-Driven Architectures to Move Away from a Monolith....32 1.2 What Are Microservices and How Do They Relate to Event-Driven....36 1.2.1 Deployment....37 1.2.2 Decoupled and Autonomously Developed....38 1.2.3 Data Ownership....38 1.3 SOA, Microservice, and Event-Driven Architectures....38 1.3.1 SOA....39 1.3.2 Microservice Architecture....40 1.3.3 Event-Driven Microservice Architecture....42 1.4 The Promise of Event-Driven Microservices....43 1.4.1 Evolving Architecture....44 1.4.2 Deployment....44 1.4.3 Team’s Autonomy....45 1.4.4 Flexible Technology Stack....46 1.4.5 Resilience and Availability....46 1.4.6 Tunable Scaling....48 1.4.7 The Past on Demand....48 1.5 When Should You Use Event-Driven Microservices?....49 1.6 Overview of the Challenges in Event-Driven Architectures....52 1.7 Summary....54 Chapter 2: Moving from a Monolith to an Event-Driven Architecture....56 2.1 Is Migrating to an Event-Driven Architecture Your Best Option?....58 2.2 Moving to an Event-Driven Architecture, How to Decide Where to Start....61 2.3 Using an Event-Driven Approach to Move Data from a Monolith....67 2.4 Using Change Data Capture (CDC) to Move Data from a Monolith....73 2.4.1 Event-Driven and Change Data Capture (CDC), a Real-World Example....77 CDC Example Using SQL Server and Kafka....79 2.5 Migrating Data from a Monolith: Event-Driven As a Source of Truth for Both Systems....83 2.6 Incremental Migration from a Monolith to an Event-Driven Architecture: Managing Dependencies....85 2.6.1 Managing a Dependency from a New Event-Driven Service to a Legacy Monolith....85 2.6.2 Managing a Dependency from a Legacy Application to a New Event-Driven Service....88 2.7 Gradually Moving Traffic to New Microservices....91 2.8 Migrating from a Monolith: Two-Way Synchronization and Living with Two Sources of Truth....94 2.9 Summary....97 Chapter 3: Defining an Event-Driven Microservice and Its Boundaries....99 3.1 Building Event-Driven Microservices....101 3.1.1 N-Tier Architectures....102 3.1.2 Clean Architecture....103 3.1.3 Event-Driven Microservices: Durable vs. Ephemeral Message Brokers and GDPR....108 The Right to Be Forgotten and Other Security Concerns....109 3.1.4 Event-Driven Message Types....112 3.1.5 Event-Driven Microservices: When to Use Documents over Events....116 3.1.6 Common Event-Driven Messaging Patterns....118 Send/Receive Pattern....119 Publish/Subscribe Pattern....119 Request/Response Pattern....120 3.1.7 Event-Driven Service Topologies....121 3.1.8 Common Event-Driven Pitfalls and Anti-patterns....125 Faking Synchronous Responses....125 Command Publishing....126 Passive-Aggressive Events....126 3.2 Organizing Event-Driven Microservice Boundaries....127 3.2.1 Organizational Composition....128 3.2.2 Likelihood of Changes....129 3.2.3 Type of Data....129 3.3 Brief and Practical Introduction to Domain-Driven Design and Bounded Contexts....131 3.3.1 How We Can Apply It in Practice....132 3.4 Event-Driven Microservices: The Impact of Aggregate Size and Common Pitfalls....136 3.5 Request-Driven vs. Event-Driven Services....138 3.6 Deciding When to Create a New Microservice or Add Functionality to an Existing One....141 3.7 Summary....144 Chapter 4: Structural Patterns and Chaining Processes....146 4.1 The Challenges of Transactional Consistency in Distributed Systems....148 4.1.1 Why Move from a Monolithic Database in the First Place?....150 4.1.2 The Limitations of Distributed Transactions....151 4.1.3 Managing Multi-step Processes with Sagas....156 4.2 Event-Driven Orchestration Pattern....159 4.3 Event-Driven Choreography Pattern....163 4.4 Event-Driven Microservices: Orchestration, Choreography, or Both?....167 4.5 Data Retrieval in Event-Driven Architectures and Associated Patterns....169 4.5.1 CQS, CQRS, and When to Use Them....173 When to Use CQRS?....176 4.5.2 The Different Flavors of CQRS....177 4.5.3 When and How to Use Event Sourcing....179 4.5.4 Concerns and When to Use Event Sourcing....184 4.5.5 Using Command Sourcing and Its Applicability....185 4.6 Building Multiple Read Models in Event-Driven Microservice Architectures....186 4.7 The Pitfall of Microservice Spaghetti Architectures and How to Avoid It....192 4.7.1 Domain Segregation and Clear Boundaries....193 4.7.2 Context Maps....195 4.7.3 Distributed Tracing....197 4.8 Summary....198 Chapter 5: How to Manage Eventual Consistency....200 5.1 The Impacts of Eventual Consistency and the Need for Alignment with the Business....203 5.1.1 Safety....206 5.1.2 Liveliness....206 5.1.3 The CAP Theorem in the Real World....208 5.2 Using Event Schema in Event-Driven Microservices to Leverage Eventual Consistency....212 5.3 Applying Microservice Domain Boundaries to Leverage Eventual Consistency....217 5.4 Handling Eventual Consistency Delays with Event Versioning....222 5.5 Saving State in Event-Driven Microservices to Avoid Eventual Consistency....224 5.5.1 Buffering State As an Alternative to Persistence....226 5.6 Tackling Eventual Consistency with the End-to-End Argument: A Real-World Use Case....228 5.7 For Most Use Cases, It’s Not Eventual If Nobody Notices....231 5.7.1 Event-Driven Autoscaling Use Case with Prometheus and Kafka....233 5.8 Discussing the Tradeoffs of Typical Eventual Consistency Handling Strategies....235 5.9 Summary....237 Chapter 6: Dealing with Concurrency and Out-of-Order Messages....239 6.1 Why Is Concurrency Different in a Monolith from an Event-Driven Architecture?....241 6.2 Pessimistic vs. Optimistic Concurrency, When and When Not to Use....248 6.2.1 Pessimistic vs. Optimistic Approaches....249 6.2.2 Solving Concurrency by Implementation and by Design....250 6.3 Using Optimistic Concurrency....252 6.4 Using Pessimistic Concurrency....256 6.4.1 Distributed Locks in Event-Driven Microservices....257 6.4.2 Database Transactions As a Concurrency Approach in Distributed Microservices....262 6.5 Dealing with Out-of-Order Events....263 6.5.1 How Can Events Lose Their Order?....264 6.5.2 Solving Out-of-Order Events with Versioning....266 6.6 Using End-to-End Message Partitioning to Handle Concurrency and Guarantee Message Ordering....268 6.6.1 Real-World Example of Event-Driven Message Routing Using Kafka....270 6.6.2 The Relevance of Message Routing and Partitioning in Event-Driven Microservices....275 6.6.3 Using End-to-End Partitioning to Handle Concurrency and Ordering....278 6.6.4 Limitations of End-to-End Partitioning in Event-Driven Microservices....281 Hotspotting....281 Momentary Hotspots....282 The Mix of Event-Driven and Synchronous APIs....283 6.7 Summary....284 Chapter 7: Achieving Resilience and Event Processing Reliability in Event-Driven Microservices....286 7.1 Common Failures in Microservice Architectures and How They Relate to Event-Driven Architectures....288 7.1.1 Cascading Failures and Event-Driven Services....290 7.1.2 Load Balancing and Rate Limiters in Event-Driven Services....295 7.2 Understanding Message Delivery Semantics....299 7.2.1 Exactly-Once Delivery Semantics in Kafka....304 7.3 Avoiding Inconsistencies When Saving State and Publishing Events in Event-Driven Microservices....304 7.3.1 Event Stream As the Only Source of Truth....306 7.3.2 Outbox Pattern in Event-Driven Microservices....309 7.3.3 Transactions and Compensating Actions to Avoid Inconsistencies in Event-Driven Microservices....311 7.4 Applying ACID 2.0 As a Resilience Strategy in Event-Driven Microservices....313 7.5 Avoiding Message Leak in Event-Driven Microservices....317 7.5.1 Poison Events....319 7.6 Applying Common Resilience Patterns in Event-Driven Microservices....319 7.6.1 Retries As a Resilience Approach in Event-Driven Microservices....320 7.6.2 Circuit Breakers in Event-Driven Microservices....322 7.7 Recovering Data and Repairing State in Event-Driven Microservices....325 7.8 Bulkhead Pattern in Event-Driven Microservices....327 7.8.1 Priority Queues....329 7.9 Conclusion....330 7.10 Summary....331 Chapter 8: Choosing the Correct Event Schema Design in Event-Driven Microservices....333 8.1 Event Storming and Event-Driven Microservices....334 8.1.1 What Are the Limitations of Event Storming?....339 8.2 Event Schema: Headers and Envelopes....339 8.2.1 Headers vs. Envelopes in Event Schema....340 8.2.2 Relevant Contextual Information in Events....342 8.3 Town Crier Events Pattern....344 8.4 Bee Events Pattern....347 8.5 The Event Schema Goldilocks Principle....350 8.6 Denormalized Event Schema....353 8.7 Schema Evolution in Event-Driven Microservices....355 8.7.1 Backward Compatibility....356 8.7.2 Forward Compatibility....357 8.7.3 Full Compatibility....357 8.7.4 No Compatibility....358 8.7.5 Managing Changes....358 8.7.6 Event Stream Versioning....361 8.7.7 Using a Downscaler/Upscaler When Evolving Event Schema....362 8.8 Summary....364 Chapter 9: How to Leverage the User Interface in Event-Driven Microservice Architectures....366 9.1 Using an Aggregating Layer to Build a UI in a Distributed Microservice Architecture....368 9.2 Backends for Frontends (BFFs)....371 9.3 UI Decomposition Pattern in Microservice Architectures....374 9.3.1 UI Application Decomposition Pattern....375 9.3.2 UI Page Decomposition Pattern....377 9.3.3 UI Section Decomposition Pattern....378 9.4 The Limitations of API Composition....380 9.5 Task-Based UIs....387 9.6 Event-Driven APIs....393 9.6.1 Event-Driven Combined with WebSockets....395 9.6.2 Event-Driven Combined with Server-Sent Events....396 9.6.3 Event-Driven Combined with WebHooks....398 9.7 Summary....400 Chapter 10: Overcoming the Challenges in Quality Assurance....402 10.1 Microservice Testing Approaches and How They Relate to Event-Driven....403 10.1.1 Unit Tests....404 10.1.2 Component Tests....406 10.1.3 Extended Component Tests....408 10.1.4 Integration Tests....411 10.1.5 Half-Breed Integration Tests....413 10.1.6 End-to-End Tests....417 Releases....419 Data and Environments....419 Governance....420 Approaches to Mitigate the End-to-End Tests Challenges....421 10.2 Applying Contract Tests and Consumer-Driven Contracts to Event-Driven....423 10.3 Test Categorization and Purpose....426 10.4 End-to-End Quality Without End-to-End Tests....428 10.5 Testing in Production....432 10.5.1 Shadowing....433 10.5.2 Canaries....438 10.5.3 Feature Flagging....440 10.5.4 Production Automated Testing....441 10.6 Summary....443 Index....445