Building Applications with AI Agents: Designing and Implementing Multiagent Systems

Copyright....6 Table of Contents....7 Preface....15 What This Book Is About....16 What This Book Is Not....17 Who This Book Is For....17 Navigating This Book....17 Conventions Used in This Book....19 Using Code Examples....19 O’Reilly Online Learning....20 How to Contact Us....20 Acknowledgments....21 Chapter 1. Introduction to Agents....23 Defining AI Agents....23 The Pretraining Revolution....24 Types of Agents....25 Model Selection....27 From Synchronous to Asynchronous Operations....28 Practical Applications and Use Cases....29 Workflows and Agents....30 Principles for Building Effective Agentic Systems....33 Organizing for Success in Building Agentic Systems....34 Agentic Frameworks....35 LangGraph....35 AutoGen....36 CrewAI....36 OpenAI Agents Software Development Kit (SDK)....36 Conclusion....37 Chapter 2. Designing Agent Systems....39 Our First Agent System....39 Core Components of Agent Systems....42 Model Selection....43 Tools....46 Designing Capabilities for Specific Tasks....46 Tool Integration and Modularity....47 Memory....47 Short-Term Memory....48 Long-Term Memory....48 Memory Management and Retrieval....48 Orchestration....49 Design Trade-Offs....49 Performance: Speed/Accuracy Trade-Offs....49 Scalability: Engineering Scalability for Agent Systems....50 Reliability: Ensuring Robust and Consistent Agent Behavior....51 Costs: Balancing Performance and Expense....52 Architecture Design Patterns....54 Single-Agent Architectures....54 Multiagent Architectures: Collaboration, Parallelism, and Coordination....54 Best Practices....56 Iterative Design....56 Evaluation Strategy....57 Real-World Testing....59 Conclusion....61 Chapter 3. User Experience Design for Agentic Systems....63 Interaction Modalities....64 Text-Based....65 Graphical Interfaces....68 Speech and Voice Interfaces....72 Video-Based Interfaces....75 Combining Modalities for Seamless Experiences....76 The Autonomy Slider....77 Synchronous Versus Asynchronous Agent Experiences....80 Design Principles for Synchronous Experiences....80 Design Principles for Asynchronous Experiences....81 Finding the Balance Between Proactive and Intrusive Agent Behavior....81 Context Retention and Continuity....82 Maintaining State Across Interactions....83 Personalization and Adaptability....84 Communicating Agent Capabilities....85 Communicating Confidence and Uncertainty....86 Asking for Guidance and Input from Users....87 Failing Gracefully....87 Trust in Interaction Design....88 Conclusion....90 Chapter 4. Tool Use....93 LangChain Fundamentals....94 Local Tools....95 API-Based Tools....97 Plug-In Tools....100 Model Context Protocol....103 Stateful Tools....106 Automated Tool Development....107 Foundation Models as Tool Makers....107 Real-Time Code Generation....108 Tool Use Configuration....109 Conclusion....110 Chapter 5. Orchestration....111 Agent Types....112 Reflex Agents....112 ReAct Agents....112 Planner-Executor Agents....113 Query-Decomposition Agents....113 Reflection Agents....113 Deep Research Agents....114 Tool Selection....115 Standard Tool Selection....116 Semantic Tool Selection....119 Hierarchical Tool Selection....123 Tool Execution....127 Tool Topologies....127 Single Tool Execution....128 Parallel Tool Execution....129 Chains....129 Graphs....131 Context Engineering....134 Conclusion....135 Chapter 6. Knowledge and Memory....137 Foundational Approaches to Memory....138 Managing Context Windows....138 Traditional Full-Text Search....139 Semantic Memory and Vector Stores....141 Introduction to Semantic Search....141 Implementing Semantic Memory with Vector Stores....141 Retrieval-Augmented Generation....143 Semantic Experience Memory....144 GraphRAG....145 Using Knowledge Graphs....145 Building Knowledge Graphs....146 Promise and Peril of Dynamic Knowledge Graphs....152 Note-Taking....155 Conclusion....156 Chapter 7. Learning in Agentic Systems....157 Nonparametric Learning....157 Nonparametric Exemplar Learning....157 Reflexion....159 Experiential Learning....163 Parametric Learning: Fine-Tuning....168 Fine-Tuning Large Foundation Models....168 The Promise of Small Models....173 Supervised Fine-Tuning....175 Direct Preference Optimization....180 Reinforcement Learning with Verifiable Rewards....183 Conclusion....184 Chapter 8. From One Agent to Many....185 How Many Agents Do I Need?....185 Single-Agent Scenarios....185 Multiagent Scenarios....192 Swarms....199 Principles for Adding Agents....200 Multiagent Coordination....202 Democratic Coordination....202 Manager Coordination....203 Hierarchical Coordination....204 Actor-Critic Approaches....204 Automated Design of Agent Systems....206 Communication Techniques....211 Local Versus Distributed Communication....211 Agent-to-Agent Protocol....211 Message Brokers and Event Buses....214 Actor Frameworks: Ray, Orleans, and Akka....217 Orchestration and Workflow Engines....221 Managing State and Persistence....223 Conclusion....224 Chapter 9. Validation and Measurement....227 Measuring Agentic Systems....227 Measurement Is the Keystone....228 Integrating Evaluation into the Development Lifecycle....229 Creating and Scaling Evaluation Sets....229 Component Evaluation....231 Evaluating Tools....231 Evaluating Planning....232 Evaluating Memory....234 Evaluating Learning....235 Holistic Evaluation....236 Performance in End-to-End Scenarios....236 Consistency....238 Coherence....239 Hallucination....240 Handling Unexpected Inputs....241 Preparing for Deployment....242 Conclusion....243 Chapter 10. Monitoring in Production....245 Monitoring Is How You Learn....246 Monitoring Stacks....248 Grafana with OpenTelemetry, Loki, and Tempo....249 ELK Stack (Elasticsearch, Logstash/Fluentd, Kibana)....249 Arize Phoenix....250 SigNoz....251 Langfuse....251 Choosing the Right Stack....252 OTel Instrumentation....252 Visualization and Alerting....254 Monitoring Patterns....257 Shadow Mode....257 Canary Deployments....257 Regression Trace Collection....258 Self-Healing Agents....258 User Feedback as an Observability Signal....258 Distribution Shifts....259 Metric Ownership and Cross-Functional Governance....261 Conclusion....263 Chapter 11. Improvement Loops....265 Feedback Pipelines....267 Automated Issue Detection and Root Cause Analysis....272 Human-in-the-Loop Review....273 Prompt and Tool Refinement....276 Aggregating and Prioritizing Improvements....281 Experimentation....282 Shadow Deployments....283 A/B Testing....284 Bayesian Bandits....285 Continuous Learning....287 In-Context Learning....287 Offline Retraining....289 Conclusion....290 Chapter 12. Protecting Agentic Systems....293 The Unique Risks of Agentic Systems....294 Emerging Threat Vectors....295 Securing Foundation Models....297 Defensive Techniques....298 Red Teaming....300 Threat Modeling with MAESTRO....303 Protecting Data in Agentic Systems....305 Data Privacy and Encryption....305 Data Provenance and Integrity....307 Handling Sensitive Data....308 Securing Agents....310 Safeguards....310 Protections from External Threats....312 Protections from Internal Failures....314 Conclusion....318 Chapter 13. Human-Agent Collaboration....319 Roles and Autonomy....319 The Changing Role of Humans in Agent Systems....320 Aligning Stakeholders and Driving Adoption....321 Scaling Collaboration....322 Agent Scope and Organizational Roles....324 Shared Memory and Context Boundaries....325 Trust, Governance, and Compliance....327 The Lifecycle of Trust....327 Accountability Frameworks....328 Escalation Design and Oversight....331 Privacy and Regulatory Compliance....332 Conclusion: The Future of Human-Agent Teams....334 Glossary....337 Index....341 About the Author....354 Colophon....354