Modern Data Architecture on Azure: Design Data-centric Solutions on Microsoft Azure

Table of Contents....5 About the Author....10 About the Technical Reviewer....11 Acknowledgments....12 Introduction....13 Chapter 1: Introduction: Fundamentals of Data Management....14 Introduction to DAMA and DMBOK....15 Essential Data Concepts....16 Types of Data....16 Qualitative Data....17 Nominal Data....17 Ordinal Data....18 Quantitative Data....18 Discrete Data....19 Continuous Data....19 Data Management Principles....19 The Data Lifecycle....20 Consistency Models....21 Data Ingestion Patterns....21 Data Platform Paradigm....22 Data Management Principles and Challenges....25 Preparing a Data Strategy....25 Defining Roles and Responsibilities....26 Data Lifecycle Management....27 Data Quality Measurements....28 Metadata....28 Maximizing Data Value for Data-Driven Decisions....30 Dealing with Substantial Volumes of Data....30 Siloed and Varied Data Sources....31 Maintaining the Quality of the Data....31 Data Integration....31 Data Governance and Security....32 Data Automation....32 Data Management Frameworks....33 The Strategic Alignment Model....33 The Amsterdam Information Model....35 The DAMA DMBOK Framework....36 The DAMA Wheel....42 Data Governance....43 Data Architecture....44 Data Modeling and Design....45 Data Storage and Operations....46 Data Security....46 Data Integration and Interoperability....47 Document and Content Management....47 Reference and Master Data....47 Data Warehousing and Business Intelligence....48 Metadata....48 Data Quality....49 Understanding the Environmental Factors Hexagon....50 Understanding the Knowledge Area Context Diagram....51 Conclusion....52 Chapter 2: Build Relational and Non-Relational Data Solutions on Azure....53 Data Integration Using ETL....54 Data Extraction....55 Data Transformation....56 Data Loading....56 Designing ELT Pipelines Using the Azure Synapse Server....57 Online Analytical Processing for Complex Analyses....59 Semantic Data Modeling....63 Challenges of Using OLAP Solutions....65 Managing Transaction Data Using OLTP....66 Managing Non-Relational Data....72 Key-Value Pair Databases....73 Column Family Databases....74 Document Databases....74 Graph Databases....75 Handling Time-Series and Free-Form Search Data....77 Working with CSV and JSON Files for Data Solutions....84 Conclusion....87 Chapter 3: Building a Big Data Architecture....88 Core Components of a Big Data Architecture....89 Data Ingestion and Processing....90 Data Analysis....92 Data Visualization....93 Data Governance....94 Using Batch Processing....94 Azure Synapse Analytics....97 Azure Data Lake Analytics....97 Azure Databricks....98 Azure Data Explorer....99 Real-Time Processing....100 Real-Time Data Ingestion....103 The Lambda Architecture....106 The Kappa Architecture....110 Internet of Things (IoT)....116 Data Mesh Principles and the Logical Architecture....118 Conclusion....123 Chapter 4: Data Management Patterns and Technology Choices with Azure....124 Data Patterns and Trends in Depth....125 CQRS Pattern....125 Event Sourcing....128 Materialized Views....128 Index Table Pattern....129 Analytical Store for Big Data Analytics....131 Azure Synapse Analytics....131 Azure Databricks....133 Data Ingestion Process....134 Data Storage....135 Data Transformation and Model Training....135 Analytics....135 Azure Data Explorer....136 Building Enterprise Data Lakes and Data Lakehouses....137 Enterprise Data Lakes....138 Enterprise Data Lakehouses....142 Data Pipeline Orchestration....144 Real-Time Stream Processing in Azure....149 Conclusion....152 Chapter 5: Data Architecture Process....153 Guide to Data Modeling....153 Conceptual Data Model....155 Logical Data Model....156 Physical Data Model....157 Focus on Business Objectives and its Requirements....157 Data Lake for Ad Hoc Queries....160 Enterprise Data Governance: Data Scrambling, Obfuscation, and DataOps....165 Data Masking Techniques....168 Data Scrambling....170 Data Encryption....171 Data Ageing....171 Data Substitution....171 Data Shuffling....171 Pseudonymization....172 Master Data Management and Storage Optimization....173 Master Data Management....174 Data Encryption Patterns....180 Conclusion....184 Chapter 6: Data Architecture Framework Explained....185 Fundamentals of Data Modeling....185 The Network Data Model....187 The Hierarchical Data Model....188 The Relational Data Model....189 The Object-Oriented Data Model....190 The Dimensional Data Model....191 The Graph Data Model....192 The Entity Relationship Data Model....193 The Open Group Architecture Framework....194 Preliminary Phase....197 Defining the Architecture Vision....197 Business Architecture....198 Information System Architecture....198 Technology Architecture....198 Opportunities and Solutions....199 Migration Planning....199 Governance Implementation....200 Architecture Change Management....200 DAMA DMBOK....200 The Zachman Framework....205 Conclusion....208 Index....209 df-Capture.PNG....1