Create an AWS Glue Job to Convert the Raw Data into a Delta Table
Go to AWS Glue and click on Go to workflows as shown in Figure 3-60. We will get navigated to the AWS Glue studio, where we can create the AWS Glue job to convert the raw CSV file into a delta table.
Figure 3-60. Go to workflows
Click on Jobs as in Figure 3-61. We need to create the workflow for the job and execute it to convert the raw CSV file into a delta table.
Figure 3-61. Click on Jobs
We will be using the delta lake connector in the AWS Glue job. We need to activate it.
Go to the Marketplace, as in Figure 3-62.
Figure 3-62. Click on Marketplace
Search for delta lake as in Figure 3-63, and then click on Delta Lake Connector for AWS Glue.
Figure 3-63. Activate delta lake connector
Click on Continue to Subscribe as in Figure 3-64.
Figure 3-64. Continue to Subscribe
Click on Continue to Configuration as in Figure 3-65.
Figure 3-65. Continue to Configuration
Click on Continue to Launch as in Figure 3-66.
Figure 3-66. Continue to Launch
Click on Usage instructions as in Figure 3-67. The usage instructions will open up. We have some steps to perform on that page.
Figure 3-67. Click Usage Instructions
Read the usage instructions once and then click on Activate the Glue connector from AWS Glue Studio, as in Figure 3-68.
Figure 3-68. Activate Glue connector
Provide a name for the connection, as in Figure 3-69, and then click on Create a connection and activate connector.
Figure 3-69. Create connection and activate connector
Once the connector gets activated, go back to the AWS Glue studio and click on Jobs, as in Figure 3-70.
Figure 3-70. Click on Jobs
Select option Visual with a source and target and depict the source as Amazon S3, as in Figure 3-71.
Figure 3-71. Configure source
Select the target as Delta Lake Connector 1.0.0 for AWS Glue 3.0, as in Figure 3-72. We activated the connector in the marketplace and created a connection earlier. Click on Create.
Figure 3-72. Configure target
Go to the S3 bucket and copy the S3 URI for the raw-data folder, as in Figure 3-73.
Figure 3-73. Copy URI
Provide the S3 URI you copied and configure the data source, as in Figure 3-74. Make sure you mark the data format as CSV and check Recursive.
Figure 3-74. Source settings
Scroll down and check the First line of the source file contains column headers option, as in Figure 3-75.
Figure 3-75. Source settings
Click on ApplyMapping as in Figure 3-76 and provide the data mappings and correct data types. The target parquet file will be generated with these data types.
Figure 3-76. Configure mappings
Click on Delta Lake Connector, and then click on Add new option, as in Figure 3-77.
Figure 3-77. Add new option
Provide Key as path and Value as URI for the delta-lake folder in the S3 bucket, as in Figure 3-78.
Figure 3-78. Configure target
Go to the Job details as in Figure 3-79 and set IAM Role as the role we created as a prerequisite with all necessary permissions.
Figure 3-79. Provide IAM role
Click on Save as in Figure 3-80 and then run the job.
Figure 3-80. Save and run
Once the job runs successfully, go to the delta-lake folder in the S3 bucket. You can see the delta table parquet file generated in Figure 3-81.
Figure 3-81. Generated delta table in the delta-lake folder
Moreover, data pipelines facilitate data integration and consolidation. Organizations often have data spread across multiple systems, databases, and applications. Data pipelines provide a means to efficiently gather, transform, and consolidate data from disparate sources into a unified and consistent format. This integrated view of data allows organizations to derive comprehensive insights and make better-informed decisions based on a holistic understanding of their data.
At its core, a data pipeline consists of the following key components:
• Data sources: The data sources are the places where the data comes from. They can be internal systems, external sources, or a combination of both.
• Data Ingestion: This is the initial stage of the data pipeline where data is collected from its source systems or external providers. It involves extracting data from various sources, such as databases, APIs, files, streaming platforms, or IOT devices. Data ingestion processes should consider factors like data volume, velocity, variety, and quality to ensure the efficient and reliable acquisition of data.
• Data Processing: Once the data is ingested, it goes through various processing steps to transform, clean, and enrich it. This stage involves applying business rules, algorithms, or transformations to manipulate the data into a desired format or structure. Common processing tasks include filtering, aggregating, joining, validating, and normalizing the data. The goal is to prepare the data for further analysis and downstream consumption.
• Data Transformation: In this stage, the processed data is further transformed to meet specific requirements or standards. This may involve converting data types, encoding or decoding data, or performing complex calculations. Data transformation ensures that the data is in a consistent and usable format for subsequent stages or systems. Transformations can be performed using tools, programming languages, or specialized frameworks designed for data manipulation.
• Data Storage: After transformation, the data is stored in a persistent storage system, such as a data warehouse, data lake, or a database. The choice of storage depends on factors such as data volume, latency requirements, querying patterns, and cost considerations. Effective data storage design is crucial for data accessibility, scalability, and security. It often involves considerations like data partitioning, indexing, compression, and backup strategies.
• Data Delivery: The final stage of the data pipeline involves delivering the processed and stored data to the intended recipients or downstream systems. This may include generating reports, populating dashboards, pushing data to business intelligence tools, or providing data to other applications or services via APIs or data feeds. Data delivery should ensure the timely and accurate dissemination of data to support decision-making and enable actionable insights.
Throughout these stages, data pipeline orchestration and workflow management play a critical role. Orchestration involves defining the sequence and dependencies of the different stages and processes within the pipeline. Workflow management tools, such as Apache Airflow or Luigi, facilitate the scheduling, monitoring, and coordination of these processes, ensuring the smooth and efficient execution of the pipeline.
It’s important to note that data pipelines can vary in complexity and scale depending on the organization’s requirements. They can range from simple, linear pipelines with a few stages to complex, branching pipelines with parallel processing and conditional logic. The design and implementation of a data pipeline should be tailored to the specific use case, data sources, processing requirements, and desired outcomes (Figure 5-4).
The stages of a data pipeline are as follows:
• Ingestion: The data is collected from the data sources and loaded into the data pipeline.
• Cleaning: The data is cleaned to remove errors and inconsistencies.
• Transformation: The data is transformed into a format that is useful for analysis.
• Storage: The data is stored in a central location.
• Analysis: The data is analyzed to extract insights.
There are two major types of data pipeline widely in use: batch and real-time data processing.
Batch processing is when data is collected over a period and processed all at once. This is typically done for large amounts of data that do not need to be processed in real-time. For example, a company might batch process their sales data once a month to generate reports.
Real-time processing is when data is processed as soon as it is received. This is typically done for data that needs to be acted on immediately, such as financial data or sensor data. For example, a company might use real-time processing to monitor their stock prices or to detect fraud.
The type of data processing that is used depends on the specific needs of the organization. For example, a company that needs to process large amounts of data might use batch processing, while a company that needs to process data in real-time might use real-time processing.
Figure 5-5. A generic batch and stream-based data processing in a modern data warehouse
Batch Processing in Detail
Batch processing is a key component of the data processing workflow, involving a series of stages from data collection to data delivery (Figure 5-5). The process begins with data collection, where data is gathered from various sources. Once collected, the data undergoes a data cleaning phase to eliminate errors and inconsistencies, ensuring its reliability for further analysis. The next step is data transformation, where the data is formatted and structured in a way that is suitable for analysis, making it easier to extract meaningful insights.
After transformation, the data is stored in a centralized location, such as a database or data warehouse, facilitating easy access and retrieval. Subsequently, data analysis techniques are applied to extract valuable insights and patterns from the data,
supporting decision-making and informing business strategies. Finally, the processed data is delivered to the intended users or stakeholders, usually in the form of reports, dashboards, or visualizations.
One of the notable advantages of batch processing is its ability to handle large amounts of data efficiently. By processing data in batches rather than in real-time, it enables better resource management and scalability. Batch processing is particularly beneficial for data that doesn’t require immediate processing or is not time-sensitive, as it can be scheduled and executed at a convenient time.
However, there are also challenges associated with batch processing. Processing large volumes of data can be time-consuming, as the processing occurs in sets or batches. Additionally, dealing with unstructured or inconsistent data can pose difficulties during the transformation and analysis stages. Ensuring data consistency and quality becomes crucial in these scenarios.
In conclusion, batch processing plays a vital role in the data processing workflow, encompassing data collection, cleaning, transformation, storage, analysis, and delivery. Its benefits include the ability to process large amounts of data efficiently and handle non-time-sensitive data. Nonetheless, challenges such as processing time and handling unstructured or inconsistent data need to be addressed to ensure successful implementation.
Example of Batch Data Processing with Databricks
Consider a retail company that receives sales data from multiple stores daily. To analyze this data, the company employs a batch data processing pipeline. The pipeline is designed to ingest the sales data from each store at the end of the day. The data is collected as CSV files, which are uploaded to a centralized storage system. The batch data pipeline is scheduled to process the data every night.
The pipeline starts by extracting the CSV files from the storage system and transforming them into a unified format suitable for analysis. This may involve merging, cleaning, and aggregating the data to obtain metrics such as total sales, top-selling products, and customer demographics. Once the transformation is complete, the processed data is loaded into a data warehouse or analytics database.
Analytics tools, such as SQL queries or business intelligence (BI) platforms, can then be used to query the data warehouse and generate reports or dashboards. For example, the retail company can analyze sales trends, identify popular products, and gain insights into customer behavior. This batch data processing pipeline provides valuable business insights daily, enabling data-driven decision-making.
In this use case, we will explore a scenario where batch processing of data is performed using CSV and flat files. The data will be processed and analyzed using Databricks, a cloud-based analytics platform, and stored in Blob storage.
Requirements:
• CSV and flat files containing structured data • Databricks workspace and cluster provisioned • Blob storage account for data storage
Steps:
Data Preparation:
• Identify the CSV and flat files that contain the data to be processed.
• Ensure that the files are stored in a location accessible to Databricks.
Databricks Setup:
• Create a Databricks workspace and provision a cluster with appropriate configurations and resources.
• Configure the cluster to have access to Blob storage.
Data Ingestion:
• Using Databricks, establish a connection to the Blob storage account.
• Write code in Databricks to read the CSV and flat files from Blob storage into Databricks’ distributed file system (DBFS) or as Spark dataframes.
Data Transformation:
• Utilize the power of Spark and Databricks to perform necessary data transformations.
• Apply operations such as filtering, aggregations, joins, and any other required transformations to cleanse or prepare the data for analysis.
Data Analysis and Processing:
• Leverage Databricks’ powerful analytics capabilities to perform batch processing on the transformed data.
• Use Spark SQL, DataFrame APIs, or Databricks notebooks to run queries, aggregations, or custom data processing operations.
Results Storage:
• Define a storage location within Blob storage to store the processed data.
• Write the transformed and processed data back to Blob storage in a suitable format, such as Parquet or CSV.
Data Validation and Quality Assurance:
• Perform data quality checks and validation on the processed data to ensure its accuracy and integrity.
• Compare the processed results with expected outcomes or predefined metrics to validate the batch processing pipeline.
Monitoring and Maintenance:
• Implement monitoring and alerting mechanisms to track the health and performance of the batch processing pipeline.
• Continuously monitor job statuses, data processing times, and resource utilization to ensure efficient execution.
Scheduled Execution:
• Set up a scheduled job or workflow to trigger the batch processing pipeline at predefined intervals.
• Define the frequency and timing based on the data refresh rate and business requirements.
Data pipelines offer numerous benefits and advantages that enable organizations to effectively manage and process their data. By leveraging data pipelines, organizations can unlock the full potential of their data assets and gain a competitive edge in the following ways:
• Improved Data Processing Speed and Efficiency: Data pipelines streamline the data processing workflow, automating repetitive tasks and reducing manual intervention. This leads to significant improvements in data processing speed and efficiency. By eliminating time-consuming manual processes, organizations can accelerate data ingestion, processing, and delivery, enabling faster insights and decision-making.
• Scalability and Handling of Large Data Volumes: With the exponential growth of data, organizations need scalable solutions to handle the increasing data volumes. Data pipelines provide a scalable architecture that can accommodate large amounts of data, ensuring efficient processing without compromising performance. They can handle data in various formats, such as structured, semi- structured, and unstructured, allowing organizations to process and analyze diverse data sources effectively.
• Standardization and Automation of Data Workflows: Data pipelines promote standardization and automation of data workflows, ensuring consistency and repeatability in data processing. By defining clear data pipeline stages, transformations, and validations, organizations can establish standardized processes for handling data. Automation reduces the risk of errors, improves data quality, and enhances productivity by eliminating manual intervention and enforcing predefined rules and best practices.
• Enables Real-Time and Near-Real-Time Analytics: Traditional batch processing methods often involve delays between data collection and analysis. Data pipelines enable real-time and near-real-time analytics by processing data as it arrives, allowing organizations to gain insights and make timely decisions. Real-time data processing is crucial in domains such as fraud detection, stock trading, IOT sensor data analysis, and customer engagement, where immediate action is required based on fresh data.
• Facilitates Data Integration and Consolidation: Organizations typically have data spread across multiple systems, databases, and applications. Data pipelines provide a mechanism for efficiently integrating and consolidating data from diverse sources into a unified view. This integration enables organizations to derive comprehensive insights, perform cross-system analysis, and make informed decisions based on a holistic understanding of their data.
• Enhanced Data Quality and Consistency: Data pipelines facilitate the implementation of data validation and cleansing techniques, improving data quality and consistency. By applying data quality checks, organizations can identify and address data anomalies, inconsistencies, and errors during the data processing stages. This ensures that downstream analytics and decision-making processes are based on accurate and reliable data.
• Enables Advanced Analytics and Machine Learning: Data pipelines play a critical role in enabling advanced analytics and machine learning initiatives. By providing a structured and automated process for data preparation and transformation, data pipelines ensure that data is the right format and quality for feeding into analytics models. This enables organizations to leverage machine learning algorithms, predictive analytics, and AI-driven insights to derive actionable intelligence from their data.
• Cost Efficiency and Resource Optimization: Data pipelines optimize resource utilization and reduce operational costs. By automating data processing tasks, organizations can minimize manual effort, streamline resource allocation, and maximize the utilization of computing resources. This helps to optimize costs associated with data storage, processing, and infrastructure, ensuring that resources are allocated efficiently based on actual data processing needs.
Data pipelines find applications in various industries and domains, enabling organizations to address specific data processing needs and derive valuable insights. Let’s explore some common use cases where data pipelines play a pivotal role:
• E-commerce Analytics and Customer Insights: E-commerce businesses generate vast amounts of data, including customer interactions, website clicks, transactions, and inventory data. Data pipelines help collect, process, and analyze this data in real-time, providing valuable insights into customer behavior, preferences, and trends. These insights can be used for personalized marketing campaigns, targeted recommendations, inventory management, and fraud detection.
• Internet of Things (IOT) Data Processing: With the proliferation of IOT devices, organizations are collecting massive volumes of sensor data. Data pipelines are essential for handling and processing this continuous stream of data in real-time. They enable organizations to monitor and analyze IOT sensor data for predictive maintenance, anomaly detection, environmental monitoring, and optimizing operational efficiency.
• Financial Data Processing and Risk Analysis: Financial institutions deal with a vast amount of transactional and market data. Data pipelines streamline the processing and analysis of this data, enabling real-time monitoring of financial transactions, fraud detection, risk analysis, and compliance reporting. By leveraging data pipelines, financial organizations can make informed decisions, detect anomalies, and mitigate risks effectively.
• Health Care Data Management and Analysis: The health-care industry generates massive amounts of data, including patient records, medical imaging, sensor data, and clinical trial results. Data pipelines assist in collecting, integrating, and analyzing this data to support clinical research, patient monitoring, disease prediction, and population health management. Data pipelines can also enable interoperability among various health-care systems and facilitate secure data sharing.
• Social Media Sentiment Analysis and Recommendation Engines:
Social media platforms generate vast amounts of user-generated content, opinions, and sentiments. Data pipelines play a critical role in collecting, processing, and analyzing this data to derive insights into customer sentiment, brand reputation, and social trends. Organizations can leverage these insights for sentiment analysis, social media marketing, personalized recommendations, and social listening.
• Supply Chain Optimization: Data pipelines are instrumental in optimizing supply chain operations by integrating data from various sources, such as inventory systems, logistics providers, and sales data. By collecting, processing, and analyzing this data, organizations can gain real-time visibility into their supply chain, optimize inventory levels, predict demand patterns, and improve overall supply chain efficiency.
• Fraud Detection and Security Analytics: Data pipelines are widely used in fraud detection and security analytics applications across industries. By integrating and processing data from multiple sources, such as transaction logs, access logs, and user behavior data, organizations can detect anomalies, identify potential security threats, and take proactive measures to mitigate risks.
• Data Warehousing and Business Intelligence: Data pipelines play a crucial role in populating data warehouses and enabling business intelligence initiatives. They facilitate the extraction, transformation, and loading (ETL) of data from various operational systems into a centralized data warehouse. By ensuring the timely and accurate transfer of data, data pipelines enable organizations to perform in- depth analyses, generate reports, and make data-driven decisions.
These are just a few examples of how data pipelines are utilized across industries. The flexibility and scalability of data pipelines make them suitable for diverse data processing needs, allowing organizations to leverage their data assets to gain valuable insights and drive innovation.
In conclusion, data pipelines offer a wide range of benefits and advantages that empower organizations to efficiently manage and process their data. From improving data processing speed and scalability to enabling real-time analytics and advanced insights, data pipelines serve as a catalyst for data-driven decision-making and innovation. By embracing data pipelines, organizations can leverage the full potential of their data assets, derive meaningful insights, and stay ahead in today’s data-driven landscape.
There are four distinct principles when it comes to implementing DataMesh, as follows:
1. Domain-driven ownership 2. Data-as-a-product
3. Self-serve data platform
4. Federated computational governance Let us discuss each of these areas in detail.
Domain-driven Ownership
As per the domain-driven ownership principle, the domain teams are owners of their respective data. For example, there can be different teams, like sales, human resources, information technology, and so on, in an enterprise. Each of these teams will generate, store, process, govern, and share their data. The ownership of the data will remain within the domain boundary. In other words, data is organized per domain. We get the following benefits by adopting this principle:
• The data is owned by domain experts who understand the data well.
This process enhances the quality of data and the way it is stored,used, and shared.
• The data structure is modeled as per the needs of the domain. So, the data schema is realistic.
Data gets decentralized when it is being managed by the corresponding domain.
This ensures better scalability and flexibility when it comes to management of data.
Data-as-a-Product
Data are often treated as a byproduct for the application. As a result, data are kept in silos, not stored properly, not utilized to the fullest, and thus the collaboration and sharing of the data is limited. However, when the data are treated as a product, they are treated at par with the application and are exposed for consumption to other teams and parties as a product. This process increases the data quality and makes data management efficient. A consumer-centric approach is followed that ensures that the data produced are consumable by the end consumers. This helps in maintaining well-defined data contracts that clearly define how the data will be accessed, shared, consumed, and integrated by other consumers in the enterprise. This makes the data teams autonomous, resulting in the team producing the data being responsible for that data and exposing it as a product.
Now, let us use the data mesh principles we discussed and design a data mesh on Azure. We will take a use case of an enterprise consisting of finance, sales, human resources, procurement, and inventory departments. Let us build a data mesh that can expose data for each of these departments. The data generated by these departments are consumed across each of them. The item details in the inventory department are used in the sales department while generating an invoice when selling items to a prospective customer. Human resources payroll data and rewards and recognition data are used in the finance department to build the balance sheet for the department. The sales data from the sales department and the procurement data from the procurement department are used in the finance department when building the balance sheet for the company.
Figure 4-2. As-is data architecture
Figure 4-2 represents the as-is data architecture for the enterprise. The data are managed by a central data store. Each of the departments stores their data in the central data store. The finance department consumes data from human resources, sales, and procurement. Inventory data is consumed by the sales department. The central data team manages the data warehouse that stores and manages the central data. Each of these departments has a dependency on the central data team for data management, storage, and governance. The data engineers in the central data team are not experts for these domains. They do not understand how this data is produced or consumed. However, the engineers in the central data team are experts in storing and managing these data. The following are the steps to design a data mesh architecture on Azure:
Figure 4-5 depicts the procurement domain product. There are different suppliers from whom the organization procures data. The data comprise procured product details, prices, and other necessary information. The data is directly ingested from the supplier system, and the data format varies from supplier to supplier. Cosmos DB database is used to store the supplier data. The Synapse pipeline reads the data from the supplier database, transforms the data into a consumable format, and then stores the data in
an Azure Data Lake Gen2. Synapse pipeline can use the Copy activity to fetch the data from HR database and put it in the Azure Data Lake Gen2 after processing the data.
Azure Data Lake Gen2 exposes the data to other domain products for consumption. Operational logs and metrics for Synapse are stored in Azure Monitor. Azure Monitor can be used to diagnose the run-time issues in the product. Azure Purview scans data capture and stores the data lineage, data schema, and other metadata information that can help in discovering the data.
Create Data Product for Sales Domain
Let us now design the sales domain product.
Figure 4-6. Sales domain product
Figure 4-6 depicts the sales domain product. There are different sales channels for the enterprise. There are distributors that sell the products. There are also B2C sales for the products in the enterprise e-commerce portal. The sales data from multiple channels are ingested into Azure Databricks through Event Hub. The volume of data that is getting ingested is huge. So, the data are stored in Azure Databricks. Data from the inventory domain droduct are consumed by the sales domain product. Spark pipelines transform the data and put the data in an Azure Data Lake Gen2, which exposes the data to other domain products for consumption. Operational logs and metrics for Synapse are stored in Azure Monitor. Azure Monitor can be used to diagnose the run-time issues in the product. Azure Purview scans data capture and stores the data lineage, data schema, and other metadata information that can help in discovering the data.
Figure 4-7 depicts the finance domain product, which consumes data from the inventory, sales, and procurement domain products. The volume of data is huge. Azure Databricks is used to manage the huge amounts of data being ingested. Spark pipelines transform the data and put it in an Azure Data Lake Gen2, which exposes the data to other domain products for consumption. Operational logs and metrics for Synapse are stored in Azure Monitor. Azure Monitor can be used to diagnose the run-time issues in the product. Azure Purview scans data capture and stores the data lineage, data schema, and other metadata information that can help in discovering the data.
Create Self-Serve Data Platform
The self-serve data platform consists of the following components, which give data producers and consumers a self-service experience:
• Data mesh experience plane
• Data product experience plane
• Infrastructure plane
Figure 4-8 depicts the Azure services that can be used to build the self-serve data platform.
Figure 4-8. Self-serve platform components
Data Mesh Experience Plane
The data mesh experience plane helps customers to discover the data products, explore metadata, see relationships among the data, and so on. The data mesh plane is used by the data governance team to ensure data compliance and best practices by giving them a way to audit and explore the data. Azure Purview can scan the data products and pull the data information from the products, like metadata, data schemas, data lineage, etc.
Data Product Experience Plane
The data product experience plane helps the data producers to add, modify, and delete data in the domain products. Azure Functions can be used in data products to allow data producers to add, modify, or delete the data in the domain product. The Purview catalog in the domain product will expose the data schema definitions and metadata, allowing the data producers to work with the data in the data domain.
Infrastructure Plane
The infrastructure plane helps in self-provisioning of the data domain infrastructure. You can use Azure Resource Manager APIs exposed through Azure Functions to create and destroy infrastructure for the data domain products.
Federated Governance
Azure Policies can be used to bring in federated governance. Data landing zones can be created using Azure Policies that can control and govern API access and activities for the data producers and consumers. Azure Functions and Azure Resource Manager APIs can also be used for governance purposes. Azure Monitor alerts can be used for generating governance alerts and notifications.
Modern data orchestration encompasses a range of techniques and practices that enable organizations to manage and integrate data effectively in today’s complex data ecosystems.
They go through cycles of data ingestion, data transformation, defining models and measures, and finally serving the data via different modes and tools based on latency, visibility, and accessibility requirements for representation (Figure 5-2).
Figure 5-2. Typical data orchestration flow in data analytics and engineering
Here are key aspects to understand about modern data orchestration:
• Data Integration and ETL: Data integration remains a fundamental component of data orchestration. It involves combining data from disparate sources, such as databases, cloud services, and third-party APIs, into a unified and consistent format. Extract, transform, load (ETL) processes are commonly employed to extract data from source systems, apply transformations or cleansing, and load it into target systems.
• Data Pipelines: Data pipelines provide a structured and automated way to process and move data from source to destination. A data pipeline typically consists of a series of interconnected steps that perform data transformations, enrichment, and validation. Modern data pipeline solutions often leverage technologies like Apache Kafka, Apache Airflow, or cloud-based services such as AWS Glue, Google Cloud Dataflow, or Azure Data Factory.
• Event-Driven Architectures: Event-driven architectures have gained popularity in data orchestration. Instead of relying solely on batch processing, event-driven architectures enable real-time data processing by reacting to events or changes in the data ecosystem. Events, such as data updates or system notifications, trigger actions and workflows, allowing for immediate data processing, analytics, and decision-making.
• Stream Processing: Stream processing focuses on analyzing and processing continuous streams of data in real-time. It involves handling data in motion, enabling organizations to extract insights, perform real-time analytics, and trigger actions based on the data flow. Technologies like Apache Kafka, Apache Flink, and Apache Spark Streaming are commonly used for stream processing.
• Data Governance and Metadata Management: Modern data orchestration also emphasizes data governance and metadata management. Data governance ensures that data is properly managed, protected, and compliant with regulations. Metadata management involves capturing and organizing metadata, which provides valuable context and lineage information about the data, facilitating data discovery, understanding, and lineage tracking.
• Cloud-Based Data Orchestration: Cloud computing platforms offer robust infrastructure and services for data orchestration. Organizations can leverage cloud-based solutions to store data, process it at scale, and access various data-related services, such as data lakes, data warehouses, serverless computing, and managed ETL/ELT services. Cloud platforms also provide scalability, flexibility, and cost-efficiency for data orchestration workflows.
• Automation and Workflow Orchestration: Automation plays a vital role in modern data orchestration. Workflow orchestration tools, such as Apache Airflow, Luigi, or commercial offerings like AWS Step Functions or Azure Logic Apps, allow organizations to define, schedule, and execute complex data workflows. These tools enable task dependencies, error handling, retries, and monitoring, providing end-to-end control and visibility over data processing pipelines.
• Data Quality and DataOps: Data quality is a critical aspect of modern data orchestration. Organizations focus on ensuring data accuracy, consistency, completeness, and timeliness throughout the data lifecycle. DataOps practices, which combine data engineering, DevOps, and Agile methodologies, aim to streamline and automate data-related processes, improve collaboration between teams, and enhance data quality.
Figure 5-3. A generic, well-orchestrated data engineering and analytics activity
A well-orchestrated data engineering model for modern data engineering involves key components and processes (Figure 5-3). It begins with data ingestion, where data from various sources, such as databases, APIs, streaming platforms, or external files, is collected and transformed using ETL processes to ensure quality and consistency. The ingested data is then stored in suitable systems, like relational databases, data lakes, or distributed file systems. Next, data processing takes place, where the ingested data is transformed, cleansed, and enriched using frameworks like Apache Spark or SQL-based transformations. They are further segregated into models and measures, sometimes using OLAP or tabular fact and dimensions to be further served to analytic platforms such as power BI for business-critical reporting.
Data orchestration is crucial for managing and scheduling workflows, thus ensuring seamless data flow. Data quality and governance processes are implemented to validate, handle anomalies, and maintain compliance with regulations. Data integration techniques bring together data from different sources for a unified view, while data security measures protect sensitive information. Finally, the processed data is delivered to end users or downstream applications through various means, such as data pipelines, reports, APIs, or interactive dashboards. Flexibility, scalability, and automation are essential considerations in designing an effective data engineering model.
One of the key aspects to grasp in data orchestration is the cyclical nature of activities that occur across data layers. These cycles of activity play a crucial role in determining the data processing layers involved in the storage and processing of data, whether it is stored permanently or temporarily.
Data processing layers and their transformation in the ETL (extract, transform, load) orchestration process play a crucial role in data management and analysis. These layers, such as work area staging, main, OLAP (online analytical processing), landing, bronze, gold, silver, and zones, enable efficient data processing, organization, and optimization.
The evolution of data processing layers has had a significant impact on data orchestration pipelines and the ETL (extract, transform, load) process. Over time, these layers have become more sophisticated and specialized, enabling improved data processing, scalability, and flexibility.
