As covered in the previous video, Apache Beam can be used to create data processing pipelines. The next step is to identify an execution engine to implement those pipelines. When choosing an execution engine for your pipeline code, it may be helpful to consider the following questions. How much maintenance overhead is involved? Is the infrastructure reliable? How is the pipeline scaling handled? How can the pipeline be monitored? Is the pipeline locked into a specific service provider? This brings us to Dataflow. Dataflow is a fully managed service for executing Apache Beam pipelines within the Google Cloud ecosystem. Dataflow handles much of the complexity relating to infrastructure setup and maintenance and is built on Google's infrastructure. This allows for reliable auto-scaling to meet the data pipeline demands. Dataflow is serverless and no-ops, which means no operations. But what does that mean exactly? A no-ops environment is one that doesn't require management from an operations team because maintenance, monitoring, and scaling our automated. Serverless computing is a Cloud computing execution model. This is when Google Cloud, for example, manages infrastructure tasks on behalf of the users. This includes tasks like resource provisioning, performance tuning, and ensuring pipeline reliability. Using a serverless and no-ops solution like Dataflow means that you can spend more time analyzing the insights from your data sets and less time provisioning resources to ensure your pipeline will successfully complete its next cycles. It's designed to be low maintenance. Let's explore the tasks dataflow performs when a job is received. It starts by optimizing a pipeline model's execution graph to remove any inefficiencies. Next, it schedules out distributed work to new workers and scales as needed. After that, it auto-heals any worker faults. From there, it automatically rebalances efforts to most efficiently use its workers. Finally, it outputs data to produce a result. BigQuery is one of many options that data can be outputted to. You'll get some more practice using BigQuery later in this course. By design, you don't need to monitor all of the compute and storage resources that dataflow manages to fit the demand of a streaming data pipeline. Even experienced Java or Python developers will benefit from using dataflow templates, which cover common use cases across Google Cloud products. The list of templates is continuously growing. They can be broken down into three categories. Streaming templates, batch templates, and utility templates. Streaming templates offer processing continuous or real-time data. For example, Pub/Sub to BigQuery, Pub/Sub to Cloud Storage, Datastream to BigQuery, and Pub/Sub to MongoDB. Batch templates offer processing bulk data or batch load data. For example, BigQuery to Cloud Storage, Bigtable to Cloud storage, Cloud Storage to BigQuery, and Cloud Spanner to Cloud Storage. Finally, utility templates address activities related to bulk compression, deletion, and conversion. For a complete list of templates, please refer to the reading list.
