Get in Touch
 Duration 21 hours

Course Outline

Introduction to Apache Airflow

  • Defining workflow orchestration
  • Primary capabilities and advantages of Apache Airflow
  • Enhancements in Airflow 2.x and ecosystem landscape

Architectural Components and Fundamental Concepts

  • Scheduler, web server, and worker processes
  • Directed Acyclic Graphs (DAGs), tasks, and operators
  • Execution models and backends (Local, Celery, Kubernetes)

Deployment and Configuration

  • Implementing Airflow in local and cloud infrastructures
  • Configuring Airflow with various executor types
  • Establishing metadata databases and external connections

Utilizing the Airflow User Interface and Command Line

  • Examining the Airflow web dashboard
  • Tracking DAG executions, task status, and logs
  • Leveraging the Airflow CLI for administrative functions

Development and Management of DAGs

  • Constructing DAGs utilizing the TaskFlow API
  • Deploying operators, sensors, and hooks
  • Governing dependencies and scheduling frequencies

Connecting Airflow with Data and Cloud Platforms

  • Linking to databases, APIs, and message queuing systems
  • Executing ETL pipelines via Airflow
  • Cloud integrations: AWS, GCP, and Azure operators

Monitoring and Observability

  • Task logging and real-time system monitoring
  • Metrics collection using Prometheus and Grafana
  • Configuring alerting and notifications via email or Slack

Securing Apache Airflow Deployments

  • Implementing Role-Based Access Control (RBAC)
  • Authentication protocols: LDAP, OAuth, and SSO
  • Secrets management using Vault and cloud-based secret stores

Scaling Apache Airflow Operations

  • Managing parallelism, concurrency, and task queues
  • Utilizing CeleryExecutor and KubernetesExecutor
  • Deploying Airflow on Kubernetes using Helm

Best Practices for Production Environments

  • Version control and CI/CD pipelines for DAGs
  • Testing and debugging DAG implementations
  • Ensuring reliability and performance at scale

Troubleshooting and Performance Optimization

  • Diagnosing failed DAGs and tasks
  • Enhancing DAG execution performance
  • Identifying common errors and prevention strategies

Conclusion and Subsequent Actions

Requirements

  • Proficiency in Python programming
  • Familiarity with data engineering or DevOps principles
  • Comprehension of ETL or workflow orchestration processes

Intended Audience

  • Data scientists
  • Data engineers
  • DevOps and infrastructure engineers
  • Software developers

Number of participants


Price per participant

Runs with a minimum of 4 + people. For 1-to-1 or private group training, request a quote.

Testimonials (7)

The instructor adapted the training to the participants’ level and responded to all questions. He was very communicative, and it was easy to interact with him. I really appreciated the format of the training, which included many practical exercises. Overall, it was a very engaging and well-organized session.

Jacek Chlopik - ZAKLAD UBEZPIECZEN SPOLECZNYCH
Course - Apache Airflow: Building and Managing Data Pipelines

The training was spot on. Very useful theory and exercices.

Vladimir - PUBLIC COURSE
Course - Apache Airflow

The training was spot on in all aspects. Usefull theoretical aspects and exercises.

Vladimir - PUBLIC COURSE
Course - Apache Airflow

The training was spot on in all aspects. Usefull theoretical aspects and exercises.

Vladimir - PUBLIC COURSE
Course - Apache Airflow

The training was spot on in all aspects. Usefull theoretical aspects and exercises.

Vladimir - PUBLIC COURSE
Course - Apache Airflow

The training was spot on in all aspects. Usefull theoretical aspects and exercises.

Vladimir - PUBLIC COURSE
Course - Apache Airflow

The training was spot on in all aspects. Usefull theoretical aspects and exercises.

Vladimir - PUBLIC COURSE
Course - Apache Airflow

Upcoming Courses

Apache Airflow: Building and Managing Data Pipelines

2026-11-06 09:30
21 hours
Louisville, KY – Regus at MET Building
$ 4054 (Online)
$ 5854 (Classroom)

Apache Airflow: Building and Managing Data Pipelines

2026-11-20 09:30
21 hours
Madison, WI - Regus - Madison East - Park Bank Plaza
$ 4054 (Online)
$ 5854 (Classroom)

Apache Airflow: Building and Managing Data Pipelines

2026-12-04 09:30
21 hours
Bangor, Maine
$ 4054 (Online)
$ 5854 (Classroom)

Apache Airflow: Building and Managing Data Pipelines

2026-12-18 09:30
21 hours
Manchester, NH – Regus at 1087 Elm Street
$ 4054 (Online)
$ 5854 (Classroom)

Apache Airflow: Building and Managing Data Pipelines

2027-01-01 09:30
21 hours
Memphis, TN - Regus at International Dr
$ 4054 (Online)
$ 5854 (Classroom)

Apache Airflow: Building and Managing Data Pipelines

2027-01-15 09:30
21 hours
Kansas City, MO - Regus – Two Pershing Square (Crown Center)
$ 4054 (Online)
$ 5854 (Classroom)

Related Courses

Advanced Apache Airflow: Custom Operators and Plugins

 21 Hours

This live, instructor-led training program, offered in US (online or onsite), is targeted at advanced-level professionals seeking to develop custom operators, sensors, and plugins within Apache Airflow. The curriculum focuses on integrating these components with established data infrastructure to achieve superior automation and monitoring capabilities for government operations.

Upon completion of this training, participants will be equipped to:

  • Engineer custom operators, hooks, and sensors aligned with specific workflow requirements.
  • Architect and deploy Airflow plugins to extend system functionality.
  • Facilitate seamless integration between Airflow workflows and external systems or services.
  • Enhance workflow optimization and oversight through advanced diagnostic tools.
  • Apply best practices for the administration of large-scale Airflow deployments.
Read more...

Mastering Apache Airflow for DevOps

 21 Hours

This instructor-led, live training in US (delivered online or onsite) targets advanced-level professionals aiming to deploy Apache Airflow in cloud environments, establish CI/CD pipelines, and implement rigorous monitoring and logging practices.

Upon completion, participants will be prepared to:

  • Deploy Apache Airflow across cloud and containerized infrastructure.
  • Configure CI/CD pipelines to automate DAG testing and deployment.
  • Integrate monitoring and logging tools to enhance system reliability.
  • Optimize Airflow configurations for performance and scalability.
  • Apply security best practices for managing workflows and environments.
Read more...

Apache Airflow for Data Science: Automating Machine Learning Pipelines

 21 Hours

This instructor-led, live training in US (online or onsite) is designed for intermediate-level professionals who require the ability to automate and manage machine learning workflows, including model training, validation, and deployment, using Apache Airflow.

Upon completion of this training, participants will be prepared to:

  • Configure Apache Airflow for machine learning workflow orchestration.
  • Automate data preprocessing, model training, and validation tasks.
  • Integrate Airflow with machine learning frameworks and tools.
  • Deploy machine learning models using automated pipelines.
  • Monitor and optimize machine learning workflows in production environments.
Read more...

Intermediate Apache Airflow: Optimizing and Scaling Workflows

 21 Hours

This instructor-led live training, conducted in US (online or onsite), is targeted at intermediate-level personnel aiming to optimize workflow performance, manage complex dependencies, and scale Apache Airflow deployments for large datasets and enterprise operations for government.

Upon completion of this training, participants will be able to:

  • Optimize Apache Airflow workflows to ensure improved performance and system reliability.
  • Manage and troubleshoot intricate workflow dependency structures.
  • Utilize advanced Airflow capabilities, including the creation of custom operators and sensors.
  • Scale Apache Airflow infrastructure to accommodate large-scale datasets and distributed system architectures.
  • Apply best practices for monitoring, logging, and security within Airflow environments.
Read more...

Related Categories

Apache Airflow
Apache Airflow