Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
Module 1: Big Data & Spark Foundations
- Examining the Big Data ecosystem and the specific function of Spark within contemporary data infrastructure
- Analyzing Spark architecture components, including the driver, executors, cluster managers, lazy evaluation, DAG construction, and execution planning
- Distinguishing between RDD and DataFrame APIs to determine appropriate usage for specific operational tasks
- Establishing SparkSession instances and reviewing the fundamentals of application configuration
Module 2: PySpark DataFrames
- Implementing data ingestion and output operations from enterprise sources, covering CSV, JSON, Parquet, and Delta formats
- Executing PySpark DataFrame operations, including transformations, actions, column expressions, filtering, joins, and aggregations
- Deploying advanced processing techniques such as window functions, timestamp management, and nested data handling
- Integrating data quality validations and developing reusable, maintainable PySpark code standards
Module 3: Kubernetes & Docker Fundamentals for Spark Users
- Explaining the relationship between Docker images and Kubernetes pods and their correlation to Spark driver and executor processes, focusing on cluster consumption rather than administration
- Detailing the execution flow of Spark jobs on Kubernetes, including pod scheduling, resource constraints, and session parameter mapping
- Defining the scope of control via Spark session parameters versus platform team cluster provisioning, and analyzing the impact on tuning strategies
- Hands-on exercise: Reviewing a live Spark-on-Kubernetes job in the sandbox environment to identify driver and executor pods
Module 4: Spark Session Configuration on Kubernetes — Deep Dive
- Evaluating the trade-offs between executor count and size, specifically regarding memory versus core allocation
- Configuring driver and executor memory allocation, overhead, and the corresponding resource requirements for pods
- Assessing dynamic allocation behavior on Kubernetes, identifying scenarios for resource efficiency and limitations
- Hands-on exercise: Executing identical jobs with varying executor and core configurations to analyze runtime and resource utilization
Module 5: Scaling Behaviour & Cost Optimisation
- Analyzing the impact of node additions or removals on job execution, completion time, and resource consumption
- Comparing performance and cost implications of utilizing numerous small executors versus fewer large ones
- Tuning shuffle behavior and partitioning strategies, including estimating the financial impact of configuration changes
- Hands-on exercise: Expanding the sandbox cluster from five to ten nodes to observe effects on job completion and resource usage
Module 6: Efficient Data Ingestion & Partitioning
- Diagnosing the small-files problem, specifically how sources fragmented into 1–5 MB Parquet files impact performance and partitioning
- Implementing repartitioning and coalescing strategies to optimize data handling
- Regulating partition sizes during both read and write operations
- Writing Parquet data efficiently to prevent the recurrence of small-files issues in downstream processes
- Hands-on exercise: Processing a dataset consisting of numerous small Parquet partitions, applying repartitioning strategies, and comparing performance metrics
Module 7: Pandas Memory Management & Failure Diagnosis
- Identifying root causes and symptoms of out-of-memory errors in Pandas environments
- Applying memory-efficient data conversion patterns between Spark and Pandas
- Preventing memory exhaustion when exporting large datasets to CSV
- Utilizing chunked processing and data type optimization for resource-constrained environments
- Hands-on exercise: Replicating a typical Pandas out-of-memory failure and resolving it through chunking and dtype optimization
Module 8: Polars as a Complementary Tool
- Comparing Polars to Pandas, focusing on performance, lazy evaluation, and memory characteristics
- Positioning Polars within modern cloud data stacks and migration strategies, including AWS environments, alongside PySpark and Pandas
- Hands-on exercise: Refactoring a Pandas-centric transformation in Polars to evaluate memory usage and processing velocity
Module 9: Applying Optimisation to ETL & ML Workloads
- Integrating configuration, partitioning, and memory management principles into a realistic ETL pipeline
- Addressing optimization requirements specific to machine learning workloads executed on shared clusters
- Applying a systematic tuning workflow to identify and resolve cost and performance issues
- Hands-on exercise: Completing an end-to-end project involving data loading, transformation, and model training, with independent tuning of Spark configurations
Requirements
Participants should have:
- Practical experience with Python programming, including functions, modules, and basic object-oriented concepts.
- Basic to intermediate experience working with Pandas and tabular data-processing workflows.
- Basic familiarity with PySpark and Spark DataFrames, including reading data, transformations, actions, joins, and aggregations.
- A general understanding of SQL and data-processing concepts, including filtering, grouping, and joining datasets.
- Basic familiarity with Docker and Kubernetes concepts, such as containers, images, and pods. No Kubernetes administration experience is required.
- Basic understanding of common data formats such as CSV, JSON, and Parquet.
Participants do not need to be Kubernetes administrators or infrastructure specialists. The course focuses on how data engineers, developers, and data scientists can understand and optimize their Spark workloads running on Kubernetes from the application and configuration perspective.
Target Audience
This course is designed for professionals who develop, maintain, or optimize data-processing and machine-learning workloads for government using Python and Spark in cloud or containerized environments.
It is particularly suitable for:
- Data Engineers working with PySpark, distributed data processing, and ETL pipelines.
- Data Scientists processing large datasets or running machine-learning workloads with Spark, Pandas, or Polars.
- Python Developers working with data-intensive applications and looking to improve memory efficiency and processing performance.
- Machine Learning Engineers managing data preparation and model-training workloads on shared Kubernetes or cloud environments.
- Analytics Engineers working with large datasets and seeking to improve data-processing efficiency.
- DevOps, Platform, and Cloud Engineers who support Spark workloads on Kubernetes and need to understand how application-level configuration affects resource usage and performance.
- Technical Leads and Solution Architects involved in designing or optimizing modern data-processing platforms.
21 Hours
Testimonials (1)
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.
