DATA ENGINEERING COURSE

DATA ENGINEERING COURSE

Data Engineering Course

Learn from the Scratch to Essential 

Course Objectives

By the end of this course, students will be able to:

  • Understand the role and responsibilities of a Data Engineer
  • Write Python scripts to read, clean, and transform data
  • Work confidently with relational databases and SQL
  • Understand ETL/ELT concepts and build a basic data pipeline
  • Pull data from APIs and work with common file formats (CSV, JSON, Parquet)
  • Understand cloud storage basics and how data lakes/warehouses work
  • Design simple database schemas using normalization and star schema principles
  • Understand workflow orchestration and build a basic Airflow DAG
  • Use Git for version control of data engineering code
  • Complete a small end-to-end data pipeline project

Course Syllabus

Day 1 — Introduction to Data Engineering. Covers what data engineering is, how the Data Engineer role differs from a Data Analyst or Data Scientist, and an overview of the modern data engineering lifecycle (source systems, ingestion, storage, transformation, serving). Students get a tour of the tools used across the course (Python, SQL, cloud storage, Airflow) and explore a sample raw dataset to understand what “engineering” data actually involves before it reaches an analyst.

Day 2 — Python Fundamentals. Covers variables, data types, operators, conditional statements (if/elif/else), loops (for/while), and writing basic functions. Students write small scripts — such as a script that loops through a list of sales records and filters values above a threshold — to get comfortable with core Python syntax before applying it to real data tasks.

Day 3 — Python for Data Handling. Covers lists, dictionaries, tuples, and sets, along with reading and writing files, including opening CSV and JSON files natively in Python. Students practice loading a raw CSV file, looping through records, and writing cleaned output back to a new file — building the foundation for automated data processing scripts.

Day 4 — Python Libraries: Pandas Basics. Introduces the Pandas library — DataFrames, Series, reading CSV/Excel/JSON into DataFrames, selecting and filtering columns/rows, handling missing values, and basic transformations (renaming columns, changing data types, adding calculated columns). Students clean and reshape a sample sales dataset using Pandas.

Day 5 — Databases & SQL Fundamentals. Covers relational database concepts (tables, records, fields, primary/foreign keys), an introduction to SQL syntax, and basic statements (SELECT, FROM, WHERE, ORDER BY, DISTINCT). Students connect to a sample database and write foundational queries to retrieve and filter records.

Day 6 — SQL Aggregation, Grouping & Joins. Covers aggregate functions (COUNT, SUM, AVG, MIN, MAX), GROUP BY and HAVING, and combining tables with INNER JOIN, LEFT JOIN, and RIGHT JOIN. Students write queries to answer real business questions across multiple related tables — e.g., total sales by region, or orders per customer.

Day 7 — Database Design & Normalization. Covers what normalization is and why it matters, 1NF/2NF/3NF explained simply, designing tables to avoid redundancy, and choosing appropriate primary/foreign keys. Students take a poorly structured spreadsheet and redesign it into a normalized set of related tables.

Day 8 — Introduction to ETL & Data Pipelines. Covers what ETL (Extract, Transform, Load) and ELT mean, batch vs. streaming processing, common pipeline patterns, and where data engineers fit into the overall data flow. Students build a simple Python script that extracts data from a CSV, applies a transformation (cleaning/renaming/filtering), and loads it into a database table — their first mini pipeline.

Day 9 — Working with APIs. Covers what a REST API is, HTTP methods (GET/POST), JSON responses, authentication basics (API keys), and using Python’s requests library to pull data from a public API. Students write a script that fetches live data from an API, parses the JSON response, and stores it as a structured table.

Day 10 — File Formats & Data Storage. Covers common data formats used in data engineering — CSV, JSON, and Parquet — along with when and why each is used, plus the difference between a data lake and a data warehouse. Students convert a dataset between formats and compare file size and read performance.

Day 11 — Introduction to Cloud Storage. Covers cloud computing basics and an introduction to cloud storage services (such as AWS S3 or Azure Blob Storage), including buckets/containers, uploading and downloading files, and basic access permissions. Students upload a cleaned dataset to cloud storage and retrieve it programmatically using Python.

Day 12 — Introduction to Data Warehousing. Covers what a data warehouse is, fact tables vs. dimension tables, and star schema design for analytics-ready data. Students design a simple star schema for a sales dataset, with a central fact table linked to Date, Product, Customer, and Region dimension tables.

Day 13 — Workflow Orchestration with Airflow (Intro). Covers what workflow orchestration is and why pipelines need scheduling, an introduction to Apache Airflow, DAGs (Directed Acyclic Graphs), tasks, and dependencies. Students build a basic Airflow DAG that runs a simple extract-transform-load script on a schedule.

Day 14 — Version Control with Git & Pipeline Assembly. Covers Git fundamentals — repositories, commits, branches, and pushing to GitHub — and how version control fits into a data engineer’s workflow. Students commit their pipeline scripts to a Git repository, then assemble the pieces built across the course (extract → clean → load → store) into a single connected pipeline script.

Day 15 — Final Data Engineering Project. Students receive a raw dataset (e.g., e-commerce order data) and build a complete end-to-end pipeline: (1) Extract — pull data from a CSV file and a sample API; (2) Transform — clean, validate, and reshape the data using Python/Pandas; (3) Load — store the cleaned data into a database table and upload a backup copy to cloud storage; (4) Organize — structure the output using a basic star schema; (5) Orchestrate — wrap the pipeline in a simple Airflow DAG. The course closes with each student walking through their pipeline and explaining each stage: “Where did this data come from, what did you do to it, and where does it end up?”

Core Topics Reference

Python

  • Basics: variables, data types, conditionals, loops, functions
  • Data structures: lists, dictionaries, tuples, sets
  • File handling: reading/writing CSV and JSON
  • Pandas: DataFrames, filtering, cleaning, transformations
  • APIs: requests library, GET requests, parsing JSON

SQL

  • Basic: SELECT, FROM, WHERE, DISTINCT, ORDER BY
  • Aggregation: COUNT, SUM, AVG, MIN, MAX
  • Grouping: GROUP BY, HAVING
  • Joins: INNER JOIN, LEFT JOIN, RIGHT JOIN
  • Design: primary keys, foreign keys, normalization (1NF–3NF)

Data Engineering Concepts

  • ETL / ELT (Extract, Transform, Load)
  • Batch vs. streaming processing
  • Data lakes vs. data warehouses
  • File formats: CSV, JSON, Parquet
  • Fact tables, dimension tables, star schema
  • Workflow orchestration, DAGs, task scheduling (Airflow)

Cloud & Tools

  • Cloud storage basics (buckets/containers, upload/download, permissions)
  • Git & GitHub: commits, branches, version control workflow

 

Final Course Outcome

By the end of the 15-day course, students should have a practical, working data engineering flow:

Raw Data → Python / Pandas (Extract & Clean) → SQL (Store & Query) → Cloud Storage → Star Schema → Orchestrated Pipeline (Airflow)

Students should be able to take a raw dataset from a file or API, clean and transform it with Python, structure it into a proper database schema, store it reliably, and automate the process end-to-end.

Recommended Session Structure

For each 2-hour session:

  • 20–30 minutes — Concept explanation
  • 60–70 minutes — Guided practical work
  • 20–30 minutes — Student exercise / assignment

This keeps the course hands-on and practical rather than theory-heavy, mirroring how data engineers actually build and maintain pipelines on the job.

Total Fees: 75,000/=

Total Duration: 30 Hrs (2 hours  x  15 Classes)
Training Mode: Individual Training, your own timetable

Live Online Classes  or Face to Face Direct Classes with our expert trainers.
Call +94 (0) 722000999 / +94 (0) 755123111 www.iss.lk. Medium : සිංහල / தமிழ் / English