GCP Data Engineering · Chapter 11

CI/CD on Google Cloud for Data Engineers

Continuous integration and deployment is a software-engineering habit that data engineering has been slow to adopt. This chapter shows what it is, which GCP pieces do what, and how to make a bad DAG unable to reach production.

0 of – checks answered

Two practices, not one

Continuous integration automatically brings many engineers' code together and checks it. Continuous deployment automatically ships the final code to production. They are related but independent.

	Continuous integration	Continuous deployment
What it automates	Testing, building images, integration checks	Pushing the built software or files to production
Main benefit	Find and locate errors quickly across many contributors	Get a version live fast and handle problems (like rollback) automatically
Common in teams	Often adopted first	Often kept manual because it feels riskier

01 Run the unit tests automatically on every push.
02 Copy the validated DAG file into the Composer environment’s dags/ folder.
03 Build a container image containing the project’s dependencies.
04 Roll back the production application automatically when an issue is detected.

The book's analogy: cooking pasta in your own restaurant needs great cooking; in a big restaurant you also follow the kitchen's system. Your data engineering skills are the cooking. CI/CD is the kitchen system, usually built by DevOps or infrastructure teams, that you need to understand and follow.

The five steps and the GCP services behind them

A typical pipeline runs source code, trigger, build, registry and deploy. Each maps to a GCP service.

Everything starts in a Git repository

Code lives in a Git repository (GitHub, GitLab or Google’s own Cloud Source Repositories). A push, a new tag or a pull request is the event that starts the process.

Book’s GCP service: Cloud Source Repositories
Also works: GitHub, GitLab, any Git host
Trigger events: Push to branch, new tag, pull request

The CI steps

Cloud Build, a serverless CI/CD platform, executes the steps you list in cloudbuild.yaml: build an image, install packages, run tests. If any step fails, the run stops and nothing after it happens.

Defined in: cloudbuild.yaml
Typical CI steps: Build, unit test, integration test
Failure: Stops the pipeline

cloudbuild.yaml key	Meaning
name	The container image the step runs inside (a public builder image, or one you built earlier)
id	A unique identifier for the step
args	The arguments, which become the command line for that image
entrypoint	Overrides the image’s default command, for example to run Python
env	Environment variables for the step, such as where Airflow should find DAGs

Push a change to a Composer repository

The pipeline builds an image with Airflow, tests the DAGs, stores the image, then deploys. Choose what you push and watch the run. Then turn off the validation step to see what CI protects you from.

What you push Validation step

Nothing pushed yet.

DAGs folder in the Composer bucket: Empty

Does CI/CD fit here?

Any code-based data application can use it. Fully UI-driven tools cannot. And you do not always need it.

01 Airflow DAG files in a repository, edited by several engineers.
02 A dashboard built entirely by clicking around in a UI-only reporting tool.
03 Terraform scripts for the data platform.
04 Apache Beam pipeline code for Dataflow and a PySpark job for Dataproc.

01 A solo data engineer building a short-term project that will rarely change.
02 A large team with hundreds of commits a day on shared pipelines.

Hands-on: your first pipeline

The details of Docker and unit tests are secondary. Focus on the shape of the pipeline and on watching a bad change get stopped.

Create a Git repository and clone it into Cloud Shell
The book uses Cloud Source Repositories. If that is unavailable to you, use GitHub or another host connected to Cloud Build.
Copy in the example project
A small Python function, its unit test, a Dockerfile and a cloudbuild.yaml. Do not worry about Docker details; the point is the pipeline.
Read cloudbuild.yaml step by step
Each step names an image to run in, has an id, and passes args like a command line. A test step can change the entrypoint to Python. If it fails, the build stops.
```
						- name: gcr.io/$PROJECT_ID/cicd-basic
  id: Validation Test
  entrypoint: python
  args: ["-m", "unittest", "tests/test_calculate.py"]
					
```
Create a Cloud Build trigger for the repository
Choose "push to a branch" and a branch pattern matching any branch.
Commit and push
Then watch the run in Build History and open each step’s log.
```
						git add .
git commit -m "initial commit"
git push
					
```
Break a unit test on purpose and push again
Change an addition to a multiplication. The Validation Test step should fail and the run stop. Read the assertion error in its log.
Repeat for a Composer DAG
Steps: build an image with Airflow installed, run a DAG test that loads every DAG, optionally push the image as a cache, then sync the dags folder into the environment’s bucket.
Sabotage a DAG and confirm it is not deployed
Replace a valid cron schedule with plain text, push, and watch validation fail with an invalid cron message. Check the bucket: the bad file must not be there.
Delete the triggers, repositories, images and any Composer environment
Environments and stored images cost money.

Key terms

CI: Continuous integration: automatically integrate and test changes.
CD: Continuous deployment: automatically release the final code.
Cloud Build: GCP’s serverless CI/CD platform.
Trigger: A rule that starts a build on a repository event.
cloudbuild.yaml: The file listing a build’s steps.
Container image / registry: A packaged environment, and where it is stored.
Unit test: A small automated check of one piece of code.
Rollback: Reverting to the previous version after a bad release.
DevOps: The practice and team that builds delivery systems like this.

Check yourself

Pick an answer to see the reasoning straight away.

What is the main benefit of continuous integration?

Do you have to implement both CI and CD?

In a Cloud Build pipeline a test step fails. What happens next?

Why is the "push image" step optional when deploying DAGs to Composer?

What does the name parameter of a Cloud Build step specify?

As a data engineer, who usually builds the whole CI/CD system?

Study notes in my own words, following the structure of Adi Widjaja's Data Engineering with Google Cloud Platform (Packt, 2022). GCP changes quickly, so check each service's current documentation before relying on names, prices or console steps.