GCP Data Engineering · Chapter 08

Machine Learning on GCP

Machine learning is not a core data engineering skill, but it is closer than it looks: it is a data process with data in and data out. This chapter gives you the vocabulary, the landscape of GCP services, and enough hands-on to talk to data scientists and to build pipelines around their models.

0 of – checks answered

ML in one page, for a data engineer

ML is a data process whose output is a generalised formula for one objective, called a model. Your data skills apply directly. The new part is the vocabulary.

	A data engineer would say	A data scientist says
Input table	A clean table	Dataset
Columns used	Input columns	Features
Column to predict	Output column	Target
How good is it	Data quality check	Accuracy / model performance

Match the term

01 Humidity and temperature: the columns used to train a rain predictor.
02 Rain / No Rain: the value you want to predict for future data.
03 The number of trees in a Random Forest, chosen by trial and error.
04 Next week’s Monday-to-Sunday rain forecast, computed in one go from this week’s readings.
05 An API takes two numbers and instantly returns "Rain" or "No Rain".

A typical script: load data, select features, split into train and test (for example 70/30), train, evaluate accuracy on the test set, save the model, then use it for batch or online prediction. The accuracy step uses predict too, but only to measure the model.

MLOps: the other 90%

Writing the model is the visible part. Everything around it is MLOps: collecting and verifying data, extracting features, configuring and provisioning resources, serving, and monitoring. It spans models, data, orchestration, containers, web services and more.

On GCP, most of that stack is available as managed services, under Vertex AI. You can use all of them or only a few: pipelines without the model registry, or training without the rest. Cost, regional availability and the learning curve decide.

Experiment on your BigQuery data

Vertex AI Notebooks is where you play with data and try models. What you write here is the "ML code". Once you are happy with it, you automate it.

Purpose: Explore and prototype
Output: ML code
Optional?: Yes

Four ways to get a model

The approaches trade off three things: development time, expertise and cost. Ordered by least to most experimenting time: pre-built, AutoML, BigQuery ML, custom.

Call an API, no training

Google has already trained models for tasks such as text in images, translation and recommendations. You call them with a client library and get predictions immediately.

Expertise needed: Almost none
Time: Least
Downside: Limited to what is offered; no edge cases; priced per call, so heavy use is costly

Which would you pick?

01 Read handwritten text from an image with one Python API call and no training.
02 A team with no data scientists needs a credit-default model from a tabular dataset this week.
03 SQL-fluent analysts want to train a model without leaving BigQuery.
04 An experienced team has time, needs a specific algorithm and wants to keep infrastructure spend low.

Try the chooser

Set your situation. This is a simplified rule of thumb based on the chapter, not a substitute for judgement.

Task exists as a Google API? Data science expertise Time available Infrastructure budget

Vertex AI Pipelines

Vertex AI Pipelines is Kubeflow Pipelines, managed. Each step is a Python function wrapped as a container, and the pipeline definition is compiled and submitted to run on Google's infrastructure, not on your machine.

Idea	What to remember
Steps are containers	Each can choose a base image and packages to install, so each step gets exactly the libraries it needs
Dependencies are implicit	Wire one task’s output to another’s input and the order follows, unlike Airflow’s explicit arrows
Steps are stateless	Do not pass in-memory objects like DataFrames; write to Cloud Storage and pass the path
Same location	The pipeline’s bucket should be regional and match where the pipeline runs
Caching	Repeated identical steps may be reused; disable it if you need fresh runs

Training or prediction?

01 Fit a Random Forest on the training rows and write a versioned .joblib file to Cloud Storage.
02 Load the saved model file and score today’s unlabeled rows into prediction.csv.
03 Query features from BigQuery and write them to a file in Cloud Storage for the next step.

Do as much aggregation and filtering as you can in BigQuery, which scales far better than pandas, and keep training and prediction as separate pipelines so they can run at different frequencies.

Hands-on: from a script to a pipeline

AutoML and pipelines cost money, so read each step's pricing and delete what you create.

Copy the credit card default table into your own dataset
Find credit_card_default in the public ML datasets and copy it to a dataset in your project (use the US multi-region).
Train a Random Forest locally
Load three feature columns and the target from BigQuery into pandas, split 70/30, fit with 100 trees, print the accuracy on the test set, and save the model with joblib.
Run a batch prediction with the saved model
Load feature-only rows, load the model, predict. Also write a tiny online-style function that takes one row.
Try a pre-built model: detect text in an image
Upload an image with text to Cloud Storage, install the Vision and Translate client libraries, and call text detection. If you hit an end-user credentials error, set up application default credentials for development.
```
						gcloud auth application-default login
					
```
Optional and paid: train with AutoML
Read the pricing guide first; a training hour is not cheap. Create a tabular Vertex AI dataset from your BigQuery table, choose classification with AutoML, set the target, keep only the features you want and start training.
Run a dummy Vertex AI pipeline
Install the pipeline SDKs, create a regional bucket in the same location as the pipeline, define four small components and their dependencies by wiring outputs to inputs, compile and submit.
Build and run the training pipeline
Step one exports data from BigQuery to a file in Cloud Storage and returns its path. Step two loads that file, trains, and writes the model file back to storage.
Build and run the prediction pipeline
Load data, load the saved model, write prediction.csv. Try turning step caching off and compare.
Delete what you created
Stop and delete endpoints, models, notebook instances and buckets you no longer need.

Key terms

ML model: A generalised formula learned from data for one objective.
Dataset / feature / target: The input table, its input columns, and the column to predict.
Hyperparameter: A setting supplied before training, often found by trial.
Train / test split: Data held out to measure accuracy honestly.
Batch / online prediction: Score many rows at once; score one request in real time.
MLOps: The practices and tools around ML code: data, pipelines, serving, monitoring.
Vertex AI: GCP’s umbrella of managed ML services.
Pre-built model: A Google-trained model you call through an API.
AutoML: Automatic search for a good model within a time budget.
BigQuery ML: Training and using models with SQL.
Kubeflow Pipelines: Open-source ML workflow tool that Vertex AI Pipelines is based on.

Check yourself

Pick an answer to see the reasoning straight away.

In a production ML system, roughly how does the ML code compare with everything around it?

Why do Vertex AI Pipeline steps pass Cloud Storage paths rather than a pandas DataFrame?

Why keep training and prediction in separate pipelines?

You call predict on the test set inside your training script. What is that step for?

Which approach is the fastest to start with, and what is its main limit?

A pipeline step shows a green arrow instead of a tick. What does it mean?

Study notes in my own words, following the structure of Adi Widjaja's Data Engineering with Google Cloud Platform (Packt, 2022). GCP changes quickly, so check each service's current documentation before relying on names, prices or console steps.