GCP Data Engineering · Chapter 05

Building a Data Lake with Dataproc

A data lake stores raw data first and asks questions later. On GCP the classic way to process it is Dataproc, Google's managed Hadoop and Spark. The big idea of the chapter is not Hadoop itself but what the cloud lets you do with it: throw the cluster away after each job.

0 of – checks answered

What makes a data lake a lake

Three abilities define it, and they line up with Hadoop, which is why the two ideas grew up together.

To a data engineer, everything is a file

Structured, semi-structured (CSV, JSON, XML) and unstructured (images, audio, logs) sound like different categories, but in a lake they are all just files. There is nothing to prepare for each kind, and no table has to exist first.

Mindset: Files, not tables
On GCP: Cloud Storage (or HDFS on a Hadoop cluster)
Contrast: A warehouse needs a schema before loading

Hadoop in one screen

You do not need the whole Hadoop zoo. Learn how HDFS stores data and how Spark processes it; the rest follows.

The distributed file system

Splits files into parts and spreads them over many disks so terabytes to petabytes fit. Worker nodes hold the data; the master node holds the index of which part lives where, so you always address the master.

Idea: One logical file system over many disks
On GCP: Usually replaced by Cloud Storage
Access: hdfs dfs -ls, -put, -mkdir

Where should this data live?

01 You want output files that BigQuery, AI tools and other GCP services can read, with the least maintenance.
02 Your output is fully structured with a clear schema and analysts will query it with SQL.
03 Job latency is critical and you are willing to run a cluster with SSD-backed disks.

The book's rule of thumb: HDFS is generally faster than Cloud Storage, and Cloud Storage faster than BigQuery, but the gap often matters less than convenience. Structured data with a clear schema goes to BigQuery; flexible files go to Cloud Storage; time-critical work can use HDFS on SSD.

Spark: RDDs, DataFrames and laziness

Spark is the framework you actually write. Two abstractions matter, and one behaviour surprises everyone.

A list spread across machines

A Resilient Distributed Dataset is like a Python list whose items are spread over several machines’ memory. "Resilient" means it survives a machine failing by recomputing from elsewhere. You cannot simply print an RDD; it is an abstraction, not a value.

Use when: No clear schema or delimiter yet
Analogy: Python list, distributed
Made with: sc.parallelize(...) or sc.textFile(...)

A distributed table

A layer on top of RDDs, like pandas but distributed. It carries a schema, can be queried with SQL, and Spark can optimise it far better than a raw RDD. Best practice: convert RDDs to DataFrames whenever you can.

Use when: You have (or can add) a schema
Analogy: pandas, distributed
Bonus: Register a temp view, then use spark.sql(...)

Lazy evaluation: try it

This is the book's example: sum column two of a small CSV. Define each transformation in order and watch what is computed. Then call the action.

RDD or DataFrame?

01 Raw web-server log lines that have no clean delimiter or schema yet, and need parsing first.
02 After parsing, you want to run SQL over the ip, date, method and url columns.
03 A well-formed CSV with a known schema.

Dataproc: managed Hadoop

Dataproc runs the machines, operating system and Hadoop install for you, and a cluster is ready in a few minutes. You still choose sizes and versions. Optional settings exist mostly for optimisation.

Add and remove workers automatically

Grows and shrinks the number of worker nodes based on YARN resource availability.

Good for: Uneven workloads
Rule: Optimise after it works

Implement first, optimise later: use these only when you find a real need.

	Where the job reads and writes	Trade-off
HDFS	On the cluster	Fastest, but data lives and dies with the cluster
Cloud Storage	Outside the cluster	Compatible with everything, and survives cluster deletion
BigQuery	Via the Spark-BigQuery connector	Best for structured, schema-clear output

Ephemeral clusters

Spark barely needs HDFS if the data lives in Cloud Storage or BigQuery. So the cluster need only exist while a job runs. Create it, run the job, delete it. That is impossible on-premises and normal on GCP.

What does it save? Try your numbers

Cluster cost / hour ($) Jobs per day Job runtime (hours) Startup (minutes)

Always-on cluster, 30 days
Ephemeral clusters, 30 days
Saving
Cluster busy

Illustrative only: real cost depends on machine types, region and current prices (use the pricing calculator). Note the startup time you pay for on every run, which is the price of throwing the cluster away.

Ephemeral or permanent?

01 A nightly Spark job reads and writes Cloud Storage. You want dedicated resources and no idle cost.
02 Analysts use Jupyter notebooks and Hue on the cluster all day.
03 Hive tables stored in HDFS are the lake’s main data store.
04 One job needs a specific OS version and library set that would conflict with other jobs.

Automating the create-run-delete cycle

01 One simple PySpark job that just needs a cluster created before it and deleted after.
02 A pipeline with many branches, backfill logic and dependencies on BigQuery and Cloud Storage tasks.

A Dataproc workflow template can define a managed cluster with a TTL, run one or more jobs as a small DAG, and delete the cluster afterwards. With Composer you chain create_cluster >> pyspark_task >> delete_cluster. Name the cluster with the run date, and set an idle-delete time so a failed DAG cannot leave a cluster running.

Hands-on: a lake on Dataproc

Clusters and Composer environments cost money while they exist. Work in one sitting and delete everything at the end.

Create a small Dataproc cluster

Costs money while it exists. Use small machine types, one master and two workers, enable the Component Gateway, and remember to delete it.


						gcloud dataproc clusters create my-cluster \
  --region us-central1 --num-workers 2 \
  --master-machine-type n1-standard-2 --worker-machine-type n1-standard-2 \
  --enable-component-gateway

SSH into the master node and list HDFS
This is the Hadoop node’s Linux, not Cloud Shell. The folders you see live in HDFS, not the local disk.
```
						hdfs dfs -ls /
					
```

Copy a file from Cloud Storage into HDFS

Pull the file to the master node, then put it into HDFS.


						gcloud storage cp gs://YOUR_BUCKET/simple_file.csv .
hdfs dfs -mkdir -p /data/simple_file
hdfs dfs -put simple_file.csv /data/simple_file/

Optional: create a Hive external table over the directory
See how SQL runs directly over a file directory. This is the lake idea in miniature.
In the pyspark shell, sum a column with an RDD
Read the file with sc.textFile, filter out the header, split on commas, map column two to an integer, and call sum(). Confirm nothing computes until the action.
Point the same code at Cloud Storage
Change only the path from an hdfs:// address to a gs:// address. Spark treats them the same.
Write a PySpark job for the web logs and submit it
Parse the log lines with an RDD, make a DataFrame, count requests per article with SQL, and write the result. Upload the file to Cloud Storage and submit it.
```
						gcloud dataproc jobs submit pyspark gs://YOUR_BUCKET/code/pyspark_job.py \
  --cluster=my-cluster --region=us-central1
					
```
Write the result to BigQuery instead
Use the Spark-BigQuery connector with a temporary bucket, and include its jar when you submit the job.
Run the job through a workflow template with a managed (ephemeral) cluster
Set a TTL, choose a single-node managed cluster, add the job, run it, and watch the cluster appear and then vanish.
Do the same with Composer: create cluster, submit job, delete cluster
Chain three Dataproc operators. Name the cluster with the run date and set an idle-delete TTL as a safety net.

Delete every cluster and environment you created

The step that protects your budget.


						gcloud dataproc clusters delete my-cluster --region us-central1

Key terms

Data lake: Raw files stored first, with storage, compute and schema separate.
Hadoop: An ecosystem of open-source projects for distributed storage and processing.
HDFS: Hadoop’s distributed file system.
Spark / PySpark: A distributed processing framework, and its Python API.
RDD: Resilient Distributed Dataset: a list spread over machines.
DataFrame: A distributed table with a schema, queryable with SQL.
Lazy evaluation: Transformations build a plan; an action runs it.
Dataproc: Google’s managed Hadoop and Spark service.
Ephemeral cluster: A cluster that exists only while jobs run.
Workflow template: A Dataproc feature for simple job dependencies and managed clusters.
Scale up / scale out: A bigger machine, versus more machines.

Check yourself

Pick an answer to see the reasoning straight away.

On GCP, what is usually the best replacement for HDFS as a lake’s storage?

What does "scale out" mean?

You write rdd.filter(...).map(...).map(...) but nothing runs until you call sum(). Why?

Which is a genuine reason NOT to use an ephemeral cluster?

Why give an ephemeral cluster a name suffixed with the run date in Airflow?

Why set an idle-delete time-to-live on a cluster created by a DAG?

Study notes in my own words, following the structure of Adi Widjaja's Data Engineering with Google Cloud Platform (Packt, 2022). GCP changes quickly, so check each service's current documentation before relying on names, prices or console steps.