Google Cloud – Professional Data Engineer Certification Exam Notes

Google Cloud Data Engineer Study Guide

Table of Contents

• Section 1: Designing Data Processing Systems
• Section 2: Ingesting and Processing the Data
• Section 3: Storing the Data
• Section 4: Preparing and Using Data for Analysis
• Section 5: Maintaining and Automating Data Workloads

Section 1: Designing Data Processing Systems (~22%)

1.1 Designing for Security and Compliance

• Identity and Access Management (e.g., Cloud IAM and organization policies)
• Data security (encryption and key management)
• Privacy (e.g., personally identifiable information, and Cloud Data Loss Prevention API)
• Regional considerations (data sovereignty) for data access and storage
• Legal and regulatory compliance

1.2 Designing for Reliability and Fidelity

• Preparing and cleaning data (e.g., Dataprep, Dataflow, and Cloud Data Fusion)
• Monitoring and orchestration of data pipelines
• Disaster recovery and fault tolerance
• Making decisions related to ACID compliance and availability
• Data validation

1.3 Designing for Flexibility and Portability

• Mapping current and future business requirements to the architecture
• Designing for data and application portability (multi-cloud, data residency)
• Data staging, cataloging, and discovery (data governance)

1.4 Designing Data Migrations

• Analyzing current stakeholder needs and planning for desired state
• Planning migration to Google Cloud (e.g., BigQuery, Datastream)
• Designing migration validation strategy
• Designing dataset and table architecture for proper governance

Data services in clouds

Cloud Service Equivalents – GCP vs AWS vs Azure

Category	GCP Service	AWS Equivalent	Azure Equivalent	Purpose
Data Warehouse	BigQuery	Amazon Redshift	Azure Synapse Analytics	Analytical SQL warehouse for large-scale queries.
Data Lake Storage	Cloud Storage (GCS)	Amazon S3	Azure Data Lake Storage (ADLS) Gen2 / Blob Storage	Object storage for raw, semi-/structured data.
Batch/Stream Processing	Dataflow (Apache Beam)	Kinesis Data Analytics / AWS Glue Streaming ETL	Azure Stream Analytics / Data Factory Mapping Data Flows	Serverless batch + streaming ETL.
Managed Hadoop/Spark	Dataproc	Amazon EMR	Azure HDInsight / Synapse Spark Pools	Managed Hadoop/Spark/Hive/Presto clusters.
Data Orchestration	Cloud Composer (Airflow)	AWS MWAA / Step Functions	Azure Data Factory	Workflow orchestration and scheduling.
Real-time Messaging	Pub/Sub	Kinesis Data Streams / SNS	Event Hubs / Service Bus	Pub/sub messaging for real-time ingestion.
ETL/ELT Service	Dataprep (Trifacta)	AWS Glue DataBrew	ADF Wrangling Data Flows	No-code data prep for analytics.
Database Migration	DMS	AWS DMS	Azure Database Migration Service	Migrate DBs to the cloud.
NoSQL Wide-Column Store	Cloud Bigtable	Amazon DynamoDB	Cosmos DB (Cassandra API)	Low-latency, high-throughput NoSQL store.
Machine Learning	Vertex AI	SageMaker	Azure Machine Learning	Managed ML platform for training, deployment, MLOps.