Senior AI Dev Kit
Back to the catalog

skill

GCP data engineering

gcp-data-engineering

Architecture choices for Google Cloud pipelines: lake or warehouse, batch or streaming, Dataform, Composer or Dataflow, and BigQuery cost.

When to use it

You design a new data pipeline on Google Cloud.

Technical description

Guides architecture decisions for data pipelines on Google Cloud — lake/warehouse/lakehouse design, batch vs. streaming, orchestration tool selection (Dataform, Cloud Composer/Airflow, Dataflow/Beam), and BigQuery cost/performance tuning (partitioning, clustering, slots). Use when designing a new pipeline, choosing a GCP data service, or optimizing BigQuery cost/performance.