skill
GCP data engineering
gcp-data-engineering
Architecture choices for Google Cloud pipelines: lake or warehouse, batch or streaming, Dataform, Composer or Dataflow, and BigQuery cost.
When to use it
You design a new data pipeline on Google Cloud.
Technical description
Guides architecture decisions for data pipelines on Google Cloud — lake/warehouse/lakehouse design, batch vs. streaming, orchestration tool selection (Dataform, Cloud Composer/Airflow, Dataflow/Beam), and BigQuery cost/performance tuning (partitioning, clustering, slots). Use when designing a new pipeline, choosing a GCP data service, or optimizing BigQuery cost/performance.