Panduan GCP Professional Data Engineer: Datastream CDC, Dataflow Beam, BigQuery Storage Write API & Vertex AI

Menyewa GCP Professional Data Engineer bersertifikat resmi memastikan data warehouse analitik Anda selalu sinkron dengan database transaksional OLTP dalam hitungan detik tanpa membebani PostgreSQL utama maupun membengkakkan biaya pemindaian BigQuery bulanan. Dengan pengalaman produksi membangun platform data Google Cloud menggunakan replikasi logis Cloud SQL / AlloyDB, Cloud Datastream CDC, Cloud Dataflow (Apache Beam), BigQuery Storage Write API, dan Vertex AI Feature Store, berikut adalah pola arsitektur dan kode Python yang saya terapkan di lingkungan produksi.

PostgreSQL WAL Replikasi Logis Datastream CDC MAX_STALENESS Dataflow Beam Watermark & DLQ Tabel BigQuery Write API APPEND_ROWS Vertex AI Feature Store
Alur pipeline streaming ujung-ke-ujung: PostgreSQL Logical Replication menuju Cloud Datastream CDC, Dataflow Apache Beam, BigQuery Storage Write API, dan Vertex AI.

1. Replikasi Logis PostgreSQL & Cloud Datastream CDC dengan BigQuery MAX_STALENESS

Melakukan polling berkala terhadap database PostgreSQL menggunakan kueri batch SELECT ... WHERE updated_at >= ? berisiko melewatkan operasi DELETE dan membebani CPU produksi. Seorang GCP Professional Data Engineer mengaktifkan slot replikasi logis (pgoutput) agar Cloud Datastream mengalirkan perubahan CDC langsung ke tabel BigQuery ber-Primary Key dengan parameter MAX_STALENESS untuk menjaga biaya pemrosesan UPSERT tetap efisien:

-- Konfigurasi sumber PostgreSQL (Cloud SQL / AlloyDB)
ALTER SYSTEM SET wal_level = 'logical';
CREATE PUBLICATION analytics_cdc_pub FOR TABLE public.orders, public.ledger_entries;
SELECT pg_create_logical_replication_slot('datastream_cdc_slot', 'pgoutput');

-- Tabel tujuan BigQuery untuk UPSERT otomatis Datastream CDC
CREATE TABLE `prod_dw.orders_cdc`
(
  order_id STRING NOT NULL,
  tenant_id STRING NOT NULL,
  currency STRING NOT NULL,
  amount_cents INT64 NOT NULL,
  status STRING NOT NULL,
  updated_at TIMESTAMP NOT NULL,
  PRIMARY KEY (order_id) NOT ENFORCED
)
PARTITION BY DATE(updated_at)
CLUSTER BY tenant_id, status
OPTIONS (
  max_staleness = INTERVAL 15 MINUTE,
  require_partition_filter = TRUE,
  description = "Tabel transaksi PostgreSQL yang direplikasi real-time melalui Datastream CDC"
);

2. Pipeline Streaming Apache Beam di Cloud Dataflow dengan Watermark & Dead-Letter Queue (DLQ)

Data event dari Pub/Sub sering datang terlambat (out-of-order) atau mengalami perubahan skema mendadak. Di Cloud Dataflow (Apache Beam), setiap transformasi streaming harus memvalidasi JSON secara defensif, mengarahkan payload rusak ke topik Dead-Letter Queue (DLQ) tanpa menghentikan worker, dan memakai FixedWindows berbasis waktu event:

import json
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions, StandardOptions
from apache_beam.transforms.window import FixedWindows
from apache_beam.transforms.trigger import AfterWatermark, AfterProcessingTime, AccumulationMode
from datetime import datetime


class ParseAndValidateEvent(beam.DoFn):
    DEAD_LETTER_TAG = "dlq"

    def process(self, raw_bytes: bytes):
        try:
            payload = json.loads(raw_bytes.decode("utf-8"))
            if "tenant_id" not in payload or "event_ts" not in payload:
                raise ValueError("Kolom wajib tenant_id / event_ts tidak ditemukan")
            ts = datetime.fromisoformat(payload["event_ts"].replace("Z", "+00:00")).timestamp()
            yield beam.window.TimestampedValue(
                {
                    "tenant_id": str(payload["tenant_id"]),
                    "amount_cents": int(payload.get("amount_cents", 0)),
                    "event_type": str(payload.get("event_type", "unknown")),
                },
                ts,
            )
        except Exception as exc:
            yield beam.pvalue.TaggedOutput(
                self.DEAD_LETTER_TAG,
                {"raw": raw_bytes.decode("utf-8", errors="replace"), "error": str(exc)},
            )


def run_streaming_pipeline(input_sub: str, dlq_topic: str, output_table: str) -> None:
    opts = PipelineOptions(save_main_session=True, streaming=True)
    opts.view_as(StandardOptions).streaming = True

    with beam.Pipeline(options=opts) as p:
        parsed = (
            p
            | "ReadPubSub" >> beam.io.ReadFromPubSub(subscription=input_sub)
            | "ValidateAndTimestamp" >> beam.ParDo(ParseAndValidateEvent()).with_outputs(
                ParseAndValidateEvent.DEAD_LETTER_TAG, main="valid"
            )
        )

        # Alirkan pesan rusak ke topik DLQ Pub/Sub agar pipeline utama tidak macet
        (
            parsed.dlq
            | "SerializeDLQ" >> beam.Map(lambda d: json.dumps(d).encode("utf-8"))
            | "WriteDLQ" >> beam.io.WriteToPubSub(topic=dlq_topic)
        )

        (
            parsed.valid
            | "Window5Min" >> beam.WindowInto(
                FixedWindows(300),
                trigger=AfterWatermark(early=AfterProcessingTime(60)),
                accumulation_mode=AccumulationMode.ACCUMULATING,
                allowed_lateness=1800,
            )
            | "KeyByTenant" >> beam.Map(lambda row: (row["tenant_id"], row["amount_cents"]))
            | "SumRevenue" >> beam.CombinePerKey(sum)
            | "WriteBigQuery" >> beam.io.WriteToBigQuery(
                output_table,
                method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API,
                write_disposition=beam.io.WriteDisposition.WRITE_APPEND,
            )
        )

3. Audit Biaya & Slot FinOps BigQuery melalui INFORMATION_SCHEMA

Kueri BI yang memindai seluruh tabel tanpa filter partisi bisa langsung melambungkan tagihan BigQuery on-demand. Gunakan audit SQL berikut pada INFORMATION_SCHEMA.JOBS_BY_PROJECT untuk menemukan kueri terboros sebelum menerapkan autoscaling slot reservations:

-- Audit 10 pola kueri BigQuery terboros dalam 7 hari terakhir
SELECT
  user_email,
  statement_type,
  COUNT(*) AS jumlah_kueri,
  ROUND(SUM(total_bytes_billed) / POW(1024, 4), 4) AS total_tib_tertagih,
  ROUND(AVG(total_slot_ms) / 1000, 2) AS rata_rata_slot_detik
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
  AND job_type = 'QUERY'
  AND state = 'DONE'
GROUP BY user_email, statement_type
ORDER BY total_tib_tertagih DESC
LIMIT 10;

4. Panduan Pilar Rekayasa Cloud, GCP & Backend Terkait

Sewa GCP Professional Data Engineer Bersertifikat

Butuh audit FinOps BigQuery, pipeline CDC Datastream ke PostgreSQL/BigQuery, Cloud Dataflow Apache Beam, atau fitur Vertex AI dalam 1 hari kerja? Pesan Layanan Pemrograman 1 Hari seharga Rp 1.000.000 (QRIS / Virtual Account Bank Indonesia) atau $100 USD (Kartu Kredit/Debit) melalui /id/1-day-service/ atau buat kontrak di Upwork.


About the Author — Hire Aldian Fazrihady (Cloud Engineer, GCP Professional Cloud Architect, GCP Professional Data Engineer, Backend, AI & Full Stack Engineer)

Aldian Fazrihady is a Top-Rated Senior Cloud Engineer, GCP Professional Cloud Architect (Google Cloud Certified), GCP Professional Data Engineer (Google Cloud Certified), Backend Engineer (Python, Django, FastAPI, Go, PostgreSQL), AI Application Engineer (LLMs, RAG, MCP, Google ADK, LangGraph), and Full Stack Engineer (React, Next.js, TypeScript) with 15+ years of production software engineering experience.

Hire Aldian Fazrihady on Upwork | Book 1-Day Programming Service ($100 USD / Rp 1.000.000) | View Live AI Portfolio | Send a Direct Message

Related Engineering Articles

Engineering Role Pillar Guides & Category Hubs