data

📊 Data Engineering live

SQL, Python/Polars, Apache Kafka KRaft (Strimzi), Spark, Airflow, dbt, ClickHouse OLAP and S3 Lakehouse (Map 19)

▸ Stage 1. Analytical SQL, Columnar Formats & Python Data Tooling live

  • Advanced analytical SQL: window functions (ROW_NUMBER, LAG/LEAD), CTEs, lateral joins, and query cost optimization
  • Vectorized data processing with pandas, Polars, and DuckDB over Parquet and Apache Arrow files

▸ Stage 2. Event Streaming with Apache Kafka KRaft & Strimzi (Map 19) live

  • Apache Kafka in KRaft mode (ZooKeeper-less Raft metadata quorum) deployed on Kubernetes via Strimzi KafkaNodePools
  • Producer/consumer reliability: acks=all, min.insync.replicas=2, idempotent producers, partitioning keys, and consumer lag control
  • Kafka Tiered Storage: offloading historical log segments to S3 while keeping hot tail data on fast local NVMe

▸ Stage 3. Orchestration & Analytics Engineering (Airflow & dbt) live

  • Idempotent DAG engineering in Apache Airflow: TaskFlow API, dynamic task mapping, backfills, sensors, and SLA alerts
  • Modular transformation layers with dbt (staging -> intermediate -> marts), incremental models, macros, and schema/data tests

▸ Stage 4. Distributed Batch/Stream Processing (Spark & Elasticsearch) live

  • Apache Spark optimization: Catalyst query planner, broadcast joins vs sort-merge joins, partition skew mitigation, and caching
  • Full-text and log search at scale with Elasticsearch: index mappings, analyzers, shard sizing, and ILM lifecycle policies

▸ Stage 5. Real-Time OLAP & S3 Lakehouse Architecture (ClickHouse) live

  • Zero-code Kafka-to-ClickHouse ingestion pipelines using Kafka table engines and Materialized Views
  • Open Lakehouse tables (Apache Iceberg / Delta Lake) on S3 with expressive executive dashboards (Plotly, Altair, Grafana)

Source: roadmaps/data.json · ETA: 2026-11