← Back to portfolio
★ Streaming · Real-time

Flight
Pipeline

Kafka Streaming · Southwest Airlines · Jan 2023 — Aug 2024
Kafka · LiveEFF stream12.4K events/min · lag 0.8s●●●
Topics
  • eff.tracking.raw
  • eff.tracking.transformed
  • eff.errors
  • eff.deadletter
Consumers
  • s3-sink · ok
  • analytics · ok
  • alerts · ok
WN1234 · DAL→LAXcruise
WN0876 · MDW→DENclimb
WN2103 · BWI→FLLdescent
WN0521 · HOU→PHXcruise
WN1890 · OAK→SANtaxi
Sai

Project Overview

Southwest's flight-ops team needed real-time visibility into the Elastic Flat Files (EFFs) that ground systems emit during every flight — currently arriving by FTP, batched, and analyzed hours later. We replaced the batch path with a Kafka-based stream that surfaces phase, position, and anomalies within seconds.

Kafka StreamsRabbitMQ MicroservicesS3 SinkPython · FlaskJava · GradleGitLab CI95% test coverageLocal-stack Kubernetes

Problem Statement

  1. Hours-late insights. EFFs landed by FTP and analytics jobs ran every 4 hours — by the time an anomaly was visible, the flight had landed.
  2. Brittle integration. Existing pipeline was a chain of cron jobs and shared filesystems. A single missed file silently dropped the next 6 hours of data.
  3. No local dev story. Engineers could only test against a shared QA environment, blocking each other constantly.
<1s
End-to-end latency from EFF arrival to dashboard visualization, versus 4-hour batches before.
95%
Test coverage on Java conversion
12K/min
Events sustained

My Role

Sr. Application Engineer / Data Scientist on a team of 12. Designed the Kafka topology, owned the EFF → S3 → Kafka pipeline, and led the Python → Java port for the production hot-path.

Kafka TopologyRabbitMQ pub/subStream TransformsLocal-stack DockerTest StrategyData Governance

The approach.

// STEP 01

Pub/sub instead of cron.

RabbitMQ between ingestion services for fan-out (analytics + cold-storage + alerting consumers). Kafka for the high-throughput tracking events. No service knows about another's existence.

// STEP 02

Local S3 in a Docker pod.

Built a local-stack S3 + Kafka inside a multi-pod Kubernetes setup that mirrors prod. Every engineer gets their own fully-isolated environment in under 30 seconds.

// STEP 03

Python prototypes, Java for hot paths.

Prototyped transforms in Python (fast iteration), then ported the proven ones to Java with Gradle builds and 95% unit-test coverage. Best of both speeds.

The outcomes.

<1s
Stream latency

From 4-hour batch to sub-second visibility on every flight event.

95%
Test coverage

Java ports landed with unit suites that catch ~95% of regressions before merge.

12 → 1
Cron jobs replaced

An entire constellation of cron + shared FS jobs collapsed into one event-driven topology.

Next case study →
Drilling-Loss NLP & Autoencoders · Shell