Modern networks process millions of events every second, from device authentication to data transmission, requiring a reliable way to capture the full story of every interaction. Apache Flink is a powerful tool for organizing and analyzing this constant flow of data in real time, ensuring that complex sequences are recorded accurately as they happen. However, scaling this process to handle massive data bursts can create hidden performance hurdles that threaten system stability.
So we rebuilt that logic as an in-memory buffer, ran it beside the original on real production data using the “A/B” development pattern for five months, and cut over by changing default topics. Here’s the pattern, the safeguards that keep it bounded, and the checklist we’d use again.
Flink and the trouble with streaming data
Apache Flink is an open-source engine for processing unbounded streams of events in real time. A Flink job reads events from a source such as Kafka, runs your logic on each one as it arrives, and writes results downstream. There is no end of the file to wait for.
That is what makes streaming harder than batch. A batch job works on a complete, finished dataset. A streaming job sees events as they come: continuously, in bursts, out of order, and sometimes late. Four ideas from Flink matter for the rest of this post:
- Event time: When an event actually happened, taken from the record itself.
- Processing time: When the event reached the machine processing it, which can be much later or in a different order.
- State: What the job remembers between events, per key (for us, per session). Flink keeps it in a state backend, either on the JVM heap (the Java process’s memory) or in RocksDB, an embedded key-value store on local disk.
- Checkpoints: Periodic snapshots of that state, so a restarted job resumes where it left off without losing or double-counting any of it.
Network connection events show why this matters. If a session’s auth step arrives after its dns step, a job that trusts arrival order reads the chain wrong. Getting it right takes stateful stream processing: remembering each session’s earlier steps and putting them in event-time order before deciding whether the connection succeeded.
The challenge: False positives at 200,000 messages per second
Our pipeline reads network connection events from Kafka. Each connection attempt is a chain of steps of different types (for example, auth, assoc, dhcp, and dns). To judge whether a connection succeeded, we have to read a session’s steps in the order they actually happened.
At 200,000 messages per second, that order is not what arrives. Traffic comes in bursts, and some events show up late. Our original Flink application struggled with both, and the result was false positives in connection-success reporting.
The flame graph pointed at RocksDB
We found the cause with the flame graph in Flink’s web UI (a chart of where the job spends its time; wider bars mean more time). It’s opt-in, so set rest.flamegraph.enabled: true to turn it on. Our session buffers lived in RocksDB, the state backend Flink uses for large, durable state.
That has a cost on the hot path. As the Flink state backends documentation explains, every read and write through RocksDB serializes and deserializes the value. In our job, every incoming event triggered that expensive state access plus a timer registration.
RocksDB wasn’t the wrong tool; it exists to hold state bigger than memory. It was the wrong place to do per-event reordering. We needed a faster layer to reorder events before committing data to durable state.
The solution: In memory reordering by event time
We built a second application in parallel called “Variant B.” It groups each session’s attempts by step type (e.g., auth, assoc, dhcp, and dns), orders them by event time, and allows late-arriving events to be sorted within each session’s buffer before parsing.
The buffer sits inside a KeyedProcessFunction, Flink’s low-level operator that exposes per-key state and timers. Events for a session go into a HashMap, and a custom comparator sorts them by event time (when the event occurred) instead of the time it was received (processing time). Late arrivals slot into their place in the session’s buffer before we parse the chain.
The hot path now touches plain heap objects, with no constant state interaction. The trade-off is memory: the buffer lives on the JVM heap, outside Flink’s managed state, so it has to be bounded. More on that below.
Checkpoint-only snapshots keep exactly-once state
Variant B still writes durable state, just not per event. At each checkpoint, we copy the buffer’s contents into a ListState (a list that Flink saves with each checkpoint), and when the application restarts, we recover the buffer from it. The Flink docs on working with state cover the snapshot and restore hooks.
Because the buffer is snapshotted as part of every checkpoint, Flink’s exactly-once guarantee still holds for the application state. End-to-end exactly-once also depends on how the sink handles replayed output, so check that side too.
Three safeguards that keep the buffer bounded
An in-memory buffer at this rate needs limits. We put three in place:
- Idle eviction: A custom three-minute idle timeout (not Flink’s built-in state TTL, or time-to-live, which only covers keyed state) removes silent clients from the buffer.
- Timer debouncing: We restricted timer registration so timers don’t fire excessively.
- Compact records: Compact POJOs (plain Java objects holding only the fields we need) replace raw records, and we parse fields only once, at ingestion.
From there, we tuned the length of the session window and the safeguard values to balance catching edge cases against resource use.
Underneath it all, we moved our Flink deployment to Kubernetes on Amazon EKS. For us, that gave the cost efficiency and resource-management flexibility this scale needs.
Five months side by side, then a cutover by configuration
We didn’t swap out the production pipeline. We ran Variant B alongside the original implementation for five months, with feature flags gating which results were visible. That let us confirm Variant B’s results against real production data while the original stayed the default.
That matters for stateful stream processing in particular. Ordering bugs show up under real traffic: bursts, late events, and edge cases a test fixture doesn’t have. Running both pipelines side by side is how you see those cases without shipping them.
When we were confident, the cutover was small:
- Update the default topics.
- Remove the branching logic from the UI, the dashboard server, and the ClickHouse query layer.
A checklist for your next stateful stream processing job
If you run a stateful Flink job at high volume, here’s what we’d do again:
- Profile first: Open the flame graph before you change your state design.
- Keep the hot path in memory: Buffer in a
KeyedProcessFunction and snapshot to ListState only at checkpoints, so restarts recover the buffer.
- Bound your buffers: Checkpoint only the state you need to rebuild the buffer, and filter out invalid records early.
- Evict idle sessions: Use tight idle eviction to prevent memory leaks.
- Validate before cutover: Use A/B development to compare downstream results in real time and catch unexpected behavior before it’s released to production.
High-throughput stateful stream processing comes down to two habits: keep state-backend access off the per-event path, and never cut over a stateful job on faith.
This week, turn on the Flink flame graph for your busiest operator and see where its time actually goes.
Instagram Engineering
Shopify Engineering