In Teardown 1, we removed synchronous RPC calls from the hot path to eliminate backpressure and prevent event loss. Once ingestion is fast, the next challenge is stream completeness.

The indexer reads PumpFun and PumpSwap trades from a live Yellowstone gRPC stream. Slots arrive sequentially. If a slot range is skipped, the system needs to detect the missing blocks and backfill them.

I built a gap detector with two parts.

First, the Carbon framework catches live stream drops. It logs the disconnect and reconnect slots to a stream_gaps table.

Second, an in-memory watermark. Every processed trade carries a slot. An atomic counter tracks the highest slot seen. If a new slot jumps more than 50 slots past this watermark, it registers a gap.

A separate recovery worker polls this table, fetches the missing blocks over standard RPC with a 64-slot overlap, and writes them. Database primary keys handle the boundary duplicates safely.

I tested this by unplugging the network for a minute. The stream dropped 938 slots. The detector caught it, the worker refetched it, and the database perfectly matched the chain state.

How it broke

Next, I tested a hard crash. I killed the process and restarted it a minute later.

A minute of live slots passed while the indexer was dead. Those trades never reached Postgres.

The gap detector recorded absolutely nothing. The system acted like the downtime never happened. I only caught the data loss by manually diffing the database against the chain.

Why it missed

The flaw is that both detection mechanisms live inside the running process. A process cannot observe its own restart.

When the process died, the atomic counter was stopped with it. Carbon could not fire a disconnect event because the framework itself was dead.

On startup, the pipeline initializes a fresh watermark starting at zero. When the gRPC connection opens, it binds directly to the current tip of the chain. The sequence checker sees a baseline of zero, assumes a fresh start, and immediately updates the watermark to the live block.

The indexer completely lost its place. Because it reconnects at the live tip instead of the crash point, that entire one-minute window is skipped. An in-memory monitor cannot track a failure that destroys its own memory.

So what’s the architectural fix?

The solution requires anchoring the runtime state to durable storage. A restart gap exists between two absolute facts: the last slot successfully written to Postgres and the first live slot the new process receives.

Instead of initializing a blank watermark on boot, the system must read its starting position from the database. The persistence layer was already committing the highest processed slot alongside the trade batches to ensure idempotent writes. I just had to query this checkpoint before opening the stream.

By seeding the watermark from the database, the system bridges the restart. The first live gRPC frame is compared against the exact stopping point. If the chain advanced while the process was down, the delta exceeds the 50-slot tolerance. The system registers the gap, the background worker fetches the missing window, and the data is restored automatically.

What we learnt?

State management in distributed systems requires separating liveness from durability.

Monitoring a live network connection is a liveness task. It belongs in process memory because the application is awake to watch the drop happen.

Surviving a hard crash is a durability task. Process memory is the wrong tool for this job because memory is exactly what the crash destroys. Any baseline required to track stream continuity across a restart must be anchored to persistent storage. The durable state was always sitting in the database. The bug was trusting an ephemeral counter to do a database’s job.