Solana Indexer Architecture: Backfill History and Switch to Live gRPC Without Missing Slots

Your indexer processed 200 million slots. Then you checked the database. Slot 187,442,201 is missing. Everything downstream that referenced it is wrong.

Not a parser bug. Not a schema issue. The slot was simply never committed.

This happens at the seam between backfill and live. Two data sources, one output store, and a window where neither covers the slots the other hasn't reached yet. That window is where data disappears.

Three properties prevent it.

01Three Invariants Your Indexer Must Never Break#

No gap. Every slot from your start slot to the current tip is represented in your store. A missing slot means missing transactions. Any query spanning that slot returns wrong results. Aggregates are wrong. Position tracking is wrong. If you're indexing a lending protocol and the liquidation transaction landed in slot 187,442,201, you didn't catch it.

No duplicate. Each slot is processed exactly once. Duplicates corrupt running totals. A token balance counted twice reports double the holdings. A DEX volume aggregate double-counts trades. Dedup isn't optional.

Monotonic order. Slot N is committed before slot N+1. Out-of-order processing breaks any stateful computation. Pool reserves, running P&L, open interest: all of these depend on seeing state changes in the order they happened on-chain. Slot 402 arriving before slot 401 produces a state that never existed.

Every section in this post builds one component that enforces one of these invariants. The three together define correctness. Violate any one and your indexer produces data you can't trust.

02Invariant 1: Backfill Without Gaps Using Historical Blocks#

This section enforces no gap.

The first mistake most teams make is fetching slots one by one with getBlock. Solana doesn't produce a block for every slot. Call getBlock on a skipped slot and it fails. Your backfill stops. Call getBlocks(start, end) first. It returns only the slots that have blocks in the range. Then iterate those.

The second mistake is not persisting state. If the process crashes at slot 187,442,000 and you restart from slot 1, you've lost nothing except time. But if you restart from slot 187,000,000 with no record of where you stopped, you're reprocessing 442,000 slots you already handled. Use a checkpoint.

import json
from pathlib import Path

CHECKPOINT_FILE = "indexer_checkpoint.json"
BATCH_SIZE = 1000

def load_checkpoint(start_slot: int) -> int:
    if Path(CHECKPOINT_FILE).exists():
        data = json.loads(Path(CHECKPOINT_FILE).read_text())
        return data["last_slot"]
    return start_slot

def save_checkpoint(slot: int) -> None:
    Path(CHECKPOINT_FILE).write_text(
        json.dumps({"last_slot": slot})
    )

def backfill(client, start_slot: int, target_slot: int) -> None:
    slot = load_checkpoint(start_slot)
    while slot < target_slot:
        batch_end = min(slot + BATCH_SIZE, target_slot)
        # Returns only slots that have blocks (skips empty slots)
        valid_slots = client.get_blocks(slot, batch_end)
        for block_slot in valid_slots:
            block = client.get_block(block_slot)
            process_block(block_slot, block)
            save_checkpoint(block_slot)
        slot = batch_end + 1

The checkpoint writes after each slot, not after each batch. A crash between writes loses at most one slot's worth of work. That's recoverable. A batch-level checkpoint crash loses the whole batch.

target_slot is a moving target. You don't need to know it upfront. Run backfill until backfill_slot is close enough to the live tip that the buffer covers the rest. The next section handles that boundary.

03Invariant 2: Buffer the Live Stream Before Backfill Finishes#

This section enforces no duplicate (and closes the gap at the handoff).

If you start your gRPC stream only after backfill finishes, there's a window between the last backfilled slot and the first streamed slot where neither source covers the data. It's small. Maybe a second or two. But it's real, and it's a gap.

Start the stream before backfill finishes. Write incoming slots to a buffer. Don't process them. Don't write them to your store. Just hold them.

import threading

buffer: dict[int, object] = {}
buffer_lock = threading.Lock()
buffer_min_slot: int | None = None

def run_grpc_buffer(endpoint: str, token: str) -> None:
    global buffer_min_slot
    for update in subscribe_stream(endpoint, token):
        slot = update.slot
        with buffer_lock:
            if buffer_min_slot is None:
                buffer_min_slot = slot  # Record where buffer starts
            buffer[slot] = update       # Store, don't process

def start_buffer_thread(endpoint: str, token: str) -> threading.Thread:
    t = threading.Thread(
        target=run_grpc_buffer,
        args=(endpoint, token),
        daemon=True,
    )
    t.start()
    return t

buffer_min_slot is the first slot the stream captured. The cutover section uses it. Don't lose it.

Why buffer and not process? Because backfill hasn't caught up yet. If the backfill cursor is at slot 187,000,000 and the live stream is at slot 187,500,000, you'd be writing live data 500,000 slots ahead of where backfill is. When backfill reaches those slots, it'd try to process them again. That's the duplicate.

The buffer holds the live data until backfill catches up. Then the reconciler drains it in order.

04Invariant 3: Slot Tracker and the Reconciler#

This section enforces monotonic order (and handles dedup).

The reconciler is the only component that writes to your output store. Backfill doesn't write directly. The buffer doesn't write directly. Everything passes through the reconciler. That's what keeps the invariant enforced in one place.

class GapError(Exception):
    pass

class SlotTracker:
    def __init__(self, start_slot: int):
        self.last_committed = start_slot - 1

    def commit(self, slot: int, data: object) -> bool:
        if slot <= self.last_committed:
            return False  # Duplicate. Skip silently.

        skipped = self._check_gap(slot)
        if skipped:
            raise GapError(
                f"Gap: expected up to {self.last_committed + 1}, got {slot}. "
                f"Intermediate blocks exist at: {skipped}"
            )

        self.last_committed = slot
        write_to_store(slot, data)
        return True

    def _check_gap(self, slot: int) -> list[int]:
        if slot == self.last_committed + 1:
            return []
        # Check if intermediate slots have blocks
        return get_blocks(self.last_committed + 1, slot - 1)

_check_gap calls getBlocks on the range between the last committed slot and the arriving slot. If it returns an empty list, those intermediate slots were skipped by Solana. Safe to advance. If it returns slots, those are real blocks you haven't processed. That's a gap. Stop and backfill them.

The return False on duplicate is silent by design. During cutover, some slots in the buffer will overlap with the last slots backfill processed. The reconciler handles that overlap without noise. No need to log every skip.

05The Cutover: Draining the Buffer to Live Without Dropping a Slot#

All three invariants converge here.

The condition to check before cutting over:

def ready_to_cut_over(backfill_slot: int) -> bool:
    with buffer_lock:
        if buffer_min_slot is None:
            return False  # Buffer hasn't started yet
        return backfill_slot >= buffer_min_slot - 1

When this returns True, the backfill cursor is one slot behind the first buffered slot. There's no gap between the two sources. You're ready.

The cutover sequence:

def cut_over(tracker: SlotTracker) -> None:
    with buffer_lock:
        # Drain buffer in slot order through the reconciler
        for slot in sorted(buffer.keys()):
            data = buffer[slot]
            tracker.commit(slot, data)  # Handles dedup if overlap
        buffer.clear()

    # From here, route new gRPC slots directly to tracker
    switch_to_live(tracker)

Sort matters. The gRPC stream doesn't guarantee slot order. Draining unsorted violates the monotonic invariant. Always sort buffer keys before draining.

After the drain, switch_to_live routes new Yellowstone gRPC updates directly to tracker.commit() instead of writing to the buffer. The indexer is now live.

At this point all three invariants hold: the checkpoint guaranteed no gap during backfill, the buffer guaranteed no gap at the handoff, the reconciler's sorted drain guaranteed monotonic order, and the slot <= self.last_committed check guaranteed no duplicate made it into the output store regardless of how much overlap existed between the two sources.

What happens at the overlap? The last few slots backfill processed may also be in the buffer. When the reconciler hits those slots during the drain, slot <= self.last_committed is true. It returns False and skips. No double write. The duplicate invariant holds.

06Skipped Slots and Re-orgs#

Solana produces roughly 400ms blocks but doesn't fill every slot. When a leader misses their window, the slot is empty. No block. No transactions. Your indexer shouldn't treat that as a gap.

# Don't do this. getBlock raises on a skipped slot.
block = client.get_block(slot)

# Do this. get_blocks returns only slots that have blocks.
valid_slots = client.get_blocks(start, end)
for slot in valid_slots:
    block = client.get_block(slot)  # Safe. Slot is guaranteed to exist.

The reconciler's _check_gap method already handles this. When it calls getBlocks on the gap range and gets back an empty list, it knows those were skipped slots. It advances safely.

Re-orgs. Solana finalizes fast but short forks happen. At processed commitment level, a block you committed may later be replaced if the validator's leader wasn't confirmed.

Subscribe at confirmed or finalized commitment level. It's the right default for almost every indexer. The latency difference is milliseconds for confirmed, seconds for finalized. Both are stable.

If you need processed level for latency reasons, keep a rollback buffer. Store the last 32 slots in memory alongside your committed data. If a slot arrives again with different transaction data, re-process from that slot forward and overwrite. Alert on it. It shouldn't happen often at confirmed. At processed, it will.

07Historical Blocks vs Trading Datasets: Which to Backfill From#

Both are viable backfill sources. The choice depends on what you need out of the data.

Use Trading Datasets if your use case maps to existing event types: DEX trades, token transfers, pool creations, pool events. You skip the parsing step entirely. Backfill takes hours instead of days. For most analytics and monitoring use cases, it's the right starting point.

Use Historical Blocks when you need raw instruction data, you're indexing a program not in the dataset, or your event schema doesn't match what Trading Datasets provides. You get full control. You also take on full responsibility for parsing.

You don't have to pick one. Start with Trading Datasets for the common event types and layer Historical Blocks on top for anything custom.

NLN Trading Datasets covers DEX trades, token transfers, and pool events going back to genesis. For the Yellowstone gRPC live stream, NLN Yellowstone gRPC is the endpoint your buffer and reconciler subscribe to.

08FAQ#

How do I backfill Solana historical data for an indexer?

Use a checkpoint file storing the last processed slot. Call getBlocks(last_slot, last_slot + batch_size) to get valid block slots in the range (Solana skips empty slots), fetch each block, process it, then save the checkpoint. On restart, resume from the checkpoint. You never reprocess and you never miss.

What is a good start slot for backfilling a Solana indexer?

It depends on the protocol you're indexing. For a specific DEX, start at the slot of the first program deployment. For general indexers, use a snapshot or pre-indexed dataset to avoid processing years of early Solana history. NLN Historical Blocks lets you specify a start slot and pull only the range you need.

How do I switch from backfilling to a live gRPC stream without missing slots?

Start the gRPC stream before backfill finishes and write incoming slots to a buffer. When the backfill cursor reaches buffer_min_slot - 1, drain the buffer through your reconciler in slot order, then switch to live processing. The cutover condition: backfill_slot >= buffer_min_slot - 1.

What is the difference between a skipped slot and a missing block in Solana?

A skipped slot has no block. Solana doesn't guarantee a block for every slot. Your indexer shouldn't treat skipped slots as gaps. Use getBlocks(start, end) to get only slots that have blocks in a range, then iterate only those. A missing block is a slot that getBlocks says has a block but your store doesn't. That's a real gap.

How large should the gRPC buffer be for indexer cutover?

A Python dict keyed by slot number is fine. For production, set a max buffer size around 5,000 slots and alert if it grows beyond that. At 400ms per slot, 5,000 slots is about 33 minutes of lag between your backfill cursor and the live tip. If the buffer keeps growing, your backfill isn't keeping up.

What happens if my indexer crashes mid-backfill?

If you write the checkpoint after each slot, the indexer resumes from the last committed slot on restart. Never batch checkpoint writes. A crash between writes loses that batch. Write to disk or database one slot at a time, after the slot is committed to your output store.

How do I handle re-orgs in a Solana indexer?

Subscribe at confirmed or finalized commitment level. Processed-level blocks can be rolled back. If you need processed-level for latency, keep a rollback window of the last 32 slots in memory. If a slot arrives again with different data, re-process from that slot forward and overwrite. For most indexers, confirmed is fast enough and removes the re-org risk entirely.

Should I use Historical Blocks or Trading Datasets for Solana backfill?

Use Trading Datasets if your use case maps to existing event types: DEX trades, token transfers, pool events. It's faster and removes the parsing overhead. Use Historical Blocks if you need raw instruction data or you're indexing a program not in the dataset. You can combine them: Trading Datasets for the common event types, Historical Blocks for anything custom.

Can I run backfill and live gRPC stream in parallel Python processes?

Yes. Run backfill in one process, gRPC buffer in another. The reconciler is the only process that commits to the output store. Coordinate via a shared checkpoint file or database row. That way the monotonic invariant is enforced in one place regardless of how many input sources feed it.

How do I detect slot gaps in a running Solana indexer?

Track last_committed_slot in your reconciler. If the next arriving slot is greater than last_committed_slot + 1, call getBlocks(last_committed_slot + 1, slot - 1) to check whether intermediate slots have blocks. If they do, you have a real gap. If getBlocks returns empty, those were skipped slots and you can advance safely.

How do I reconnect a Yellowstone gRPC stream without losing slots?

Store the last slot you received. On reconnect, include that slot in your SubscribeRequest so Yellowstone replays from that point. While disconnected, log the gap window. After reconnect, call getBlocks on the gap and backfill any blocks you missed before resuming live processing.

What is the cutover condition for switching from backfill to live?

The condition is backfill_slot >= buffer_min_slot - 1. buffer_min_slot is the first slot your gRPC buffer captured when the stream started. When the backfill cursor is one slot behind the buffer's start, there's no gap between the two sources. Drain the buffer through your reconciler in slot order, then route new gRPC slots directly to the reconciler.

If you're sourcing the backfill data, NLN Historical Blocks and NLN Trading Datasets cover the historical side. Raw blocks if you need custom parsing, pre-indexed events if your use case fits the schema. For the live stream your buffer connects to, NLN Yellowstone gRPC is the endpoint.