Detecting Replay Stream Completion Using Main Stream Disconnect Timestamp

Overview

When working with Validic's Inform Streaming API, you may need to coordinate between a mainstream and a replay stream. The replay stream captures up to 30 days of back data. This article describes determining when a replay stream has caught up to real-time using the main stream's disconnection timestamp.

Challenge

Customers need a reliable way to:

• Temporarily disconnect their mainstream

• Run a replay stream

• Detect when the replay stream has caught up

• Reconnect the mainstream without data loss

Solution

Use the mainstream's disconnection timestamp as a precise marker to determine when the replay stream has caught up.

How It Works

1. Capture Disconnect Time

   • When disconnecting the mainstream, store the exact timestamp of disconnection

   • This timestamp serves as your completion marker

2. Monitor Replay Events

   • For each event in the replay stream, check the created_at timestamp

   • Ignore poke events (sent every 5 seconds)

   • Compare each event's created_at against your stored disconnect timestamp

3. Detect Completion

   • When you see an event with created_at >= disconnect timestamp

   • This indicates the replay stream has caught up to the point where the mainstream disconnected

4. Reconnect Main Stream

   • The mainstream will automatically resume from its disconnection point

   • Validic maintains this checkpoint position for 7 days

   • No data loss occurs during the handoff

Benefits

• Precise: Uses exact timestamp comparison

• Simple: Straightforward implementation

• Reliable: Leverages built-in Validic functionality

• Clean: Ensures no data gaps during stream handoff

Important Notes

• Store disconnect timestamp in ISO format for consistent comparison

• Remember the main stream's state (connected/disconnected)

• Reconnect mainstream within 7 days to ensure checkpoint retention

• Consider keeping the replay stream active briefly after mainstream reconnection

• This solution works with any SSE client implementation

Need Help?

Think of the Streaming API like a baseball game - Validic pitches, and your SSE listener catches. Validic Support can help with any "pitching" issues related to the Streaming API itself, such as data flow, event structure, or connection management. However, selecting and implementing an SSE listener (the "catching" part) is the customer's responsibility - you'll need to research and implement the SSE listener that best fits your environment and requirements. Validic Support cannot guide specific SSE listener client implementations or programming languages.