Managing Connections to a Stream

Owned by Matt Dougherty
Last updated: Feb 02, 2024
3 min read

The primary method for getting data from the Inform API is via the Streaming API. By default, every Validic Organization Access Token is allowed up to 5 individual streams.

For any Validic Organization created after January 2023, each of stream is allowed a maximum of 3 HTTP connections.

Knowing this, there are probably a few questions you’ll be asking as you implement your solution to work with the stream.

Do I need to use all 5 streams, or will just 1 be enough?

In most cases, you’ll probably only need 1 stream. We have found that additional streams are generally used for testing or backup of data. That being said, it’s no coincidence that Validic has 5 main data resource types and allow 5 unique streams. We have seen some orgs create 1 stream for Summary data, 1 stream for Fitness data, etc.

In general, you will know, based on your system architecture, if you need to use more than 1 stream.

Do I need to use all 3 connections to my stream?

The answer to this question mostly is determined by the volume of data coming through your stream. As above, you’ll most likely be just fine with 1 connection to a stream.

The reason you would want or need more than 1 connection is if you need to utilize the automatic load balancing that is engaged as soon as there is more than 1 connection (more below on this). As an example, here are some data points representing data volumes that we have seen for some Validic production orgs. This is just as a reference to help you decide if you need more than 1 connection:

~350K connected users, peaking at 500K records/hour - 2 connections on their stream

~200K connected users, peaking at 340K records/hour - 3 connections on their stream

In both of those examples, the full data volume is evenly split (balanced) across their 2 or 3 connections

So, again, much like how many streams you will need, this will be mostly decided by your system and how much volume a single connection can handle without missing records. If you start noticing regular missing records and have only 1 connection and high number of users we’d recommend adding an additional HTTP client and making another connection for load balancing purposes.

If you have the max number of allowed HTTP clients connected to a stream, a large number of connected users and a high volume of records/hour and are noticing regular missing records then please contact support@validic.com for additional options.

What other things should I be aware of when working with the streams?

  • It’s noted above, but important enough to state again…each stream is limited to a maximum of 3 connections.

  • ANY HTTP connection is counted as one of those 3 connections. This includes requests you may have in Postman as well as web browser connections. That seemingly innocent time you dropped the stream URL into a browser tab just to check something? Yeah, that counts and affects load balancing.

  • If you start noticing regular missing records and have only one connection and high number of users we’d recommend adding an additional HTTP client and making another connection for load balancing purposes.

  • Replay streams can be used to reprocess missing records