Understanding Expected Data Latency (Cloud Sources)

For developers using the Inform API with cloud-based data sources like Fitbit, Garmin, Withings, Omron, etc., it's critical to understand the concept of expected data latency. Expected data latency is the period from when the data is generated until Validic can collect it. Unfortunately, this period cannot be clearly defined, but you can take steps to mitigate potential data delivery disruptions. This article will help you understand why this "natural" data latency occurs and how to help mitigate the impact.

What is Expected (or Natural) Data Latency

Simply put, the time passes before Validic can consume data. As you know, once our platform receives data, it streams it to you in near-real time. However, the data must make several hops before we can do this.

Hop One: Wearable Device to Transmitting Device

The first step of our journey starts with the actual wearable device. In this case, a Fitbit device collects fitness data. For this data to make the first hop, we need the wearable device to connect to the device, ultimately connecting us to the cloud. In this case, a smartphone. That is typically achieved by connecting to the device using Bluetooth.

Open

Possible Disruptions

• The wearable device has disabled BLE, gone into Airplane Mode, or is turned off.

• The transmitting device has disabled BLE, gone into Airplane Mode, or is turned off.

• The BLE pairing has become corrupted or been forgotten.

Hop Two: Transmitting Device/Wearable Device to Application

Open

Our next hop in the connection chain moves the data from the wearable into the mobile application. This step is required as the application stores the data and passes it onto the cloud. Suppose the data cannot be inserted into the application. In that case, the wearable device will continue to look for the application and pass the data once the application is open on the transmitting device.

Possible Disruptions

• Closing the application without enabling background sync in your phone settings.

• Hard closing the application on the phone. This closes the app and turns off background sync.

Hop Three: Application to Cloud

With the application receiving the data, we're almost done with external latency influencers. When the application has received the data, it will transmit it to the cloud once a connection is established. This will require an internet or cellular connection. During this phase of the journey, we find that most applications send data to their cloud as soon as it's received. That said, it is possible some applications may batch data and transmit data at time intervals.

Possible Disruptions

• The transmitting device may have disconnected from the internet, turned off cellular service, or be outside a service area.

• Weak internet or cellular service may cause data transmission to fail.

• In rare cases, data sources can experience issues in the cloud that prevent data transmission for a short period. These are typically related to system-down situations.

Hop Four: Cloud Collection

Once the data source receives the data, the last step is to wait for the system to make the data available.

In most cases, that happens immediately, with the data sources sending us a notification that data exists. Within a couple of seconds, we collect the data, convert it to our standard format, commit that data to your API data stream, and store it for collection in our RESTful API. Sometimes, the data source may not notify us of new data, and we must check for new records at a set interval. This is usually every 15-30 mins, but may be hours. Reviewing the data source information on our support site is essential to understand this delay. There, you'll find a table outlining expected delays for each integration.

Possible Disruptions

• Natural delays while we wait for data sources to make data available

• In rare cases, data sources can experience issues in the cloud that prevent data collection for a short period. These are generally related to system-down situations.

Now what?

These hops represent the time it takes to get data into Validic. The next leg of the journey depends on how you get data from us: RESTful API, Streaming API, or Push service. The fastest of these services, the Streaming API, will deliver the data within a few seconds of collection. We suggest spending time with our API documentation to learn more about this. It’s not that we don’t want to tell you more, it’s just, this article is about the front end of your data journey, and we can’t have you hanging out here all day. People will wonder where you are and you have data to deliver.

How to Mitigate Data Delivery Disruptions

Preventing these natural data latency issues is nearly impossible. There are just too many variables that you cannot control. However, now that you know why it happens, you can educate your end users to help prevent data latency issues. Below are our top tips to keep data flowing:

1. Keep the application open on the phone! Without the application collecting data, it can't be passed to the data source.

2. Using the wearable device only with the device manufacturer application is crucial. Some devices, such as blood pressure cuffs, can be connected to multiple mobile applications. But, doing so can lead to applications competing for data and hindering data transmission to the cloud. To avoid this potential issue, it is highly recommended to instruct users to only pair their device with the application provided by the device manufacturer.

3. Stay connected! Encourage your users to keep their devices connected to the internet and do not disable BLE connections.

4. To ensure a smooth user experience, it's important to understand the data sources you support and set realistic expectations. Before launching, it's recommended to test wearable devices and document any possible delays. Creating potential end-user delays, such as disabling BLE, can also be helpful. Ensure to note how the device manufacturer's application responds and document the recovery steps and time. Providing this information to your support team can help prevent user frustration and encourage engagement.

5. Review the integration page for each data source and note expected delays in collecting data.