Enhance Instana or Instana agents capability to cache data, during Instana backend upgrade

Hello Team,

we are reaching out to request an enhancement to improve data retention capabilities during Instana backend infrastructure upgrades. Recently, during our self-host air-gapped environment Instana backend upgrade, we observed that all Instana agents stopped collecting data. Although data collection resumed after the upgrade, we experienced a data gap of approximately 20 to 25 minutes. We Open IBM support case, support team and IBM Technical Sales team have suggested to open enhancement request.

To prevent data loss during above scenarios, we propose the following enhancement options (option 1 or 2 or both):

1. Persistent Agent-Backend Connection During Upgrades: We suggest enhancing the Instana infrastructure to allow agents to maintain a persistent connection to the backend, even during upgrades. This would enable agents to continue collecting and transmitting data without interruption.

2. Agent-Level Caching Mechanism for Data Retention: As an alternative, we propose implementing a caching mechanism at the agent level. If an agent loses connection with the backend, it could temporarily cache data locally (for a configurable duration, such as 30 minutes). When the connection is restored, the agent would forward the cached data to the backend. This approach is similar to the IBM Tivoli Monitoring (ITM) agent's cache feature, which has proven effective in minimizing data loss during transient connection issues.

Business Value of This Enhancement

1. Data Continuity: Continuous data collection without interruption, even during maintenance windows, would prevent monitoring gaps.

2. Enhanced Reliability for Alerts and Analytics: Maintaining uninterrupted data flow improves the accuracy and reliability of alerts, trends, and analytical insights.

3. Improved Troubleshooting: Eliminating data gaps facilitates troubleshooting and allows for a more complete view of our environment’s health.

Implementation Considerations

To ensure a smooth experience, we recommend:

1. Allowing configurable settings for cache retention duration and size at the agent level to suit various infrastructure needs.

2. Ensuring a reliable synchronization mechanism for agents to send cached data once connection to the backend is re-established.

We believe that this enhancement would add significant value to the Instana platform and greatly enhance its reliability during upgrade processes. We kindly request that this feature be considered for future releases and are happy to provide any additional information or feedback needed.

Thank you and Best Regards