Support Violation Count-Based Alert Triggering Within a Lookback Window

Problem Statement

Instana Smart Alerts currently evaluate alert conditions based on the number of consecutive evaluation windows that contain violations rather than the number of violations that occur within a specified time period.

This limits the ability to detect recurring or intermittent issues that occur multiple times within a short period of time.

For example, a service may experience numerous spikes or repeated violations over a five-minute period, yet those violations may not occur across enough consecutive evaluation windows to trigger an alert. Conversely, a small number of isolated violations spread across multiple windows may trigger an alert even when there is little operational impact. The second radio button in the alert config doesn't let you adjust the number of violations in a short time frame, seems there is a limitation.

This behavior makes it difficult to accurately identify flapping conditions, recurring performance degradation, or repeated transient failures while minimizing false positives.

Proposed Enhancement

Introduce an alerting option that allows users to trigger alerts based on the number of violations occurring within a configurable lookback period.

The enhancement could be offered as an alternative alert evaluation method alongside the existing consecutive-window approach.

Business Value

Faster Detection of Real Issues

- Repeated violations occurring within a short time window can indicate a developing incident even when those violations are not distributed across consecutive evaluation windows. 

Reduced Alert Fatigue

- Users can better distinguish between isolated metric spikes and recurring operational problems, reducing unnecessary alerts.

Improved Alert Accuracy

- Alerts become more representative of actual application impact by allowing customers to define meaningful thresholds based on the frequency of violations rather than only evaluation timing. 

Greater Alerting Flexibility

- Different applications have different tolerance levels for failures. Violation-count thresholds allow organizations to tailor alerts to their specific workloads and business requirements.

Example Use Case

A service occasionally experiences a brief latency spike that does not require action.

However, if latency exceeds the threshold 5 times within 10 minutes, the issue is likely impacting users and should trigger an alert.

Today, Instana evaluates based on consecutive violation windows rather than the total number of violations within a lookback period, making this type of alerting difficult to implement.