Enable selective integration of otel_client collected logs into Application/Log Messages for enhanced APM experience
## Problem Statement
Currently, when using **otel_client** to collect application logs (such as WAS SystemOut.log, IHS access_log, and Odoo application logs), these logs successfully appear in **Analytics > Logs** with proper filtering, grouping, and analysis capabilities. However, these same logs do NOT appear in the **Application > Log Messages** view within the APM perspective.
This creates a **significant gap in the APM experience** because:
1. **Broken Correlation**: Logs cannot be correlated with application traces, calls, and performance metrics in the unified APM view
2. **Context Switching**: Users must switch between Analytics and Application perspectives to get complete observability
3. **Limited APM Value**: The full potential of Application Performance Monitoring is not realized when logs are isolated from traces
4. **Custom Log Visibility**: Applications without native Instana sensors (like Odoo, custom apps) cannot benefit from integrated APM log analysis
5. **Incomplete Troubleshooting**: When investigating issues in Application view, users see "No log messages available" despite logs being collected and visible in Analytics
## Current Behavior
### What Works Today
- ✅ otel_client successfully collects logs from configured sources
- ✅ Logs appear in **Analytics > Logs** with full functionality:
- Filtering by log file path, custom tags, hostname
- Grouping and aggregation
- Time-series visualization
- Log level analysis (DEBUG, INFO, WARN, ERROR)
- Full log message content and metadata
### What's Missing
- ❌ Logs do NOT appear in **Application > Log Messages** tab
- ❌ Application view shows "No log messages available" despite active log collection
- ❌ No correlation between collected logs and application traces
- ❌ No unified view of logs alongside service calls, dependencies, and errors
- ❌ Cannot leverage APM context (service, endpoint, trace ID) for log analysis
## Desired Behavior
Provide an **easy-to-configure mechanism** to selectively enable logs collected via **otel_client** to appear in the **Application > Log Messages** view, providing:
1. **Selective Log Integration**: Allow users to choose which log sources should appear in Application/Log Messages (e.g., application logs like Odoo) vs. remain only in Analytics (e.g., OS logs like /var/log/secure, /var/log/messages)
2. **Simple Configuration**: Easy opt-in mechanism (e.g., tag-based, attribute-based, or configuration file) to mark specific log sources for APM integration
3. **Unified APM Experience**: View selected application logs alongside traces, calls, and metrics in a single application context
4. **Correlation with Analytics**: Enable correlation between Application/Log Messages and Analytics/Logs for comprehensive troubleshooting
5. **Context-Aware Filtering**: Filter logs by service, endpoint, trace ID, and other APM dimensions
6. **Enhanced Troubleshooting**: Quickly identify root causes by correlating log messages with performance anomalies
7. **Sensor-Independent Monitoring**: Enable full APM capabilities for applications without native Instana sensors
**Rationale for Selective Integration:**
Infrastructure teams often collect various log types via otel_client:
- **Application Logs** (should appear in APM): `/var/log/odoo/odoo18.log`, WAS SystemOut.log, IHS access_log
- **Infrastructure Logs** (should remain in Analytics only): `/var/log/secure`, `/var/log/messages`, `/var/log/audit/audit.log`
Not all collected logs are relevant to application performance monitoring. The solution should allow easy selection of which logs integrate with APM while keeping infrastructure logs in Analytics for security and compliance purposes.
## Business Value
### Improved Troubleshooting Efficiency
- **Faster MTTR**: Reduce mean time to resolution by eliminating context switching between Analytics and Application views
- **Better Root Cause Analysis**: Correlate log messages directly with slow transactions, errors, and service dependencies
- **Unified Investigation**: Single pane of glass for all observability data related to an application
- **Complete Context**: No more "No log messages available" when investigating application issues
### Extended APM Coverage
- **Custom Applications**: Enable APM log integration for applications without native Instana sensors (Odoo, custom-built apps)
- **Legacy Systems**: Bring WAS, IHS, and other traditional middleware into modern APM workflows
- **Flexible Log Sources**: Support any log source that can be collected via OpenTelemetry
- **Broader Platform Support**: Monitor diverse technology stacks with consistent APM experience
### Enhanced User Experience
- **Intuitive Workflow**: Users expect logs to appear in the Application perspective where they analyze performance
- **Consistent Interface**: Maintain familiar APM navigation patterns without forcing users to learn Analytics perspective
- **Contextual Insights**: Automatically enrich logs with APM metadata (service name, trace ID, span ID)
- **Reduced Training**: Single interface for all APM activities
## Use Cases
### Use Case 1: Odoo Application Monitoring (No Native Sensor)
**Scenario**: Monitoring Odoo ERP application that has no native Instana sensor
**Current Workflow**:
1. otel_client collects `/var/log/odoo/odoo18.log` successfully
2. Logs visible in Analytics > Logs with DEBUG level messages
3. User investigates application performance in Application view (EAL-005852_SD-LOG)
4. Clicks on "Log messages" tab → sees "No log messages available"
5. Must switch to Analytics > Logs to view Odoo application logs
6. Cannot correlate Odoo logs with service calls or performance metrics
7. Loses application context when analyzing logs
**Desired Workflow**:
1. otel_client collects `/var/log/odoo/odoo18.log`
2. Logs automatically appear in Application > Log Messages
3. User investigates performance in Application view
4. Clicks "Log messages" tab → sees Odoo logs with full context
5. Logs correlated with service calls and performance data
6. Can filter by log level, time range, and APM dimensions
7. One-click navigation between logs, traces, and calls
### Use Case 2: WAS Application Troubleshooting
**Scenario**: A WebSphere Application Server experiences intermittent errors
**Current Workflow**:
1. User notices errors in Application > Error Messages
2. Switches to Analytics > Logs to search SystemOut.log
3. Manually correlates timestamps between views
4. Loses trace context when analyzing logs
**Desired Workflow**:
1. User sees errors in Application > Error Messages
2. Clicks on Log Messages tab in same application view
3. Sees correlated SystemOut.log entries with trace IDs
4. Clicks log entry to jump to related trace
### Use Case 3: IHS Access Log Analysis
**Scenario**: Investigating slow HTTP requests to a web application
**Current Workflow**:
1. Identify slow calls in Application > Calls view
2. Switch to Analytics > Logs to find IHS access_log entries
3. Filter by timestamp and URL pattern
4. Cannot directly link to the slow trace
**Desired Workflow**:
1. Identify slow call in Application > Calls view
2. View Log Messages tab showing correlated IHS access_log entries
3. See request/response details enriched with trace context
4. One-click navigation between logs and traces
### Use Case 4: Custom Application Logs
**Scenario**: Custom-built application without native Instana sensor needs log integration
**Current Workflow**:
1. Configure otel_client to collect custom application logs
2. Logs appear only in Analytics > Logs
3. No APM integration or trace correlation
4. Limited value from APM investment
**Desired Workflow**:
1. Configure otel_client to collect custom application logs
2. Logs automatically appear in Application > Log Messages
3. Full APM integration with automatic trace correlation
4. Complete observability without custom sensor development
## Technical Context
### Current Setup
- **Log Collection**: otel_client configured with file receivers for WAS, IHS, and Odoo logs
- **Log Destination**: Logs successfully ingested and visible in Analytics > Logs
- **Log Attributes**: Custom tags, hostname, log file path properly captured
- **Log Levels**: DEBUG, INFO, WARN, ERROR properly parsed and displayed
- **Application Monitoring**: Applications instrumented and visible in APM perspective
- **Gap**: Application > Log Messages shows "No log messages available"
### Example Log Sources
- WAS: `/web/logs/websphere/pac/SystemOut.log`
- IHS: `/web/logs/httpd/access_log`
- Odoo: `/var/log/odoo/odoo18.log`
- Custom applications: Any text-based log file
### Expected Integration Points
- **Selective Opt-In Configuration**: Simple mechanism to mark log sources for APM integration
- Option 1: Tag-based (e.g., `apm.enabled=true` in otel_client configuration)
- Option 2: Attribute-based (e.g., `log.destination=apm` or `log.destination=analytics`)
- Option 3: Configuration file mapping (e.g., specify which log paths integrate with APM)
- **Trace Correlation**: Use trace ID, span ID from log context to link logs to traces
- **Service Association**: Map logs to services based on hostname, tags, or configuration
- **Log Level Mapping**: Preserve log levels (ERROR, WARN, INFO, DEBUG) in APM view
- **Timestamp Alignment**: Ensure log timestamps align with trace timeline
- **Metadata Preservation**: Maintain custom tags and attributes from otel_client
### Example Configuration Approach
```yaml
# otel_client configuration example
receivers:
filelog/odoo:
include: ["/var/log/odoo/odoo18.log"]
attributes:
apm.enabled: true # Opt-in to Application/Log Messages
service.name: "odoo-erp"
filelog/system:
include:
- "/var/log/secure"
- "/var/log/messages"
- "/var/log/audit/audit.log"
attributes:
apm.enabled: false # Remain in Analytics only
```
## Competitive Context
Other APM solutions (Dynatrace, New Relic, Datadog) provide unified log integration where logs collected via OpenTelemetry or custom collectors automatically appear in the application monitoring context with full trace correlation. This is considered a standard feature in modern APM platforms.
## Success Criteria
1. **Selective Integration**: Users can easily configure which log sources appear in Application > Log Messages
2. **Simple Configuration**: Clear, documented mechanism (tags, attributes, or config file) to opt-in logs to APM
3. **Application Logs in APM**: Selected logs (Odoo, WAS, IHS) appear in Application > Log Messages view
4. **Infrastructure Logs Separate**: OS/security logs remain in Analytics only unless explicitly configured
5. **No Log Messages Fixed**: "No log messages available" replaced with actual log content for configured sources
6. **Trace Correlation**: Logs correlated with traces using trace/span IDs where available
7. **APM Filtering**: Users can filter logs by APM dimensions (service, endpoint, trace ID)
8. **Direct Navigation**: Log entries link directly to related traces and calls
9. **Analytics Preserved**: No degradation in Analytics > Logs functionality
10. **Sensor Independence**: Works for applications with and without native Instana sensors
## Priority Justification
**HIGH PRIORITY** because:
- Addresses fundamental gap in APM user experience
- Enables APM value for applications without native sensors (Odoo, custom apps)
- Reduces troubleshooting time and improves operational efficiency
- Aligns with OpenTelemetry standards and modern observability practices
- Requested by Instana Support as a common customer need
- Eliminates confusing "No log messages available" message when logs are actually being collected
- Critical for customers with diverse technology stacks
## Additional Notes
This enhancement would significantly improve the value proposition of Instana for customers with:
- **Modern Applications**: Odoo, custom Python/Node.js/Go applications without native sensors
- **Traditional Middleware**: WAS, IHS, WebLogic, Apache, etc.
- **Hybrid Environments**: Mixing modern and legacy systems
- **Diverse Technology Stacks**: Requirements for unified observability across multiple platforms
- **OpenTelemetry Adoption**: Organizations standardizing on OTel for log collection
### Key Design Principles
**Selective, Not Automatic**: The solution should provide an easy opt-in mechanism rather than automatically routing all collected logs to Application/Log Messages. This is critical because:
- Infrastructure teams collect various log types (application, OS, security, audit)
- Not all logs are relevant to application performance monitoring
- OS and security logs (`/var/log/secure`, `/var/log/messages`, `/var/log/audit/audit.log`) should remain in Analytics for compliance and security purposes
- Application logs (`/var/log/odoo/odoo18.log`, WAS SystemOut.log, IHS access_log) should have an easy path to APM integration
**Configuration Flexibility**: The integration should:
- Respect existing log collection configurations
- Not require customers to reconfigure their entire otel_client setup
- Allow granular control at the log source level
- Work seamlessly with current Analytics > Logs functionality
- Extend visibility into Application perspective only for selected sources
### Implementation Considerations
- Maintain backward compatibility with existing log collection
- Default behavior: logs remain in Analytics only (no breaking changes)
- Opt-in mechanism: simple attribute, tag, or configuration to enable APM integration
- Support both trace-correlated and standalone log entries
- Provide clear documentation on configuration options
- Ensure performance at scale with high log volumes
- Support log sampling and filtering policies
- Allow per-log-source configuration (some to APM, others to Analytics only)
---
## References
- **Current Implementation**: otel_client with file receivers
- **Log Sources**:
- WAS SystemOut.log
- IHS access_log
- Odoo odoo18.log
- Extensible to other sources
- **Analytics View**: Fully functional in Analytics > Logs
- **APM View**: Missing from Application > Log Messages (shows "No log messages available")
- **Support Case**: Recommended by Instana Support to submit as AHA idea
- **Screenshots**: Available showing logs in Analytics but not in Application view
