Feature Request: Kafka Cost Monitoring for AWS MSK and Confluent Kafka

Executive Summary

We propose adding comprehensive cost monitoring capabilities for Kafka infrastructure, supporting both AWS MSK (Managed Streaming for Apache Kafka) and Confluent Kafka platforms.

Business Case & Value Proposition

Why This Feature is Critical:

Kafka has become a foundational component of our enterprise architecture, particularly within our payments domain, which processes mission-critical transactions. Currently, Kafka infrastructure represents one of our largest AWS expenditures, yet we lack granular visibility into cost drivers and optimization opportunities.

Without dedicated Kafka cost monitoring, organizations face:

• Difficulty attributing costs to specific teams, applications, or business units
• Limited ability to identify cost anomalies or inefficiencies
• No proactive alerting for cost spikes or resource waste
• Challenges in rightsizing Kafka clusters and optimizing configurations

Who Will Benefit

1. FinOps Teams: Gain visibility into Kafka spending patterns, track cost trends, and enforce budget controls
2. Platform Engineering Teams: Identify overprovisioned resources, optimize cluster configurations, and reduce waste
3. Application Teams: Understand the cost impact of their Kafka usage (topics, partitions, throughput)
4. Finance & Leadership: Access accurate cost allocation for chargeback/showback models and budget planning
5. Payments Domain Teams: Monitor costs for high-volume, business-critical Kafka workloads

Proposed Functionality

Core Features:

Multi-Platform Support

• AWS MSK cost tracking (broker costs, storage, data transfer)
• Confluent Kafka cost monitoring (CKUs, storage, support tiers)
• Unified dashboard for hybrid Kafka environments

Cost Attribution & Tagging

• Per-cluster cost breakdown
• Topic-level cost estimation
• Application/team attribution via tags
• Environment segregation (dev, QA, prod)

Usage Metrics Correlation

• Link costs to actual usage (throughput, storage, partition count)
• Identify idle or underutilized clusters
• Track cost-per-message or cost-per-GB metrics

Alerting & Optimization

• Budget threshold alerts
• Anomaly detection for unexpected cost spikes
• Recommendations for rightsizing (broker types, partition optimization)
• Retention policy optimization suggestions

Reporting & Analytics

• Cost trends over time (daily, monthly, quarterly)
• Comparison across clusters and environments
• Export capabilities for financial reporting
• Forecasting based on historical patterns

Implementation Approach

Data Sources:

• Integration with AWS Cost Explorer API for MSK costs
• Confluent Cloud API for Confluent Kafka metrics
• CloudWatch/Prometheus metrics for usage correlation

Visualization:

• Dedicated Kafka cost dashboards
• Drill-down capabilities (organization → team → cluster → topic)
• Cost allocation reports

Integration Points:

• Existing FinOps platforms (Cloudability, CloudHealth, etc.)
• ITSM tools for budget approval workflows
• Slack/Teams notifications for alerts

Expected Outcomes

• Cost Reduction: 15-25% savings through identification of waste and optimization opportunities
• Visibility: Complete transparency into Kafka spending across the organization
• Accountability: Accurate cost attribution to teams and applications
• Proactive Management: Early detection of cost anomalies before they impact budgets
• Better Planning: Data-driven decisions for Kafka infrastructure scaling and investment

Conclusion

Given Kafka's critical role in our payments infrastructure and its significant contribution to our AWS costs, dedicated cost monitoring is essential for financial governance and operational efficiency. This feature would provide immediate value to multiple stakeholders and deliver measurable ROI through cost optimization and improved visibility.