Cloud Management and AIOps


This is an IBM Automation portal for Cloud Management, Technology Cost Management, Network Automation and AIOps products. To view all of your ideas submitted to IBM, create and manage groups of Ideas, or create an idea explicitly set to be either visible by all (public) or visible only to you and IBM (private), use the IBM Unified Ideas Portal (https://ideas.ibm.com).

Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:

Search existing ideas

Start by searching and reviewing ideas and requests to enhance a product or service. Take a look at ideas others have posted, and add a comment, vote, or subscribe to updates on them if they matter to you. If you can't find what you are looking for,

Post your ideas

  1. Post an idea.

  2. Get feedback from the IBM team and other customers to refine your idea.

  3. Follow the idea through the IBM Ideas process.

Specific links you will want to bookmark for future use

Welcome to the IBM Ideas Portal (https://www.ibm.com/ideas) - Use this site to find out additional information and details about the IBM Ideas process and statuses.

IBM Unified Ideas Portal (https://ideas.ibm.com) - Use this site to view all of your ideas, create new ideas for any IBM product, or search for ideas across all of IBM.

ideasibm@us.ibm.com - Use this email to suggest enhancements to the Ideas process or request help from IBM for submitting your Ideas.

Add a new idea Subscribe
Status Future consideration
Workspace Instana
Categories Sensor
Created by Guest
Created on May 15, 2024

RELATED IDEAS

Kafka Cluster Consumer Group Metrics at the Partition Level

I would like to monitor Kafka consumer group metrics such as current offset, log end offset, lag, throughput, etc., at the partition level.

This would be similar to the metrics provided by the kafka-consumer-groups.sh CLI tool:

▶ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group

 

GROUP               TOPIC       PARTITION   CURRENT-OFFSET  LOG-END-OFFSET  LAG
my_consumer_group   my_topic    3           466552579140    466555101712    2522572
my_consumer_group   my_topic    2           465441201347    465443973791    2772444
my_consumer_group   my_topic    7           832170815467    832171950535    1135068
my_consumer_group   my_topic    0           467681966829    467684854884    2888055
my_consumer_group   my_topic    4           635344417235    635346807110    2389875
my_consumer_group   my_topic    5           608012034778    608016294905    4260127
my_consumer_group   my_topic    6           786214393587    786217043441    2649854
my_consumer_group   my_topic    1           466268145640    466270648921    2503281
my_consumer_group   my_topic    12          614493787790    614493794408    6618
...


This feature would be extremely helpful in identifying partition imbalances, such as when a single partition is receiving a disproportionate amount of data.

Idea priority High
  • Admin
    Aysha Kanakkovil
    Dec 5, 2025

    Based on the discussion had with @Guest for customer NTT Data:

    Customer would also like to obtain 2 metrics for Last Offset in a Partition and Records-Lag.

    1.kafka.log:type=Log,name=LogEndOffset,topic=([-.\w]+),partition=([0-9]+)This does not check the numerical value at a specific point in time for a specific partition, but rather checks the increase over a certain period of time, whether the increase amounts are similar between multiple partitions (i.e., whether only a specific partition has increased extremely).

    2. kafka.consumer:type=consumer-fetch-manager-metrics,partition="{partition}",topic="{topic}",client-id="{client-id}"

    This metric relates to the consumer (Kafka AP) rather than the broker (Kafka server), and is used to check an estimate of how many unprocessed messages there are on the Kafka server per partition.

    If there is a processing delay, this number will be larger than expected, so it should be checked and monitored individually on a partition/consumer basis.

    Screenshot 2025...
    Screenshot 2025...
    Screenshot 2025-12-05 at 3.56.09 PM.png
    Open full size
    Screenshot 2025-12-05 at 3.55.38 PM.png
    Open full size
    Screenshot 2025-12-05 at 3.56.09 PM.png Screenshot 2025-12-05 at 3.55.38 PM.png
    Reply
  • Admin
    Rodoljub Radivojevic
    Aug 16, 2024

    This feature is not on the roadmap for 2H 2024.

    Will be considered for 1H 2025 according to its priority and priority of other features.

    2 replies
15 MERGED

Request for Kafka Partition-Level Metrics Collection in Instana

Merged
Currently, Instana does not provide the ability to collect Kafka metrics at the partition level. For example, it is not possible to monitor the backlog of messages for each partition. If such information cannot be obtained directly in Instana, it ...
3 months ago in Instana / Metric Store 0 Future consideration

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.