Implementing Monitoring of log ingestion for Log Anomaly Detection using AIOps

Hi Team.

Customer has following questions regarding the Implementing the Log Anomaly Detection using AIOps.

1. Is there any self monitoring feature available in AIOps UI to monitor the Log ingestion rate like how many log lines are received and ingested into AIOps & also about the working log ingestion and its threshold values?
Description: In AIOps for Log Anomaly Detection when logs are getting ingested through kafka/ELK/Mezmo interface of AIOps, customer need to wait for the golden signal template formation to know either log are getting ingested or not. For this golden signal template formation minimum 100k logs are needed or need 3.5 days to wait. Until this customer doesn't get to know either logs are coming to the respective interface of AIOps or not and it is being processed by AIOps or not. There is no such facility in AIOps GUI to track the incoming logs and no way to check the number of logs that are being processed for Log Anomaly Detection. Once the integration is done for logs, it becomes totally blank spot for user to track the logs and monitor it .
Though there are few manual complex command line steps to get only little info from the AIOps Elasticsearch api, but it is not relevant to the user, because it requires openshift commands, tokens and a cli terminal and interference of openshift Admin person. Hence it is necessary to have a monitoring dashboard for AIOps user/admin to track how many logs are being processed by the AIOps, log ingestion rate per second and the volume of logs.

2. I found out there are few Command line tools available to check the functionality of AIOps which can be used as a temporarily solution for the self monitoring functionality. Is there is any way to implement RBAC mechanism to allow access to application owners and allow them to monitor the working functionality of AIOps.
Description: Customer always concern about the data privacy. When we are ingesting data/logs to the AIOps it also contains sensitive info like user mail, username. So only few authorised people should have the access to view/edit the logs which AIOps is processing. So there we need some specific RBAC policy to grant access to a specific group of people to see all LOG related activities/Tiles in AIOps.
Once we have monitoring, RBAC is must to be provided with that and what ever logs that can be viewed through elastic search through CLI that also need to be restricted through RBAC.

3. Impact:
Description: There is no such visibility of Logs that are being ingested and processed in AIOps and customer doesn't have the confidence on Log Anomaly Detection. The visibility and the monitoring solution must required to identify AIOps acceptable log format, Log ingestion rate per second, and proceeded log count.
If i will specifically talk about the impact, if log ingestion rate goes more than the limit or volume is more then it will highly impact to the total AIOps performance. As it is for a telco customer (T-system) , it is highly required.

4. Any losses of revenue for this idea?
Yes. It is the basic requirements which was not taken care. Right now it is difficult for me to quantify the losses, but lack of monitoring solution may lead to the performance issue of AIOps. Hence customer is not getting the confidence about Log Anomaly detection capability of AIOps.
We are at POC stage, but to move to the production we need to have these solutions in place.

Customer: T-system, German